Title: Graph-Structured Gaussian Processes for Transferable Graph Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph learning

transfer learning

Gaussian process
Abstract: Transferable graph learning involves knowledge transferability from a source graph to a relevant target graph. The major challenge of transferable graph learning is the distribution shift between source and target graphs induced by individual node attributes and complex graph structures. To solve this problem, in this paper, we propose a generic graph-structured Gaussian process framework (GraphGP) for adaptively transferring knowledge across graphs with either homophily or heterophily assumptions. Specifically, GraphGP is derived from a novel graph structure-aware neural network in the limit on the layer width. The generalization analysis of GraphGP explicitly investigates the connection between knowledge transferability and graph domain similarity. Extensive experiments on several transferable graph learning benchmarks demonstrate the efficacy of GraphGP over state-of-the-art Gaussian process baselines.

Graph-Structured Gaussian Processes for Transferable Graph Learning

Abstract

Transferable graph learning involves knowledge transferability from a source graph to a relevant target graph. The major challenge of transferable graph learning is the distribution shift between source and target graphs induced by individual node attributes and complex graph structures. To solve this problem, in this paper, we propose a generic graph-structured Gaussian process framework ( GraphGP ) for adaptively transferring knowledge across graphs with either homophily or heterophily assumptions. Specifically, GraphGP is derived from a novel graph structure-aware neural network in the limit on the layer width. The generalization analysis of GraphGP explicitly investigates the connection between knowledge transferability and graph domain similarity. Extensive experiments on several transferable graph learning benchmarks demonstrate the efficacy of GraphGP over state-of-the-art Gaussian process baselines.

1 Introduction

Transfer learning [40, 54] aims at studying the transfer of knowledge or information from a source domain to a relevant target domain under distribution shifts. The knowledge transferability across domains has been investigated from various aspects. For example, [3, 15, 67] proposed to empirically estimate the data-level domain discrepancy for measuring the transferability across graphs. [8, 49] adopted adaptive Gaussian processes with transfer kernels, which formulated the task relatedness as a hyper-parameter for tuning. [22, 36, 63] instead evaluated the transferability of a pre-trained source model in the target domain. However, most existing works followed the IID assumption that samples are independent and identically distributed

Source

Target

(a)

(b)

(c)

in each domain. This might limit their capacities in understanding the knowledge transferability across domains with non-IID data, e.g., cross-domain recommendation [55], cross-network role identification [69], etc.

In this paper, we focus on the problem of transferable graph learning over non-IID graph data. The challenge of transferable graph learning induced by the graph structure information can be explained in Figure 1. It can be seen that Figure 1(a) provides source and target samples associated with input features and output class labels (indicated by node colors). Suppose that source and target

distributions have been aligned (e.g., optimized by domain discrepancy minimization algorithms [15, 67]), Figure 1(a) implies that source and target domains have similar data distributions in the feature space, thereby enabling feasible knowledge transfer across domains. Compared to Figure 1(a), Figure 1(b)(c) introduce additional graph structure information. In Figure 1(b), source and target graphs follow similar structures where nodes within the same class tend to be connected (i.e., the homophily assumption [59]). In contrast, the structures of source and target graphs fundamentally differ in Figure 1(c). That is, though the target graph follows the homophily assumption, the source graph holds the heterophily assumption that connected nodes may have different classes and dissimilar node attributes [68]. Therefore, compared to standard transfer learning [31, 40] with IID data, transferable graph learning is much more challenging because knowledge transferability across graphs can be determined by both individual node features and complex graph structures.

In recent years, transferable graph learning has been studied [55, 58, 66, 69] where both source and target graphs follow the homophily assumption (Figure 1(b)). However, little effort has been devoted to investigating the knowledge transferability between homophilic and heterophilic graphs (Figure 1(c)). To bridge this gap, in this paper, we focus on studying the knowledge transferability across graphs with either the same (e.g., homophily or heterophily) or different assumptions.

We start by introducing a universal structure-aware neural network to build the input-output relationship between structure-aware input sample ( v, G ) and its associated output value y v for tackling homophilic and heterophilic graphs. This model encodes both local node representation (sample-level) and global graph representation (domain-level) simultaneously. The crucial idea is to automatically select relevant neighborhoods by adding learnable weights to neighborhood aggregation. Thus, it enables knowledge transferability between homophilic and heterophilic graphs by adaptively selecting neighborhoods for each graph. The intuition is that source and target graphs might have common knowledge from the selected neighborhoods, e.g., nearby neighborhoods for the homophilic graph and high-order neighborhoods for the heterophilic graph. Then, we show the equivalence between structure-aware neural network and graph Gaussian process in the limit on the layer width. This observation sheds light on building graph-structured Gaussian processes ( GraphGP ) for transferable graph learning. Moreover, the generalization analysis of GraphGP shows the connection between knowledge transferability and graph domain similarity. Compared to previous works [45, 58, 69], GraphGP benefits from (i) feasible knowledge transferability between homophily and heterophily graphs, and (ii) flexible incorporation with existing graph neural networks and graph kernels. Extensive experiments on node regression tasks demonstrate the effectiveness of GraphGP over state-of-the-art Gaussian process baselines. The main contributions of this paper are summarized as follows.

A generic graph-structured Gaussian process framework GraphGP is proposed for transferable graph learning. It is derived from a structure-aware neural network encoding local node representation (sample-level) and global graph representation (domain-level) simultaneously.

We show that GraphGP tackles the knowledge transferability in homophily and heterophily graphs using a simple neighborhood selection strategy. In addition, we theoretically analyze the knowledge transferability across graphs from the perspective of graph domain similarity.

Experimental results on a variety of graph learning benchmarks demonstrate the efficacy of GraphGP over state-of-the-art Gaussian process baselines.

The rest of this paper is organized as follows. Section 2 summarizes the related work, and Section 3 provides the problem definition of transferable graph learning. In Section 4, we present the graphstructured Gaussian process ( GraphGP ) for transferable graph learning. The experimental results are provided in Section 5. Finally, we conclude the paper in Section 6.

2 Related Work

Transfer Learning: Transfer learning [40, 54] tackles the transfer of knowledge or information from a source domain to a relevant target domain. It is theoretically shown [3, 67] that the generalization performance of a learning algorithm can be improved by leveraging latent knowledge from the source domain. Specifically, the generalization error of a learning algorithm on the target domain can be bounded in terms of the source knowledge and the distribution discrepancy across domains. Recently, transfer learning has been explored in non-IID graph data [7, 55, 58, 64, 69]. However, most existing works focus on either investigating the transferability of a pre-trained graph neural

network [27, 45, 69] or designing graph transfer learning algorithms [58, 66] with relaxed assumptions (e.g., k -hop ego-graphs are assumed to be independent and identically distributed [69]).

Gaussian Process: Gaussian process [9, 42] provides a principled nonparametric framework for learning stochastic functions from observations. It has been applied to transfer learning when samples in each domain are independent and identically distributed [8, 49, 56]. The crucial idea is to infer the task relatedness between source and target domains using available labeled source and target examples. More recently, Gaussian processes have also been investigated in graph learning tasks, e.g., link prediction [14, 38, 65] and node regression [6, 21, 28, 35, 37, 39]. However, little effort has been devoted to understanding transferable graph learning from Gaussian process perspectives.

3 Preliminaries

3.1 Notation and Problem Definition

Suppose that a graph is represented as G = ( V, E ) , where V = { v 1 , · · · , v | V | } is the set of n nodes and E ⊆ V × V is the edge set in the graph. Each node v ∈ V can be associated with a D -dimensional feature vector x v ∈ R D . In the node regression tasks, the ground-truth output of each node is denoted by y v ∈ R . The structure of a graph G can also be represented by an adjacency matrix A ∈ R | V |×| V | , where A ij is the edge weight between v i and v j within the graph. Following [55, 69], we introduce a generic problem setting of transferable graph learning as follows.

Definition 3.1 ( Transferable Graph Learning) . Given a source graph G s = ( V s , E s ) and a target graph G t = ( V t , E t ) , transferable graph learning aims to improve the prediction performance of a graph learning algorithm in the target graph using latent knowledge from a relevant source graph.

3.2 Message-Passing Graph Neural Networks

Weisfeiler-Lehman (WL) graph subtree kernel [46] measures the similarity of two input graphs, inspired by the Weisfeiler-Lehman test of isomorphism [53]. The crucial component of WL subtree kernel is to recursively represent the subtree structure rooted at each node. This has motivated a variety of message-passing graph neural networks (GNN), which recursively learn the node presentation of node v by aggregating the feature vectors from v 's local neighborhood [17, 18, 20, 57, 59]. Generally, a graph convolutional layer of message-passing GNNs can be summarized as follows.

where COMBINE ( l ) function compresses the representations from the node itself and its neighbors at the l th graph convolutional layer into a single representation, and AGGREGATE ( l ) j ( j = 1 , · · · , k ) function aims to aggregate message from j -order neighborhood N j ( v ) of node v . It indicates that there are two major components for designing graph convolutional layers in GNNs: neighborhood selection and aggregation. More specifically, neighborhood aggregation allows compressing the graph structure and node attributes simultaneously, while neighborhood selection provides the flexibility of GNNs in tackling graphs with different assumptions (e.g., homophily [20] or heterophily [68]).

Homophily graphs: Homophily graph holds the assumption that nodes within the same class tend to be connected in the graph, e.g., citation networks [13], social networks [33], etc. This assumption has motivated various instantiations of message-passing graph neural networks over first-order neighborhoods, e.g., GCN [25], GraphSAGE [20], GAT [50], GIN [59].

Heterophily graphs: Heterophily graph holds that connected nodes may have different class labels and dissimilar node attributes, e.g., different classes of amino acids are more likely to connect within protein structures [30, 68]. Recently, by adaptively exploring the potential homophily in a high-order local neighborhood (e.g., k > 1 ), message-passing graph neural networks have been proposed for heterophily graphs, e.g., MixHop [1], H2GCN [68], GPR-GNN [11], HOG-GCN [52], GloGNN [29].

4 Methodology

In this section, we propose the graph-structured Gaussian processes for transferable graph learning.

4.1 Structure-Aware Neural Network

We start by showing the graph sampling process from a probability distribution space P . A graph G = ( V, E ) is sampled by first sampling a graph domain distribution P G from P , and then sampling a specific graph G from P G . More specifically, one realization from P is a graph domain distribution P G (also denoted as P G = ( P V , P E ) for node and edge sampling distribution) characterizing a twostage graph sampling as follows. A set of nodes V is sampled from the graph distribution P G (more specifically, sampled from P V ), and then the edge weight is sampled from P G (more specifically, sampled from P E ) over any pair of nodes in V . In addition to the graph structure (induced by the node dependence), the output label of a node can also be sampled accordingly in the context of node regression tasks. That is, a label-informed graph domain distribution P G , Y is sampled from P . Next, a graph G = ( V, E ) is sampled from P G = ( P V , P E ) , and then the output labels of nodes are sampled from P Y|G . The goal of node regression task [55, 69] is to learn a prediction function f ( · ) that predicts ˆ y v = f ( v, G ) for each node v ∈ V .

In this paper, we consider the covariate shift assumption [40] over graphs. That is, the source and target graphs share conditional distribution P Y|G ( y | ( v, G )) but different marginal distributions P G ( v, G ) . The aforementioned graph sampling process motivates us to learn the input-output relationship between an input sample ( G,v ) and its associated output y v for transferable graph learning. To this end, for each input graph G (e.g., source G s or target graph G t in transferable graph learning), we define a structure-aware neural network f ( · , · ) as follows.

where M is the layer width 1 . µ ( l ) ( v | G ) denotes the node (sample-level) representation of v given the graph G at the l th layer, and ν ( l ) ( G ) denotes the graph (domain-level) representation of the graph G . Here READOUT function compresses a set of node representations into a single vector representation [59, 62]. This definition indicates that the embedding representation of the input ( v, G ) (e.g., a pair of a node v and its associated graph G ) is given by two crucial components: an individual node embedding representation µ ( l ) ( v | G ) and a global graph embedding representation ν ( l ) ( G ) . h ( l ) v and ˜ h ( l ) v represent graph neural networks to learn node and graph representations separately.

The intuition behind this definition is explained below. The graph G is a realization of a graph domain distribution P G . Then, the embedding representation of the graph G can be explained as an empirical estimate of the distribution P G , i.e., ν ( G ) ≈ ν ( P G ) . Moreover, when the size of the graph G goes to infinity (i.e., associated with an infinite number of nodes), G approximates the true distribution P G and the embedding representation of the graph G will be able to recover the true embedding of probability distribution P G [19, 48]. Besides, the embedding representation of a node v can also be explained as an approximation of P G at the location v , i.e., µ ( v | G ) ≈ µ ( v | P G ) .

Remark. The structure-aware neural network Eq. (2) is defined over message-passing graph neural networks commonly used for various single-domain graph learning tasks [18, 20, 23, 59]. Different from previous works, we focus on transferable graph learning scenarios across graphs. This motivates us to consider the task relatedness between source and target graphs, which can be explicitly measured by the domain distribution similarity [15, 56]. To this end, we design the structure-aware neural network Eq. (2) for learning domain-aware node presentation, i.e., the integration of the node (sample-level) representation and the entire graph (domain-level) representation. Specifically, we can show the connection between graph domain distribution and graph representation learning in the reproducing kernel Hilbert space for cross-graph learning tasks (see Corollary 4.4).

4.2 Graph-Structured Gaussian Process

Inspired by [11, 52, 68], we derive a universal message-passing graph neural network Eq. (1) for both homophily and heterophily graphs. More specifically, the message-passing graph convolutional layer of Eq. (1) for node v within an input graph G can be defined as follows.

where x (0) u = x u ∈ R D , N 0 ( v ) = { v } denotes the seed node, W ( l ) SANN , b ( l ) SANN are the weight and bias parameters, and α i explicitly indicates the importance of the i -order neighborhood in finding relevant neighbors around node v . Similarly, we define the domain-level graph neural layer ˜ h ( l ) v parameterized by different ˜ W ( l ) SANN , ˜ b ( l ) SANN and shared α i .

Generally, it is flexible to apply different model architectures to learn the node (sample-level µ ( l ) ( v | G ) ) and graph (domain-level ν ( l ) ( G ) ) representations. For simplicity, we adopt the same model architecture but different model parameters to learn node-level and graph-level representations. It is noteworthy that the neighborhood importance α i is shared because both node and graph representation would share the same assumption (homophily or heterophily) for a given graph. Then, the structure-aware neural network Eq. (2) can be given by

where the READOUT function of Eq. (2) is instantiated with mean pooling [62]. Different from previous works [11, 52, 68], we would take the neighborhood importance scores α i as hyper-parameters for building transferable graph Gaussian process model as follows.

Theorem 4.1. Assuming that all the parameters of structure-aware graph neural network f ( v, G ) are independent and randomly drawn from Gaussian distributions, i.e., W ( l ) ∼ N ( 0 , σ 2 w I ) , b ( l ) SANN ∼ N ( 0 , ς 2 b I ) , W ( l ) SANN ∼ N ( 0 , ς 2 w I ) , ˜ b ( l ) SANN ∼ N ( 0 , ˜ ς 2 b I ) , ˜ W ( l ) SANN ∼ N ( 0 , ˜ ς 2 w I ) , when the layer width M goes to infinity, the output function f ( l ) i in Eq. (5) follows a Gaussian process with f ( l ) i ∼ GP (0 , K ( l ) ) , where the covariance function K ( l ) is given by

where

Here P ( i ) ( P ′ ( j ) ) denotes the adjacency matrix given by the i -order neighborhood from graph G ( j -order neighborhood from graph G ′ ). K ( l -1) ab = ∑ k i,j =0 α i α ′ j ( b 2 + a 2 P ( i ) C ( l -2) ( a, b )( P ′ ( j ) ) T ) and C (0) uu ′ ( a, b ) = ⟨ x u , x u ′ ⟩ for any u ∈ V, u ′ ∈ V ′ .

4.2.1 Implications of Theorem 4.1

In the following, we show that the sample-level covariance/kernel K ( l ) µ of Theorem 4.1 can be explained as a message-passing operation in the kernel space. Following [4, 37], when using ReLU as activation function ϕ ( · ) (i.e., ϕ ( x ) = max { 0 , x } ), C ( l -1) is the arc-cosine kernel [12] as follows.

Then, the following corollary derives the feature map of K ( l ) µ ( u, u ′ | G,G ′ )

in the kernel space. Corollary 4.2. Let φ 1 : H → H 1 denote the kernel mapping from a pre-activation RKHS H to postactivation RKHS H 1 , i.e., ⟨ φ 1 ( s ) , φ 1 ( s ′ ) ⟩ = 1 2 || s || · || s ′ || · κ 1 ( ⟨ s,s ′ ⟩ || s ||·|| s ′ || ) . Given the sample-level kernel K ( l ) µ ( v, v ′ | G,G ′ ) in Theorem 4.1, if the graph convolutional layer of Eq. (3) has no bias term ( ς b = 0 ), the feature map of this kernel is given by

Corollary 4.2 shows that the feature map Ψ ( l ) v of sample-level kernel K ( l ) µ is a kernelized version of the graph convolutional layer in Eq. (3), which recursively aggregates message from nodes' neighborhood in the kernel space. The neighborhood can be automatically selected by optimizing the hyper-parameters α i and α ′ i .

Next, we show the implication of domain-level kernel K ( l ) ν ( G,G ′ ) . As discussed in Subsection 4.1, a graph G , empirically drawn from graph domain distribution P G , can approximate the true distribution P G when increasing the size of graph G . This motivates us to understand the connection between empirical domain representation ν ( l ) ( G ) and the embedding of probability distribution P G [19, 48].

Definition 4.3 (Mean Embedding of Probability Distributions [19]) . Given an RKHS H induced by a kernel k ( · , · ) , the mean embedding of a probability distribution P is defined as τ P = ∫ k ( · , x ) d P ( x ) , i.e., E x ∼ P [ f ( x )] = ⟨ f, τ P ⟩ H for all f ∈ H . Furthermore, the empirical mean embedding of P is given by ˆ τ P = 1 m ∑ m i =1 k ( · , x i ) , where m samples are independent and identically drawn from P .

Analogously, the mean embedding of a graph domain distribution P G can also be defined as τ P G = ∫ k G ( · , v | P G ) d P G ( v ) given an RKHS H G induced by a kernel k G ( · , · ) . In the context of graph learning, v | P G indicates the individual node attributes of v and graph structure information around node v induced by P G . It subsumes the conventional mean embedding [19] of a probability distribution τ P with independent and identically distributed (IID) samples x ∼ P , if there are no edges in the graph (i.e., no graph structure is involved around node v ). It is observed that the additional graph structure challenges empirical estimation of graph domain distribution P G , thereby resulting in a nontrivial domain similarity estimator between source and target graphs in transferable graph learning. The following corollary states that the graph representation ν ( l ) ( G ) can be considered as the empirical mean embedding of τ P G in the RKHS induced by K ( l ) ν ( · , · ) .

Corollary 4.4. With the same conditions in Theorem 4.1, for each l , in the limit on the layer width, the graph representation ν ( l ) ( G ) recovers the empirical mean embedding ˆ τ ( l ) P G of the domain distribution P G in the reproducing kernel Hilbert space induced by K ( l ) ν , and ˆ τ ( l ) P G is given by

where ˜ Ψ ( l ) v is the feature map of K ( l ) ν with ˜ Ψ ( l ) v = ˜ ς w ∑ k i =0 α i ∑ u ∈ N i ( v ) φ 1 ( ˜ Ψ ( l -1) u ) .

Corollary 4.4 shows that K ( l ) ν ( G,G ′ ) in Theorem 4.1 can be explained as the similarity of the empirical mean embeddings of domain distributions P G and P G ′ in the kernel space e.g., K ( l ) ν ( G,G ′ ) = ⟨ ˆ τ ( l ) P G , ˆ τ ( l ) P G ′ ⟩ H K ( l ) ν . A similar metric over distributions is the maximum mean discrepancy (MMD) [19]

defined over the distance of empirical mean embeddings, i.e., MMD ( l ) ν = || ˆ τ ( l ) P G -ˆ τ ( l ) P G ′ || H K ( l ) ν , where H K ( l ) ν denotes the RKHS induced by K ( l ) ν . MMDhas been widely applied to transfer learning [31, 32, 61] for measuring the distribution shift across image domains, under the covariate shift assumption [40] (i.e., source and target domains share conditional distribution p ( y | x ) but different marginal distributions p ( x ) ). In contrast, in this paper, we focus on measuring the domain distribution similarity for transferable graph learning.

4.2.2 Homophily vs. Heterophily

As illustrated in Section 3.2, there are two assumptions in graph learning [20, 68]: homophily and heterophily. We show the special cases of Theorem 4.1 in tackling homophily or heterophily based transferable graph learning. Given homophily source graph G and target graph G ′ where connected nodes have similar output values, it is revealed [20, 25, 59] that each node aggregates message from itself and its 1-order neighborhood, e.g., k = 1 and α 0 = α 1 = α ′ 0 = α ′ 1 = 1 . Then the sample-level kernel K ( l ) µ ( v, v ′ | G,G ′ ) ∝ ∑ u ∈{ v ∪ N 1 ( v ) } ∑ u ′ ∈{ v ′ ∪ N 1 ( v ′ ) } C ( l -1) uu ′ . In contrast, if source and target graphs follow heterophily that connected nodes are not similar in the output space, nodes might aggregate message from high-order neighborhood [1, 52, 68], e.g., k = 2 and α 0 = α 2 = 1 , α 1 = 0 . Then it holds that K ( l ) µ ( v, v ′ | G,G ′ ) ∝ ∑ u ∈{ v ∪ N 2 ( v ) } ∑ u ′ ∈{ v ′ ∪ N 2 ( v ′ ) } C ( l -1) uu ′ . These results indicate that by learning the neighborhood importances α i and α ′ i , Gaussian process led by f ( v, G ) can adaptively select the neighborhood for message aggregation from homophily or heterophily graphs. Moreover, α i = α ′ i implies that source and target graphs follow the same assumption. The flexibility of α i , α ′ i allows us to transfer knowledge across different types of graphs. For example, using k = 2 , α 0 = α 1 = 1 , α 2 = 0 , α ′ 0 = α ′ 2 = 0 , α ′ 1 = 0 , it enables the knowledge transferability from a homophily source graph to a heterophily target graph.

4.3 Proposed Algorithms

The goal of transferable node regression is to learn the prediction function for the target graph, by leveraging the input-output relationship from a relevant source graph. Given a source graph G s = ( V s , E s ) with fully labeled nodes (e.g., y v ∈ R is associated with each node v ∈ V s ) and a target graph G t = ( V t , E t ) with a limited number of labeled nodes (e.g., | V la t | ≪ | V s | where V la t ⊂ V t is the set of labeled target nodes), we propose the adaptive graph Gaussian process algorithm (termed as GraphGP ) as follows.

For notation simplicity, we let f v = f ( L ) ( v, G ) (given by L graph convolutional layers in Eq. (5)) be the function value at node v . Let f s = [ f 1 , f 2 , · · · , f | V s | ] T be a vector of latent function values over labeled source nodes, and y s = [ y 1 , y 2 , · , y | V s | ] T be the associated ground-truth output values. Similarly, we can define f t and y t over target nodes. Then, the GP prior over function values can be defined as f ∼ GP (0 , K ( L ) ( · , · )) (defined in Theorem 4.1), and its instantiation at labeled training nodes is given by p ( f | V s ∪ V la t ) = N ( 0 , K ( s + t )( s + t ) ) where K ( s + t )( s + t ) is a block

a, b ∈ { s, t } . For the likelihood, we consider the noisy scenarios where p ( y a | f a ) = N ( f a , ϱ 2 a I ) for a ∈ { s, t } ( ϱ a measures the noisy magnitude). Then, following standard Gaussian process [42], the posterior distribution of GraphGP over testing nodes V ∗ ⊂ V t has a closed-form expression, i.e., p ( f ∗ | V ∗ , f , V s ∪ V la t ) = N ( γ , Γ ) where

Here, y = [ y s y t la ] denotes the ground-truth output values of labeled nodes V s ∪ V la t from source and target graphs. Each entry of K ∗ represents the covariance between testing target node and training node, and K ∗∗ denotes the covariance matrix over testing target nodes.

The posterior distribution of GraphGP has the following hyper-parameters: σ w , ς w , ς b , α s i , α t j , and noise variances ϱ s , ϱ t . The hyper-parameters of the Gaussian process can be optimized by maximizing the marginal likelihood p ( y ) over all the training samples [42]. However, in the context of transferable graph learning, the number of labeled target nodes is much smaller than the number of labeled source nodes. By directly maximizing the marginal likelihood over all labeled source and target nodes, GraphGP might be biased towards the source domain. Therefore, in this paper, we propose to optimize the marginal distribution p ( y t la ) over labeled target nodes by considering all the nodes V s ∪ V la t as the inducing points [47] (see more efficiency analysis in Appendix A.6). The objective function is given by maximizing the following log marginal likelihood

4.4 Generalization Analysis

We define the generalization error of the target graph for transferable node regression. Given the ground-truth labeling function f ∗ in the target domain and the estimator f ( L ) learned from observed source and target graphs, the generalization error is given by

where γ ( · ) is the posterior mean of f ( L ) ( · ) in Eq. (6), L is the mean squared loss function, and p ( v t , G t ) denotes the sampling probability 2 of a target input ( v t , G t ) . It is shown [9, 42] that if the GP prior is correctly specified, i.e., the predictor f ( L ) has the same prior as f ∗ , the generalization error is given by the predictive variance of the GP. That is, ϵ t = ∫ Γ( v t , G t ) p ( v t , G t ) d ( v t , G t ) where Γ( v t , G t ) is the posterior variance at ( v t , G t ) in Eq. (6). Furthermore, the following theorem shows that the generalization error can be upper bounded in terms of the variance that takes all inputs ( v s , G s ) of the source graph as the target samples, i.e., both ( v s , G s ) and ( v t , G t ) are assumed to follow the same target distribution for single-domain graph learning.

Theorem 4.5. Let K µ ss be the sample-level covariance matrix over source nodes, i.e., [ K µ ss ] ( v s ,v ′ s ) = K ( L ) µ ( v s , v ′ s | G s ) , ν ss = K ( L ) ν ( G s , G s ) , ν tt = K ( L ) ν ( G t , G t ) be the intra-graph kernels and ν st = K ( L ) ν ( G s , G t ) be the inter-graph kernel. Suppose ¯ ϱ 2 s ∆ = ( ν ss · ν tt ν st · ν st -1 ) σ 2 w ν tt ¯ λ ss + ν tt · ν tt ν st · ν st ϱ 2 s where ¯ λ is the maximum eigenvalue of K µ , for any ( v , G ) the generalization error is bounded by ss ss t t

where Γ t ( v t , G t ; ¯ ϱ 2 s , ϱ 2 t ) is the variance assuming that all source examples are observed in the target domain with respect to noises ¯ ϱ 2 s , ϱ 2 t .

This theorem confirms that compared to the single-domain Gaussian process (assuming all inputs are drawn from the same distribution), GraphGP -based transferable graph learning enables the reduced generalization error. In addition, the following corollary considers a special scenario where no labeled nodes are available in the target graph. In this case, the generalization error is determined by the K ( L ) ν ( G s ,G t )

Corollary 4.6. When there are no labeled nodes in the target graph, i.e., V la t = ∅ , we have

where ν ss = K ( L ) ν ( G s , G s ) , ν tt = K ( L ) ν ( G t , G t ) and ν st = K ( L ) ν ( G s , G t ) .

Model

RBFGP [42]

DINO [56]

GGP [35]

SAGEGP [37]

GINGP [37]

GraphGP

Model

RBFGP

DINO

MixHopGP

H2GCNGP

GraphGP

→

RU

0.4241

±

0.6717

0.3699

0.5958

0.6803

0.7520

±

±

±

±

±

→

RU

0.4962

±

0.7641

0.5173

0.6921

0.6910

0.7909

±

±

±

±

±

PT

→

0.5507

EN

±

0.5920

0.0024

±

0.4237

±

0.5720

±

0.6349

±

0.6745

±

PT

0.0117

0.0140

0.0028

0.0029

0.0294

0.0109

ES

0.0024

0.0114

0.0087

0.0076

0.0198

0.0144

PT

0.0055

0.0082

0.0018

0.0174

0.0540

0.0382

0.0195

0.0037

0.0187

0.0090

0.0127

PT

→

0.4324

RU

±

0.6498

0.0087

±

0.3167

±

0.5655

±

0.5782

±

0.7069

±

0.0059

0.0079

0.0196

0.0218

0.0055

EN

→

0.5430

±

0.7565

±

0.5689

±

0.5191

±

0.7525

±

0.8013

±

Wikipedia graphs

SQ

→

CH

CH

→

-0.0383

±

0.2938

0.0526

±

0.4100

0.4050

0.4165

0.4938

WebKB graphs

CO

TX

WS

→

→

0.3089

±

0.3536

±

0.3836

±

0.3264

±

0.3816

0.0533

0.0442

0.0530

0.0569

0.0430

±

0.0253

±

±

±

±

±

0.0205

0.0456

0.0559

0.0352

-0.0119

SQ

±

0.1157

±

0.0838

0.0288

0.0145

±

0.3509

±

0.2652

±

0.0307

0.0102

0.0106

0.3214

±

0.4146

0.0402

0.0080

5 Experiments

Data Sets: We use the following graph learning benchmarks with node regression tasks. (1) Twitch [44]: It has 6 different domains ("DE", "EN", "ES", "FR", "PT", and "RU"). (2) Agriculture [34, 60]: It has 3 different domains ("Maize" (MA), "Sorghum" (SG), and "Soybean" (SY)). (3) Airports [43]: It has 3 different domains ("USA" (US), "Brazil" (BR), and "Europe" (EU)). (4) Wikipedia [44]: It has 3 different domains ("chameleon" (CH), "crocodile" (CR), and "squirrel" (SQ)). (5) WebKB [41]: It has 3 different domains ("Cornell" (CO), "Texas" (TX), and "Wisconsin" (WS)).

Baselines: We consider the following Gaussian process baselines. (1) RBFGP [42] and DINO [56] are feature-Only Gaussian processes without using graph structures. (2) GGP [35], SAGEGP [37], and GINGP [37] are graph Gaussian processes by considering source and target graphs as a large disjoint graph. (3) LINKXGP, MixHopGP, and H2GCNGP are graph Gaussian processes derived from LINKX [30], MixHop [1], H2GCN [68] respectively. It is notable that recent work [37] shows the equivalence between graph neural networks (e.g., GraphSAGE [20], GIN [59]) and graph Gaussian processes. Similarly, we can derive the corresponding graph Gaussian processes for LINKX [30], MixHop [1] and H2GCN [68] in Appendix A.5, which are termed as LINKXGP, MixHopGP, and H2GCNGP, respectively.

Model Configuration: In the experiments, we use GPyTorch [16] to build the graph Gaussian process models and optimize the hyperparameters with gradient descent optimizer. For the proposed GraphGP algorithm, we adopt k = 2 and L = 2 for all the experiments. The hyperparameters are optimized using Adam [24] with a learning rate of 0.01 and a total number of training epochs of 500. All the experiments are performed on a Windows machine with four 3.80GHz Intel Cores, 64GB RAM, and two NVIDIA Quadro RTX 5000 GPUs 3 .

5.1 Results

0.2756

±

0.2537

±

0.2898

±

0.3062

±

0.3036

±

0.3301

±



































2

R

ES

→

RU

0.4738

±

0.5636

0.0059

±

0.4237

±

0.5960

±

0.5997

±

0.6789

±

0.0072

0.0035

0.0074

0.0185

0.0172

TX

0.0204

0.0232

0.0176

0.0337

0.0442

0.0585

Table 1 and Table 2 provide the transferable node regression results on Airport, Agriculture, and Twitch graphs. Following [37], we report the coefficient of determination R 2 (mean and standard deviation with 5 runs) on the testing target nodes for performance comparison. The experimental results verify the effectiveness of GraphGP over Gaussian process baselines. It is notable that the graphs in Table 1 and Table 2 follow the homophily assumption. In contrast, Wikipedia and WebKB graphs in Table 3 hold the heterophily assumption [68]. It can be seen that under heterophily assumption,























R

2





























R

2









GraphGP also archives much better performance in most cases. In addition, we evaluate the knowledge transferability across graphs with different assumptions in Figure 3. It is observed that compared to GINGP (or H2GCNGP) over both labeled source and target nodes, GINGP-T (or H2GCNGP-T) learns the Gaussian process over only target nodes, thereby enabling much better transfer performance from WebKB to Airport in Figure 3a (or from Airport to WebKB in Figure 3b). Moreover, GraphGP can outperform those target-only graph Gaussian processes in this scenario.

We also investigate the correlation between transfer performance and normalized graph domain simi- larity K ( L ) ν ( G s ,G t ) √ K ( L ) ν ( G s ,G s ) · K ( L ) ν ( G t ,G t ) in Corollary 4.6. Figure 2 visualizes the estimated normalized graph

domain similarity of GraphGP and R 2 over the target testing nodes for RU → PT on Twitch, where the graph domain similarity changes when injecting noise to the target graph (see Appendix A.7.1). It shows that the transfer performance is positively correlated with the normalized graph domain similarity. This is consistent with our observation in Corollary 4.6.

5.2 Analysis

Flexibility of GraphGP : As illustrated in Subsection 4.1, it is flexible to instantiate structure-aware neural networks of Eq. (2) with existing message-passing graph neural networks. Table 4 provides the results of GIN [59] induced GraphGP algorithm. Furthermore, it is feasible to simply instantiate K ( L ) ν ( G,G ′ ) of GraphGP with existing graph kernels, e.g., Shortest Path kernel [5]. It is observed that the variants GraphGP _GIN and GraphGP _ShortestPath of

GraphGP achieve comparable performance. This highlights that GraphGP is flexible to incorporate with existing GNNs and graph kernels for transferable graph learning tasks.

Comparison with Transferable GNNs: In addition to Gaussian processes, we also compare GraphGP with state-of-theart transferable graph neural networks. Table 5 shows the competitive performance of GraphGP over transferable GNN baselines. This is because GraphGP explicitly maximizes the marginal likelihood p ( y t la ) .

In contrast, transferable GNN baselines minimize the prediction loss over all the labeled nodes ( | V s | ≫ | V la t | ) and domain discrepancy, and thus they are more likely to bias towards the source graph.

6 Conclusion

This paper studies the problem of transferable graph learning involving knowledge transfer from a source graph to a relevant target graph. To solve this problem, we propose a graph Gaussian process ( GraphGP ) algorithm, which is derived from a structure-aware neural network encoding both sample-level node representation and domain-level graph representation. The efficacy of GraphGP is verified theoretically and empirically in various transferable node regression tasks.

Acknowledgments and Disclosure of Funding

This work is supported by National Science Foundation under Award No. IS-1947203, IIS-2117902, IIS-2137468, IIS-2002540, and Agriculture and Food Research Initiative (AFRI) grant no. 202067021-32799/project accession no.1024178 from the USDA National Institute of Food and Agriculture. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.

References

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In International Conference on Machine Learning , pages 21-29. PMLR, 2019.

Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society , pages 337-404, 1950.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning , 79:151-175, 2010.

Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems , 2019.

Karsten M Borgwardt and Hans-Peter Kriegel. Shortest-path kernels on graphs. In Fifth IEEE international conference on data mining (ICDM'05) , pages 8-pp. IEEE, 2005.

Viacheslav Borovitskiy, Iskander Azangulov, Alexander Terenin, Peter Mostowsky, Marc Deisenroth, and Nicolas Durrande. Matérn gaussian processes on graphs. In International Conference on Artificial Intelligence and Statistics , pages 2593-2601. PMLR, 2021.

Bin Cao, Nathan N Liu, and Qiang Yang. Transfer learning for collective link prediction in multiple heterogenous domains. In International Conference on Machine Learning , pages 159-166, 2010.

Bin Cao, Sinno Jialin Pan, Yu Zhang, Dit-Yan Yeung, and Qiang Yang. Adaptive transfer learning. In proceedings of the AAAI Conference on Artificial Intelligence , volume 24, pages 407-412, 2010.

Kian Chai. Generalization errors and learning curves for regression with multi-task gaussian processes. Advances in Neural Information Processing Systems , 2009.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning , pages 1725-1735. PMLR, 2020.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations , 2021.

Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in Neural Information Processing Systems , 2009.

Valerio Ciotti, Moreno Bonaventura, Vincenzo Nicosia, Pietro Panzarasa, and Vito Latora. Homophily and missing links in citation networks. EPJ Data Science , 5:1-14, 2016.

Jinyuan Fang, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang. Gaussian process with graph convolutional kernel for relational learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 353-363, 2021.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research , 17(1):2096-2030, 2016.

Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. GPyTorch: blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems , 2018.

Floris Geerts, Filip Mazowiecki, and Guillermo Perez. Let's agree to degree: Comparing graph convolutional networks in the message-passing framework. In International Conference on Machine Learning , pages 3640-3649, 2021.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning , pages 1263-1272. PMLR, 2017.

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems , 30, 2017.

Jilin Hu, Jianbing Shen, Bin Yang, and Ling Shao. Infinitely wide graph convolutional networks: semi-supervised learning via gaussian processes. arXiv preprint arXiv:2002.12168 , 2020.

Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy transferability estimation. In International Conference on Machine Learning , pages 9201-9225, 2022.

Junteng Jia and Austion R Benson. Residual correlation in graph neural network regression. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 588-598, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as gaussian processes. In International Conference on Learning Representations , 2018.

Ron Levie, Wei Huang, Lorenzo Bucci, Michael Bronstein, and Gitta Kutyniok. Transferability of spectral graph convolutional neural networks. The Journal of Machine Learning Research , 22(1):12462-12520, 2021.

Naiqi Li, Wenjie Li, Jifeng Sun, Yinghua Gao, Yong Jiang, and Shu-Tao Xia. Stochastic deep gaussian processes over graphs. Advances in Neural Information Processing Systems , 33:5875-5886, 2020.

Xiang Li, Renyu Zhu, Yao Cheng, Caihua Shan, Siqiang Luo, Dongsheng Li, and Weining Qian. Finding global homophily in graph neural networks when meeting heterophily. In International Conference on Machine Learning , pages 13242-13256. PMLR, 2022.

Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. Advances in Neural Information Processing Systems , pages 20887-20902, 2021.

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning , pages 97-105. PMLR, 2015.

Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning , pages 2208-2217. PMLR, 2017.

Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of a feather: Homophily in social networks. Annual review of sociology , 27(1):415-444, 2001.

Christopher M Montes, Carolyn Fox, Álvaro Sanz-Sáez, Shawn P Serbin, Etsushi Kumagai, Matheus D Krause, Alencar Xavier, James E Specht, William D Beavis, Carl J Bernacchi, et al. High-throughput characterization, correlation, and mapping of leaf photosynthetic and functional traits in the soybean (glycine max) nested association mapping population. Genetics , 221(2), 2022.

Yin Cheng Ng, Nicolò Colombo, and Ricardo Silva. Bayesian semi-supervised learning with graph gaussian processes. Advances in Neural Information Processing Systems , 31, 2018.

Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. LEEP: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning , pages 7294-7305, 2020.

Zehao Niu, Mihai Anitescu, and Jie Chen. Graph neural network-inspired kernels for gaussian processes in semi-supervised learning. In International Conference on Learning Representations , 2023.

Felix Opolka and Pietro Lio. Bayesian link prediction with deep graph convolutional gaussian processes. In International Conference on Artificial Intelligence and Statistics , pages 4835-4852. PMLR, 2022.

Felix Opolka, Yin-Cong Zhi, Pietro Lio, and Xiaowen Dong. Adaptive gaussian processes on graphs via spectral graph wavelets. In International Conference on Artificial Intelligence and Statistics , pages 4818-4834. PMLR, 2022.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering , 22(10):1345-1359, 2010.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. In International Conference on Learning Representations , 2020.

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning . MIT Press, 2006.

Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node representations from structural identity. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining , pages 385-394, 2017.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks , 9(2), 2021.

Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the transferability of graph neural networks. Advances in Neural Information Processing Systems , 33:1702-1712, 2020.

Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research , 12(9), 2011.

Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. Advances in neural information processing systems , 2005.

Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert RG Lanckriet. Hilbert space embeddings and metrics on probability measures. The Journal of Machine Learning Research , 11:1517-1561, 2010.

Petru Tighineanu, Kathrin Skubch, Paul Baireuther, Attila Reiss, Felix Berkenkamp, and Julia Vinogradska. Transfer learning with gaussian processes for bayesian optimization. In International Conference on Artificial Intelligence and Statistics , pages 6152-6181. PMLR, 2022.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations , 2021.

Tao Wang, Di Jin, Rui Wang, Dongxiao He, and Yuxiao Huang. Powerful graph convolutional networks with adaptive propagation mechanism for homophily and heterophily. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 4210-4218, 2022.

Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. NTI, Series , 2(9):12-16, 1968.

Jun Wu and Jingrui He. Trustworthy transfer learning: Transferability and trustworthiness. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 5829-5830, 2023.

Jun Wu, Jingrui He, and Elizabeth Ainsworth. Non-IID transfer learning on graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10342-10350, 2023.

Jun Wu, Jingrui He, Sheng Wang, Kaiyu Guan, and Elizabeth Ainsworth. Distribution-informed neural networks for domain adaptation regression. Advances in Neural Information Processing Systems , 35:10040-10054, 2022.

Jun Wu, Jingrui He, and Jiejun Xu. DEMO-Net: Degree-specific graph neural networks for node and graph classification. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 406-415, 2019.

Man Wu, Shirui Pan, Chuan Zhou, Xiaojun Chang, and Xingquan Zhu. Unsupervised domain adaptive graph convolutional networks. In Proceedings of The Web Conference 2020 , pages 1457-1467, 2020.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019.

Craig R Yendrek, Tiago Tomaz, Christopher M Montes, Youyuan Cao, Alison M Morse, Patrick J Brown, Lauren M McIntyre, Andrew DB Leakey, and Elizabeth A Ainsworth. Highthroughput phenotyping of maize leaf physiological and biochemical traits using hyperspectral reflectance. Plant physiology , 173(1):614-626, 2017.

Wei Ying, Yu Zhang, Junzhou Huang, and Qiang Yang. Transfer learning via learning to transfer. In International Conference on Machine Learning , pages 5085-5094. PMLR, 2018.

Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in Neural Information Processing Systems , 31, 2018.

Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. LogMe: Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning , pages 12133-12143, 2021.

Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. Graph domain adaptation via theory-grounded spectral regularization. In International Conference on Learning Representations , 2023.

Kai Yu and Wei Chu. Gaussian process models for link analysis and transfer learning. Advances in Neural Information Processing Systems , 2007.

Yizhou Zhang, Guojie Song, Lun Du, Shuwen Yang, and Yilun Jin. DANE: domain adaptive network embedding. In IJCAI International Joint Conference on Artificial Intelligence , pages 4362-4368, 2019.

Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning , pages 7404-7413. PMLR, 2019.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems , 33:7793-7804, 2020.

Qi Zhu, Carl Yang, Yidan Xu, Haonan Wang, Chao Zhang, and Jiawei Han. Transfer learning of graph neural networks with ego-graph information maximization. Advances in Neural Information Processing Systems , 2021.

A Appendix

In the appendix, we provide more details regarding the proposed GraphGP for transferable graph learning in the paper, including

In Subsections A.1 and A.2, we discuss the broader impacts and limitations of this paper.

In Subsection A.3, we prove the theorems and corollaries stated in the paper.

In Subsection A.4, we discuss the instantiations of our GraphGP algorithm with existing graph neural networks.

In Subsection A.5, we derive the graph Gaussian Processes of existing Heterophilic GNNs, including LINKX [30], MixHop [1], H2GCN [68]. These derived Gaussian Processes are used as the baselines in our experiments (see Table 3).

In Subsection A.6, we discuss the model efficiency of GraphGP and provide a computationally efficient approximation solution for GraphGP based on Nyström approximation.

In Subsection A.7, we provide detailed experimental setups and additional evaluation results.

A.1 Broader Impacts

This paper focuses on the fundamental research problem of transferable graph learning. The goal is to effectively understand a target graph by leveraging latent knowledge from a relevant source graph. Generally, there are no negative societal impacts involved in this work.

A.2 Limitations

This paper develops a generic framework to build graph Gaussian processes for transferable graph learning under covariate shift assumption [40] over graphs. In addition to covariate shift, label shift (i.e., P Y shifts across domains) is also commonly considered in transfer learning scenarios. It is much more challenging to extend the developed transferable graph Gaussian processes to tackle label shift scenarios. This is because it is difficult to accurately estimate the graph domain distribution with limited label information from the target domain. In addition, following standard transfer learning [40], after model training, we simply apply the learned posterior function over testing target nodes for predicting their output values. It might be feasible to incorporate GraphGP with test-time adaptation techniques [51] to further improve the transfer performance.

A.3 Proof of Theorems and Corollaries

A.3.1 Proof of Theorem 4.1

Theorem 4.1 states that assume all the parameters of structure-aware graph neural network f ( v, G ) are independent and randomly drawn from Gaussian distributions, i.e., W ( l ) ∼ N ( 0 , σ 2 w I ) , b ( l ) SANN ∼ N ( 0 , ς 2 b I ) , W ( l ) SANN ∼ N ( 0 , ς 2 w I ) , ˜ b ( l ) SANN ∼ N ( 0 , ˜ ς 2 b I ) , ˜ W ( l ) SANN ∼ N ( 0 , ˜ ς 2 w I ) , when the layer width M goes to infinity, the output function f ( l ) i in Eq. (5) follows a Gaussian process with f ( l ) i ∼ GP (0 , K ( l ) ) , where the covariance function K ( l ) is given by

where

Here P ( i ) ( P ′ ( j ) ) denotes the adjacent matrix given by the i -order neighborhood from graph G ( j -order neighborhood from graph G ′ ). K ( l -1) ab = ∑ k i,j =0 α i α ′ j ( b 2 + a 2 P ( i ) C ( l -2) ( a, b )( P ′ ( j ) ) T ) and C (0) uu ′ ( a, b ) = ⟨ x u , x u ′ ⟩ for any u ∈ V, u ′ ∈ V ′ .

Proof. Given the definition of f ( v, G ) , we have

where x (0) u = ˜ x (0) u = x u denote the initial node attributes. The l -th layer of the function f ( v, G ) is defined as

Moreover,

where [ a ] i denotes the i -th entry of vector a . Besides,

which completes the proof.

A.3.2 Proof of Positive Definiteness

With the assumptions in Theorem 4.1, the covariance kernel K ( l ) is positive definite.

Proof. This corollary can be proven using the Schur Product theorem that the Hadamard product of two positive semidefinite matrices is also a positive semidefinite matrix. Based on the definition of ( v, v ′ | G,G ′ ) and K ( l ) ν ( G,G ′ ) , the covariance matrix of these kernels over all nodes in G,G ′ are positive semidefinite. Then as explained in [37], a kernel function is positive definite (resp. strictly positive definite) if the corresponding kernel matrix is positive semi-definite (resp. positive definite) for any collection of distinct points. Therefore, the kernel function K ( l ) (( v, G ) , ( v ′ , G ′ )) is symmetric and positive definite. Then based on Moore-Aronszajn theorem [2], K ( l ) defines a unique reproducing kernel Hilbert space (RKHS).

A.3.3 Proof of Corollary 4.2

Corollary 4.2 states let φ 1 : H → H 1 denote the kernel mapping from a pre-activation RKHS H to post-activation RKHS H 1 , i.e., ⟨ φ 1 ( s ) , φ 1 ( s ′ ) ⟩ = 1 2 || s || · || s ′ || · κ 1 ( ⟨ s,s ′ ⟩ || s ||·|| s ′ || ) . Given the sample-level kernel K ( l ) µ ( v, v ′ | G,G ′ ) in Theorem 4.1, if the graph convolutional layer of Eq. (3) has no bias term ( ς b = 0 ), the feature map of this kernel is given by 4

Proof. Based on Theorem A.1, we have

Then, with ς b = 0 , we have

Therefore, it holds that

This result provides the feature map for nodes from graph G and G ′ separately. But we show that it is equivalent to the following unified feature mapping on a large graph integrating G and G ′ .

More specifically, graphs G = ( V, E ) and G ′ = ( V ′ , E ′ ) can be considered as a single large graph G ′′ = ( V ∪ V ′ , E ∪ E ′ ) with two disjoint components. Then, for any node v ′′ ∈ V ′′ in graph G ′′ , we have

where v ′′ can be a node from either G or G ′ , and I [ a ] = 1 if an event a is true, I [ a ] = 0 otherwise.

A.3.4 Proof of Corollary 4.4

Corollary 4.4 states that with the same conditions in Theorem 4.1, for each l , in the limit on the layer width, the graph representation ν ( l ) ( G ) recovers the empirical mean embedding ˆ τ ( l ) P G of the domain distribution P G in the reproducing kernel Hilbert space induced by K ( l ) ν , given.

where ˜ Ψ ( l ) v is the feature map of K ( l ) ν with ˜ Ψ ( l ) v = ˜ ς w ∑ k i =0 α i ∑ u ∈ N i ( v ) φ 1 ( ˜ Ψ ( l -1) u ) .

Proof. We derive the feature map of domain-level kernel K ( l ) ν as follows. Similarly, we consider the case with ˜ ς b = 0 , we have

That is, given the node representation ˜ Ψ ( l ) v = ˜ ς w ∑ k i =0 α i ∑ u ∈ N i ( v ) φ 1 ( ˜ Ψ ( l -1) u ) in the kernel space, the mean pooling in the READOUT function (see Eq.(2)) results in the mean embedding of empirical node distribution over { ˜ Ψ ( l ) v } v ∈ V , i.e.,

Moreover, when the sample size of graph G goes to infinity, the expected mean embedding of node distribution over { ˜ Ψ ( l ) v } v ∈ V is given by

which corresponds to expression of graph domain distribution P G in the reproducing kernel Hilbert space induced by K ( l ) ν (given by the kernel mapping ˜ Ψ ( l ) v ).

A.3.5 Proof of Theorem 4.5

Theorem 4.5 states that let K µ ss be the sample-level covariance matrix over source nodes, i.e., [ K µ ss ] ( v s ,v ′ s ) = K ( L ) µ ( v s , v ′ s | G s ) , ν ss = K ( L ) ν ( G s , G s ) , ν tt = K ( L ) ν ( G t , G t ) be the intra-graph kernels and ν st = K ( L ) ν ( G s , G t ) be the inter-graph kernel. Suppose ¯ ϱ 2 s ∆ = ( ν ss · ν tt ν st · ν st -1 ) σ 2 w ν tt ¯ λ ss + ν tt · ν tt ν st · ν st ϱ 2 s where ¯ λ ss is the maximum eigenvalue of K µ ss , for any ( v t , G t ) the generalization error is bounded by

where Γ t ( v t , G t ; ¯ ϱ 2 s , ϱ 2 t ) is the variance assuming that all source examples are observed in the target domain with respect to noises ¯ ϱ 2 s , ϱ 2 t .

Proof. The covariance matrix K over training samples can be rewritten as

Then

Following [9], we have

where

Thus, it holds

Thus, the minimum of the upper bound is given by

A.3.6 Proof of Corollary 4.6

Corollary 4.6 states that when there are no labeled nodes in the target graph, i.e., V la t = ∅ , it holds

where ν ss = K ( L ) ν ( G s , G s ) , ν tt = K ( L ) ν ( G t , G t ) are intra-graph kernels and ν st = K ( L ) ν ( G s , G t ) is inter-graph kernel.

Proof. It is given

For all ( v t , G t ) , it holds

where Γ s ( v t , G t ) is the posterior variance by considering ( v t , G t ) as a virtual source sample. Then when the number of source samples goes to infinity, we have

where ν ss = K ( L ) ν ( G s , G s ) , ν tt = K ( L ) ν ( G t , G t ) denote the intra-graph kernels and ν st = K ( L ) ν ( G s , G t ) is the inter-graph kernel.

The generalization error is determined by graph domain similarity K ( L ) ν ( G s ,G t ) √ K ( L ) ν ( G s ,G s ) · K ( L ) ν ( G t ,G t ) .

A.4 Model Discussion

In this section, we discuss several instantiations of the proposed structure-aware neural networks in Eq. (2) and their induced graph Gaussian processes. This shows the flexibility of the proposed algorithms in incorporating existing graph neural networks and graph kernels.

A.4.1 Intantiations of Message-Passing Graph Neural Networks

Table 6 summarizes several instantiations of message-passing graph neural networks (GNNs) [18] of Eq. (1) with different neighborhood selection and message aggregation strategies.

It can be seen that homophily-based GNNs are more likely to aggregate messages from nearby neighbors, while heterophily-based GNNs have to adaptively aggregate messages from distant neighbors within the graph. These GNNs can be applied to design the proposed structure-aware neural networks in Eq. (2).

Thus,

Table 6: Instantiations of message-passing graph neural networks (* initial residual connection is adopted)

A.4.2 Instantiations of Structure-Aware Neural Networks

It can be seen that the definition of structure-aware neural network Eq. (2) is flexible to incorporate existing graph neural networks [10, 20, 25, 59] by instantiating µ ( l ) ( v | G ) and ν ( l ) ( G ) . For example, we can use Graph Isomorphism Network (GIN) [25] to define the structure-aware neural network Eq. (2) as follows. The graph convolutional layer of GIN is

̸

where a uv = 1 for u = v and a uv = 1 + ϵ ( l ) for u = v . W ( l ) GIN and b ( l ) GIN denote the weight and bias parameters in GIN, respectively. In this case, we use GIN to instantiate both µ ( l ) ( v | G ) and ν ( l ) ( G ) for learning h ( l ) v and ˜ h ( l ) v . In addition, the READOUT function of Eq. (2) can be instantiated with mean pooling [62], i.e., ˜ h ( l ) G = 1 | V | ∑ v ∈ V ˜ h ( l ) v . It is revealed [21, 37] that GIN is equivalent to the Gaussian process in the limit on the layer width. Following this observation, the following theorem shows the equivalence between structure-aware neural network and adaptive graph Gaussian process when using GIN to instantiate Eq. (2).

Theorem A.1. Suppose the graph convolutional layers of structure-aware neural network Eq. (2) are instantiated with GIN and the READOUT function of Eq. (2) is instantiated with mean pooling. Assume that the layer width the network width goes to infinity, and the model parameters of all neural layers are independently and randomly drawn from Gaussian distribution, then for each i and l , the output function f ( l ) i in Eq. (2) follows a Gaussian process with f ( l ) i ∼ GP (0 , K ( l ) ) , where the covariance function K ( l ) is given by

where

Here, a and a ′ T are the adjacency matrices with [ a ] uv = a uv and [ a ′ ] uv = a ′ uv respectively.

Proof. The sample covariance is given by

When learning the node representation, graphs G = ( V, E ) and G ′ = ( V ′ , E ′ ) can be considered as a single large graph G ′′ = ( V ∪ V ′ , E ∪ E ′ ) with two disjoint components. The adjacent matrix of G ′′ is then given by a ′′ = [ a 0 0 a ′ ] . In this case, learning node representation of v from G is equivalent to learning that from G ′′ , due to the disconnection of two components G and G ′ within the large graph G ′′ . Then, using the results from [37], given a single graph G ′′ , the outputs of each layer in GCN over all nodes are equivalent to the Gaussian process with zero mean and covariance matrix

Considering the block matrix form of C ( l -1) G ′′ = [ C ( l -1) G C ( l -1) ( C ( l -1) ) T C ( l -1) G ′′ ] ( C ( l -1) G ′′ is a symmetric matrix), we have

where

Therefore, it holds that

where [ B ] vv ′ denotes the entry of a matrix B at the v -th row and v ′ -th column.

In addition, the domain covariance is given by

which completes the proof.

Similarly, we can also design the adaptive graph Gaussian processes for other message-passing graph neural networks. More generally, we have the following observations.

Corollary A.2. Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. If µ ( l ) j ( v | G ) and ν ( l ) j ( G ) are Gaussian processes, i.e., µ ( l ) j ( v | G ) ∼ GP (0 , K ( l ) µ ) and ν ( l ) j ( G ) ∼ GP (0 , K ( l ) ν ) , then, for each i and l , the output function f ( l ) i ( v, G ) in Eq. (2) is a Gaussian process with f ( l ) i ( v, G ) ∼ GP (0 , K ( l ) ) , where the covariance matrix K ( l ) is given by

where v ∈ V and v ′ ∈ V ′ denote nodes within the graphs G and G ′ , respectively.

Proof. Since both µ ( l ) j ( v | G ) and ν ( l ) j ( G ) are Gaussian processes and the weight and bias parameters are independent and identically distributed (IID), the output f ( l ) i ( v, G ) at the l th layer can be considered as a sum of IID terms. Based on the Central Limit Theorem, in the limit of infinite network width N l →∞ , f ( l ) i ( v, G ) is Gaussian distributed. Moreover, using the multidimensional Central Limit Theorem, any finite collections { f ( l ) i ( v 1 , G ) , f ( l ) i ( v 2 , G ) , · · · , f ( l ) i ( v m , G ) } have a joint multivariate Gaussian distribution. Therefore, we conclude that f ( l ) i ( v, G ) forms a Gaussian process, i.e., f ( l ) i ( v, G ) ∼ GP ( ω ( l ) , K ( l ) ) . The mean function ω ( l ) is given by

and the covariance function is given by

Moreover, we see that for any i > 0 , f ( l ) i ( v, G ) can form a Gaussian process with identical mean and covariance functions.

Remark. Corollary A.2 shows that the transferable graph Gaussian process can be derived when both µ ( l ) j ( v | G ) ∼ GP (0 , K ( l ) µ ) and ν ( l ) j ( G ) ∼ GP (0 , K ( l ) ν ) are Gaussian processes. It is flexible to adopt different GNNs to define µ ( l ) j and ν ( l ) j . Moreover, optimizing the selection of kernel spaces over µ ( l ) j and ν ( l ) j might lead to better transferable graph learning performance. We would like to leave it as our future work.

A.5 Graph Gaussian Processes of Heterophilic GNNs

In Section 5, in order to tackle the knowledge transferability over heterophilic graphs, we compare the proposed GraphGP algorithm with LINKXGP, MixHopGP, and H2GCNGP derived from LINKX [30], MixHop [1], H2GCN [68] respectively. The graph convolutional layer of MixHop [1] is defined as

where || denotes column-wise concatenation. For example, Table 6 shows the MixHop with P = { 0 , 1 , 2 } , which corresponds to the neighbors from at most 2-order neighborhoods. The graph convolutional layer of MixHop [1] can also be rewritten as

where N j 1 ( v ) denotes the first-order neighborhood of node v under adjacent matrix ˆ A j . The following theorem shows the graph Gaussian process (termed as MixHopGP) derived from MixHop.

Corollary A.3 (Graph Gaussian Process of MixHop) . Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. Let h ( l ) v ( i ) be the i th entry of the output vector h ( l ) v . Then for any i , h ( l ) v ( i ) of MixHop over all nodes { v 1 , · · · , v | V | } ∈ V follows Gaussian process with h ( l ) v ( i ) ∼ GP (0 , K ( l ) MixHop ) where the covariance function K ( l ) MixHop is given by

where v, v ′ ∈ V are two nodes within graph G = ( V, E ) .

Similarly, we can show the graph Gaussian processes (terms as LINKXGP and H2GCNGP) for LINKX [30] and H2GCN [68] as follows.

Corollary A.4 (Graph Gaussian Process of LINKX) . Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. Let h ( l ) v ( i ) be the i th entry of the output vector h ( l ) v . Then for any i , h ( l ) v ( i ) of LINKX over all nodes { v 1 , · · · , v | V | } ∈ V follows Gaussian process with h ( l ) v ( i ) ∼ GP (0 , K ( l ) LINKX ) where the covariance function K ( l ) LINKX is given by

where C A , C X corresponds the NNGP kernel [26] induced by the node attributes and adjacent matrix respectively.

Proof. Following [26], the MLPs over node attributes and adjacent matrix are equivalent to Gaussian processes C A , C X . Then the graph Gaussian process of LINKX over C A , C X can be derived using y = MLP f (W[ h A || h X ] + h A + h X ) where h A , h X are adjacent and feature hidden representation respectively.

Corollary A.5 (Graph Gaussian Process of H2GCN) . Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. Let h ( l ) v ( i ) be the i th entry of the output vector h ( l ) v . Then for any i , the output layer h ( L ) v ( i ) of H2GCN over all nodes { v 1 , · · · , v | V | } ∈ V follows Gaussian process with h ( L ) v ( i ) ∼ GP (0 , K ( L ) H2GCN ) where the covariance function K ( L ) H2GCN is given by

where C (0) is NNGP kernel [26] induced by ego-feature embedding layers. A j H2GCN is the adjacent matrix of the j th neighborhood defined in H2GCN.

Proof. The jumping knowledge is leveraged in designing the H2GCN for heterophilic graphs. Thus, we show the graph Gaussian process of H2GCN at the output layer. The output layer is given by h final v = || j ∈{ 0 , 1 , ··· ,L } h ( l ) v . Following Corollary A.3, the Gaussian process of each layer can be derived and then combined in the output layer.

A.6 Efficiency Analysis

As illustrated in Subsection 4.3 and Algorithm 1, in the context of transferable graph learning, the hyperparameters of GraphGP can be optimized by maximizing the following log marginal likelihood

The crucial idea is to consider all the nodes V s ∪ V la t as the inducing points [47].

However, the objective function involves the inversion of the covariance matrix with time complexity O (( | V s | + | V la t | ) 3 ) , thereby resulting in the intractable computation of GraphGP in large graphs. To this end, we provide a computationally efficient approximation in both model training and inference. Following [37, 47], we choose a subset of landmarks V LM from the labeled source and target nodes. First, all the labeled target nodes are selected. Second, another q · | V la t | landmarks are randomly selected from the labeled source nodes. Thus, the landmarks V LM have ( q +1) · | V la t | nodes in total. In practice, a small value of q leads to ( q +1) · | V la t | ≪| V s | + | V la t | . Using the landmark nodes, the objective function of Eq. (7) can be approximated by

Algorithm 1 GraphGP

Input: Source graph G s = ( V s , E s ) with labeled nodes V s , target graph G t = ( V t , E t ) with labeled nodes V la t ⊂ V t , number of neighborhoods k , layers of structure-aware neural network L . Output: Predicted output values on target test nodes V ∗ ⊂ V t .

1: Initialize the hyper-parameters of GraphGP ;

2: while Stopping criterion is not satisfied do

3: Calculate the covariance matrices K t ( s + t ) , K ( s + t )( s + t ) ;

4: Maximize the log marginal likelihood of Eq. (7);

5: end while

6: Estimate the posterior distribution of Eq. 6;

7: Output the prediction results γ ( V ∗ , G t ) .

where K (LM)(LM) denotes the covariance matrix over landmarks, and K t (LM) is the covariance matrix between labeled target nodes and landmarks. Following [37], the Nyström approximation of the posterior distribution in GraphGP can also be calculated over the selected landmarks. The proposed approximation strategy would reduce the overall time complexity from O (( | V s | + | V la t | ) 3 ) to O ( | V s || V la t | 2 ) .

A.7 Additional Experiments

In this section, we provide more details and additional results for our experiments.

A.7.1 Setup

Data Sets: We use the following graph learning benchmarks with node regression tasks.

Twitch [44]: It has 6 different domains ("DE", "EN", "ES", "FR", "PT", and "RU"). In each domain, the graph represents the friendships among Twitch users, where each node corresponds to a Twitch user and each edge represents the mutual friendship of two users. Each node is also associated with an attribute vector encoding the game, location, and streaming habit information. Different from previous work [44] which considers the binary classification of whether a streamer uses explicit language, in this paper, we focus on the node regression task. Thus, we use the number of views in the original data 5 as the output value for each node/user.

Agriculture [34, 60]: It has 3 different domains ("Maize" (MA), "Sorghum" (SG), and "Soybean" (SY)). The goal of agriculture analysis is to predict diverse traits (e.g., specific leaf area) of plants related to the plants' growth using leaf hyperspectral reflectance. The knowledge transferability across different species allows us to effectively infer the biochemical traits in real scenarios. Each data point represents a plant associated with a 1901-dimensional feature vector (e.g., spectral wavelengths 500-2400 nm). Following [55], we use k -NN to build the graph based on the feature similarity.

Airports [43]: It has 3 different domains ("USA" (US), "Brazil" (BR), and "Europe" (EU)). Those domains involve the airport networks where each node corresponds to an airport and each edge represents the existence of commercial fights. Each node/airport is assigned a label based on their level of activity, measured in fights or people. Following [69], we use the degree-guided feature vector for each node.

Wikipedia [44]: It has 3 different domains ("chameleon" (CH), "crocodile" (CR), and "squirrel" (SQ)). As introduced in [44], those domain data are given by Wikipedia page-page networks on different topics: chameleons, crocodiles, and squirrels. Each node corresponds to an article and each edge represents mutual links of two articles. The node feature is given by the presence of particular nouns in the articles and the average monthly traffic. It aims to predict the log average monthly traffic.

WebKB [41]: It has 3 different domains ("Cornell" (CO), "Texas" (TX), and "Wisconsin" (WS)). It is collected from computer science departments of various universities by Carnegie

Mellon University. Each node corresponds to a web page, and each edge represents the hyperlink of two web pages. The node features are given by the bag-of-words representation of web pages. The web pages are manually classified into five categories, student, project, course, staff, and faculty. In contrast to previous works [41], we would like to consider the regression task where each node is associated with an output value from [0 , 4] , e.g., 4.0 for nodes from class 4.

The train/validation/test split in our experiments is given below. As discussed in Subsection 4.3, for transferable node regression, it is given a source graph G s = ( V s , E s ) with fully labeled nodes and a target graph G t = ( V t , E t ) with a limited number of labeled nodes (e.g., | V la t | ≪| V s | where V la t ⊂ V t is the set of labeled target nodes). Therefore, for Airport, Wikipedia, and WebKB data sets, we randomly select 10% of target nodes for the training set, 10% for the validation set, and 80% for the testing set. For Agriculture and Twitch data sets, we randomly select 1% of target nodes for the training set, 1% for the validation set, and 98% for the testing set.

In Subsection 5.1, we verify the positive correlation between transfer performance and graph domain similarity on Twitch (RU → PT). To this end, we gradually add some noise to the target graph such that the graph domain similarity between the source and target graphs can change accordingly. More specifically, given the feature vector (node attribute) x v of each node v , we simply multiply the feature vector by a constant δ 0 , i.e., ˆ x v = δ 0 · x v , where δ 0 denotes the perturbation magnitude. In the experiments, we choose δ 0 from 0.1 to 1 with the interval of 0.1.

Baselines: We consider the following Gaussian process baselines. (1) RBFGP [42] and DINO [56] are feature-Only Gaussian processes without using graph structures. (2) GGP [35], SAGEGP [37], and GINGP [37] are graph Gaussian processes by considering source and target graphs as a large disjoint graph. (3) LINKXGP, MixHopGP, and H2GCNGP are graph Gaussian processes derived from LINKX [30], MixHop [1], H2GCN [68] respectively.

RBFGP [42] is a standard Gaussian process algorithm designed for IID data. DINO [56] considers the transfer learning scenarios by designing the adaptive transfer kernel between source and target domains. By incorporating the graph structures, GGP [35], SAGEGP [37], and GINGP [37] design the graph convolutional kernels. For example, recent work [37] shows the equivalence between graph neural networks (e.g., GraphSAGE [20], GIN [59]) and graph Gaussian processes. Similarly, we can derive the corresponding Gaussian processes for LINKX [30], MixHop [1] and H2GCN [68] in Appendix A.5, which are termed as LINKXGP, MixHopGP, and H2GCNGP, respectively. It is notable that for a fair comparison, all the baselines are trained over both labeled source and target nodes. For example, graph Gaussian processes can take the input source and target graphs as a large disjoint graph. In this case, the distribution shift between source and target graphs is not considered.

A.7.2 Additional Results

Learned Weight α i : In Table 7, we report the learned weight α i of GraphGP in the target domain for different data sets. For homophilic graphs, e.g., EU → BR of Airport, we see that the node itself and its first-order neighborhood have higher importance weights. In contrast, for heterophilic graphs, e.g., CO → TX of WebKB, the first-order neighborhood is less important for transferable graph learning. Furthermore, when source and target graphs follow different assumptions, e.g., TX (WebKB) → BR (Airport) or EU (Airport) → TX (WebKB), the learned weights on the target graph can also capture the homophily or heterophily properties of the target graph.

Impact of L : We further investigate the impact of the number of layers L in the proposed GraphGP algorithm. As shown in Table 8, GraphGP achieves relatively better results when L ∈ { 2 , 3 , 4 } . This observation is also consistent with the over-smoothing issues of conventional graph neural networks [20, 59].

Efficiency Analysis: We investigate the computational efficiency of GraphGP and its approximation GraphGP -E . Table 9 shows the performance of GraphGP -E on the Twitch data set 6 . It can be seen that GraphGP -E consistently outperforms GINGP on transferable graph learning. Moreover, Table 10 shows the effectiveness and efficiency of GraphGP -E with different values of q on RU → PT of Twitch, where Full indicates the original GraphGP algorithm and the running time indicates the seconds per epoch. It can be seen that compared to GraphGP , GraphGP -E can achieve comparable performance and much higher computational efficiency.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a generic graph-structured Gaussian process framework (GraphGP) to investigating the knowledge transferability between homophilic and heterophilic graphs. GraphGP uses a structure-aware neural network to encode local node representation and global graph representation (domain-level) simultaneously. In addition, a simple neighborhood selection strategy is designed to tackle the knowledge transferability in homophily and heterophily graphs. Experimental results demonstrate superior performance compared to baseline models.\n\nSTRENGTHS:\n(1) Concerning on the problem of transferable graph learning over non-IID graph data is interesting.\n\n(2) The paper is clearly written and well formulated.\n\n(3) This paper theoretically discussed the expressive power of GraphGP and analyze the knowledge transferability across graphs from the perspective of graph domain similarity.\n\nWEAKNESSES:\n(1) In the methodology section, the algorithm flow description of GraphGP should be clearer. It is recommended to add some brief descriptions of the algorithm flow, draw the algorithm table, and draw the overall framework figure.\n\n(2) algorithm complexity analysis and convergence analysis should be described more specific.\n\n(3) The theoretical analysis of the algorithm is sufficient but the experiment is weak. In order to verify the generalization of the model, more transfer learning tasks should be designed in the experiment, such as classification tasks.\n\n(4) The experimental part should contain several real-world datasets to verify the effectiveness of the algorithm.\n\nQUESTIONS:\n(1) There are many hyperparameters in the GraphGP algorithm, and whether the performance of the model is sensitive to these hyperparameters?\n\n(2) There are many network structures for graph representation learning in existing research. Why the author uses message-passing graph neural network to extract graph features? Can he be well applied in the task of transfer learning? Can it be well applied to transfer learning tasks?\n\n(3) What are the limitations of the proposed methods?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deal with the ttransferable graph learning problem, especially between the homophily and heterophily graphs. To solve this problem, they propose a graph Gaussian process (GraphGP) algorithm, which is derived from a structure-aware neural network encoding both sample-level node representation and domain-level graph representation. The effectiveness of GraphGP is verified both theoretically and experimentally in various transferable node regression tasks.\n\nSTRENGTHS:\n1.\tComprehensive theoretical analysis.\n2.\tClear problem definition.\n3.\tNew techniques to consider the transfer learning between homophily and heterophily graphs.\n\nWEAKNESSES:\n1.\tThis paper has poor organization/presentation. There are two problem formulation and several proposed methods, not clear which is the major contribution. It is also not clear what is the relationship between the different mentioned techniques in this paper.\n2.\tSome comparison experiments are missing.\n\nQUESTIONS:\n1.\tIn introduction part, the author says “However, most existing works followed the IID assumption”. However, this seems not correct,  since graph data are not IID, and the existing transfer learning for graphs should also consider non-IID assumption. Could the author give more explanations on this?\n2.\tIn line 52, the author mentions “structure-aware node”. It is better to provide the explanation of what kind of node can be named as “structure-aware”.\n3.\tIn line 88, they author claims that “However, most existing works focus on either investigating the transferability of a pre-trained graph neural network”. It is not clear why the current existing transfer learning techniques for pretrained model cannot solve the problem formulated in this paper. This is a very important question to judge the contribution of the proposed techniques.\n4.\tIn equation 2, what “i” and “j” refers to? And what is the meaning of “layer width”?\n5.\tIn Section 4, the author introduces several methods, including “Structure-Aware Neural Network”. It is not clear if this method is proposed by author? If it is, the novelty is largely limited, since it is just the typical existing message passing network with a different theoretical analysis. \n6.\tThe author formalized problems both in Section 3.1 and 4.3. It is confusing that which one is the goal of this paper.\n7.\tThis paper only compares the proposed model with Gaussian models, while the existing graph transfer learning techniques are not compared.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
The paper studies transferable graph learning over non-IID graph data. In order to adapt the knowledge from source graphs to target graphs, the paper proposes a graph-structured Gaussian Process (GraphGP). The GraphGP is derived from a structure-aware neural network and due to the flexibility of the hyperparameters, GraphGP is able to transfer knowledge across different types of graphs, such as homophily graphs and heterophily graphs. Experimental results on five datasets show that the proposed GraphGP achieve better performance than Gaussian Process baselines.

STRENGTHS:
1. The motivation of the proposed GraphGP is clear and the idea of using the Gaussian Process to address the graph transfer learning problem is novel and interesting. 

2. Theoretical analyses are provided to justify the rationale of the proposed method. 

3. Codes are provided for reproducibility.

WEAKNESSES:
1. It is not very clear from the paper how to implement the kernel function $K^{(L)}$ of GraphGP in the experiments. What kind of kernel function is used in the experiments?

2. The proposed GraphGP can only be used in the regression task. It seems that it is not easy to adapt the model to other graph learning tasks such as the classification task. 

3. The paper only uses Gaussian Process models as baselines. Why not compare against some graph neural network models which are considered to be more effective in graph learning tasks?  

4. The statistics of experimental datasets are missing. It’s unclear how the proposed GraphGP performs when dealing with graphs of different sizes.

QUESTIONS:
Please see the questions in the Weaknesses section.

LIMITATIONS:
The authors have discussed the limitations and broader impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of transferable graph learning involving knowledge transfer from a source graph to a relevant target graph. To solve this problem, the authors propose a graph Gaussian process (GraphGP) algorithm, which is derived from a structure-aware neural network encoding both sample-level node representation and domain-level graph representation. The efficacy of GraphGP is verified theoretically and empirically in various transferable node regression tasks.\n\nSTRENGTHS:\n1. This paper is well-organized and the presentation is good. \n2. The authors propose a generic graph-structured Gaussian process framework, which  encodes local node representation (sample-level) and global graph representation (domain-level) simultaneously.\n3. The proposed framework tackles the knowledge transferability in homophily and heterophily graphs using a simple neighborhood selection strategy.\n\nWEAKNESSES:\n1. Related work is inadequate. Graph transfer learning has also been studied in some important literature [1,2,3], but they are not discussed in this paper and should be adopted as baselines to compare with the proposed method.\n2. The experimental part lacks the study of each component involved in the method, and the contribution of the proposed component to this paper cannot be proved, such as local node representation (sample-level), global graph representation (domain-level) and the neighborhood selection strategy.\n3. The caption of Table 5 is too brief. It should contains more information to explain the table content.\n\n[1] Zhu, Qi, et al. \"Transfer learning of graph neural networks with ego-graph information maximization.\" Advances in Neural Information Processing Systems 34 (2021): 1766-1779.\n[2] Han, Xueting, et al. \"Adaptive transfer learning on graph neural networks.\" Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.\n[3] Wu, Jun, Jingrui He, and Elizabeth Ainsworth. \"Non-IID Transfer Learning on Graphs.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 9. 2023.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nIn addition to covariate shift, label shift is also commonly considered in transfer learning scenarios. It is much more challenging to extend the developed transferable graph Gaussian processes to tackle label shift scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}