Title: Learning Invariant Representations of Graph Neural Networks via Cluster Generalization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph neural networks

network representation learning

deep learning
Abstract: Graph neural networks (GNNs) have become increasingly popular in modeling graph-structured data due to their ability to learn node representations by aggregating local structure information. However, it is widely acknowledged that the test graph structure may differ from the training graph structure, resulting in a structure shift. In this paper, we experimentally  find that the performance of GNNs drops significantly when the structure shift happens, suggesting that the learned models may be biased towards specific structure patterns. To address this challenge, we propose the Cluster Information Transfer (\textbf{CIT}) mechanism, which can learn invariant representations for GNNs, thereby improving their generalization ability to various and unknown test graphs with structure shift. The CIT mechanism achieves this by combining different cluster information with the nodes while preserving their cluster-independent information. By generating nodes across different clusters, the mechanism significantly enhances the diversity of the nodes and helps GNNs learn the invariant representations. We provide a theoretical analysis of the CIT mechanism, showing that the impact of changing clusters during structure shift can be mitigated after transfer. Additionally, the proposed mechanism is a plug-in that can be easily used to improve existing GNNs. We comprehensively evaluate our proposed method on three typical structure shift scenarios, demonstrating its effectiveness in enhancing GNNs' performance.

Learning Invariant Representations of Graph Neural Networks via Cluster Generalization

Abstract

Graph neural networks (GNNs) have become increasingly popular in modeling graph-structured data due to their ability to learn node representations by aggregating local structure information. However, it is widely acknowledged that the test graph structure may differ from the training graph structure, resulting in a structure shift. In this paper, we experimentally find that the performance of GNNs drops significantly when the structure shift happens, suggesting that the learned models may be biased towards specific structure patterns. To address this challenge, we propose the Cluster Information Transfer ( CIT ) mechanism 2 , which can learn invariant representations for GNNs, thereby improving their generalization ability to various and unknown test graphs with structure shift. The CIT mechanism achieves this by combining different cluster information with the nodes while preserving their cluster-independent information. By generating nodes across different clusters, the mechanism significantly enhances the diversity of the nodes and helps GNNs learn the invariant representations. We provide a theoretical analysis of the CIT mechanism, showing that the impact of changing clusters during structure shift can be mitigated after transfer. Additionally, the proposed mechanism is a plug-in that can be easily used to improve existing GNNs. We comprehensively evaluate our proposed method on three typical structure shift scenarios, demonstrating its effectiveness in enhancing GNNs' performance.

1 Introduction

Graphs are often easily used to model individual properties and inter-individual relationships, which are ubiquitous in the real world, including social networks, e-commerce networks, citation networks. Recently, graph neural networks (GNNs), which are able to effectively employ deep learning on graphs to learn the node representations, have attracted considerable attention in dealing with graph data [13, 24, 32, 36, 9, 5]. So far, GNNs have been applied to various applications and achieved remarkable performance, e.g. , node classification [13, 32], link prediction [33, 14] and graph classification [6, 34].

Message-passing mechanism forms the basis of most graph neural networks (GNNs) [13, 24, 5, 9]. That is, the node representations are learned by aggregating feature information from the neighbors in each convolutional layer. So it can be seen that the trained GNNs are highly dependent on the local structure. However, it is well known that the graph structure in the real world always changes. For instance, the paper citations and areas would go through significant change as time goes by in citation network [11]. In social networks, nodes represent users and edges represent activity between users, because they will be changed dynamically, the test graph structure may also change [7]. This change in the graph structure is a form of distribution shift, which we refer to as graph structure shift. So

the question naturally arises: when the test graph structure shift happens, can GNNs still maintain stability in performance?

We present experiments to explore this question. We generate graph structure and node features. Subsequently we train GNNs on the initially generated graph structure and gradually change the structures to test the generalization of GNNs (more details are in Section 2). Clearly, the performance consistently declines with changes (shown in Figure 1), implying that the trained GNNs are severely biased to one typical graph structure and cannot effectively address the structure shift problem.

This can also be considered as Out-Of-Distribution problem (OOD) on graph. To ensure good performance, most GNNs require the training and test graphs to have identically distributed data. However, this requirement cannot always hold in practice. Very recently, there are some works on graph OOD problem for node classification. One idea assumes knowledge of the graph generation process and derives regularization terms to extract hidden stable variables [28, 8]. However, these methods heavily rely on the graph generation process, which is unknown and may be very complex in reality. The other way involves sampling unbiased test data to make their distribution similar to that of the training data [37]. However, it requires sampling test data beforehand, which cannot be directly applied to the whole graph-level structure shift, e.g. , training the GNNs on a graph while the test graph is another new one, which is a very typical inductive learning scenario [24, 17].

Therefore, it is still technically challenging to learn the invariant representations which is robust to the structure shift for GNNs. Usually, the invariant information can be discovered from multiple structure environments [28], while we can only obtain one local structure environment given a graph. To avoid GNNs being biased to one structure pattern, directly changing the graph structure, e.g. , adding or removing edges, may create different environments. However, because the graph structure is very complex in the real world and the underlying data generation mechanism is usually unknown, it is very difficult to get the knowledge on how to generate new graph structures.

In this paper, we propose to learn the invariant representations of GNNs by transferring the cluster information of the nodes. First, we apply GNNs to learn the node representations, and then combine it with spectral clustering to obtain the cluster information in this graph. Here, we propose a novel Cluster Information Transfer (CIT) mechanism, because the cluster information usually captures the local properties of nodes and can be used to generate multiple local environments. Specifically, we characterize the cluster information using two statistics: the mean of cluster and the variance of cluster, and transfer the nodes to new clusters based on these two statistics while keeping the cluster-independent information. After training GNNs on these newly generated node representations, we aim to enhance their generalization ability across various test graph structures by improving the generalization ability on different clusters. Additionally, we provide insights into the transfer process from the embedding space and theoretically analyze the impact of changing clusters during structure shift can be mitigated after transfer. The contributions of our work are summarized as follows:

We study the problem of structure shifts for GNNs, and propose a novel CIT mechanism to improve the generalization ability of GNNs. Our proposed mechanism is a friendly plug-in, which can be easily used to improve most of the current GNNs.

Our proposed CIT mechanism enables that the cluster information can be transferred while preserving the cluster-independent information of the nodes, and we theoretically analyze that the impact of changing clusters during structure shift can be mitigated after transfer.

We conduct extensive experiments on three typical structure shift tasks. The results well demonstrate that our proposed model consistently improves generalization ability of GNNs on structure shift.

2 Effect of structure distribution shift on GNN performance

In this section, we aim to explore the effect of structure shift on GNN performance. In the real world, the tendency of structure shift is usually unknown and complex, so we assume a relatively simple scenario to investigate this issue. For instance, we train the GNNs on a graph with two community structures, and then test the GNNs by gradually changing the graph structures. If the performance of GNNs drops sharply, it indicates that the trained GNNs are biased to the original graph, and cannot be well generalized to the new test graphs with structure shift.

We generate a network with 1000 nodes and divide all nodes into two categories on average, that is, 500 nodes are assigned label 0 and the other 500 nodes are assigned label 1. Meanwhile, node features, each of 50 dimensions, are generated by two Multivariate Gaussian distributions. The node features with same labels are generated by same Gaussian distribution. We employ the Stochastic Blockmodel (SBM) [12] to generate the graph structures. We set two community and set the generation edges probability on inter-community is 0.5% and intro-community is 0.05% . We randomly sample 20 nodes per class for training, and the rest are used for testing. Then we train GCN [13] and GAT [24] on this graph. The test graph structures are generated as follows: we decrease the inter-community edge probability from 0.5%

GNN

GCN

0.95

GAT

APPNP

0.90

0.85

0.80

0.75

0.70

0.5 0.05

0.45 0.1

0.4 0.15

0.350.2

0.3 0.25

0.25 0.3

Probability ofedges

to 0.25% and increase the intra-community probability from 0.05% to 0.3%.

The accuracy is shown in Figure 1. The x-axis is the probability of edges, where the first number is the edge probability of inter-community and the second number is the edge probability of introcommunity. As we can see, because of the structure shift, the performance of GNN declines significantly. It shows that once the test graph pattern shifts, the reliability of GNNs becomes compromised.

3 Methods

Notations: Let G = ( A , X ) represent a training attributed graph, where A ∈ R n × n is the symmetric adjacency matrix with n nodes and X ∈ R n × d is the node feature matrix, and d is the dimension of node features. Specifically, A ij = 1 represents there is an edge between nodes i and j , otherwise, A ij = 0 . We suppose each node belongs to one of C classes and focus on semi-supervised node classification. Considering that the graphs always change in reality, here we aim to learn the invariant representations for GNNs, where the overall framework is shown in Figure 2 and the whole algorithm is shown in A.1.

3.1 Clustering process

We first obtain the node representations through GNN layers:

where Z ( l ) is the node representations from the l -th layer, Z (0) = X , ˜ A = A + I , ˜ D is the degree matrix of ˜ A , σ is a non-linear activation and W ( l -1) GNN is the trainable parameters of GNNs. Eq. (1) implies that the node representations will aggregate the information from its neighbors in each layer, so the learned GNNs are essentially dependent on the local structure of each node. Apparently, if the structure shift happens in the test graph, the current GNNs may provide unsatisfactory results. Therefore, it is highly desired that the GNNs should learn the invariant representations while structure changing, so as to handle the structure shift in test graphs.

One alternative way is to simulate the structure change and then learn the invariant representations from different structures, which is similar as domain generalization [26, 35]. Motivated by this, we consider that the local properties of a node represents its domain information. Meanwhile, different clusters can capture different local properties in a graph, so we can consider cluster information as domain information of nodes. Based on this idea, we first aim to obtain the cluster information using spectral clustering [2]. Specifically, we compute the cluster assignment matrix S of node using a multi-layer perceptron (MLP) with softmax on the output layer:

GraphNeuralNetworks

Input

z(）.….z()

Z'(l)

→Lf =Le(fe(Z'(l),r)

↑

Cluster InformationTransfer(CIT)mechanism

Every k epochs

H(H)

Lu=Lc+AL

T

H(H)

Spectral clustering

Transfercluster

Newrepresentations

where W MLP and b are trainable parameters of MLP. For assignment matrix S ∈ R n × m , and m is the number of clusters. s ij represents the weight that node i belongs to cluster j . The softmax activation of the MLP guarantees that the weight s ij ∈ [0 , 1] and S T 1 M = 1 N .

For S , we want to cluster strongly connected nodes. So we adopt the cut loss which evaluates the mincut given by S :

where Tr is the trace of matrix. Minimizing L c encourages the nodes which are strongly connected to be together.

However, directly minimizing L c will make the cluster assignments are equal for all nodes, which means all nodes will be in the same cluster. So in order to avoid the clustering collapse, we can make the the clusters to be of similar size, and we use orthogonality loss to realize it:

where ∥ · ∥ F indicates the Frobenius norm, M is the number of clusters. Notably, when S T S = I M , the orthogonality loss L o will reach 0. Therefore, minimizing it can encourage the cluster assignments to be orthogonal and thus the clusters can be of similar size.

Overall, we optimize the clustering process by these two losses:

where λ 1 is a coefficient to balance the mincut process and orthogonality process.

After obtaining the assignment matrix S , we can calculate cluster representations H c by average the node representations Z ( l ) :

3.2 Transfer cluster information

Now we obtain the cluster representations H c , and each cluster captures the information of local properties. Originally, given a graph, a node can be only in one domain, i.e. , one cluster. Next, we aim to transfer the node to different domains. Specifically, the cluster information can be characterized by two statistics, i.e. , the center of the cluster ( H c ) and the standard deviation of the cluster ( σ ( H c k ) ):

where H c k is the representation of the k -th cluster and Z ( l ) i is the representation of node i .

Then we can transfer the node i in the k -th cluster to the j -th cluster as follows:

where k is the cluster that node i belongs to, and j is the cluster which is randomly selected from the remaining clusters.

Here, we explain the Eq. (8) in more details. As shown in Figure 3, the center of cluster k is H c k , and node i belongs to this cluster, where the representation of node i is Z ( l ) i . It can be seen that H c k is obtained by aggregating and averaging the local structure information, which captures the cluster information. Therefore, Z ( l ) i -H c k represents the cluster-independent information. Then, the CIT mechanism can be seen as we transfer the node to a new position from embedding space. The standard deviation is the weighted distance of nodes from the center, which is the aggregating scope of the clusters. After the transfer, the target node surrounds a new cluster with new cluster information, while keeping the cluster-independent information.

The above process only transfers the nodes on the original domain. Moreover, in order to improve the robustness of the model for unknown domain, we increase the uncertainty and diversity of model [16, 21]. Based on this, we add Gaussian perturbations to this process. The whole transfer becomes:

The statistics of Gaussian is determined by the whole features:

Here, with the Gaussian perturbations, we can generate new clusters based on the original one and the nodes can be further transferred to more diverse domains.

The proposed transfer process offers a solution to overcome the limitations of changing graph structures when generating nodes in different domains. Traditional approaches require changing the graph structure by adding or deleting edges, which can be challenging due to the lack of prior knowledge on how the changes in graph structure may affect the domain. Here, our proposed CIT mechanism addresses this challenge by directly transferring a node to another domain through the manipulation of cluster properties of nodes in the embedding space. The CIT method is implemented before the classifier in GNNs training, making it backbone agnostic and compatible with any GNN last layer.

3.3 Objective function

With CIT mechanism, we have the generated representation Z ′ ( l ) i in other clusters for node i . In traditional GNNs training, we input the representation Z ( l ) , where the i -th row vector represents the representation of node i , to the cross-entropy loss [13]. Here, we randomly select a part of nodes, and replace original representation Z ( l ) i with the generated representation Z ′ ( l ) i . Then we can obtain the new representation matrix Z ′ ( l ) , and optimize the cross-entropy loss based on it as follows:

where L e is cross-entropy loss function, f θ is the classifier, and Y is the labels.

Finally, the overall optimization function is as follows:

where λ 2 is a coefficient to balance the classification process and clustering process. In the optimization process, we randomly select n × p nodes and then conduct Eq. (9) every k epochs.

3.4 Theoretical analysis

In this section, we theoretically analyze our CIT mechanism from the perspective of domain adoption theory. Since the Eq. (8) is the cornerstone of our CIT mechanism and Eq. (9) is an extending from it, we use Eq. (8) to represent our CIT mechanism in the following proof for convenience.

We analyze that the classifier using the new generated representation Z ′ ( l ) in Eq. (8) has better generalization on the structure shift. As mentioned before, cluster information can capture local properties of nodes, so we convert this problem to analyzing the classifier has better generalization on cluster shift. To achieve this, following [31], we take the binary classification task as the example, and analyze the relationship between the decision boundary with cluster information. For analysis, we follow [31] using a fisher classifier and analyse the relationship between them. According to Fisher's linear discriminant analysis [1, 23, 3], the decision boundary of fisher classifier depends on two statistics V ar ( Z ) and Cov ( Z, Y ) .

Theorem 1. The decision boundary of fisher classifier is affected by the cluster information.

The proof is given in Appendix A.3, where we calculate the mathematical expression of the classification boundary. Theorem 1 indicates that the cluster information will 'help' the classifier to make decisions. But when the graph structure changes, the cluster information of the nodes also changes. Therefore, the classifier using cluster information to make decisions is not reliable.

Theorem 2. Let Z R represent the node representations in cluster R . Assume that there are p percent of nodes are transferred from cluster R to cluster D by Σ D Z R -µ R Σ R + µ D . After the transfer, the impact of changing clusters during structure shift can be mitigated.

The proof is given in Appendix A.4. Comparing the expression of classifier before and after our CIT mechanism, we can find that we mitigate the impact of changing clusters during structure shift, enhancing the robustness of the model against such changes.

4 Experiment

Datasets and baselines. To comprehensively evaluate the proposed CIT mechanism, we use six diverse graph datasets. Cora, Citeseer, Pubmed [20], ACM, IMDB [27] and Twitch-Explicit [17]. Details of datasets are in Appendix B.1. Our CIT mechanism can be used for any other GNN backbones. We plug it in four well-known graph neural network methods, GCN [13], GAT [24], APPNP [9] and GCNII [5]. Meanwhile, we compare it with two graph OOD methods which are also used for node-classification, SR-GNN [37] and EERM [28]. We combine them with four methods mentioned before. Meanwhile, we use CIT-GNN(w/o) to indicate that no Gaussian perturbations are added.

Experimental setup. For Cora, Citeseer and Pubmed, we construct structure shift by making perturbations on the original graph structure. For ACM and IMDB, we test them using two relation structures. For Twitch-Explicit, we use six different networks. We take one network to train, one network to validate and the rest of networks to test. So we divide these datasets into three categories: perturbation on graph structures, Multiplex networks and Multigraph. The implementation details are given in Appendix B.1.

4.1 Perturbation on graph structures data

In this task, we use Cora, Citeseer and Pubmed datasets. We train the model on original graph. We create new graph structures by randomly adding 50%, 75% and deleting 20%, 50% edges of original graph. To comprehensively evaluate our model on random structure shift, we test our model on the

new graphs. We follow the original node-classification settings [13] and use the common evaluation metrics, including Macro-F1 and classification accuracy. For brief presentation, we show results of deleting edges in Appendix B.2.

The results are reported in Table 1. From the table we can see that the proposed CIT-GNN generally achieves the best performance in most cases. Especially, for Acc and Macro-f1, our CIT-GNN achieves maximum relative improvements of 5.35% and 4.1% respectively on Citesser-Add-0.75. The results demonstrate the effectiveness of our CIT-GNN. We can also see that CIT-GNN improves the four basic methods, so the results show that our method can improve the generalization ability of the basic models. Meanwhile, our mechanism is to operate the node representation at the embedding level, which can be used for any GNN backbones.

4.2 Multiplex networks data

In this task, we use Multiplex networks datasets ACM and IMDB to evaluate the capabilities of our method on different relational scenarios. Both of them have two relation structures. We construct structure shift by taking the structure of one relation for training and the other for testing respectively. The partition of data follows [27]. The results are reported in Table 2 and the relation structure shown in the table is the training structure. Different from the first experiment, the structure shift is not random, which is more intense because the new graph is not based on the original graph. As can be seen, our proposed CIT-GNN improves the four basic methods in most cases, and outperforms

RoC-Auc

Method

ACM

IMDB

PAP

PLP

MDM

MAM

Acc

Macro-f1

Acc

Macro-f1

Acc

Macro-f1

Acc

Macro-f1

GCN

64.65±1.91

60.66±1.88

80.26±1.98

79.71±1.94

52.36±1.40

48.55±1.60

58.98±1.11

57.01±1.72

SR-GCN

67.75±1.20

68.51±1.10

82.14±2.10

81.88±2.32

51.94±0.97

50.76±0.85

59.84±1.08

58.21±1.11

EERM-GCN

66.85±1.87

67.84±1.54

82.16±1.87

82.16±1.67

54.07±1.83

51.80±1.62

57.21±1.93

56.52±1.66

CIT-GCN(w/o)

67.53±1.52

65.32±1.93

81.30±1.58

80.98±1.82

53.67±1.79

50.71±1.56

57.93±1.32

56.23±1.24

CIT-GCN

68.06±1.13

∗∗

68.79±1.27

∗∗

82.67±1.55

∗

82.56±1.66

∗

55.42±1.88

∗∗

52.75±1.65

∗∗

56.68±1.45

54.66±1.97

GAT

66.35±1.81

64.23±2.25

82.48±1.73

82.50±1.65

51.59±1.13

48.26±1.27

58.64±1.87

57.72±1.94

SR-GAT

67.20±1.87

67.89±2.13

84.61±1.34

84.49±1.65

50.81±1.92

46.62±1.73

59.05±1.57

57.42±1.84

EERM-GAT

67.67±1.17

68.22±1.25

79.25±1.27

78.84±0.99

52.24±1.28

50.86±1.35

58.20±1.65

57.25±1.59

CIT-GAT(w/o)

67.15±1.23

67.34±1.43

83.45±1.54

83.01±1.46

53.91±0.96

∗

51.90±1.21

∗

57.18±1.55

56.82±1.23

CIT-GAT

68.49±1.32

∗

68.15±1.39

∗∗

85.75±1.76

∗∗

85.64±1.32

∗∗

52.86±0.98

51.06±1.01

59.51±1.73

58.40±1.46

APPNP

78.49±1.33

79.00±1.56

86.76±1.22

86.74±1.85

51.78±1.01

46.57±1.32

62.01±1.21

61.34±1.56

SR-APPNP

77.60±1.28

76.25±1.77

86.16±1.37

86.16±1.52

55.02±1.98

51.74±2.03

60.70±1.08

60.14±1.32

80.89±1.82

80.43±1.65

83.58±1.58

83.46±1.84

54.32±0.96

51.03±1.07

61.27±1.75

60.30±1.87

CIT-APPNP(w/o)

81.66±1.12

81.35±1.01

86.60±0.98

86.52±0.87

56.37±1.41

∗∗

54.13±1.78

∗∗

61.81±0.92

60.98±1.10

CIT-APPNP

81.70±1.58

∗∗

81.60±1.47

∗

87.19±1.21

∗

87.16±1.02

∗

55.86±1.54

52.32±1.73

62.50±1.45

62.00±1.54

77.92±1.64

76.73±1.77

81.88±1.05

81.21±1.32

52.71±1.86

47.08±1.77

52.45±2.01

47.61±2.35

SR-GCNII

78.91±1.73

78.77±1.76

84.32±0.89

83.22±1.01

53.52±1.56

49.87±1.36

54.20±2.33

49.00±1.98

78.82±1.23

79.24±1.76

83.81±1.21

83.61±1.02

53.56±1.23

50.27±1.42

54.32±1.98

49.70±1.87

CIT-GCNII(w/o)

78.10±1.54

78.80±1.64

84.85±1.41

84.62±1.54

53.42±2.01

49.10±1.58

52.30±1.87

48.01±2.10

CIT-GCNII

79.73±1.61

∗

79.26±1.32

∗∗

85.23±1.93

∗∗

85.06±1.89

∗∗

54.20±1.88

∗

50.91±1.95

∗

53.69±1.89

48.84±2.01

(a) DE

(b) ENGB

(c) ES

(d) RU

Original

CTT(w/o)

Original

CTT(w/o)

Original

CTT(w/o)

Original

CTT(w/o)

EERM

CIT

EERM

CIT

EERM

CIT

EERM

CIT

73

65

69

58

64

68

S

71

Roc-Auc

DC-AuC

67

Roc-Auc

E9

66

56

70

9

65

55

69

64

68

61

63

54

67

GCN

GAT

APPNP

09

GCN

GAT

APPNP

62

GCN

GAT

APPNP

53

GCN

GAT

APPNP

SR-GNN and EERM, implying that our CIT-GNN can improve the generalization ability of the basic models.

4.3 Multigraph data

In this task, we use Multigraph dataset Twitch-Explicit, and its each graph is collected from a particular region. In this dataset, the node features also change but they still share the same input feature space and output space. The graphs are collected from different regions. So different structures are determined by different regions. To comprehensively evaluate our method, we take the FR for training, TW for validation, and test our model on the remaining four graphs (DE, ENGB, RS, EU). We choose ROC-AUC score for evaluation because it is a binary classification task. Since SR-GNN is not suitable for this dataset, only the base model and EERM are compared. The results are reported in Figure 4. Comparing the results on four graph, our model makes improvement in most cases, which verifies the effectiveness about generalizing on multigraph scenario.

4.4 Analysis of hyper-parameters

Analysis of p . The probability of transfer p determines how many nodes to transfer every time. It indicates the magnitude of the CIT mechanism. We vary its value and the corresponding results are shown in Figure 5. With the increase of p , the performance goes up first and then declines, which indicates the performance benefits from an applicable selection of p .

Analysis of k . We make transfer process every k epochs. We vary k from 1 to 50 and and plot the results in Figure 6. Zero represents the original model accuracy without out CIT mechanism. Notably, the performance remains consistently stable across the varying values of k , indicating the robustness of the model to this parameter.

(a) cora-add-0.5

(b) citeseer-add-0.5

CIT-GCN

CIT-APPNP

CIT-GCN

CIT-APPNP

CIT-GAT

CIT-GCNII

CIT-GAT

CIT-GCNII

0.82

0.74

0.80

0.72

0.78

0.70

0.76

0.66

99'0

0.74

0.64

0.72

0.62

0.70

0

0.05

The Probability of transfer

0.1

0.15

0.2

0.25

E°0

0.60

0.05

The Probability of transfer

0.1

0.15

0.2

0.25

E'0

Analysis of m . The number of clusters is the most important parameter in the clustering process. We choose silhouette coefficient to measure the clustering performance and vary m from 20 to 100 unevenly. Then we calculate the accuracy and Silhouette Coefficient. The corresponding results are shown in Figure 7. As we can see, accuracy changes synchronously with silhouette coefficient. We infer that the performance of our model is related to the clustering situation, and when the clustering process performs well, our model also performs well.

5 Related work

Graph neural networks. Recently, Graph Neural Networks have been widely studied. GCN [13] proposes to aggregate the node features from the one-hop neighbors. GAT [24] designs an attention mechanism to aggregate node features from neighbors. PPNP [9] utilizes PageRank's node propagation way to aggregate node features and proposes an approximate version. GCNII [5] extends GCN by introducing two effective techniques: initial residual and identity mapping, which make the network deeper. This is a fast growing research field, and more detailed works can be found in [4, 22].

OOD generalization of GNNs. Out-Of-Distribution (OOD) on graph has attracted considerable attention from different perspectives. For node-classification, [8] shows node selection bias drastically affects the performance of GNNs and investigates it from point view of causal theory. [37] explores the invariant relationship between nodes and proposes a framework to account for distributional differences between biased training data. [28] handles it by minimizing the mean and variance of risks from multiple environments which are generated by adversarial context generators. For graph classification, [15] proposes to capture the invariant relationships between predictive graph structural information and labels in a mixture of latent environments. [29, 18] find invariant subgraph structures from a causal perspective to improve the generalization ability of graph neural networks. [10] builds a graph OOD bench mark including node-level and graph-level methods and two kinds of distribution shift which are covariate shift and concept shift.

Graph clustering with graph neural networks. As graph neural networks continue to perform better in modeling graph data, some GNN-based graph clustering methods have been widely applied. Deep attentional embedded graph clustering [25] uses an attention network to capture the importance of the neighboring nodes and employs the KL-divergence loss in the process of graph clustering. [2] achieves the graph clustering process by obtaining the assignment matrix through minimizing optimizing its spectral objective.[19] uses a new objective function for clustering combining graph spectral modularity maximization and a new regularization method.

6 Conclusion

In this paper, we explore the impact of structure shift on GNN performance and propose a CIT mechanism to help GNNs learn invariant representations under structure shifts. We theoretically analyze that the impact of changing clusters during structure shift can be mitigated after transfer.

(a) cora

(b) citeseer

CIT-GCN

CIT-APPNP

-X--

CIT-GCN

CIT-APPNP

CIT-GAT

CIT-GCNII

CIT-GAT

CIT-GCNII

0.82

0.74

0.80

0.72

0.78

0.70

0.76

0.68

0.66

0.74

0.64

0.72 -

0.62

0.70

0

The Epochtimes

5

10

20

50

0.60

0

The Epochtimes

5

10

20

50

(a) cora

(b) citeseer

0.80

0.40

0.70

0.40

0.35

0.35

0.78

0.30

0.68

0.30

0.76

0.25

0.66

0.25

0.20

0.20

0.74

0.15

0.64

0.15

0.72

0.10

0.62

0.10

0.05

0.05

0.70

20

40

50

100

0.00

0.60

20

40

50

100

0.00

The number ofclusters

The number ofclusters

Moreover, the CIT mechanism is a friendly plug-in, and the comprehensive experiments well demonstrate the effectiveness on different structure shift scenarios.

Limitations and broader impact. One potential limitation lies in its primary focus on node-level tasks, while further investigation is needed to explore graph-level tasks. Although our CIT mechanism demonstrates significant advancements, certain theoretical foundations remain to be fully developed. Our work explores the graph from the perspective of the embedding space, thereby surpassing the limitations imposed by graph topology, and offers a fresh outlook on graph analysis.

Acknowledgments and Disclosure of Funding

This work is supported in part by the National Natural Science Foundation of China (No. U20B2045, 62192784, U22B2038, 62002029, 62172052, 62322203).

References

Theodore Wilbur Anderson. An introduction to multivariate statistical analysis. Technical report, Wiley New York, 1962.

Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In International Conference on Machine Learning , pages 874-883. PMLR, 2020.

T Tony Cai and Linjun Zhang. A convex optimization approach to high-dimensional sparse quadratic discriminant analysis. The Annals of Statistics , 49(3):1537-1568, 2021.

Hongxu Chen, Hongzhi Yin, Tong Chen, Quoc Viet Hung Nguyen, Wen-Chih Peng, and Xue Li. Exploiting centrality information with graph convolutions for network representation learning. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) , pages 590-601. IEEE, 2019.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning , pages 1725-1735. PMLR, 2020.

Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. arXiv preprint arXiv:1912.09893 , 2019.

Shobeir Fakhraei, James Foulds, Madhusudana Shashanka, and Lise Getoor. Collective spammer detection in evolving multi-relational social networks. In Proceedings of the 21th acm sigkdd international conference on knowledge discovery and data mining , pages 1769-1778, 2015.

Shaohua Fan, Xiao Wang, Chuan Shi, Kun Kuang, Nian Liu, and Bai Wang. Debiased graph neural networks with agnostic label selection bias. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribution benchmark. arXiv preprint arXiv:2206.08452 , 2022.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Brian Karrer and Mark EJ Newman. Stochastic blockmodels and community structure in networks. Physical review E , 83(1):016107, 2011.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of-distribution generalization. In Advances in Neural Information Processing Systems , 2022.

Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling-Yu Duan. Uncertainty modeling for out-of-distribution generalization. arXiv preprint arXiv:2202.03958 , 2022.

Derek Lim, Xiuyu Li, Felix Hohne, and Ser-Nam Lim. New benchmarks for learning on non-homophilous graphs. arXiv preprint arXiv:2104.01404 , 2021.

Siqi Miao, Mia Liu, and Pan Li. Interpretable and generalizable graph learning via stochastic attention mechanism. In International Conference on Machine Learning , pages 15524-15543. PMLR, 2022.

Emmanuel Müller. Graph clustering with graph neural networks. Journal of Machine Learning Research , 24:1-21, 2023.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Yichun Shi and Anil K Jain. Probabilistic face embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6902-6911, 2019.

Damien Teney, Lingqiao Liu, and Anton van Den Hengel. Graph-structured representations for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9, 2017.

T Tony Cai and Linjun Zhang. High dimensional linear discriminant analysis: optimality, adaptive algorithm and missing data. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 81(4):675-705, 2019.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Chun Wang, Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, and Chengqi Zhang. Attributed graph clustering: A deep attentional embedding approach. arXiv preprint arXiv:1906.06532 , 2019.

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering , 2022.

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In The world wide web conference , pages 2022-2032, 2019.

Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. arXiv preprint arXiv:2202.02466 , 2022.

Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. arXiv preprint arXiv:2201.12872 , 2022.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning , pages 25407-25437. PMLR, 2022.

Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems , 33:17009-17021, 2020.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems , 31, 2018.

Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization in vision: A survey. arXiv preprint arXiv:2103.02503 , 2021.

Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021 , pages 1215-1226, 2021.

Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems , 34:27965-27977, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to handle the structure shift problem in GNN. The paper proposed a novel approach -- Cluster Information Transfer (CIT) to address this challenge. CIT first computes cluster center representation during training. CIT then extract the cluster center from the node embedding to retrieve the cluster-independent information. Finally CIT transfers the cluster-independent information to another random cluster center, allowing the model to learn structure-independent knowledges.\n\nSTRENGTHS:\n+ Previous works like EERM, SR-GNN majorly focus on data distribution shifts on graphs, while this paper focus on graph structural shift which is of potential research interest.\n+ The method proposed in the paper can be applied on various existing GNN backbones, the method can also fit into different type of tasks and graphs.\n+ This paper is well-written and is easy to follow.\n\nWEAKNESSES:\n- The CIT method proposed in the paper is not well supported by theorem.  Theoretical analysis only shows that CIT weakens the affect of cluster information. However, the paper still need to answer why it is necessary to transfer nodes across clusters using Eq.8 (If the cluster information is harmful, why not just remove it?) and why can the model learn invariant representations from this process. \n- The method introduces a lot of additional hyper-parameters which are sensitive to the results, which are all tuned by grid-search according to the experiments settings. Finally, the accuracy improvements in the experiments is not remarkable compared with other baselines.\n- The method is only evaluated on small graphs with several thousand nodes, making it unknown whether the method can be applied to larger graphs.\n\nQUESTIONS:\nQ1：What is the relationship between invariant representations and cluster information and why is transferring a node to another cluster reasonable ?\nQ2: Can CIT possibly be scaled to larger graphs?\n\nLIMITATIONS:\nThe reviewer does not see potential negative social impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new mechanism called Cluster Information Transfer (CIT) to improve the generalization ability of GNNs to various and unknown test graphs with structure shift. The CIT mechanism enhances the diversity of nodes by combining different cluster information with the nodes, which helps GNNs learn invariant representations. The paper presents experimental results on several benchmark datasets, demonstrating that the proposed CIT-GNN model outperforms existing GNN models in terms of accuracy and macro-F1 score.\n\nSTRENGTHS:\n- The paper is easy to understand.\n- While the theory analysis is straightforward, it provides theoretically support for the proposed method.\n- The experiments are extensive and the results look promising.\n\nWEAKNESSES:\n- **Fair Novelty:** The proposed method is essentially an embedding transformation function that aims to eliminate the statistic variance between clusters that causes bias and hinders generalization. The novelty comes from the clustering mechanism and the augmented training using adjusted node embeddings. However, the spectral clustering mechanism has been introduced previously, and the transformation is relatively straightforward.\n- **Assumptions and scope of the solution:** My second concern is about the assumptions of this paper. The paper generally assumes that there exist cluster / structural shifts between training and testing datasets. While structure could change across environments, the node labels remain invariant. The paper also implicitly assumes the node embeddings in different clusters are subject to Multivariate Gaussian distributions. These assumptions should be made explicit and formal. Moreover, the scope of structural shifts needs to be clarified. I understand that the variant of graph density or node degree can be one factor, but do other factors, e.g., edge homophily, also contribute to the shifts? A more general scope should be discussed in order to ensure its applicability.\n- **More validation about the claim:** The examples in the introduction considers the shifts between sparse and dense graphs. One simple baseline to mitigate it is to augment the datasets or clusters via DropEdge and AddEdge compared to modifying node embeddings. While I understand that the graph structure is complex in the real world, as the authors claimed (Line 57-59), it would be good to experimentally validate the effectiveness of the method with a simple baseline that modifies the graph structure.  \n- **Lack of real datasets with natural shifts**: The experiments are somewhat synthetic in the sense that the training and testing sets are splitted with strong structural shifts. I wonder the performance of this method on datasets with more natural shifts, e.g., ogbn-arxiv, where the training and testing sets are splitted based on the time difference.\n- **Missing Related Works:** It is unclear how the proposed clustering objective differs from the existing group clustering methods, e.g., Zhou et al. [1].\n\n**Minor:**\n\n- I feel Section 2 is not very informative since the setting is simple and the conclusion is not surprising. It would be good to demonstrate the structural shifts in more real settings or just combine this section to the introduction.\n- In Line139 and 149, $m$ and $M$ are both used to indicate the number of clusters. It's better to make it consistent.\n\nIn general, the angle of mitigating structure shifts is interesting. However, several things above should be solved or clarified to understand its generality and effectiveness.\n\n\n[1] Towards Deeper Graph Neural Networks with Differentiable Group Normalization. Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, Xia Hu.\n\nQUESTIONS:\n- The method considers the embedding space only. Thus, it seems that the method can be extended to other domains, e.g., for images. If that is the case, are there any graph-specific properties that make this solution remains on the graph domain rather than other general domains?\n- What's the method's performance when removing/adding 5%, 20% edges and not removing/adding any edges? Does the improvement of the method decline with the decreasing ratios?\n\nLIMITATIONS:\nThe authors discussed the limitations on the fact that the method could only apply to node-level tasks directly and the theory remain to be fully developed, as well as the limitation that the transfer remains on embedding space but graph topology.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper focuses on the invariant learning of graph neural networks and proposes a cluster information transfer mechanism with two statistics: the mean of cluster and the variance of cluster. The authors prove that with CIT mechanism, the model is able to capture the cluster independent information, so as to improve the generalization ability across different structure shifts. Experiments on different graph benchmarks show promising results of CIT mechanism.

STRENGTHS:
Introduction and background parts are clearly written and motivate this study well. The research problem is important, the ideas are clearly clarified and all the technical steps are easy to follow. The paper provides the comprehensive experiments on different kinds of graphs to evaluate the performance of the proposed model.

WEAKNESSES:
No specific discussions on structure shift. Time complexity analysis is not provided. The CIT mechanism is not explained sufficiently.

QUESTIONS:
1. It seems that most of the graphs used here are not large enough, so what is the time complexity of the proposed model?
3. The authors design several experiments to evaluate the model, e.g., perturbate edges, multiplex graphs. Do all these changes belong to the structure shift? Can the authors provide more discussions on what changes can be considered as structure shift?
4. The paper proves that with CIT mechanism, the cluster information has less influence on the classifier, but what if the cluster information is related with the labels?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The paper tackles an important question of learning invariant representations of GNNs. The authors show that once the test graph pattern shifts, the reliability of GNNs becomes compromised. Then they propose the cluster information transfer mechanism, which can be easily combined with current GNNs to improve their robustness on structure shift. The authors present numerical experiments that suggest that the proposed mechanism is effective.

STRENGTHS:
1.	I enjoyed reading the paper. The presentation is clear and mostly easy to follow. 

2.	Clear results on different scenarios, well-backed by experiments.

3.	The CIT mechanism is interesting and with technical analysis.

WEAKNESSES:
1.	In section 2, why GAT performs worse than GCN?

2.	In fig.2, the model uses the node representation learned by GNNs to obtain clusters, so does the representation learned by different layers affect the clustering process?

3.	The authors mentioned that the graph OOD benchmark is built, so why the datasets used in the experiments are still the traditional graphs?

QUESTIONS:
It would be beneficial to answer the questions in weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel Cluster Information Transfer (CIT) mechanism to enhance the generalization capability of Graph Neural Networks (GNNs) in the presence of structural shifts. The authors provide theoretical analysis, showing that the impact of cluster information on the classifier diminishes during transfer. They demonstrate that the CIT mechanism can be easily incorporated into existing models and showcase its effectiveness through extensive experiments. Overall, the paper highlights the value of the CIT mechanism in improving GNN generalization and supports its claims with comprehensive empirical evidence.\n\nSTRENGTHS:\n- The paper includes both theoretical analysis and numerical results, providing a well-rounded evaluation of the proposed approach.\n- The idea presented in the paper is clear and well-motivated, addressing an important problem.\n- The experiments conducted in the paper are extensive and provide convincing evidence to support the claims made by the authors.\n\nWEAKNESSES:\n- The experiments in section 2 are not sufficient.\n- The descriptions of the experimental settings are unclear and could benefit from more detailed and precise explanations.\n- There are some typos present in the paper that should be addressed for improved clarity and readability.\n\nQUESTIONS:\n- In Section 2, the authors focus on presenting results specifically for the GCN and GAT models, but they do not mention the evaluation of other GNN models. It would be beneficial for the authors to clarify if they have considered or conducted experiments with other GNN models as well.\n- The authors introduce different structure shifts in their experiments, but it would be helpful to provide a clearer distinction between their approach and the OOD graph benchmarks proposed in [1]. Additionally, the paper could benefit from providing standard settings for structure shifts and including information on the label rate used in training the models.\n- There are several typos in the paper, such as \"less affect\" in line 321, \"bulids\" in line 315, and \"Initial residual and Identity mapping two effective techniques on GCN\" in line 305. These errors should be addressed to improve the clarity and accuracy of the paper.\n\n[1] Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribution benchmark. arXiv preprint arXiv:2206.08452, 2022.\n\nLIMITATIONS:\nThe authors have discussed the limitation and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}