SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The results are reported in Table 1. From the table we can see that the proposed CIT-GNN generally achieves the best performance in most cases. Especially, for Acc and Macro-f1, our CIT-GNN achieves maximum relative improvements of  $5.35\%$  and  $4.1\%$  respectively on Citesser-Add-0.75. The results demonstrate the effectiveness of our CIT-GNN. We can also see that CIT-GNN improves the four basic methods, so the results show that our method can improve the generalization ability of the basic models. Meanwhile, our mechanism is to operate the node representation at the embedding level, which can be used for any GNN backbones.
4.2 Multiplex networks data
In this task, we use Multiplex networks datasets ACM and IMDB to evaluate the capabilities of our method on different relational scenarios. Both of them have two relation structures. We construct structure shift by taking the structure of one relation for training and the other for testing respectively. The partition of data follows [27]. The results are reported in Table 2 and the relation structure shown in the table is the training structure. Different from the first experiment, the structure shift is not random, which is more intense because the new graph is not based on the original graph. As can be seen, our proposed CIT-GNN improves the four basic methods in most cases, and outperforms
Table 2: Quantitative results  $(\% \pm \sigma)$  on node classification for multiplex networks data and the relation in table is for training, while the superscript refers to the results of paired t-test between original model and CIT-GNN (* for 0.05 level and ** for 0.01 level).
Method	ACM	IMDB
PAP	PLP	MDM	MAM
Acc	Macro-f1	Acc	Macro-f1	Acc	Macro-f1	Acc	Macro-f1
GCN	64.65±1.91	60.66±1.88	80.26±1.98	79.71±1.94	52.36±1.40	48.55±1.60	58.98±1.11	57.01±1.72
SR-GCN	67.75±1.20	68.51±1.10	82.14±2.10	81.88±2.32	51.94±0.97	50.76±0.85	59.84±1.08	58.21±1.11
EERM-GCN	66.85±1.87	67.84±1.54	82.16±1.87	82.16±1.67	54.07±1.83	51.80±1.62	57.21±1.93	56.52±1.66
CIT-GCN(w/o)	67.53±1.52	65.32±1.93	81.30±1.58	80.98±1.82	53.67±1.79	50.71±1.56	57.93±1.32	56.23±1.24
CIT-GCN	68.06±1.13**	68.79±1.27**	82.67±1.55*	82.56±1.66*	55.42±1.88**	52.75±1.65**	56.68±1.45	54.66±1.97
GAT	66.35±1.81	64.23±2.25	82.48±1.73	82.50±1.65	51.59±1.13	48.26±1.27	58.64±1.87	57.72±1.94
SR-GAT	67.20±1.87	67.89±2.13	84.61±1.34	84.49±1.65	50.81±1.92	46.62±1.73	59.05±1.57	57.42±1.84
EERM-GAT	67.67±1.17	68.22±1.25	79.25±1.27	78.84±0.99	52.24±1.28	50.86±1.35	58.20±1.65	57.25±1.59
CIT-GAT(w/o)	67.15±1.23	67.34±1.43	83.45±1.54	83.01±1.46	53.91±0.96*	51.90±1.21*	57.18±1.55	56.82±1.23
CIT-GAT	68.49±1.32*	68.15±1.39**	85.75±1.76**	85.64±1.32**	52.86±0.98	51.06±1.01	59.51±1.73	58.40±1.46
APPNP	78.49±1.33	79.00±1.56	86.76±1.22	86.74±1.85	51.78±1.01	46.57±1.32	62.01±1.21	61.34±1.56
SR-APPNP	77.60±1.28	76.25±1.77	86.16±1.37	86.16±1.52	55.02±1.98	51.74±2.03	60.70±1.08	60.14±1.32
EERM-APPNP	80.89±1.82	80.43±1.65	83.58±1.58	83.46±1.84	54.32±0.96	51.03±1.07	61.27±1.75	60.30±1.87
CIT-APPNP(w/o)	81.66±1.12	81.35±1.01	86.60±0.98	86.52±0.87	56.37±1.41**	54.13±1.78**	61.81±0.92	60.98±1.10
CIT-APPNP	81.70±1.58**	81.60±1.47*	87.19±1.21*	87.16±1.02*	55.86±1.54	52.32±1.73	62.50±1.45	62.00±1.54
GCNII	77.92±1.64	76.73±1.77	81.88±1.05	81.21±1.32	52.71±1.86	47.08±1.77	52.45±2.01	47.61±2.35
SR-GCNII	78.91±1.73	78.77±1.76	84.32±0.89	83.22±1.01	53.52±1.56	49.87±1.36	54.20±2.33	49.00±1.98
EERM-GCNII	78.82±1.23	79.24±1.76	83.81±1.21	83.61±1.02	53.56±1.23	50.27±1.42	54.32±1.98	49.70±1.87
CIT-GCNII(w/o)	78.10±1.54	78.80±1.64	84.85±1.41	84.62±1.54	53.42±2.01	49.10±1.58	52.30±1.87	48.01±2.10
CIT-GCNII	79.73±1.61*	79.26±1.32**	85.23±1.93**	85.06±1.89**	54.20±1.88*	50.91±1.95*	53.69±1.89	48.84±2.01
(b) ENGB
(c) ES
Figure 4: ROC-AUC on Twitch where we compare different GNN backbones.
(d) RU
SR-GNN and EERM, implying that our CIT-GNN can improve the generalization ability of the basic models.
4.3 Multigraph data
In this task, we use Multigraph dataset Twitch-Explicit, and its each graph is collected from a particular region. In this dataset, the node features also change but they still share the same input feature space and output space. The graphs are collected from different regions. So different structures are determined by different regions. To comprehensively evaluate our method, we take the FR for training, TW for validation, and test our model on the remaining four graphs (DE, ENGB, RS, EU). We choose ROC-AUC score for evaluation because it is a binary classification task. Since SR-GNN is not suitable for this dataset, only the base model and EERM are compared. The results are reported in Figure 4. Comparing the results on four graph, our model makes improvement in most cases, which verifies the effectiveness about generalizing on multigraph scenario.
4.4 Analysis of hyper-parameters
Analysis of  $p$ . The probability of transfer  $p$  determines how many nodes to transfer every time. It indicates the magnitude of the CIT mechanism. We vary its value and the corresponding results are shown in Figure 5. With the increase of  $p$ , the performance goes up first and then declines, which indicates the performance benefits from an applicable selection of  $p$ .
Analysis of  $k$ . We make transfer process every  $k$  epochs. We vary  $k$  from 1 to 50 and plot the results in Figure 6. Zero represents the original model accuracy without out CIT mechanism. Notably, the performance remains consistently stable across the varying values of  $k$ , indicating the robustness of the model to this parameter.
(a) cora-add-0.5
(b) citeser-add-0.5
(a) cora
(b) citeseeer
Analysis of  $m$ . The number of clusters is the most important parameter in the clustering process. We choose silhouette coefficient to measure the clustering performance and vary  $m$  from 20 to 100 unevenly. Then we calculate the accuracy and Silhouette Coefficient. The corresponding results are shown in Figure 7. As we can see, accuracy changes synchronously with silhouette coefficient. We infer that the performance of our model is related to the clustering situation, and when the clustering process performs well, our model also performs well.
Figure 5: Analysis of the probability of transfer.
(a) cora
Figure 7: The number of clusters is varying on cora and citeseer. The accuracy corresponds to the right vertical axis, while the Silhouette Coefficient values correspond to the left vertical axis.
Figure 6: Analysis of the epochs.
(b) citeseeer
Graph neural networks. Recently, Graph Neural Networks have been widely studied. GCN [13] proposes to aggregate the node features from the one-hop neighbors. GAT [24] designs an attention mechanism to aggregate node features from neighbors. PPNP [9] utilizes PageRank's node propagation way to aggregate node features and proposes an approximate version. GCNII [5] extends GCN by introducing two effective techniques: initial residual and identity mapping, which make the network deeper. This is a fast growing research field, and more detailed works can be found in [4, 22].
OD generalization of GNNs. Out-Of-Distribution (OOD) on graph has attracted considerable attention from different perspectives. For node-classification, [8] shows node selection bias drastically affects the performance of GNNs and investigates it from point view of causal theory. [37] explores the invariant relationship between nodes and proposes a framework to account for distributional differences between biased training data. [28] handles it by minimizing the mean and variance of risks from multiple environments which are generated by adversarial context generators. For graph classification, [15] proposes to capture the invariant relationships between predictive graph structural information and labels in a mixture of latent environments. [29, 18] find invariant subgraph structures from a causal perspective to improve the generalization ability of graph neural networks. [10] builds a graph OOD bench mark including node-level and graph-level methods and two kinds of distribution shift which are covariate shift and concept shift.
Graph clustering with graph neural networks. As graph neural networks continue to perform better in modeling graph data, some GNN-based graph clustering methods have been widely applied. Deep attentional embedded graph clustering [25] uses an attention network to capture the importance of the neighboring nodes and employs the KL-divergence loss in the process of graph clustering. [2] achieves the graph clustering process by obtaining the assignment matrix through minimizing optimizing its spectral objective.[19] uses a new objective function for clustering combining graph spectral modularity maximization and a new regularization method.
In this paper, we explore the impact of structure shift on GNN performance and propose a CIT mechanism to help GNNs learn invariant representations under structure shifts. We theoretically analyze that the impact of changing clusters during structure shift can be mitigated after transfer.
Moreover, the CIT mechanism is a friendly plug-in, and the comprehensive experiments well demonstrate the effectiveness on different structure shift scenarios.
Limitations and broader impact. One potential limitation lies in its primary focus on node-level tasks, while further investigation is needed to explore graph-level tasks. Although our CIT mechanism demonstrates significant advancements, certain theoretical foundations remain to be fully developed. Our work explores the graph from the perspective of the embedding space, thereby surpassing the limitations imposed by graph topology, and offers a fresh outlook on graph analysis.
This work is supported in part by the National Natural Science Foundation of China (No. U20B2045, 62192784, U22B2038, 62002029, 62172052, 62322203).
[1] Theodore Wilbur Anderson. An introduction to multivariate statistical analysis. Technical report, Wiley New York, 1962.
[2] Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In International Conference on Machine Learning, pages 874-883. PMLR, 2020.
[3] T Tony Cai and Linjun Zhang. A convex optimization approach to high-dimensional sparse quadratic discriminant analysis. The Annals of Statistics, 49(3):1537-1568, 2021.
[4] Hongxu Chen, Hongzhi Yin, Tong Chen, Quoc Viet Hung Nguyen, Wen-Chih Peng, and Xue Li. Exploiting centrality information with graph convolutions for network representation learning. In 2019 IEEE 35th International Conference on Data Engineering (ICDE), pages 590-601. IEEE, 2019.
[5] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pages 1725-1735. PMLR, 2020.
[6] Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. arXiv preprint arXiv:1912.09893, 2019.
[7] Shobeir Fakhraei, James Foulds, Madhusudana Shashanka, and Lise Getoor. Collective spammer detection in evolving multi-relational social networks. In Proceedings of the 21th acm SIGkdd international conference on knowledge discovery and data mining, pages 1769-1778, 2015.
[8] Shaohua Fan, Xiao Wang, Chuan Shi, Kun Kuang, Nian Liu, and Bai Wang. Debiased graph neural networks with agnostic label selection bias. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[9] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Gunnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997, 2018.
[10] Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribution benchmark. arXiv preprint arXiv:2206.08452, 2022.
[11] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22133, 2020.
[12] Brian Karrer and Mark EJ Newman. Stochastic blockmodels and community structure in networks. Physical review  $E$ , 83(1):016107, 2011.
[13] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
[33] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
[34] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[35] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization in vision: A survey. arXiv preprint arXiv:2103.02503, 2021.
[36] Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021, pages 1215-1226, 2021.
[37] Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems, 34:27965-27977, 2021.