SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We investigate the following questions to verify the effectiveness of the proposed method. Q1: How does the proposed GNNEvaluator perform in evaluating well-trained GNNs' node classification accuracy (Sec. 4.2)? Q2: How does the proposed GNNEvaluator perform when conducting an ablation study regarding the DiscGraph set component? (Sec. 4.3) Q3: What characteristics does our constructed DiscGraph set have (Sec. 4.4)? Q4: How many DiscGraphs in the set are needed for the proposed GNNEvaluator to perform well (Sec. 4.5)?
Datasets. We perform experiments on three real-world graph datasets, i.e., DBLPv8, ACMv9, and Citationv2, which are citation networks from different original sources (DBLP, ACM, and Microsoft Academic Graph, respectively) by following the processing of [29]. Each dataset shares the same label space with six categories of node class labels, including 'Database', 'Data mining', 'Artificial intelligent', 'Computer vision', 'Information Security', and 'High-performance computing'. We evaluate our proposed GNNEvaluator with the following six cases, i.e., A→D, A→C, C→A, C→D, D→A, and D→C, where A, C, and D denote ACMv9, DBLPv8, and Citationv2, respectively. Each arrow denotes the estimation of GNNEvaluator trained by the former observed graph and tested on the latter unseen graph. Note that, in all stages of GNNEvaluator training and inference, we do not access the labels of the latter unseen graph. More detailed statistical information of the used dataset can be found in Appendix B.
GNN Models and Evaluation. We evaluate four commonly-used GNN models, including GCN [17], GraphSAGE [12] (abbr. SAGE), GAT [28], and GIN [31], as well as the baseline MLP model that is prevalent for graph learning. For each model, we train it on the training set of the observed graph under the transductive setting, until the model achieves the best node classification on its validation set following the standard training process, i.e., the 'well-trained' GNN model. To minimize experimental variability, we train each type of GNN with five random seeds. That means, for instance, we have five well-trained GCN models on the same fully-observed graph with the same hyper-parameter space but only five different random seeds. More details of these well-trained GNN models are provided in Appendix D. We report the Mean Absolute Error (MAE), the average absolute difference between the ground truth accuracy (in percentage) on the unseen graph and the estimated accuracy from ours and baselines on the same unseen graph, across different random seeds for each evaluated GNN. The smaller MAE denotes better GNN model evaluation performance.
**Baseline Methods.** As our method is the first GNN model evaluation approach, there is no available baseline for direct comparisons. Therefore, we evaluate our proposed approach by comparing it to three convolutional neural network (CNN) model evaluation methods applied to image data. Note that existing CNN model evaluation methods are hard to be directly applied to GNNs, since GNNs have entirely different convolutional architectures working on different data types. Compared with Euclidean image data, graph structural data distributions have more complex and wider variations due to their non-independent node-edge interconnections. Therefore, we make necessary adaptations to
Table 1: Mean Absolute Error (MAE) performance of different GNN models across different random seeds. (GNNs are well-trained on the ACMv9 dataset and evaluated on the unseen and unlabeled Citationv2 and DBLPv8 datasets, i.e., A→C and A→D, respectively. Best results are in bold.)
Methods	ACMv9→Citationv2	ACMv9→DBLPv8
GCN	SAGE	GAT	GIN	MLP	Avg.	GCN	SAGE	GAT	GIN	MLP	Avg.
ATC-MC [9]	4.49	8.40	4.37	18.40	34.33	14.00	21.96	24.20	30.30	24.06	26.62	25.43
ATC-MC-c [9]	2.41	5.74	4.67	22.00	51.41	17.25	31.15	30.55	30.18	29.71	45.81	33.48
ATC-NE [9]	3.97	8.02	4.28	17.35	38.87	14.50	22.93	24.78	30.50	23.74	31.13	26.62
ATC-NE-c [9]	4.44	6.09	3.30	23.95	44.62	16.48	34.42	28.31	27.02	30.28	39.28	31.86
Thres. (τ = 0.7) [6]	32.64	35.81	33.63	50.76	35.28	37.63	9.59	12.14	14.30	32.67	39.72	21.68
Thres. (τ = 0.8) [6]	26.30	29.60	26.18	49.25	35.87	33.44	2.63	7.44	14.47	32.20	40.31	19.41
Thres. (τ = 0.9) [6]	17.56	21.34	16.38	46.53	36.08	27.58	8.20	7.42	16.07	31.47	40.56	20.74
AutoEval-G [6]	18.94	26.19	26.12	50.86	32.40	30.90	2.77	2.54	7.25	48.68	29.95	18.24
GNNEvaluator (Ours)	4.85	4.11	12.23	10.14	22.20	10.71	11.80	14.88	6.36	13.78	17.49	12.86
Table 2: Mean Absolute Error (MAE) performance of different GNN models across different random seeds. (GNNs are well-trained on the Citationv2 dataset and evaluated on the unseen and unlabeled ACMv9 and DBLPv8 datasets, i.e., C→A and C→D, respectively. Best results are in bold.)
Methods	Citationv2→ACMv9	Citationv2→DBLPv8
GCN	SAGE	GAT	GIN	MLP	Avg.	GCN	SAGE	GAT	GIN	MLP	Avg.
ATC-MC [9]	9.50	13.40	8.28	35.51	43.40	22.02	22.57	1.37	21.87	29.24	35.20	22.05
ATC-MC-c [9]	6.93	11.75	6.70	38.93	57.43	24.35	33.67	4.92	28.23	30.89	52.59	30.06
ATC-NE [9]	8.86	13.04	7.87	34.88	47.49	22.42	23.97	1.86	23.74	28.96	39.72	23.65
ATC-NE-C [9]	7.73	13.94	7.63	41.17	62.96	26.69	37.16	4.66	29.43	31.66	58.95	32.37
Thres. (τ = 0.7) [6]	37.33	36.61	31.68	58.91	34.33	39.77	10.70	23.05	12.74	34.60	38.29	23.88
Thres. (τ = 0.8) [6]	29.62	28.95	22.77	57.48	34.53	34.67	5.65	15.01	7.61	34.36	38.43	20.21
Thres. (τ = 0.9) [6]	19.59	19.06	11.37	55.72	34.56	28.06	10.65	8.28	8.07	34.00	38.44	19.89
AutoEval-G [6]	23.01	31.24	26.74	59.66	35.02	28.28	2.57	16.52	6.96	19.20	32.24	24.59
GNNEvaluator (Ours)	5.45	8.53	9.61	29.77	28.52	16.38	11.64	7.02	5.58	6.46	22.87	10.71
enable these methods to work on graph-structured data for GNN model evaluation. Specifically, we compare: (1) Average Thresholded Confidence (ATC) score [9], which learns a threshold on CNN's confidence to estimate the accuracy as the fraction of unlabeled images whose confidence scores exceed the threshold. We consider its four variants, denoting as ATC-NE, ATC-NE-c, ATC-MC, and ATC-MC-c, where ATC-NE and ATC-MC calculate negative entropy and maximum confidence scores, respectively, and  $-c$  denotes their confidence calibrated versions. In our adaptation, we use the training set of the observed graph under the transductive setting to calculate these confidence scores, and use the validation set for their calibrated versions. (2) Threshold-based Method, which is introduced by [6] and determines three threshold values for  $\tau = \{0.7, 0.8, 0.9\}$  on the output softmax logits of CNNs and calculates the percentage of images in the entire dataset whose maximum entry of logits are greater than the threshold  $\tau$ , which indicates these images are correctly classified. We make an adaptation by using the GNN softmax logits and calculating the percentage of nodes in a single graph. (3) AutoEval Method [6], which conducts the linear regression to learn the CNN classifier's accuracy based on the domain gap features between the observed training image data and unseen real-world unlabeled image data. We adapt it to GNN models on graph structural data by calculating the Maximum Mean Discrepancy (MMD) measurement as the graph domain gap features extracted from our meta-graph set, followed by the linear regression, which we name as AutoEval-G variant. More details of baseline methods can be found in the Appendix C.
4.2 GNN Model Evaluation Results
In Table 1, Table 2, and Table 3, we report MAE results of evaluating different GNN models across different random seeds. In general, we observe that our proposed GNNEvaluator significantly outperforms the other model evaluation baseline methods in all six cases, achieving the lowest average MAE scores of all GNN model types, i.e., 10.71 and 12.86 in  $\mathrm{A}\rightarrow \mathrm{C}$  and  $\mathrm{A}\rightarrow \mathrm{D}$  cases, 16.38 and 10.71 in  $\mathrm{C}\rightarrow \mathrm{A}$  and  $\mathrm{C}\rightarrow \mathrm{D}$  cases, and 6.79 and 7.80 in  $\mathrm{D}\rightarrow \mathrm{A}$  and  $\mathrm{D}\rightarrow \mathrm{C}$  cases, respectively.
More specifically, we observe that some baseline methods achieve the lowest MAE for a certain GNN model type on a specific case. Taking GCN as an example in Table 1, ATC-MC-c performs best with 2.41 MAE under  $\mathrm{A}\rightarrow \mathrm{C}$  case. However, it correspondingly has 31.15 worst MAE under  $\mathrm{A}\rightarrow \mathrm{D}$  case. That means, for a GCN model that is well-trained by ACMv9 dataset, ATC-MC-c provides model evaluation results with significant performance variance under different unseen graphs. Such high
Table 3: Mean Absolute Error (MAE) performance of different GNN models across different random seeds. (GNNs are well-trained on the DBLPv8 dataset and evaluated on the unseen and unlabeled ACMv9 and Citationv2 datasets, i.e., D→A and D→C, respectively. Best results are in bold.)
Methods	DBLPv8→ACMv9	DBLPv8→Citationv2
GCN	SAGE	GAT	GIN	MLP	Avg.	GCN	SAGE	GAT	GIN	MLP	Avg.
ATC-MC [9]	4.98	31.49	37.08	28.74	30.42	26.54	5.47	29.72	37.43	32.07	36.42	28.22
ATC-MC-C [9]	3.42	32.83	40.31	37.68	34.72	29.79	4.66	28.67	43.89	38.13	41.64	31.40
ATC-NE [9]	2.83	27.67	40.90	35.88	36.03	28.66	4.25	26.04	39.38	38.38	45.15	30.64
ATC-NE-C [9]	12.95	20.86	39.34	43.41	38.85	31.08	16.87	15.77	40.26	47.64	47.61	33.63
Thres. (τ = 0.7) [6]	21.66	32.69	39.19	30.10	27.80	30.29	27.22	35.48	34.35	37.99	30.36	33.08
Thres. (τ = 0.8) [6]	18.63	29.34	37.25	25.80	20.34	26.27	22.70	30.86	32.48	32.73	22.36	28.23
Thres. (τ = 0.9) [6]	16.12	23.81	43.26	23.73	12.98	23.98	15.63	26.07	35.88	27.15	12.20	23.39
AutoEval-G [6]	7.28	9.72	12.05	14.17	22.07	13.06	21.13	15.11	5.65	12.33	31.90	17.22
GNNEvaluator (Ours)	2.46	10.27	6.94	8.86	5.42	6.79	11.68	7.83	3.97	9.62	5.92	7.80
variance evaluation performance would significantly limit the practical applications of ATC-MC-c, as it only performs well on certain unseen graphs, incurring more uncertainty in GNN model evaluation. Similar limitations can also be observed in Table 2 for ATC-MC-c on evaluating GAT model, with 6.70 lowest MAE in C→A but 28.23 high MAE in C→D. Moreover, in terms of AutoEval-G method on GraphSAGE model evaluation in Table 3, it achieves the lowest MAE of 9.72 in D→A case, but a high MAE of 15.11 in C→A case. In contrast, our proposed GNNEvaluator has smaller MAE variance for different unseen graphs under the same well-trained models. For instance, our GNNEvaluator achieves 4.85 MAE on GCN under A→C case, and 11.80 MAE on GCN in A→D case, which is better than ATC-MC-c with 2.41 and 31.15 on the same cases. All these results verify the effectiveness and consistently good performance of our proposed method on diverse unseen graph distributions for evaluating different GNN models.
We conduct an ablation study to verify the effectiveness of the proposed DiscGraph set component for our proposed GNN model evaluation pipeline. Concretely, we replace the DiscGraph set with the meta-graph set for GNNEvaluator training, which does not involve discrepancy node attributes calculated based on specific well-trained GNNs' node embedding space with the discrepancy measurement function. The output node classification accuracy  $(\%)$  of the proposed GNNEvaluator for evaluating five GAT models under different seeds are shown in Fig. 3 with lines for the proposed DiscGraph set (w/ DiscAttr) in green and the meta-graph set (w/o DiscAttr) in blue, respectively. And we also list the ground-truth node classification accuracy in histograms for comparison.
Figure 3: Ablation study of the proposed DiscGraph set with discrepancy node attributes (w/DiscAttr) in green lines vs. the meta-graph set without discrepancy node attributes (w/oDiscAttr) in blue lines. And the ground-truth node classification accuracy is in histograms for reference.
Generally, we observe that the proposed DiscGraph set (w/ DiscAttr) trained GNNEvaluator has better performance for GAT model evaluation than that trained by the meta-graph set (w/o DiscAttr), as all green lines are closer to the ground-truth histograms than the blue line, reflecting the effectiveness of the proposed DiscGraph set, especially the discrepancy node attributes with the discrepancy measurement function. Moreover, the meta-graph set (w/o DiscAttr) trained GNNEvaluator almost produces the same accuracy on a certain GNN type, i.e., its performance stays the same on all five GATs, making it fail to evaluate a certain GNN model type under different optimal model parameters. That is because, compared with the proposed DiscGraph set, the meta-graph set lacks the exploration of the latent node embedding space of well-trained GNNs and only utilizes the output node class predictions from the well-trained GNNs. In contrast, the proposed DiscGraph set fully exploits both latent node embedding space and output node class predictions of well-trained GNNs for comprehensively modeling the discrepancy between the observed training graph and unseen test graph, enabling its superior ability for GNN model evaluation.
Figure 4: Visualizations on discrepancy node attributes in the proposed DiscGraph set for GCN model evaluation. Darker color denotes a larger discrepancy.
4.4 Analysis of Discrepancy Meta-graph Set
We visualize the discrepancy node attributes in the proposed DiscGraph set on GCN model evaluation trained on ACMv9 and tested on unseen DBLPv8 in Fig. 4 (a). Taking this as a reference, we present visualizations of discrepancy node attributes derived between ACMv9 and arbitrary two meta-graphs from our created meta-graph set in Fig. 4 (c) and (d). Darker color close to black indicates a larger discrepancy in the output node embedding space of a well-trained GNN, and lighter color close to white indicates a smaller discrepancy. As can be observed, the discrepancy pairs of (ACMv9, unseen DBLPv8), (ACMv9, Meta-graph-1), and (ACMv9, Meta-graph-2) generally show similar heat map patterns in terms of discrepancy. This indicates the effectiveness of the proposed meta-graph set simulation strategy, which could capture potential unseen graph data distributions by synthesizing diverse graph data with various graph augmentations. More detailed statistical information about the proposed DiscGraph set, including the number of graphs, the average number of nodes and edges, and the accuracy label distributions are provided in Appendix E.
4.5 Hyper-parameter Analysis of GNNEvaluator
Figure 5: Effects of different numbers of DiscGraphs  $(K)$  for GNNEvaluator training. Histograms denote MAE for  $C\to A$  and  $C\rightarrow D$  cases over five GCNs under different  $K$ , and the line indicates the average MAE of such two evaluation cases.
We test the effects of different numbers of discrepancy meta-graphs  $(K)$  for GNNEvaluator training in Fig. 5. Concretely, we utilize the number of DiscGraphs for  $K = [100,200,300,400]$  to train the proposed GNNEvaluator for evaluating five GCN models that are well-trained on Citationv2 dataset and make inferences on unseen ACMv9 and DBLPv8 datasets. Considering the average performance under two transfer cases reflected in the line, despite  $K = 100$  having a higher MAE, the average MAE over two cases does not change much with the number  $K$  for training the proposed GNNEvaluator. This indicates that the proposed GNNEvaluator can effectively learn to accurately predict node attributes by training on an appropriate number of DiscGraphs, without the need for a significantly excessive amount of training examples.
In this work, we have studied a new problem, GNN model evaluation, for understanding and evaluating the performance of well-trained GNNs on unseen and unlabeled graphs. A two-stage approach is proposed, which first generates a diverse meta-graph set to simulate and capture the discrepancies of different graph distributions, based on which a GNNEvaluator is trained to predict the accuracy of a well-trained GNN model on unseen graphs. Extensive experiments with real-world unseen and unlabeled graphs could verify the effectiveness of our proposed method for GNN model evaluation. Our method assumes that the class label space is unchanged across training and testing graphs though covariate shifts may exist between the two. We will look into relaxing this assumption and address a broader range of natural real-world graph data distribution shifts in the future.
In this work, S. Pan was supported by an Australian Research Council (ARC) Future Fellowship (FT210100097), M. Zhang was supported by National Natural Science Foundation of China (NSFC) grant (62306084), and C. Zhou was supported by the CAS Project for Young Scientists in Basic Research (YSBR-008).
International Conference on Data Mining (ICDM), pages 783-792. IEEE, 2022.
[43] Xin Zheng, Miao Zhang, Chunyang Chen, Quoc Viet Hung Nguyen, Xingquan Zhu, and Shirui Pan. Structure-free graph condensation: From large-scale graphs to condensed graph-free data. arXiv preprint arXiv:2306.02664, 2023.
[44] Xin Zheng, Miao Zhang, Chunyang Chen, Qin Zhang, Chuan Zhou, and Shirui Pan. Auto-heg: Automated graph neural network on heterophilic graphs. In Proceedings of the ACM Web Conference (WWW), pages 611-620, 2023.
[45] Yizhen Zheng, Vincent CS Lee, Zonghan Wu, and Shirui Pan. Heterogeneous graph attention network for small and medium-sized enterprises bankruptcy prediction. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pages 140–151. Springer, 2021.
[46] Yizhen Zheng, Shirui Pan, Vincent Cs Lee, Yu Zheng, and Philip S Yu. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[47] Yizhen Zheng, He Zhang, Vincent Lee, Yu Zheng, Xiao Wang, and Shirui Pan. Finding the missing-half: Graph complementary learning for homophily-prone and heterophily-prone graphs. International Conference of Machine Learning (ICML), 2023.
[48] Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems (NeurIPS), 34:27965–27977, 2021.
This is the appendix of our work: 'GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labels'. In this appendix, we present additional information regarding the GNN model evaluation problem and corresponding solutions, including in-depth discussions that distinguish our method from related works, dataset statistics used in our experiments, comprehensive details about our compared baseline methods, training details with the performance of well-trained GNN models on both the test set of the observed graph and the unseen test graphs (for ground-truth reference), and more experimental results with detailed analysis.
Predicting Model Generalization Error. Our work is relevant to the line of research on predicting model generalization error, which aims to develop a good estimator of a model's classifier performance on unlabeled data from the unknown distributions in the target domain, when the estimated models' classifier has been trained well in the source domain with labeled data [6, 9, 34, 5, 10, 4]. Typically, Guillory et al. [10] proposed to estimate a classifier's performance of convolutional neural network (CNN) models on image data based on a designed criterion, named difference of confidences (DoC), that estimates and reflects model accuracy. And Garg et al. [9] proposed to learn a score-based Average Thresholded Confidence (ATC) by leveraging the softmax probability of a CNN classifier, whose accuracy is estimated as the fraction of unlabeled images that receive a score above that threshold. In contrast, Deng et al. [6, 4] directly predicted CNN classifier accuracy by deriving distribution distance features between training and test images with a linear regression model.
However, these existing methods mostly focus on evaluating CNN model classifiers on image data in computer vision, and the formulation of evaluating GNNs for graph structural data still remains under-explored in graph machine learning. Concretely, conducting the model evaluation on GNNs for graph structural data has two critical challenges: (1) different from Euclidean image data, graph structural data lies in non-Euclidean space with complex and non-independent node-edge interconnections, so that its node contexts and structural characteristics significantly vary under wide-range graph data distributions, posing severer challenges for GNN model evaluation when serving on unknown graph data distributions. (2) GNNs have entirely different convolutional architectures from those of CNNs, when GNN convolution aggregates neighbor node messages along graph structures. Such that GNNs trained on the observed training graph might well fit its graph structure, and due to the complexity and diversity of graph structures, serving well-trained GNNs on unlabeled test distributions that they have not encountered before would incur more performance uncertainty.
Hence, in this work, we first investigate the GNN model evaluation problem for graph structural data with a clear problem definition and a feasible solution, taking a significant step towards understanding and evaluating GNN models for practical GNN deployment and serving. Our proposed two-stage GNN model evaluation framework directly estimates the node classification accuracy results of specific well-trained GNNs on practical unseen graphs without labels. Our method could not only facilitate model designers to better evaluate well-trained GNNs' performance in practical serving, but also provide users with confidence scores or model selection guidance, when using well-trained GNNs for inferring on their own test graphs.
Unsupervised Graph Domain Adaption. Our work is also relevant to the unsupervised graph domain adaption (UGDA) problem, whose goal is to develop a GNN model with both labeled source graphs and unlabeled target graphs for better generalization ability on target graphs. Typically, existing UGDA methods focus on mitigating the domain gap by aligning the source graph distribution with the target one. For instance, Yang et al. [33] and Shen et al. [27] optimized domain distance loss based on the statistical information of datasets, e.g., maximum mean discrepancy (MMD) metric. Moreover, DANE [39] and UDAGCN [29] introduced domain adversarial methods to learn domain-invariant embeddings across the source domain and the target domain. Therefore, the critical distinction between our work and UGDA is that our work is primarily concerned with evaluating the GNNs' performance on unseen graphs without labels, rather than improving the GNNs' generalization ability when adapting to unlabeled target graphs. Besides, different from UGDA which uses the unlabeled target graphs in their model design and training stage, the GNN model evaluation problem
explored by our work is not allowed to access the unlabeled test graphs, i.e., unseen in the whole GNN model evaluation process.