Title: GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labels
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks

GNN model evaluation

node classification accuracy
Abstract: Evaluating the performance of graph neural networks (GNNs) is an essential task for practical GNN model deployment and serving, as deployed GNNs face significant performance uncertainty when inferring on unseen and unlabeled test graphs, due to mismatched training-test graph distributions. In this paper, we study a *new* problem, **GNN model evaluation**, that aims to assess the performance of a specific GNN model trained on labeled and observed graphs, by precisely estimating its performance (e.g., node classification accuracy) on unseen graphs without labels. Concretely, we propose a two-stage GNN model evaluation framework, including (1) DiscGraph set construction and (2) GNNEvaluator training and inference. The DiscGraph set captures wide-range and diverse graph data distribution discrepancies through a discrepancy measurement function, which exploits the GNN outputs of latent node embeddings and node class predictions. Under the effective training supervision from the DiscGraph set, GNNEvaluator learns to precisely estimate node classification accuracy of the to-be-evaluated GNN model and makes an accurate inference for evaluating GNN model performance. Extensive experiments on real-world unseen and unlabeled test graphs demonstrate the effectiveness of our proposed method for GNN model evaluation.

GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labels

Abstract

Evaluating the performance of graph neural networks (GNNs) is an essential task for practical GNN model deployment and serving, as deployed GNNs face significant performance uncertainty when inferring on unseen and unlabeled test graphs, due to mismatched training-test graph distributions. In this paper, we study a new problem, GNNmodel evaluation , that aims to assess the performance of a specific GNN model trained on labeled and observed graphs, by precisely estimating its performance ( e.g. , node classification accuracy) on unseen graphs without labels. Concretely, we propose a two-stage GNN model evaluation framework, including (1) DiscGraph set construction and (2) GNNEvaluator training and inference. The DiscGraph set captures wide-range and diverse graph data distribution discrepancies through a discrepancy measurement function, which exploits the outputs of GNNs related to latent node embeddings and node class predictions. Under the effective training supervision from the DiscGraph set, GNNEvaluator learns to precisely estimate node classification accuracy of the to-be-evaluated GNN model and makes an accurate inference for evaluating GNN model performance. Extensive experiments on real-world unseen and unlabeled test graphs demonstrate the effectiveness of our proposed method for GNN model evaluation.

1 Introduction

As prevalent graph data learning models, graph neural networks (GNNs) have attracted much attention and achieved great success for various graph structural data related applications in the real world [36, 38, 35, 41, 42, 14, 21, 46, 44]. In practical scenarios of GNN model deployment and serving [37, 40, 43], understanding and evaluating GNN models' performance is a vital step [3, 13, 22], where model designers need to determine if well-trained GNN models will perform well in practical serving, and users want to know how the in-service GNN models will perform when inferring on their own test graphs [10]. Conventionally, model evaluation utilizes well-annotated datasets for testing to calculate certain model performance metrics ( e.g. , accuracy and F1-score) [3, 24, 23, 20]. However, it may fail to work well in real-world GNN model deployment and serving, where the unseen test graphs are usually not annotated, making it difficult to obtain essential model performance metrics without ground-truth labels [4, 5]. As shown in Fig. 1 (a-1), taking node classification accuracy metric as an example, it is typically calculated as the percentage of correctly predicted node labels. However, when ground-truth node class labels are unavailable, we can not verify whether the predictions of GNNs are correct, and thus cannot get the overall accuracy of the model.

In light of this, a natural question, referred to GNNmodel evaluation problem, arises: in the absence of labels in an unseen test graph, can we estimate the performance of a well-trained GNN model?

∗

Unseen Test Graph T

1

well-trained&fixed

GNN'

inference

ACC=

#total nodes

s√nodes

Unseen Test Graph T

ACC(T,GCN)

(a-1) Conventional model evaluation(w/unseen test graphlabels).

GCN's

71.63%

UnseenTestGraphT

GAT'

GNNEvaluator

70.01%

ACC(J,GAT)

well-trained&fixed

GraphSAGE'

ACC (T,GraphSAGE)

GNN'

GNNEvaluator

estimation

ACC≈71.63%

?Withoutnodelabels

69.54%

(a-2) The proposed GNN model evaluation (w/ounseen test graph labels).

(b)Anapplicablecaseoftheproposed GNNEvaluator.

In this work, we provide a confirmed answer together with an effective solution to this question, as shown in Fig.1 (a-2). Given a well-trained GNN model and an unseen test graph without labels, GNN model evaluation directly outputs the overall accuracy of this GNN model. This enables users to understand their GNN models at hand, benefiting many GNN deployment and serving scenarios in the real world [2, 8, 30, 25, 47, 45, 16, 15]. For example, given a GNN model in service ( e.g. , a GNN model trained and served with Amazon DGL and SageMaker [1]), GNN model evaluation can provide users with a confidence score based on its estimated node classification accuracy, so that users know how much faith they should place in GNN-predicted node labels on their own unlabeled test graphs. Moreover, as shown in Fig. 1 (b), given a set of available well-trained GNNs in service, GNN model evaluation can provide users with reliable guidance to make an informed choice among these deployed GNN models on their own unlabeled test graphs.

In this paper, we focus on the node classification task, with accuracy as the primary GNN model evaluation metric. Developing effective GNN model evaluation methods for this task faces threefold challenges: Challenge-1: The distribution discrepancies between various real-world unseen test graphs and the observed training graph are usually complex and diverse, incurring significant uncertainty for GNN model evaluation. Challenge-2: It is not allowed to re-train or fine-tune the practically in-service GNN model to be evaluated, and the only accessible model-related information is its outputs. Hence, how to fully exploit the limited GNN outputs and integrate various training-test graph distribution differences into discriminative discrepancy representations is critically important. Challenge-3: Given the discriminative discrepancy representations of training-test graph distributions, how to develop an accurate GNN model evaluator to estimate the node classification accuracy of an in-service GNN on the unseen test graph is the key to GNN model evaluation.

To address the above challenges, in this work, we propose a two-stage GNN model evaluation framework, including (1) DiscGraph set construction and (2) GNNEvaluator training and inference. More specifically, in the first stage, we first derive a set of meta-graphs from the observed training graph, which involves wide-range and diverse graph data distributions to simulate (ideally) any potential unseen test graphs in practice, so that the complex and diverse training-test graph data distribution discrepancies can be effectively captured and modeled ( Challenge-1 ). Based on the meta-graph set, we build a set of discrepancy meta-graphs (DiscGraph) by jointly exploiting latent node embeddings and node class predictions of the GNN with a discrepancy measurement function, which comprehensively integrates GNN output discrepancies to expressive graph structural discrepancy representations involving discrepancy node attributes, graph structures, and accuracy labels ( Challenge-2 ). In the second stage, we develop a GNNEvaluator composed of a typical GCN architecture and an accuracy regression layer, and we train it to precisely estimate node classification accuracy with effective supervision from the representative DiscGraph set ( Challenge-3 ). During the inference, the trained GNNEvaluator could directly output the node classification accuracy of the in-service GNN on the unseen test graph without any node class labels.

In summary, the contributions of our work are listed as follows:

Problem. We study a new research problem, GNN model evaluation, which opens the door for understanding and evaluating the performance of well-trained GNNs on unseen real-world graphs without labels in practical GNN model deployment and serving.

Solution. We design an effective solution to simulate and capture the discrepancies of diverse graph data distributions, together with a GNNEvaluator to estimate node classification accuracy of the in-service GNNs, enabling accurate GNN model evaluation.

Evaluation. We evaluate our method on real-world unseen and unlabeled test graphs, achieving a low error ( e.g. , as small as 2.46%) compared with ground-truth accuracy, demonstrating the effectiveness of our method for GNN model evaluation.

Prior Works . Our research is related to existing studies on predicting model generalization error , which aims to estimate a model's performance on unlabeled data from the unknown and shifted distributions [6, 9, 34, 10, 4]. However, these researches are designed for data in Euclidean space ( e.g. , images) while our research is dedicatedly designed for graph structural data. Our research also significantly differs from others in unsupervised graph domain adaption [33, 39, 29], out-ofdistribution (OOD) generalization [18, 48], and OOD detection [19, 26], in the sense that we aim to estimate well-trained GNN models' performance, rather than improve the generalization ability of new GNN models. Detailed related work can be found in Appendix A.

2 Problem Definition

Preliminary. Consider that we have a fully-observed training graph S = ( X , A , Y ) , where X ∈ R N × d denotes N nodes with d -dimensional features, A ∈ R N × N denotes the adjacency matrix indicating the edge connections, and Y ∈ R N × C denotes the C -classes of node labels. Then, training a GNN model on S for node classification objective can be denoted as:

where θ S denotes the parameters of GNN trained on S , Z S ∈ R N × d 1 is the output node embedding of graph S from GNN θ S , and ˆ Y ∈ R N × C denotes GNN predicted node labels. By optimizing the node classification loss function L cls between GNN predictions ˆ Y and ground-truth node labels Y , i.e. , cross-entropy loss, we could obtain a well-trained GNN with optimal weight parameters θ ∗ S , denoted as GNN ∗ S . Then, the node classification accuracy can be calculated to reflect GNN performance as: Acc ( S ) = ∑ N i =1 (ˆ y i == y i ) /N , which indicates the percentage of correctly predicted node labels between the GNN predicted labels ˆ y i ∈ ˆ Y and ground truths y i ∈ Y . Given an unseen and unlabeled graph T = ( X ′ , A ′ ) including M nodes with its features X ′ ∈ R M × d and structures A ′ ∈ R M × M , we assume the covariate shift between S and T , where the distribution shift mainly lies in node numbers, node context features, and graph structures, but the label space of T keeps the same with S , i.e. , all nodes in T are constrained in the same C -classes. Due to the absence of ground-truth node labels, we could NOT directly calculate the node classification accuracy to assess the performance of the well-trained GNN ∗ S on T . In light of this, we present a new research problem for GNNs as:

Definition of GNN Model Evaluation. Given the observed training graph S , its well-trained model GNN ∗ S , and an unlabeled unseen graph T as inputs, the goal of GNN model evaluation aims to learn an accuracy estimation model f ϕ ( · ) parameterized by ϕ as:

where f ϕ : ( GNN ∗ S , T ) → a and a ∈ R is a scalar denoting the overall node classification accuracy Acc ( T ) for all unlabeled nodes of T . When the context is clear, we will use f ϕ ( T ) for simplification.

3 The Proposed Method

3.1 Overall Framework of GNN Model Evaluation

As shown in Fig. 2, the proposed overall framework of GNN model evaluation contains two stages: (1) DiscGraph set construction; and (2) GNNEvaluator training and inference.

Stage 1. Given the observed training graph S , we first extract a seed graph from it followed by augmentations, leading to a set of meta-graphs from it for simulating any potential unseen graph distributions in practice. Then, the meta-graph set G meta and the observed training graph S are fed into the well-trained GNN for obtaining latent node embeddings, i.e. , Z ( i, ∗ ) meta and Z ∗ S , respectively. After, a

GraphS

DiscrepancyFunction

GNN'

Zs

fixed

Z(l)

D()

1

train

GNNEvaluator

GNN's

'meta

extract

ACC

ydisc

2

inference

Ps

A, (Edge-Drop)

A (Sub-Graph)

disc

3

Unseen Graph T

Estimated Accuracy(T)

Seed Graph Sseed

A (Attr-Mask)

A, (Node-Mix)

Meta-graphSet Gmeta

Discrepancy attributes

Accuracylabels

Structures

Stage-1 DiscGraph set construction

Stage-2 GNNEvaluator training & inference

discrepancy measurement function D ( · ) works on these two latent node embeddings, leading to 1 discrepancy node attributes X i disc . Meanwhile, the output node class predictions of well-trained GNN on each meta-graph are used to calculate the node classification accuracy according to its groundtruth node labels, leading to the new 2 scalar accuracy labels y i disc . Along with 3 graph structures A i disc from each meta-graph, these three important components composite the set of discrepancy meta-graphs (DiscGraph) in expressive graph structural discrepancy representations.

Stage 2. In the second stage, the representative DiscGraph set is taken as effective supervision to train a GNNEvaluator through accuracy regression. During the inference, the trained GNNEvaluator could directly output the node classification accuracy of the in-service GNN on the unseen test graph T without any node class labels.

3.2 Discrepancy Meta-graph Set Construction

The critical challenge in developing GNN model evaluation methods is complex and diverse graph data distribution discrepancies between various real-world unseen test graphs and the observed training graph. As the in-service GNN in practice fits the observed training graph well, making inferences on various and diverse unseen test graphs would incur significant performance uncertainty for GNN model evaluation, especially when the node classification accuracy can not be calculated due to label absence.

In light of this, we propose to construct a set of diverse graphs to simulate wide-range discrepancies of potential unseen graph distributions. Specifically, such a graph set should have the following characteristics: (1) sufficient quantity: it should contain a relatively sufficient number of graphs with diverse node context and graph structure distributions; (2) represented discrepancy: the node attributes of each graph should indicate its distribution distance gap towards the observed training graph; (3) known accuracy: each graph should be annotated by node classification accuracy as its label.

To address the characteristic of (1) sufficient quantity , we introduce a meta-graph simulation strategy to synthesize a wide variety of meta-graphs for representing any potential unseen graph distributions that may be encountered in practice. Specifically, we extract a seed sub-graph S seed from the observed training graph S . The principle of seed sub-graph selection strategy is that S seed involves the least possible distribution shift within P S from the observed training graph, and shares the same label space with S , satisfying the assumption of covariate shift. Then, S seed is fed to a pool of graph augmentation operators as

which belongs to the distribution P A ( p i , | ϵ ) with { p i } 4 i =1 ∈ (0 , 1) represent the augmentation ratio on S seed for each augmentation type, and ϵ determines the probability of sampling a particular augmentation type. For instance, p 1 = 0 . 5 means dropping 50% edges in S seed . In this way, we can obtain a set of meta-graphs G meta = { g i meta } K i =1 with K numbers of graphs, and we have G meta ∼ P A × P S : S seed → g i meta . For each meta-graph g i meta = { X i meta , A i meta , Y i meta } , where

X i meta ∈ R M i × d , A i meta ∈ R M i × M i , and Y i meta ∈ R M i × C have M i number of nodes, d -dimensional features, and known node labels from S seed belonging to C classes.

To incorporate the characteristic of (2) represented discrepancy , we fully exploit the outputs of latent node embeddings and node class predictions from well-trained GNN ∗ S , and integrate various training-test graph distribution differences into discriminative discrepancy representations. Specifically, given the GNN ∗ S learned latent node embeddings Z ∗ S = GNN ∗ S ( X , A ) , and Z ( i, ∗ ) meta , ˆ Y ( i, ∗ ) meta = GNN ∗ S ( X i meta , A i meta ) , where Z ∗ S ∈ R N × d 1 and Z ( i, ∗ ) meta ∈ R M i × d 1 , we derive a distribution discrepancy measurement function D ( · ) to calculate the discrepancy meta-graph node attributes as:

where X i disc ∈ R M i × N reflects the node-level distribution discrepancy between each g i meta and S with the well-trained GNN ∗ S . Each element in x i u,v ∈ X i disc denotes the node embedding discrepancy gap between a certain node u in the i -th meta-graph and a node v in the observed training graph. Taking this representation as node attributes could effectively integrate representative node-level discrepancy produced by well-trained GNN's node embedding space.

Meanwhile, the characteristic of (3) known accuracy can be involved with the outputs of node class predictions produced by GNN ∗ S on meta-graph ˆ Y ( i, ∗ ) meta = { ˆ y j meta ( i, ∗ ) } M i j =1 . We calculate the node classification accuracy on the meta-graph given its ground-truth node class labels Y i meta = { y j meta ( i ) } M i j =1 as:

where y j meta ( i ) and ˆ y j meta ( i, ∗ ) denote the ground truth label and GNN ∗ S predicted label of j -th node in the i -th graph of the meta-graph set, respectively. Note that y i disc ∈ R is a continuous scalar denoting node classification accuracy under specific GNN ∗ S within the range of (0 , 1) .

In this way, by incorporating all these characteristics with the discrepancy node attributes ( X i disc , the scalar node classification accuracy label y i disc , and the graph structure from meta-graph as A i disc = A i meta for indicating discrepancy node interconnections, we could derive the final discrepancy meta-graph set, i.e. , DiscGraph set, as

The proposed DiscGraph set G dist contains a sufficient number of graphs with diverse node context and graph structure distributions, where each discrepancy meta-graph contains the latent node embedding based discrepancy node attributes and the node class prediction based accuracy label, according to the well-trained GNN's outputs, along with diverse graph structures. All these make the proposed DiscGraph set a discriminative graph structural discrepancy representation for capturing wide-range graph data distribution discrepancies.

3.3 GNNEvaluator Training and Inference

Training. Given our constructed DiscGraph set G disc = { g i disc } K i =1 and g i disc = ( X i disc , A i disc , y i disc ) in Eq. (6), we use it to train a GNN regressor with f ϕ : g i disc → a for evaluating well-trained GNNs, which we name as GNNEvaluator . Specifically, the proposed GNNEvaluator takes a two-layer GCN architecture as the backbone, followed by a pooling layer to average the representation of all nodes of each g i disc , and then maps the averaged embedding to a scalar node classification accuracy on the whole graph. The objective function of our GNN regressor can be written as:

where L reg is the regression loss, i.e. , the mean square error (MSE) loss.

Inference. During the inference in the practical GNN model evaluation, we have: (1) to-be-evaluated GNN ∗ S , and (2) the unseen test graph T = ( X ′ , A ′ ) without labels. The first thing is to calculate

the discrepancy node attributes on the unseen test graph T towards the observed training graph S according to Eq. (4), so that we could obtain X T disc = D ( Z ∗ T , Z ∗ S ) , where Z ∗ T = GNN ∗ S ( X ′ , A ′ ) . Along with the unseen graph structure A T disc = A ′ , the proposed GNNEvaluator could directly output the node classification accuracy of GNN ∗ S on T as:

where ϕ ∗ denotes the optimal GNNEvaluator weight parameters trained by our constructed DiscGraph set G dist .

4 Experiments

This section empirically evaluates the proposed GNNEvaluator on real-world graph datasets for node classification task. In all experiments, to-be-evaluated GNN models have been well-trained on observed training graphs and keep FIXED in GNN model evaluation. We only vary the unseen test graph datasets and evaluate various different types and architectures of well-trained GNNs. Throughout our proposed entire two-stage GNN model evaluation framework, we are unable to access the labels of unseen test graphs. We only utilize these labels for the purpose of obtaining true error estimates for experimental demonstration.

We investigate the following questions to verify the effectiveness of the proposed method. Q1: How does the proposed GNNEvaluator perform in evaluating well-trained GNNs' node classification accuracy (Sec. 4.2)? Q2: How does the proposed GNNEvaluator perform when conducting an ablation study regarding the DiscGraph set component? (Sec. 4.3) Q3: What characteristics does our constructed DiscGraph set have (Sec. 4.4)? Q4: How many DiscGraphs in the set are needed for the proposed GNNEvaluator to perform well (Sec. 4.5)?

4.1 Experimental Settings

Datasets. We perform experiments on three real-world graph datasets, i.e. , DBLPv8, ACMv9, and Citationv2, which are citation networks from different original sources (DBLP, ACM, and Microsoft Academic Graph, respectively) by following the processing of [29]. Each dataset shares the same label space with six categories of node class labels, including 'Database', 'Data mining', 'Artificial intelligent', 'Computer vision', 'Information Security', and 'High-performance computing'. We evaluate our proposed GNNEvaluator with the following six cases, i.e. , A → D, A → C, C → A, C → D, D → A, and D → C, where A, C, and D denote ACMv9, DBLPv8, and Citationv2, respectively. Each arrow denotes the estimation of GNNEvaluator trained by the former observed graph and tested on the latter unseen graph. Note that, in all stages of GNNEvaluator training and inference, we do not access the labels of the latter unseen graph. More detailed statistical information of the used dataset can be found in Appendix B.

GNN Models and Evaluation. We evaluate four commonly-used GNN models, including GCN [17], GraphSAGE [12] ( abbr. SAGE), GAT [28], and GIN [31], as well as the baseline MLP model that is prevalent for graph learning. For each model, we train it on the training set of the observed graph under the transductive setting, until the model achieves the best node classification on its validation set following the standard training process, i.e. , the 'well-trained' GNN model. To minimize experimental variability, we train each type of GNN with five random seeds. That means, for instance, we have five well-trained GCN models on the same fully-observed graph with the same hyper-parameter space but only five different random seeds. More details of these well-trained GNN models are provided in Appendix D. We report the Mean Absolute Error (MAE), the average absolute difference between the ground truth accuracy (in percentage) on the unseen graph and the estimated accuracy from ours and baselines on the same unseen graph, across different random seeds for each evaluated GNN. The smaller MAE denotes better GNN model evaluation performance.

Baseline Methods. As our method is the first GNN model evaluation approach, there is no available baseline for direct comparisons. Therefore, we evaluate our proposed approach by comparing it to three convolutional neural network (CNN) model evaluation methods applied to image data. Note that existing CNN model evaluation methods are hard to be directly applied to GNNs, since GNNs have entirely different convolutional architectures working on different data types. Compared with Euclidean image data, graph structural data distributions have more complex and wider variations due to their non-independent node-edge interconnections. Therefore, we make necessary adaptations to

enable these methods to work on graph-structured data for GNN model evaluation. Specifically, we compare: (1) Average Thresholded Confidence (ATC) score [9], which learns a threshold on CNN's confidence to estimate the accuracy as the fraction of unlabeled images whose confidence scores exceed the threshold. We consider its four variants, denoting as ATC-NE, ATC-NE-c, ATC-MC, and ATC-MC-c, where ATC-NE and ATC-MC calculate negative entropy and maximum confidence scores, respectively, and '-c' denotes their confidence calibrated versions. In our adaptation, we use the training set of the observed graph under the transductive setting to calculate these confidence scores, and use the validation set for their calibrated versions. (2) Threshold-based Method , which is introduced by [6] and determines three threshold values for τ = { 0 . 7 , 0 . 8 , 0 . 9 } on the output softmax logits of CNNs and calculates the percentage of images in the entire dataset whose maximum entry of logits are greater than the threshold τ , which indicates these images are correctly classified. We make an adaptation by using the GNN softmax logits and calculating the percentage of nodes in a single graph. (3) AutoEval Method [6], which conducts the linear regression to learn the CNN classifier's accuracy based on the domain gap features between the observed training image data and unseen real-world unlabeled image data. We adapt it to GNN models on graph structural data by calculating the Maximum Mean Discrepancy (MMD) measurement as the graph domain gap features extracted from our meta-graph set, followed by the linear regression, which we name as AutoEval-G variant. More details of baseline methods can be found in the Appendix C.

4.2 GNN Model Evaluation Results

In Table 1, Table 2, and Table 3, we report MAE results of evaluating different GNN models across different random seeds. In general, we observe that our proposed GNNEvaluator significantly outperforms the other model evaluation baseline methods in all six cases, achieving the lowest average MAE scores of all GNN model types, i.e. , 10.71 and 12.86 in A → C and A → D cases, 16.38 and 10.71 in C → A and C → D cases, and 6.79 and 7.80 in D → A and D → C cases, respectively.

More specifically, we observe that some baseline methods achieve the lowest MAE for a certain GNN model type on a specific case. Taking GCN as an example in Table 1, ATC-MC-c performs best with 2.41 MAE under A → C case. However, it correspondingly has 31.15 worst MAE under A → D case. That means, for a GCN model that is well-trained by ACMv9 dataset, ATC-MC-c provides model evaluation results with significant performance variance under different unseen graphs. Such high-

variance evaluation performance would significantly limit the practical applications of ATC-MC-c, as it only performs well on certain unseen graphs, incurring more uncertainty in GNN model evaluation. Similar limitations can also be observed in Table 2 for ATC-MC-c on evaluating GAT model, with 6.70 lowest MAE in C → A but 28.23 high MAE in C → D. Moreover, in terms of AutoEval-G method on GraphSAGE model evaluation in Table 3, it achieves the lowest MAE of 9.72 in D → A case, but a high MAE of 15.11 in C → A case. In contrast, our proposed GNNEvaluator has smaller MAE variance for different unseen graphs under the same well-trained models. For instance, our GNNEvaluator achieves 4.85 MAE on GCN under A → C case, and 11.80 MAE on GCN in A → D case, which is better than ATC-MC-c with 2.41 and 31.15 on the same cases. All these results verify the effectiveness and consistently good performance of our proposed method on diverse unseen graph distributions for evaluating different GNN models.

4.3 Ablation Study

We conduct an ablation study to verify the effectiveness of the proposed DiscGraph set component for our proposed GNN model evaluation pipeline. Concretely, we replace the DiscGraph set with the meta-graph set for GNNEvaluator training, which does not involve discrepancy node attributes calculated based on specific well-trained GNNs' node embedding space with the discrepancy measurement function. The output node classification accuracy (%) of the proposed GNNEvaluator for evaluating five GAT models under different seeds are shown in Fig. 3 with lines for the proposed DiscGraph set (w/ DiscAttr) in green and the meta-graph set (w/o DiscAttr) in blue, respectively. And we also list the ground-truth node classification accuracy in histograms for comparison.

Generally, we observe that the proposed DiscGraph set (w/ DiscAttr) trained GNNEvaluator has better performance for GAT model evaluation than that trained by the meta-graph set (w/o DiscAttr), as all green lines are closer to the ground-truth histograms than the blue line, reflecting the effectiveness of the proposed the proposed DiscGraph set, especially the discrepancy node attributes with the discrepancy measurement function. Moreover, the meta-graph set (w/o DiscAttr) trained GNNEvaluator almost produces the same accuracy on a certain GNN type, i.e. , its performance stays the same on all five GATs, making it fail to evaluate a certain GNN model type under different optimal model parameters. That is because, compared with the proposed DiscGraph set, the meta-graph set lacks the exploration of the latent node embedding space of well-trained GNNs and only utilizes the output node class predictions from the well-trained GNNs. In contrast, the proposed DiscGraph set fully exploits both latent node embedding space and output node class predictions of well-trained GNNs for comprehensively modeling the discrepancy between the observed training graph and unseen test graph, enabling its superior ability for GNN model evaluation.

0

1000

2000

3000

4000

5000

Nodes in unseen graphs

1000

2000

0.75

0.50

0.25

Axuedauosio

#

3000

4000

0.00

0.25

0.50

0.75

5000

# Nodes in observed training graphs

(a)Node Attributes inDiscGraph of ACMv9→unseen DBLPv8(GCN)

1000

2000

3000

4000

5000

# Nodes in our simulated meta-graphs

200

400

600

0.75

0.50

008

0.25

1200

1000

0.00

0.25

0.50

0.75

1400

1600

# Nodes in observed training graphs

(b) Node Attributes in DiscGraph of ACMv9-→Meta-graph-1 (GCN)

1000

2000

3000

4000

5000

Nodes in our simulated meta-graphs

200

400

600

- 0.75

0.50

800

0.25

7

1000

1200

0.00

0.25

0.50

0.75

1400

1600

Nodes in observed training graphs

(c) Node Attributes in DiscGraph of ACMv9—→Meta-graph-2 (GCN)

4.4 Analysis of Discrepancy Meta-graph Set

We visualize the discrepancy node attributes in the proposed DiscGraph set on GCN model evaluation trained on ACMv9 and tested on unseen DBLPv8 in Fig. 4 (a). Taking this as a reference, we present visualizations of discrepancy node attributes derived between ACMv9 and arbitrary two meta-graphs from our created meta-graph set in Fig. 4 (c) and (d). Darker color close to black indicates a larger discrepancy in the output node embedding space of a well-trained GNN, and lighter color close to white indicates a smaller discrepancy. As can be observed, the discrepancy pairs of (ACMv9, unseen DBLPv8), (ACMv9, Meta-graph-1), and (ACMv9, Meta-graph-2) generally show similar heat map patterns in terms of discrepancy. This indicates the effectiveness of the proposed meta-graph set simulation strategy, which could capture potential unseen graph data distributions by synthesizing diverse graph data with various graph augmentations. More detailed statistical information about the proposed DiscGraph set, including the number of graphs, the average number of nodes and edges, and the accuracy label distributions are provided in Appendix E.

4.5 Hyper-parameter Analysis of GNNEvaluator

MAEPerformanceoverFiveGCNs

(Citationv2→ACMv9,DBLPv8)

18

14

MAE

10

6

2

100

200

300

400

#Num of DiscGraphs (K)

Citationv2→ACMv9

Citationv2-→DBLPv8

t-Avg.

5 Conclusion

In this work, we have studied a new problem, GNN model evaluation, for understanding and evaluating the performance of well-trained GNNs on unseen and unlabeled graphs. A two-stage approach is proposed, which first generates a diverse meta-graph set to simulate and capture the discrepancies of different graph distributions, based on which a GNNEvaluator is trained to predict the accuracy of a well-trained GNN model on unseen graphs. Extensive experiments with real-world unseen and unlabeled graphs could verify the effectiveness of our proposed method for GNN model evaluation. Our method assumes that the class label space is unchanged across training and testing graphs though covariate shifts may exist between the two. We will look into relaxing this assumption and address a broader range of natural real-world graph data distribution shifts in the future.

We test the effects of different numbers of discrepancy meta-graphs ( K ) for GNNEvaluator training in Fig. 5. Concretely, we utilize the number of DiscGraphs for K = [100 , 200 , 300 , 400] to train the proposed GNNEvaluator for evaluating five GCN models that are well-trained on Citationv2 dataset and make inferences on unseen ACMv9 and DBLPv8 datasets. Considering the average performance under two transfer cases reflected in the line, despite K = 100 having a higher MAE, the average MAE over two cases does not change much with the number K for training the proposed GNNEvaluator . This indicates that the proposed GNNEvaluator can effectively learn to accurately predict node attributes by training on an appropriate number of DiscGraphs, without the need for a significantly excessive amount of training examples.

1

1

1

1

1

1

1

1

1

1

1

1

1

Acknowledgment

In this work, S. Pan was supported by an Australian Research Council (ARC) Future Fellowship (FT210100097), M. Zhang was supported by National Natural Science Foundation of China (NSFC) grant (62306084), and C. Zhou was supported by the CAS Project for Young Scientists in Basic Research (YSBR-008).

References

How AWS uses graph neural networks to meet customer needs. https://www.amazon. science/blog/how-aws-uses-graph-neural-networks-to-meet-customer-needs . Accessed: 2023-05-16.

Real-time fraud detection with gnns. https://github.com/awslabs/ realtime-fraud-detection-with-gnn-on-dgl .

Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning (ICML) , pages 1617-1629. PMLR, 2021.

Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell us about classifier accuracy under varying testing environments? In International Conference on Machine Learning (ICML) , pages 2579-2589. PMLR, 2021.

Weijian Deng, Stephen Gould, and Liang Zheng. On the strong correlation between model invariance and generalization. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15069-15078, 2021.

DC Dowson and BV666017 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis , 12(3):450-455, 1982.

Lukas Galke, Iacopo Vagliano, and Ansgar Scherp. Can graph neural networks go 'online '? an analysis of pretraining and inference. In Representation Learning on Graphs and Manifolds: ICLR 2019 Workshop , 2019.

Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. In International Conference on Learning Representations (ICLR) , 2022.

Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1134-1144, 2021.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML) , pages 1321-1330. PMLR, 2017.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems (NeurIPS) , 2017.

Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter. Assessing generalization of sgd via disagreement. In International Conference on Learning Representations (ICLR) , 2022.

Ming Jin, Yuan-Fang Li, and Shirui Pan. Neural temporal walks: Motif-aware representation learning on continuous-time dynamic graphs. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 , 2023.

Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, Shirui Pan, Vincent S. Tseng, Yu Zheng, Lei Chen, and Hui Xiong. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196 , 2023.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR) , 2017.

Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Out-of-distribution generalization on graphs: A survey. arXiv preprint arXiv:2202.07987 , 2022.

Yixin Liu, Kaize Ding, Huan Liu, and Shirui Pan. Good-d: On unsupervised graph out-ofdistribution detection. In Proceedings of the Sixteenth ACM International Conference on Web

Search and Data Mining (WSDM) , pages 339-347, 2023.

Yixin Liu, Kaize Ding, Jianling Wang, Vincent Lee, Huan Liu, and Shirui Pan. Learning strong graph neural networks with weak information. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) , 2023.

Yixin Liu, Yizhen Zheng, Daokun Zhang, Vincent Lee, and Shirui Pan. Beyond smoothing: Unsupervised graph representation learning with edge heterophily discriminating. In Proceedings of the Association for the Advanced of Artificial Intelligence (AAAI) , 2023.

Yuzhe Lu, Zhenlin Wang, Runtian Zhai, Soheil Kolouri, Joseph Campbell, and Katia P Sycara. Predicting out-of-distribution error with confidence optimal transport. In ICLR 2023 Workshop on Pitfalls of Limited Data and Computation for Trustworthy ML , 2023.

Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Normalizing flow-based neural process for few-shot knowledge graph completion. In The 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2023.

Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arxiv:2310.01061 , 2023.

Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. arXiv preprint arxiv:2306.08302 , 2023.

Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations (ICLR) , 2021.

Xiao Shen, Quanyu Dai, Sitong Mao, Fu-lai Chung, and Kup-Sze Choi. Network together: Node classification via cross-network deep network embedding. IEEE Transactions on Neural Networks and Learning Systems (TNNLS) , 32(5):1935-1948, 2020.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR) , 2017.

Man Wu, Shirui Pan, Chuan Zhou, Xiaojun Chang, and Xingquan Zhu. Unsupervised domain adaptive graph convolutional networks. In Proceedings of The Web Conference (WWW) , pages 1457-1467, 2020.

Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. In International Conference on Learning Representations (ICLR) , 2022.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR) , 2019.

Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 , 2021.

Shuwen Yang, Guojie Song, Yilun Jin, and Lun Du. Domain adaptive classification on heterogeneous information networks. In Proceedings of International Conference on International Joint Conferences on Artificial Intelligence (IJCAI) , pages 1410-1416, 2021.

Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-ofdistribution error with the projection norm. In International Conference on Machine Learning (ICML) , pages 25721-25746. PMLR, 2022.

He Zhang, Bang Wu, Shuo Wang, Xiangwen Yang, Minhui Xue, Shirui Pan, and Xingliang Yuan. Demystifying uneven vulnerability of link stealing attacks against graph neural networks. In International Conference on Machine Learning (ICML) , pages 41737-41752. PMLR, 2023.

He Zhang, Bang Wu, Xingliang Yuan, Shirui Pan, Hanghang Tong, and Jian Pei. Trustworthy graph neural networks: Aspects, methods and trends. arXiv preprint arXiv:2205.07424 , 2022.

He Zhang, Xingliang Yuan, Quoc Viet Hung Nguyen, and Shirui Pan. On the interaction between node fairness and edge privacy in graph neural networks. arXiv preprint arXiv:2301.12951 , 2023.

He Zhang, Xingliang Yuan, Chuan Zhou, and Shirui Pan. Projective ranking-based gnn evasion attacks. IEEE Transactions on Knowledge and Data Engineering (TKDE) , 2022.

Y Zhang, G Song, L Du, S Yang, and Y Jin. Dane: Domain adaptive network embedding. In Proceedings of International Conference on International Joint Conferences on Artificial Intelligence (IJCAI) , pages 4362-4368, 2019.

Xin Zheng, Yixin Liu, Zhifeng Bao, Meng Fang, Xia Hu, Alan Wee-Chung Liew, and Shirui Pan. Towards data-centric graph machine learning: Review and outlook. arXiv preprint arXiv:2309.10979 , 2023.

Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. Graph neural networks for graphs with heterophily: A survey. arXiv preprint arXiv:2202.07082 , 2022.

Xin Zheng, Miao Zhang, Chunyang Chen, Chaojie Li, Chuan Zhou, and Shirui Pan. Multirelational graph neural architecture search with fine-grained message passing. In 2022 IEEE

International Conference on Data Mining (ICDM) , pages 783-792. IEEE, 2022.

Xin Zheng, Miao Zhang, Chunyang Chen, Quoc Viet Hung Nguyen, Xingquan Zhu, and Shirui Pan. Structure-free graph condensation: From large-scale graphs to condensed graph-free data. arXiv preprint arXiv:2306.02664 , 2023.

Xin Zheng, Miao Zhang, Chunyang Chen, Qin Zhang, Chuan Zhou, and Shirui Pan. Auto-heg: Automated graph neural network on heterophilic graphs. In Proceedings of the ACM Web Conference (WWW) , pages 611-620, 2023.

Yizhen Zheng, Vincent CS Lee, Zonghan Wu, and Shirui Pan. Heterogeneous graph attention network for small and medium-sized enterprises bankruptcy prediction. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD) , pages 140-151. Springer, 2021.

Yizhen Zheng, Shirui Pan, Vincent Cs Lee, Yu Zheng, and Philip S Yu. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Yizhen Zheng, He Zhang, Vincent Lee, Yu Zheng, Xiao Wang, and Shirui Pan. Finding the missing-half: Graph complementary learning for homophily-prone and heterophily-prone graphs. International Conference of Machine Learning (ICML) , 2023.

Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems (NeurIPS) , 34:27965-27977, 2021.

Appendix

This is the appendix of our work: 'GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labels' . In this appendix, we present additional information regarding the GNN model evaluation problem and corresponding solutions, including in-depth discussions that distinguish our method from related works, dataset statistics used in our experiments, comprehensive details about our compared baseline methods, training details with the performance of well-trained GNN models on both the test set of the observed graph and the unseen test graphs (for ground-truth reference), and more experimental results with detailed analysis.

A Related Work

Predicting Model Generalization Error. Our work is relevant to the line of research on predicting model generalization error, which aims to develop a good estimator of a model's classifier performance on unlabeled data from the unknown distributions in the target domain, when the estimated models' classifier has been trained well in the source domain with labeled data [6, 9, 34, 5, 10, 4]. Typically, Guillory et al. [10] proposed to estimate a classifier's performance of convolutional neural network (CNN) models on image data based on a designed criterion, named difference of confidences (DoC), that estimates and reflects model accuracy. And Garg et al. [9] proposed to learn a score-based Average Thresholded Confidence (ATC) by leveraging the softmax probability of a CNN classifier, whose accuracy is estimated as the fraction of unlabeled images that receive a score above that threshold. In contrast, Deng et al. [6, 4] directly predicted CNN classifier accuracy by deriving distribution distance features between training and test images with a linear regression model.

However, these existing methods mostly focus on evaluating CNN model classifiers on image data in computer vision, and the formulation of evaluating GNNs for graph structural data still remains underexplored in graph machine learning. Concretely, conducting the model evaluation on GNNs for graph structural data has two critical challenges: (1) different from Euclidean image data, graph structural data lies in non-Euclidean space with complex and non-independent node-edge interconnections, so that its node contexts and structural characteristics significantly vary under wide-range graph data distributions, posing severer challenges for GNN model evaluation when serving on unknown graph data distributions. (2) GNNs have entirely different convolutional architectures from those of CNNs, when GNN convolution aggregates neighbor node messages along graph structures. Such that GNNs trained on the observed training graph might well fit its graph structure, and due to the complexity and diversity of graph structures, serving well-trained GNNs on unlabeled test distributions that they have not encountered before would incur more performance uncertainty.

Hence, in this work, we first investigate the GNN model evaluation problem for graph structural data with a clear problem definition and a feasible solution, taking a significant step towards understanding and evaluating GNN models for practical GNN deployment and serving. Our proposed two-stage GNN model evaluation framework directly estimates the node classification accuracy results of specific well-trained GNNs on practical unseen graphs without labels. Our method could not only facilitate model designers to better evaluate well-trained GNNs' performance in practical serving, but also provide users with confidence scores or model selection guidance, when using well-trained GNNs for inferring on their own test graphs.

Unsupervised Graph Domain Adaption. Our work is also relevant to the unsupervised graph domain adaption (UGDA) problem, whose goal is to develop a GNN model with both labeled source graphs and unlabeled target graphs for better generalization ability on target graphs. Typically, existing UGDA methods focus on mitigating the domain gap by aligning the source graph distribution with the target one. For instance, Yang et al. [33] and Shen et al. [27] optimized domain distance loss based on the statistical information of datasets, e.g. , maximum mean discrepancy (MMD) metric. Moreover, DANE [39] and UDAGCN [29] introduced domain adversarial methods to learn domain-invariant embeddings across the source domain and the target domain. Therefore, the critical distinction between our work and UGDA is that our work is primarily concerned with evaluating the GNNs' performance on unseen graphs without labels, rather than improving the GNNs' generalization ability when adapting to unlabeled target graphs. Besides, different from UGDA which uses the unlabeled target graphs in their model design and training stage, the GNN model evaluation problem

explored by our work is not allowed to access the unlabeled test graphs, i.e. , unseen in the whole GNN model evaluation process.

Graph Out-of-distribution (OOD) Generalization & Detection. Out-of-distribution (OOD) generalization [18, 48] on graphs aims to develop a GNN model given several different but related source domains, so that the developed model can generalize well to unseen target domains. Li et al. [18] categorized existing graph OOD generalization methodologies into three conceptually different branches, i.e. , data, model, and learning strategy, based on their positions in the graph machine learning pipeline. We would like to highlight that, even ODD generalization and our proposed GNN model evaluation both pay attention to the general graph data distribution shift issue, we have different purposes: our proposed GNN model evaluation aims to evaluate well-trained GNNs' performance on unseen test graphs, while ODD generalization aims to develop a new GNN model for improve its performance or generalization capability on unseen test graphs. Moreover, OOD detection [32], aims to detect test samples drawn from a distribution that is different from the training distribution, with the definition of distribution to be well-defined according to the application in the target domain. Hence, the primary difference between OOD detection and our GNN model evaluation is, OOD detection works on detecting OOD test samples at the data level and our GNN model evaluation works on evaluating well-trained GNN's performance on ODD data at the model level.

B Dataset Statistics

We provide the details of dataset statistics used in our experiments in Table. A1, where these datasets meet the covariate shift assumption between any pair of test cases.

C Baseline Methods Details

Average Thresholded Confidence (ATC) & Its Variants. This metric [9] learns a threshold on CNN's confidence to estimate the accuracy as the fraction of unlabeled images whose confidence scores exceed the threshold as:

where f θ ∗ ( · ) denotes the well-trained CNN's classifier with the optimal parameter θ ∗ , and s ( · ) denotes the score function working on the softmax prediction of f θ ∗ ( · ) . When the context is clear, we will use f ( · ) for simplification. We adopted two different score functions, deriving two variants as: (1) Maximum confidence variant ATC-MC with s ( f ( x ′ j )) = max k ∈Y f k ( x ′ j ) ; and (2) Negative Entropy variant ATC-NE with s ( f ( x ′ j )) = ∑ k f k ( x ′ j ) log ( f k ( x ′ j ) ) , where Y = { 1 , 2 , . . . , C } is the label space. And for t in Eq. (9), it is calculated based on the validation set of the observed training set ( x val u , y val u ) ∈ S val :

̸

where p ( · ) denotes the predicted labels of samples. For all the calibration variants with '-c' in our experiments, they conduct standard Temperature Scaling [11] with following equations:

where z u denotes the network output logits before softmax and T is the temperature scaling factor.

Threshold-based Method. This is an intuitive solution introduced by [6], which is not a learningbased method. It follows the basic assumption that a class prediction will likely be correct when it has a high softmax output score. Then, the threshold-based method would provide the estimated accuracy of a model as:

where τ is the pre-defined thresholds as τ = { 0 . 7 , 0 . 8 , 0 . 9 } on the output softmax logits of CNNs. This metric calculates the percentage of images in the entire dataset whose maximum entry of logits are greater than the threshold τ , which indicates these images are correctly classified.

AutoEval & Our Adaption AutoEval-G. This is a learning-based method of estimating classifier accuracy by utilizing dataset-level feature statistics [6]. AutoEval synthesizes a meta image dataset D (a dataset comprised of many datasets) from the observed training dataset D ori, and conducts the linear regression to learn the CNN classifier's accuracy based on the dataset-level feature statistics between the observed training image data and unseen real-world unlabeled image data. The accuracy linear regression with the dataset statistic feature f linear is written as:

where FD is the Fréchet distance [7] to measure the domain gap with the mean feature vectors µ ori and µ , the covariance matrices Σ ori and Σ of D ori and D , respectively. And Tr ( · ) denotes the trace of the matrix, w 1 and w 0 denote the parameters of linear regression.

We make the following necessary adaptions to expand the AutoEval method to graph structural data on GNN model evaluation, deriving AutoEval-G by: (1) we synthesize a meta-graph set G meta as demonstrated in Sec. 3.2 of the main manuscript; (2) we calculate the Maximum Mean Discrepancy (MMD) distance as AutoEval-G's graph dataset discrepancy representation f ′ linear , which measures the graph distribution gap between the observed training graph S and each meta-graph g meta ∈ G meta with their node embeddings from the well-trained GNN S as:

D Well-trained GNN Model Details

In this section, we provide the details of well-trained GNN models in terms of their architecture parameters and training details.

For all GNN and MLP models, the default settings are : (a) the number of layers is 2; (b) the hidden feature dimension is 128; (c) the output feature dimension before the softmax operation is 16. The hyperparameters of training these GNNs and MLP are shown in Table A2. The five seeds for training each type of models (GCN, GraphSAGE, GAT, GIN, MLP) are { 0 , 1 , 2 , 3 , 4 } , and the node classification performance (accuracy) on the test set of each observed training graph and the ground-truth accuracy on the unseen test graphs without labels are shown in Table A3, Table A4, Table A5, Table A6, and Table A7, respectively.

E More Experimental Analysis and Results.

E.1 DiscGraph Statistics

As a vital component of our proposed two-stage GNN model evaluation framework, DiscGraph set captures wide-range and diverse graph data distribution discrepancies. By fully exploiting the latent node embeddings and node class predictions from well-trained GNNs, the DiscGraph set involves representative node attributes incorporating diverse discrepancies through a discrepancy measurement function, leading to effective instructions for the training of GNNEvaluator . Hence, in the following, we show the statistical information of our derived DiscGraph set in Table A8. Note that, each generated DiscGraph set is related to a specific dataset and a specific well-trained GNN, as it considers the outputs of node embeddings and predictions on a well-trained GNN trained by a specific graph dataset.

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

Nodes in unseen graphs

2000

1000

0.75

0.50

Sdesb uaasun ui sopon #

2000

1000

0.50

0.75

Nodes in unseen graphs

2000

1000

0.8

0.6

0.25

Aouedassia

- 0.25

0.4

3000

0.00

3000

0.00

3000

0.2

0.25

-0.25

0.0

#

4000

0.50

0.75

4000

0.50

0.75

4000

0.2

0.4

5000

5000

5000

#Nodesin observed traininggraphs

#Nodes inobserved traininggraphs

#Nodes in observed training graphs

(a)Node AttributesinDiscGraph ofACMv9-→unseenDBLPv8(GAT)

AEPerformanceonCitationv2-→ACMv9

AEPerformanceonCitationv2-→DBLPv8

MAEPerformanceoverFiveGCNs

12

18

（Citationv2→ACMv9,DBLPv8)

10

16

18

8

14

14

6

12

MAE

10

4

2

10

6

0

8

2

100

200

300

400

100

200

300

400

100

200

300

400

#Num ofDiscGraphs(K)

#Num of DiscGraphs (K)

# Num of DiscGraphs (K)

GCN-2

GCN-4

GCN-5

GCN-2

GCN-4

GCN-5

Citationv2-→ACMv9

Citationv2-DBLPv8

E.2 More Visualization Results of Discrepancy Node Attributes

In Fig. A1, we present more visualization results on discrepancy node attributes in the proposed DiscGraph set for different GNN models, i.e. , (a) GAT, (b) GraphSAGE, and (c) GIN, under ACMv9 → DBLPv8 case.

As can be observed, for different GNNs, the node attributes in the proposed DiscGraph set show significant differences, denoting the effectiveness of the proposed discrepancy measurement function, which could capture model-specific discrepancy representations effectively. And such discrepancy representations could instruct our proposed GNNEvaluator to learn graph data distribution discrepancies with different well-trained GNNs for accuracy regression.

E.3 More Results on The Number of DiscGraphs

We provide more results for illustrating the effects of different numbers of discGraphs for training the proposed GNNEvaluator in Fig. A2. Along with the MAE results reported in Fig. 5 in the main manuscript, we report the Absolute Error (AE) results for each GCN model under C → A and C → D cases. Lower AE denotes better performance. It can be observed that different GCN models have different appropriate K values for GNNEvaluator training, but they show similar trends for each unseen test graph in the left and the middle figures. For instance, the proposed GNNEvaluator has highest AE when it is trained with K = 100 DiscGraphs, and comparable AEs with K = 300 and K = 400 for evaluating all five GCNs under the C → D case. Nevertheless, in general, as shown in the red 'Avg.' line of the right figure, the performance of the proposed GNNEvaluator would not significantly vary along the number of DiscGraphs when evaluating all GCNs that are well-trained on Citationv2 dataset and served on unseen DBLPv8 dataset.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of evaluating the performance of well-trained Graph Neural Networks (GNNs) on unseen test graphs without ground-truth labels. Traditional evaluation methods that rely on annotated datasets are not applicable in real-world scenarios where test graphs are unlabeled. The paper proposes a two-stage evaluation framework: DiscGraph set construction and GNNEvaluator training and inference. The framework effectively captures the distribution discrepancies between training and test graphs, models graph structural discrepancies, and estimates node classification accuracy without labeled data. Experimental results indicate the success of the proposed method.\n\nSTRENGTHS:\n1. I really appreciate the practical value of this paper since it often requires a long time for evaluation a model in the real-world scenario. The paper introduces the novel problem of evaluating GNN models on unseen test graphs without labels, which is essential for real-world GNN deployment and serving. \n\n2. The proposed framework first constructs a set of meta-graphs to simulate potential unseen test graphs, captures graph data distribution discrepancies, and models graph structural discrepancies using latent node embeddings and node class predictions. The GNNEvaluator is then trained to estimate node classification accuracy based on the representative DiscGraph set. \n\n3. The paper evaluates the proposed method on real-world unlabeled test graphs and demonstrates its effectiveness in estimating node classification accuracy.\n\nWEAKNESSES:\n1.  In section 3.3, the author introduces extracting a seed sub-graph Sseed from the observed training graph S, Nonetheless, the details about how to extract the seed subgraph lack of introduction. \n\n2.  The estimation of the represented discrepancy is too simple. I am concerned that not be generalized to larger graph in more domains, especially those larger datasets, e.g., ogbn-arxiv, ogbn-products. Those graphs show more diverse patterns and the proposed method may not work in this scenario. \n\n3. The graph argumentation method is to provide sufficient quantity. Nonetheless, the graph argumentation algorithms seem to only disturb the graph very small. I am wondering how those methods can be generalized to other scenarios with larger gap. Moreover, I am wondering whether it can help to predict the performance on the model trained with those graph argumentation techniques, e.g., dropedge.\n\n4.  I am wondering why the GNN evaluator can have the transferability, e.g., the learned GNN evaluator on SAGE can also help to predict the performance of the GCN.\n\nQUESTIONS:\nSee the above weakness\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel problem called GNN model evaluation, aiming to assess the performance of a Graph Neural Network (GNN) on unseen graphs without labels. The authors propose a two-stage GNN model evaluation framework, which includes DiscGraph set construction and GNNEvaluator training and inference. The DiscGraph set is designed to capture diverse graph data distribution discrepancies using a discrepancy measurement function that exploits GNN outputs. The GNNEvaluator, composed of a GCN architecture and an accuracy regression layer, learns to estimate node classification accuracy with effective supervision from the DiscGraph set. The method demonstrates effectiveness in evaluating GNN performance on real-world, unlabeled test graphs, achieving low errors compared to ground-truth accuracy.\n\nSTRENGTHS:\nThis paper is the first to evaluate GNN performance on out-of-distribution (OOD) graphs, which may inspire future studies. The meta-graph set construction and the three characteristics are intriguing and useful. Additionally, the proposed method demonstrates significant performance improvements over the baselines.\n\nWEAKNESSES:\nThis paper has a few areas that could be improved upon. I would consider raising the score if the authors addressed the first two points. \n1. The paper lacks a discussion of the use cases for the proposed method, considering the prediction error remains relatively high (Please refer to Question 1).\n2. The experiments involve only four datasets, and there is a lack of comprehensive study on node classification in different settings (Please refer to Question 2).\n3. The paper does not provide ablation studies for the predictor in GNNEvaluator (Please refer to Question 3).\n\nQUESTIONS:\nTo enhance the manuscript, the authors may consider addressing the following points:\n\n1. Although the proposed method outperforms the baselines, the prediction error remains high. For example, in ACMv9 -> DBLPv8, the performance of GCN, SAGE, GAT, GIN, and MLP are 45.51%, 45.49%, 44.84%, 50.37%, and 33.28%, respectively, and the evaluation errors are 4.85%, 4.11%, 12.23%, 10.14%, 22.20%, respectively. The relative evaluation errors for various GNNs are quite significant. With such high relative errors, it is challenging for GNNEvaluator to distinguish the performance of any two GNNs. In light of this, what are the practical applications of the proposed method? Including a discussion of compelling real-world applications could make this paper more appealing.\n\n2. While the paper focuses on node classification settings, which is an important application of GNNs, the experiments only evaluate inductive node classification on citation networks. To make the experiments more comprehensive, I suggest that the authors investigate (1) inductive node classification on heterophilous graphs, and (2) transductive node classification.\n\n3. The paper employs a two-layer GCN as a predictor, but detailed ablation studies are missing. For example, (1) what is the variance of the GCN predictor? (2) could a simpler model (e.g., SGC) or a more complex model achieve better performance? It would be beneficial to include such analysis in the paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper studies the problem of evaluating a GNN model, to estimate the accuracy on unseen unlabeled data for a well trained model. The goal of this paper is not to improve the generalization error of a GNN model, rather estimate the well-trained model error. This is a novel track for model generalization error, where before only studied in Euclidian data. It contains two parts, DiscGraph, that extracts a set of meta-graphs from the original training graph to simulate and capture the discrepancies of diverse graph data distributions for the aforementioned unseen test graphs. In addition, it proposes GNNEvaluator, a model to estimate the trained model accuracy  on the unseen test data set.

STRENGTHS:
This is a very well written paper, and it is aiming at a new domain of research for GNNs. The proposed idea is very well presented, in addition to detailed analysis of each part of the GNNEvaluator. Due to lack of related work for GNN model, the authors introduced and adapt recent works from recent CNN algorithm. Overall, I found this paper very novel.

WEAKNESSES:
I found the paper very interesting, however, the experimental studies is very limited (understandable due to the novelty of the problem). However it is not very convincing to see the result only on one set of application (for limited number of labeled class).
Also it would be beneficial to see the performance on different architectures (deeper GNN) as right now the GNNEvaluator only uses 2 layers GCN.

QUESTIONS:
- Related to the problem mentioned above, how we can be certain about the number of DiscGraph generated for first stage?
- What is the time and memory requirement for these additional stages for GNN training? Given the limited and small evaluated graphs, how scalable is current approach?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new research problem named GNN model evaluation, to evaluate well-trained GNNs on observed training graphs for testing on real-world unobserved test graphs without labels. To achieve this goal, this work (1)constructs a DiscGraph set to model the distribution differences of graph datasets, and (1) designs a GNNEvaluator to learn on the constructed DiscGraph set and directly output the overall node classification accuracy by a regression model.\nExperimental results on some shifted-distribution graph datasets verify the proposed method and questions' effectiveness.\n\nSTRENGTHS:\n(1) This work first defines the GNN model evaluation problem, which is a very practical problem in real-world gnn model deployment and selection for user sides. This work has good originality and explores a new research problem for real-world GNN deployment and applications. It could inspire some interesting explorations in terms of GNN model service and deployment.\n\n\n(2) The challenges of the GNN model evaluation, the corresponding technical solutions proposed by this work and experimental settings are clearly clarified. And the DiscGraph set constructed for capturing the graph dataset discrepancy is novel with sufficient quantity, represented discrepancy, and known accuracy. The calculated distance of two graph dataset distribution as the node attributes sound rational.\nAnd a new trained GNN regressor on this constructed DiscGraph set for predicting GNN's classification result over the whole dataset is reasonable. \n\n(3) The experimental setting could align with the real-world application scenario, and the result parts could support this work's claims. Appropriate ablation study, DiscGraph set analysis, and hyper-parameter analysis provides adequate information for verifying the method's effectiveness with small errors from the ground truth accuracy.\n\n(4) This work is well-structured and clearly written with logic.\n\nWEAKNESSES:\nSee Questions\n\nQUESTIONS:\n(1) How to obtain the seed graph for augmentation to generate the meta-graph? it would be better to show the difference between the observed graph used for training and the created meta-graph for constructing the DiscGraph set.\n\n(2) Do the results in each table (tables 1 2 and 3), share the same trained models? for instance, in table 2, a gcn model is well-trained on citationv2 but evaluated the same model on both acmv9 and dblpv8? \n\n(3) The authors should provide more experimental analysis and discussions to demonstrate why such a method design could benefit gnn model evaluation compared with other baselines. Besides, there might be a typo in table 1 first six columns? should be acmv9 to citationv2 dataset?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}