Kati8/ICLR_2023_2025 huggingface dataset:
authors: Xin Zheng
authors: Dongjin Song
authors: Qingsong Wen
authors: Bo Du
authors: Shirui Pan
reviews: 
Comments: This work proposed a metric named LEBED, learning behavior discrepancy score, for a new research problem "online GNN evaluation" in terms of the graph distribution shift issue during test time. By imitating the learning behavior in GNN training parameters, this metric could indicate the GNN model performance without the test graph labels through the parameter space distance between the training and test graphs. Moreover, this work contains a parameter-free optimality criterion which introduces graph structure information to guide GNN retraining on the test-time graphs. Experiments under different training-test graph shifts could verify its effectiveness.
Strengths: The strength of the work:

(1) The overall idea and research topic of "online GNN evaluation" is interesting and novel for understanding the real-world GNN model performance, especially the graph distribution shift issue is also considered, making it more applicable.  This can be a new research direction that inspires future work on GNN model understanding and real-world deployment.

(2) The solution with the proposed LEBED metric for estimating test-time generalization error on graph data appears logical to me. With retraining GNN on the test unlabeled graphs with distribution shifts, its parameter space metric with the context of learning behavior discrepency is reasonable for GNN model evaluation, especially given its graph structure discrepency-related criterion.

(3) The experiments involve graphs with diverse distribution shift cases, domains, and scales, making this metric relatively convincing. And the performance of this method seems effective enough as they argued to indicate the test-time GNN performance.

(4) The paper is well-structured, and easy to understand. Its logic and writing look good to me.
Weaknesses: Q1: How to explain that some baseline methods with some cases positive values but other cases negative values, more explanations of the experimental results should be provided here.

Q2: How easily can the proposed LEBED be integrated into existing GNN frameworks or pipelines, considering it seems a general GNN model evaluation method covering different models and test graph data?

Q3: Does retraining the GNN on unlabeled test graphs introduce significant computational overhead, and what about the time complexity?
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper studies the training-test graph data distribution shift problem along with GNN model evaluation problem. 
Its method contains a GNN retraining part and a parameter-free optimality criterion. It develops a metric LEBED to estimate the test-time errors of trained GNN models, and this proposed metric can be taken as an indicator to show the performance of GNN on real-world test graphs without labels. Its experimental parts involves many evaluation models and graph datasets with distribution shifts covering node feature, domain, and temporal, the experimental results could demonstrate the metric's effectiveness under the online test-time GNN evaluation scenario.
Strengths: 1. The proposed online GNN evaluation problem under test-time graph distribution shift is insightful. It attempts to understand the well-trained GNN models' performance in the practical GNN model deployment and service scenario, which could be useful to real-world GNN development.

2. The introduction of the proposed evaluation metric is thoughtful with a relatively clear problem definition. The main idea of leveraging the GNN retraining discrepency to align the training-test graph discrepancy is comprehensible when the online scenario does not allow to access the training graphs.

3. The paper tests and evaluates their proposed LEBED metric on a wide range of real-world potential graph distribution shift datasets, it could provide empirical evidence of its practical utility as they claimed. Their findings can be interesting in that some existing CNN evaluation metrics would not commit consistent performance for GNNs.
Weaknesses: 1. In the dataset part, could you provide more details about how the graph distribution shifts are simulated in the experiments? And can these test graph datasets be publicly released?

2. Given the proposed GNN re-training process on test-time graphs, is the parameter-free criterion in Eq.(6) used for stopping the retraining but not being optimized in retraining? More explanations are needed here.

3. How is the ground-truth performance (test error) of these distribution-shifted test graphs on the online GNN models?
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good