Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies an interesting question in terms of which graph datasets should be selected for GNN pre-training tasks. The authors propose a novel graph selector that is able to provide the most instructive data for the model. The criteria in the graph selector include predictive uncertainty and graph properties (graph entropy, density, degree, etc). Besides, they propose a data-active graph pre-training (APT) framework, which integrates the graph selector and the pre-training model into a unified framework.

strength_and_weaknesses: **Strength**

- Overall, the problem studied seems to be a novel angle in the context of GNN pre-training. Efficient pre-training is currently a new trend in other domains such as NLP and CVs with a focus on how to select informative training instances. It is important to shed some light in the context of how to better use the pre-training graphs data. 

- The core idea the author proposed intuitively makes sense. Indeed, graphs with certain characteristics such as higher network entropy, larger density, higher average degree, higher degree variance, or a smaller scale-free exponent will contain a larger amount of information. Even though the authors do not provide theoretical justification, empirically, this work does provide good insight into GNN pre-training. 

**Weaknesses**

- The intuition figure 1 is a bit hard to fully understand. For the first row, when scaling up the sample size, does it mean we keep the same number of pre-train graphs but we vary the same portion of sample size from each pre-train graph dataset? For the bottom row, the total number of pre-train graphs in the paper is eleven and the number of graphs in the figure is only up to ten. The author should provide what is the benchmark results if we use all the possible pre-train datasets to properly compare 

- Directly using contrastive loss to define and measure predictive uncertainty is not questionable. 

- The notation of this paper is a bit messy. For example, both scaler and vector use non-bolded lowercase (degree vector and node degree) which is not conventional. 

- The selection mechanism is a bit complicated and heavily handcrafted. Currently, the graph selection policy uses a combination of 6 loss terms in total. Besides, loss terms work on a very different scale and are combined together with an additional time-adaptive parameter. I am not sure if this design is reasonable. Empirically, there is no proper ablation study on the choice of which graph property to be included in the pre-train graph selection criteria. 

- There is an additional proximal regularization term in the model design, which aims to better preserve the knowledge or information contained in previous input data when we train on new incoming data, a similar design component in the continue learning paradigm. I am not able to fully follow the rationale of this design. The pre-training problem is completely different from the catastrophic forgetting setting since we will normally shuffle the training order of the samples after each epoch. The claim of "previous input data will be forgotten or covered by new incoming data" is invalid if we shuffle the data training order. Besides, the $\mathbf{\theta}$ parameters learned from the first $j$ graphs, does it mean in terms of the memory complexity of the model will be $j$ times larger since we need to store the previous training iteration of the model?

- Experimental results-wise, I have several concerns: 1) It seems the work is directly established on top of the GCC paper, but with completely different suits of pre-training datasets and downstream datasets. The choice of dataset section seems arbitrary and suspicious. Could the author directly work on the precious experiment setting in GCC? or what's the reason behind the complete switch of datasets? 
 2) The baseline uses are quite outdated, using baselines only coming before the year 2020. A lot of recent work in terms of GNN pre-training or graph data augmentation should be included [1] - [8].  3) One suggestion to the authors, since the paper uses a lot of downstream tasks and lots of numbers in the result table, a better way to present the results can be considered. For example, providing an average rank number across all the datasets for each method will be informative and helpful to deliver the message to readers.  3) I will suggest including the full pre-training dataset results in the experiment table to see the effectiveness of the pre-train graph selection scheme. Or should I interpret the GCC results as the full pre-training dataset results?  4) Can the author provide the training time comparison? It seems that to fully  

- Can the author comment on the relationship of this work to some recent adaptive graph positive sample generation work? It seems they all achieve a similar end goal by finding the proper training samples [3] [5] [6] [7].


[1] Hu, Ziniu, et al. "GPT-GNN: Generative pre-training of graph neural networks." KDD 2020.

[2] Xu, Dongkuan, et al. "Infogcl: Information-aware graph contrastive learning." NeurIPS 2021.

[3] Zhu, Yanqiao, et al. "Graph contrastive learning with adaptive augmentation." Proceedings of the Web Conference 2021. 2021.

[4] Zhu, Yanqiao, et al. "An Empirical Study of Graph Contrastive Learning." NeurIPS 2021.

[5] You, Yuning, et al. "Graph contrastive learning automated." ICML 2021.

[6] Lee N, Lee J, Park C. Augmentation-free self-supervised learning on graphs. In Proc. of the AAAI Conference on Artificial Intelligence 2022.

[7] Han et al. "G-Mixup: Graph Data Augmentation for Graph Classification." ICML 2022.

[8] Hou, Zhenyu, et al. "GraphMAE: Self-Supervised Masked Graph Autoencoders." KDD 2022.








clarity,_quality,_novelty_and_reproducibility: Please refer to Strength And Weaknesses.
summary_of_the_review: Please refer to Strength And Weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a novel framework for cross-domain graph pre-training using fewer training samples. The paper first demonstrates the phenomenon of the “curse of big data” - more training and graph datasets do not always lead to better performance in downstream node and graph classification tasks. Next, it presents a data-active graph pre-training (APT) framework that consists of two parts.  (1) The graph selector uses a loss that combines the predictive uncertainty of a data sample and its graph properties (entropy, density, etc.) to make an informed decision on the most instructive data samples for the model. (2) The pre-training model that learns from the graph selector and provides it guidance for better selection.  The paper implements the framework for different datasets and tasks to show that the proposed method improves performance for downstream tasks in different domains. The proposed method is compared with existing graph-based pre-trained models. The paper also includes results on the ablation of the different components of the graph selector, which is an important analysis for such a framework. 
strength_and_weaknesses: Strengths:

- The paper is overall well presented, and the methods are well described
- It aims to tackle an important problem in the field and proposes an interesting strategy to do so
- The experiments and results are thorough and demonstrate better downstream performance on a variety of tasks with fewer training samples
- The presentation of the phenomenon of large training samples not adding to the performance is useful 

Weakness:

It would  be helpful if the following points regarding experimental settings, results, and implementation could be clarified:

- Why were different models used for node (logistic regression) versus graph classification (SVMs)?
- The ProNE model seems to outperform the APT model significantly for certain datasets. It would be helpful to discuss this result.
- How is F (the maximal period of training one graph) different from the training epochs, or is F measured in seconds? 
- How sensitive is the model to the predictive uncertainty thresholds Tg and Ts (set to 3 and 2, respectively)? The decision choices for F, Tg, and Ts could be better described. 
- Are the baseline methods being re-trained on new datasets? If so, does it make sense to use the default hyperparameters for the same? 

Minor Comments:
- I may have missed this, but what was the selection strategy for pre-training versus test data? 
- In appendix C, Table 3, the description of the ‘msu’ datasets implies it belongs to the protein category instead of the social category. The authors should make corrections where necessary.
- In appendix C, specifically in Tables 3,4, and 5, it would be helpful to define the abbreviations |V| , |E|, and |G| in the captions. 

clarity,_quality,_novelty_and_reproducibility: Clarity: The motivation, explanation, and results are clear. The clarity of the experiment section could be improved with reduced reliance on the Appendix. 
 
Quality: The paper is well-written, and the results support the claims. 
 
Novelty: Some of the ideas are present in the existing literature, but their implementation in the graph pretraining setting is new. The writing does a good job of placing the work appropriately with respect to existing literature.
 
Reproducibility: An anonymous code repository has been provided. 

summary_of_the_review: The studied problem in this paper is important, the idea is new, and the experimental results are comprehensive. Specifically, it is demonstrated that the APT framework can achieve SOTA performance on downstream tasks by using less pre-training cross-domain data. The paper also shows this framework reduces the time complexity for pre-training. It would be great if some of the experimental choices and results could be described more clearly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on pre-training graph neural networks (GNNs), aiming to identify a few most important examples for pre-training. For this purpose, a graph selector is designed, which uses the predictive uncertainty from a pre-trained GNN model as features for data selection. Extensive experiments on a few datasets prove the effectiveness of the proposed framework.
strength_and_weaknesses: Strengths:
1. The problem of GNN pre-training is drawing growing attention in the graph machine learning field. Most works for GNN pre-training aim to collect more data or design better pre-training objectives to boost the performance. By contrast, this paper proposes a new idea, i.e., using data in a more efficient way by picking up those informative examples for training. Overall, I think the idea is interesting and novel.
2. The proposed approach is quite intuitive.
3. The experiment is extensive, where 13 graphs are used for pre-training. The results show that the proposed approach APT outperforms many existing methods.

Weaknesses:
1. In Sec. 3.1, this paper defines the predictive uncertainty by using the InfoNCE loss. Although this definition is intuitive, I feel like the design is heuristic and is lack of theoretical guarantee. It would be helpful to further discuss the advantage of the InfoNCE loss for measuring the sample importance.
2. In terms of graph properties, this paper considers a few features of graphs, including network entropy, density, average degree, and, degree variance. Although they are very important to describe a graph, I feel like these properties are insufficient to yield a comprehensive characterization of the graph. I wonder whether it is possible to automatically learn some features as the structural properties of graphs.
clarity,_quality,_novelty_and_reproducibility: See above comments.
summary_of_the_review: See above comments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to select some graph data to improve the pre-training of GNNs. Existing works use all data to conduct pre-training, and it is observed that more data does not necessarily leads to better accuracy. The authors propose to use prediction uncertainty and graph properties to select data for training. Empirical results show that the proposed methods improve the accuracy of downstream tasks.
strength_and_weaknesses: Strength
1. The observation that more data does not necessarily leads to better accuracy is neat.
2. Using uncertainty and graph properties to select data makes sense.
3. The empirical results are strong.


Weakness
1. Discussions of related works are shallow, which make the contributions unclear. For instance, in Section 5, the Pre-training in CV and NLP part is nothing more than saying that CV and NLP have used pre-training. However, the readers are interested in what methods they use for pre-training, especially for selecting data, and what are the differences from this work. The same applies to the Graph pre-training part. My suggestion is that when you discuss some closely related works, you should make the readers understand how these works on the methodology level, instead of only describing their targets. To my knowledge, uncertainty is widely used for sample selection is many areas. What are the differences from this work?

2. Method design lacks explanation and does not make sense in some places. Eq. (4) calculates the mean of several graph property metrics (i.e., network entropy, density, degree). However, the appendix shows that these metrics differ in scale, which means that a metric may dominate the others when computing the mean. It also needs to be explained why we prefer larger values of these metrics (the network entropy part is clear). My guess that the intuition is that we prefer complex graphs that are likely to contain a diverse set of graph structures, and the proposed method tends to select graph with larger average degrees.

3. Experiment and analysis need to be significantly improved. (1) What is the time used for data selection? Will it be a concern? (2) In Tables 1-2, different variants of APR perform the best on different datasets, what are the reasons? Is it related to the properties of the test graph? (3) Why not compare with some well-known GNN baselines, such as GraphSAGE, which can be trained in an unsupervised manner? (4) How does different graph properties affect accuracy? How does the weight of uncertainty and graph properties affect performance? 

clarity,_quality,_novelty_and_reproducibility: Clarity needs to be significantly improved;
Quality moderate;
Novelty limited;
Reproducibility good as code is provided.

summary_of_the_review: This paper proposes to select data to pre-train GNNs. The idea makes sense but there are significant problems in method design and experiment analysis. Novelty is also unclear from the presentation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates how to choose the most effective graph datasets for pre-training Graph Neural Network (GNN) models. The authors propose a graph selector that utilizes predictive uncertainty and various graph properties, such as graph entropy and density, to select the most instructive data. They also present a data-active graph pre-training (APT) framework that combines the graph selector and the pre-training model into a single system.

The reviewers raised two major concerns on the effectiveness of the proposed pre-training approach for Graph Neural Networks (GNNs) and the practicality of the heavily handcrafted training objective: (1) The authors did not provide evidence of the benefits of the pre-training approach under the proper experimental setting, in which the encoder is frozen during fine-tuning for downstream tasks. (2) The handcrafted training objective heavily replies on certain graph properties，while the downstream graphs may have very different beneficial graph statistics. Therefore, the pre-training can be less effective without extensive tuning.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject