Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work proposes a novel benchmark for continual graph learning. The benchmark factorizes continual graph learning tasks into node- and graph-level tasks, and task-incremental and class-incremental settings. They also releases a toolkit for training, evaluating and visualizing CGL methods. Finally, the authors performs a systematic comparison of state-of-the-art CGL methods.

Contributions:

- A novel and comprehensive benchmark for continual graph learning methods.

- An easy-to-use toolkit for training, evaluating and visualizing CGL methods.

- A systematic comparison of existing CGL methods.

strengths: - *Presentation.* The paper is well-written and its main idea is easy to flow.

- *Benchmark.* The benchmark is well-designed and it addresses an important topic in graph learning with significant real-world application.

- *Analysis of existing methods.* Using the proposed benchmark and toolkit, the authors perform a systematic comparison of existing CGL methods.

weaknesses: - The benchmark contains implementation of a variety of existing CGL methods. However, it does not properly document how to implement and adapt new CGL methods to use this benchmark. This would limit the usability of the benchmark and the companying toolkit.

correctness: The paper does not have apparent technical errors. The claims are supported by experiments.

clarity: The paper is well-written. The main idea is easy-to-follow.

relation to prior work: The related work is comprehensively discussed. The authors not only relate their work to continual learning, but also compares it with existing benchmarks for CGL.

additional feedback: ---
review_title: A good benchmark for continual graph learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: CGLB, provide a comprehensive toolkit for training, evaluation, and visualization of different CGL
models, which could greatly alleviate the burden/risks to investigate CGL problems. Finally, with
CGLB and the toolkit, we conduct comprehensive studies on the existing state-of-the-art methods,
which reveal the effectiveness of these methods in different scenarios and point out challenges of the
existing models under certain settings.
strengths: - Graph Learning is an interesting topic
- Results seem reasonable
weaknesses: - It seems like it doesnt fit the dataset track as its more like a technique from my understanding. 
- From the benchmarking side of view its unclear why this method should be the benchmark (explanation or clear differentation is missing)
correctness: Seems correct
clarity: Yes
relation to prior work: Yes.
additional feedback: None
review_title: Continual Graph Learning

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work researches the experimental protocols in continual graph learning and develops the CGLB benchmark. The authors categorize continual graph learning tasks into node-level and graph-level, and both tasks contain task incremental and class incremental settings. This work also explores the state-of-the-art baselines on CGLB to benchmark the current progress in continual graph learning and release a toolkit for preprocessing the datasets.
strengths: 1. This work is overall well-motivated. Benchmarking the experimental settings and configuration is very essential in the ML area. Besides, a more difficult and practical task class-incremental setting can also benefit the continual graph learning area.
2. This work provides analyses and insights into the experimental results.
3. The submission is accompanied by a code repo containing the preprocessing and evaluating code.
weaknesses: 1. The authors mention in line 255: "Due to the message passing mechanism of GNNs, the inter-task edges also break the restriction on the access to the information of the previous tasks". Since the message passing can leverage multi-hop neighbor information, a deeper GNN model can access more information from the previous tasks. I recommend the authors discuss the layer of the GNN model to make the comparison fairer.
2. The presened toolkit (GitHub repo) should contain a detailed document to make it more user-friendly.
correctness: I think the authors' claim in this paper is correct, with a clear description of the dataset construction process and the experiment settings.
clarity: Yes, I think the paper is overall well-organized and clearly written.
relation to prior work: This paper has a clear discussion of related work on continual learning and its application to graphs. I think the authors should discuss more about the relations between this work and existing datasets.
additional feedback: I don't have moe comments.
review_title: Review for CGLB paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper studies continual learning scenarios on graph data (CGL), which have rarely been discussed under a unified framework previously. It generally categorizes the CGL into node-level and graph-level under task-incremental and class-incremental settings. Some graph-specific settings (e.g., inter-connection among tasks in N-CGL) are detailed to help understand the graph-specific properties in CGL. The authors have shown the attached analysis by implementing baselines and evaluating them on some public graph datasets. The corresponding datasets and training/evaluation toolkits are also provided.
strengths: - It’s an up-to-date benchmark that strives to provide fair comparisons for CGL methods, which is meaningful for the development of the related fields.
- The settings in this benchmark are elaborated by presenting the connections with the classical continual learning settings and demonstrating the graph-specific settings. 
- Comprehensive evaluations are conducted based on 8 continual learning methods, and some insights across different datasets have been provided.
- The released toolkit is well-organized and easy to follow.

weaknesses: - A key concern is that the evaluation metric, i.e., performance matrix $M^p$ only considers sequences of two tasks, rather than a sequence of tasks stated in the section 3.1. Consequently, the experiments are limited in terms of continual learning settings. 
- It’s would be better to discuss the difference between the current benchmark and the previous benchmark [1][2] in detail. 
- The description “In contrast, CGL targets the forgetting problem, therefore the data from previous tasks are inaccessible. Few-shot graph learning aims at fast model adaptation to new tasks. In the training, few-shot learning models have access to all tasks simultaneously (not available in CGL).” in section 2.2 should be more careful, since N-CGL (with inter-task edges) proposed in this paper actually leverages the data from existing tasks.
- When dividing classes into tasks, K is supposed to be set as multiple choices to provide more insights.
- It’s weird that most continual learning methods fail to bring improvements in Arxiv-CL according to Table 5. The authors are expected to present additional analysis or demonstrate individual task performances.

[1] Antonio Carta, Andrea Cossu, Federico Errica, and Davide Bacciu. Catastrophic forgetting in deep graph networks: an introductory benchmark for graph classification. arXiv preprint arXiv:2103.11750, 2021.

[2] Stanley, Megan, et al. "Fs-mol: A few-shot learning dataset of molecules." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.

correctness: The benchmark seems technologically sound and the related details are sufficient.
clarity: This paper is well written.
relation to prior work: Related works are properly discussed.
additional feedback: As has been discussed:
- More evaluations and analysis are supposed to be presented.
- Some descriptions should be more careful.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper is a benchmark paper that focuses on both node-level and graph-level graph continual learning. In particular, this paper studies two settings, i.e., task-incremental setting and class-incremental setting. Based on the established taxonomy, the authors construct the Continual Graph Learning Benchmark and evaluate a list of baselines under different evaluation settings. This paper also provides the toolkit for training, evaluation, and visualization, which can better promote the future research in this field.
strengths: 1. Graph continual learning is largely underexplored and requires a benchmark to promote future research.

2. This paper formulates different settings and also have the corresponding evaluations. 

3. The paper is overall well-written and easy-to-follow.
weaknesses: 1. The training label ratio is relatively large (60% for node-level evaluation and 80% for graph-level evaluation). However, this is not practical since new tasks or new classes only contain fewer labeled data than existing ones. It would be better to show the evaluation results with less training labels in the experiments. 

2. The results seem quite random in task incremental setting (with inter-edges or without inter-edges), though the authors provide some possible explanations, more theoretical analysis is needed to give deeper understanding for this problem.

2. It would be much better if the authors could propose a simple and effective method after having the analysis on the existing methods. Based on the experiment results, what are the most important factor for addressing continual graph learning, especially for the class incremental setting? New insights for promoting the future research is somehow missing in the paper.



correctness: The claims in the paper are overall correct. The experiment design is reasonable and comprehensive.  
clarity: Yes, this paper is well written and easy-to-follow.
relation to prior work: Yes, this is the first benchmark paper in continual graph learning.
additional feedback: Please refer to the weaknesses.
review_title: Could be a good benchmark paper by adding new method and more insights

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces a new benchmark dataset called Continual Graph Learning Benchmark (CGLB). The paper first builds a systematic taxonomy of different CGL scenarios including node-level CGL (N-CGL) and graph-level CGL (G-CGL). Then it further introduces two learning settings: task incremental learning (task-IL) and class incremental learning (class-IL). Next, the paper introduces how the dataset was built and also proposes some evaluation metrics to evaluate algorithms. Finally, several empirical studies are provided.
strengths: 1. This paper defines the two tasks (node-level and graph-level continual learning), and also two learning settings (task/class incremental learning).
2. The experimental results show that there is some room for improvement.
3. The documentation in the Github repo looks good in general.
weaknesses: 1. The rationale behind the designed metrics Average Performance (AP) and Average Forgetting (AF) are not provided. For AP, why not take an average over the entire matrix?
2. It looks like the performance matrix $M^p$ only considers sequences of two tasks, rather than a sequence of $T$ tasks stated in the section 3.1.
3. Some details are not documented well. For example, 
- Is the order of different tasks fixed?
- What does $M_{i,i}^p$ mean?
4. This paper does not provide statistics for each task. For example, the average number of nodes/edges for each task; maybe also the difficulty level of different tasks. It will be better if the information can be included.

correctness: Not sure about the design of the performance matrix and evaluation metrics. More details should be provided.
clarity: No, the rationale of the design for the performance matrix and evaluation metrics are not provided. 
relation to prior work: Yes. 
additional feedback: No.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present a number of contributions related to the CGL:
- a systematic taxonomy of different CGL scenarios that include node-level CGL and graph-level CGL;
- the differences between the task-incremental learning setting and the class-incremental learning setting;
- a useful toolkit for training, evaluation, and visualization, with a maintenance plan;
- a comprehensive study of state-of-the-art methods.

strengths: Significance: the contribution is significant as it lowers the barrier to exploring the CGL problem to a large extent. The constructed standard datasets, experimental setting and evaluation toolkit are beneficial for the development of CGL.

Relevance: the benchmark is relevant to the continual graph learning community. 

Accessibility and Accountability: the documentation is mostly clear.

weaknesses: One concern is that the results of some existing SOTAs across different datasets is inconsistent. For example, Tables 3 and 4 show that NAS performs significantly better than LwF on the CoraFull-CL and Arxiv-CL datasets, whereas the opposite is true on the Reddit-CL and Products-CL datasets. Some detailed analysis is necessary.
correctness: The evaluation methods and experimental designs are appropriate.
clarity: The paper is well written and easy to understand. 
relation to prior work: The discussion on prior work is clear and sufficient.
additional feedback: Please refer to the weakness above.
review_title: A comprehensive benchmark and a useful toolkit for Continual Graph Learning

Meta-review: metareview: This paper had 7 reviews. The reviewers’ comments were overall favorable from the point of view of quality, clarity (i.e., the paper is well-written and its main idea is easy to flow’), originality (this work proposes a novel benchmark for continual graph learning), and significance (a benchmark for continual graph learning (CGLB) is an important contribution). Overall, I believe the authors have addressed the reviewers’ comments satisfactorily, including comments from H68Y. There are no issues with Correctness, Documentation or Ethics. The authors have included a GitHub repository to support reproducibility of their work with detailed instructions
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept