Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents GraphGT, a large collection of 36 datasets for graph generation and graph transformation tasks, filling a need which is unmet by existing benchmarks for graph representation learning. These datasets span many fields, from biology to electrical engineering. In addition to the datasets, there is a Python package which provides a set of dataloaders and evaluation functions for each dataset and task. These datasets and tools provide a useful resource to researchers looking for graph datasets across many disparate fields.
strengths: - This fills a need, supplementing widely used representation learning benchmarks with datasets more suited to generation and transformation. This is largely due to the lack of existing datasets with large numbers of graphs (as opposed to individually large graphs)
- The number and variety of datasets is impressive, and provides great opportunities for machine learning research across many fields that use graph-structured data
- The Python API provides a very useful set of processing and evaluation tools for graph generation and transformation, which should enhance accessibility and reproducibility
weaknesses: - Motivation for including each dataset is not clear in many cases. For example, statements like "it is attractive for deep generative models to model protein structures" should be backed up by examples or references. Some datasets have even less motivation stated.
- Detail on the datasets is lacking. Recognizing that is an inherent disadvantage of including such a large number of datasets with limited space, it nonetheless makes it difficult to understand the structure of each dataset and what types of tasks one would use each for. The descriptions are also inconsistent in the main text -- for example, sometimes the number of graphs and graph sizes are discussed and sometimes they are not. The paper would benefit from a clear parallel structure across task descriptions.
- Benchmarking is only performed on one task (molecule generation). There is no benchmarking on any graph transformation datasets. The authors' argument that they only benchmark one task because graph generation is domain specific is at odds with the goal of creating a unified benchmark with standardized evaluations. Without defining domain-specific evaluation criteria for each task, it is much harder for researchers to compare models consistently, harming the utility of the benchmark as a whole. 
- Data format is not described, nor are the hyperparameters or experiment settings for the benchmark.
correctness: There is little detail of dataset processing and construction for each task, and each domain has specific requirements, making it difficult to assess the quality of the datasets from the paper.
clarity: The structure of the paper is clear, but clarity is hindered by the lack of detail due to the large number of tasks (particularly in Section 4). The description of graph generation could also be clearer.
relation to prior work: Yes
additional feedback: This is an ambitious project and impressive array of datasets. I also think the Python toolkit will be useful for practitioners. However, the lack of detail in dataset construction and processing, combined with the lack of systematic benchmarking or even specification of evaluation criteria for each task, limit the impact of the work as currently presented.
review_title: A large collection of datasets and useful API, but falls short on task-specific details and benchmarking

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes GraphGT, containing 36 datasets from 9 domains across 6 subjects, which assists the research on graph generation and graph transformation tasks. The authors provide a systematic review and classification of the datasets and provide APIs for the graph generation pipeline, which simplifies data loading, experimental setup, and evaluation. 
strengths: * A vast range of domains are considered, including biology, engineering, social science, etc.
* Different tasks of graph generation and graph transformation are covered by the collected datasets.
* Various metrics are provided to evaluate graph generation and transformation.
weaknesses: * It seems that all datasets are collected from previous works (Table 1), there is no newly built dataset. It is just a simple summarization of datasets in previous works. There is no discussion on why these datasets are chosen and how significant they are for graph generation and transformation. It remains unclear what kind of datasets are really useful for these tasks or any criteria when building related datasets.
* The detailed description of all datasets takes a major part of the paper (which also appears in the Appendix), however, they are all from existing works which do not cover the contribution of this work.
* The benchmark part is too thin, without any detail about the considered methods or experiment setup (even in appendix), which makes the results less convincing. There isn’t any analysis on the results, e.g. how do they differ from or agree with results in previous works?
* Overall, this paper is more like a summarization of datasets in previous works with limited contribution for future research in graph generation and transformation. 
correctness: All datasets are collected from previous works. Some of them are processed, but none of them are entirely new. The benchmark part is poorly represented without any detail of experiments and code, thus the results are not guaranteed to be correct. 
clarity: There are several obvious errors. In line 111, the dimension of edges is N, equal to the dimension of nodes, which is obviously wrong. In line 135, there is a clear error in the vector space of the target domain. 
relation to prior work: The authors only discuss datasets in other domains like node classification and discuss that they are not appropriate for graph generation and transformation. However, there is no discussion about how the collected datasets differ from those used in previous works of graph generation and transformation.
additional feedback: My suggestion to the authors is to think first what is actually the challenges in graph generation and transformation. A simple summarization of datasets in previous works doesn't make much sense for the community. It would be more important to find out how datasets or benchmarks can cover limitations in previous works of graph generation and transformation, and contribute to further research in this domain.
review_title: More like a summarisation of datasets in previous works with limited contribution for graph generation and transformation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper systematically studies the problem of graph generation and transformation. Specifically, 36 datasets from various domains are collected and organized into a handy library; benchmarking studies on generation for chemical datasets are conducted to demonstrate the effectiveness of existing state-of-the-arts.

strengths: + Graph generation and transformation are two important topics for many real-world problems.

+ A handy library featuring dataset loading and evaluation is provided.

+ Experiments on state-of-the-art methods of graph generation with various evaluation metrics are conducted.


weaknesses: - Experiments are limited to one task, i.e. chemical molecular graph generation, only.

- More discussions should be given to better illustrate the background, real-world impact, challenges, and status of existing studies. See my detailed comments below.

- The development of graph generation and transformation models should be outlined in the related work part.

correctness: No technical flaws are spotted.
clarity: The paper reads somewhat quirky but is easy to follow overall. More background knowledge of real-world applications may better demonstrate the impact of this problem.

relation to prior work: The relation to prior datasets in relevant fields have been examined in the related work section.
additional feedback: 1. As the authors point out in the main text, the problem of graph generation and transformation may differ in contexts and domains. Therefore, it is suggested to compare the implementations of graph generation and transformation in different domains, so as to allow readers to understand the limitations and difficulties. In the current version, some baselines methods target at specific domains, e.g., MoFlow for molecule graph generation, while some others are for general domains, e.g., GraphRNN. A comparative study may be helpful for better showing the difference in different domains.

2. Deeper discussions on the experimental results should be provided, in particular, how do existing work differs from each other and how different metrics affect model performance. It is interesting for me to discuss the choice of evaluation metrics, which seems are most based on simple statistics and heuristics. From Table 2, it is observed that many existing baselines have already achieved quite promising performance in terms of the concerned three metrics; does it mean that the generation quality is good enough, or may other domain-specific metrics be adopted? The authors may also outline several future research directions that allow for improvements.

3. Minor edits:
  - In Line 322, what is "label graph"?
  - In Line 352, "Fig. 2" should be Table 2.
review_title: Technically sound; rooms for improvements

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work presents a benchmark called GraphGT for graph generation and transformation tasks for the first time. The collected datasets cover various domains and subjects and are released with useable APIs. It also benchmarks two datasets with three evaluation metrics.
strengths: 1. The collected datasets are diverse in terms of domains and subjects;
2. The task formulations are clear with motivating examples;
3. The evaluation pipeline is carefully normalized;
4. A python library and website are provided for easy usage;
weaknesses: 1. There're only two datasets benchmarked in the paper, looking forwards to see more and willing to increase my rating if authors do so;
2. The benchmarked datasets seem to be easy because most of the baselines got 100% on evaluation metrics;
3. It might be better to use figures to illustrate the task formulation for easy understanding;
correctness: I do not have any concerns with the claims made in the paper.
clarity: The paper is overall well written but could be more concise on the dataset descriptions.
relation to prior work: The paper clearly contextualizes its contributions over previous work.
additional feedback: All my feedback is outlined in the above sections.
review_title: A nice collection of datasets for new tasks

Meta-review: All reviewers voted for acceptance. I recommend the authors to take into account the reviewers' comments and improve the paper for its camera-ready version.
Decision: Accept