Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work provides a unified benchmark of graph neural networks across multiple models, attack vectors, and datasets. It further enables deeper insights by analyzing results through a lens of hardness in attacking different nodes (easy→hard). Further its modular architecture supports multiple different types of attack vectors, external datasets, graph development frameworks.
strengths: This paper provides a very rigorous benchmarking of the robustness of graph neural networks. . The benchmark is well documented and leaderboards are easily accessible (both in paper and on the webpage). The paper is also well written and easily accessible.
weaknesses: This paper does a very good job of benchmarking the robustness of graphs neural networks. Following are my comments to further improve it. (Note that some of them are minor changes, thus do not amount to a weakness. I am listing all of them together to arange them in their order of appearance in the paper )

The majority of research work in adversarial graph learning (consequentially this benchmark too) consider the task of node classification. However, graph neural networks also work well for tasks such as link prediction, clustering, graph classification. It will be helpful if authors can discuss their thoughts on expanding the benchmark to a wider range of tasks.

While the grb framework supports a wider range of configurations, the leaderboard only considers one specific configuration (Table 1). Was there a particular reason authors made this particular choice? Or is it a random selection?

I agree with the authors arguments (section 2.4) about imperceptibility in poisoning attacks in graphs neural networks. However, this section reads more like a discussion, in particular since the benchmark follows a similar setup as previous works. I encourage authors to move this part to discussion.

It's good to have support for external datasets. I also highly encourage authors to integrate open-graph-benchmark/SNAP making it seamless to test attacks/defenses on an even broader range of datasets.

One metric: It's great that authors aggregate the performance of different attacks/defenses with avg/3-min/weighted accuracy. Is it feasible to further reduce it to one number? Having a single number makes it much easier to rank methods (e.g., it unclear which one of three accuracy metrics determines the rank on the leaderboard).

Can authors also provide details on the runtime of different attacks/defenses? This will be helpful to judge whether the grb-aminer dataset is the highest scale for moderate gpu clusters, or whether we can scale up testing robustness of graph nets to an even larger scale.

Disentangle benchmark and algorithmic improvements: I find that the GRB is well designed and well executed. However, authors also propose using layer normalization and adversarial training to improve the robustness of graphs neural networks. Both methods work well, however, I find both their presentation and evaluation pretty thin. For example, the paper is missing a discussion of related work showing how both techniques differ from them, a concrete comparison with previous works, and a more detailed discussion of its results. For example, it's not discussed why adversarial training leads to lower performance in one case (Fig. 9, APPNP). I encourage authors to either 1) move both methods to appendix/discussion 2) Or provide a more detailed discussion on these methods.

correctness: I believe that the evaluation is done correctly and rigorously and the claims are correct.
clarity: Yes, the paper is well written.
relation to prior work: Yes, authors provide adequate references to the related work.
additional feedback: I've already discussed it in an earlier section.
review_title: This paper provides a unified, scalable, and extensive benchmark for the robustness of deep neural networks.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper Graph Robustness Benchmark (GRB) to evaluate the performance of adversarial attacks and defenses for GNNs. It first discusses the limitations of the current comparisons, then defines a realistic threat model which is the the object of the benchmark, and finally introduces the structure of the new benchmark and its entries. GRB provides tools to compare new methods in a standardized way. In addition to existing methods, the authors also propose two novel baselines as defenses, which improve the robustness across datasets and models.
strengths: - The authors clearly present the different threat models of adversarial robustness in graph ML, and discuss the current limitations about both the constraints imposed on the attacks (the notion of unnoticeable) and the methodology in the evaluation. Based on such analysis, they propose a well-defined scenario, with reasonable assumptions for both attackers and defenders, which allows to compare different techniques in a standardized way. I think such discussion is important as a clear task on which comparing methods might lead to better understanding of existing methods and promote development of new ones.

- GRB provides tools to reproduce the results on the leaderboards and to use attacks and defenses to test new methods. This, together with the new splitting schemes of the datasets, should facilitate the evaluation of new methods and comparison to SOTA.

- The authors propose two new baselines for defenses, applying adversarial training and layer normalization to the existing models, achieving higher robustness the current techniques.

- The interactive leaderboards allow to easily compare attacks and defenses along different metrics.
weaknesses: - Currently, the benchmark is limited to a single threat model (black-box, evasion, inductive, injection). Although this seems the easiest for a standardized evaluation, I'm not sure how popular or established it is: in fact, according to Table 1 none of the existing attacks focuses on it, and the baselines are adapted (L250). Moreover, the fact that a simple baseline as adversarial training, which is the standard in image classification tasks, hasn't been used in prior works while achieving the best results suggests that this threat model hasn't received much attention.

- The constraints for the attacks should be clearly stated in the main part (I think they're only mentioned in the appendix), especially given the discussion in Section 2. How are they chosen?

- From Figures 6 and 7 and leaderboards it seems that 1) the standard models are already quite robust, 2) the difference in robustness from the most robust model GAT+AT and the standard counterpart GAT is not large (~11% if I read the leaderboard right), 3) the robust accuracy is not particularly lower than the clean one for many models. Then, wouldn't be more appropriate to have stronger attacks, e.g. with the possibility of injecting more nodes or edges, for a more challenging task? It is not clear currently how large the margin for possible improvements is.
correctness: The evaluation methods seem appropriate.
clarity: The presentation is clear and easy to follow.
relation to prior work: The authors discuss existing works and how different those are from the proposed GRB.
additional feedback: As mentioned above, my main concerns are about the significance of the chosen threat model.
review_title: Well designed benchmark, concerns about the significance of the chosen task

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper outlines flaws in existing measures of adversarial robustness in graph neural networks, namely relating to unrealistic assumptions that are typically made about the knowledge of the attacker and the defender. The authors propose a new measure of adversarial robustness which appears to be more suited for graph learning tasks.

In this model, the attackers are allowed to inject new nodes and edges into the network, but the attacker cannot modify existing nodes or node attributes or edges. This is to reflect the realistic attack of creating spam accounts to manipulate the defender's model toward various ends. The defenders have full knowledge of their training and validation set, but not the test set.

The authors then train several standard graph neural network models on a number of standard datasets using their measure of adversarial robustness, and summarize the results.
strengths: The paper is well-written, and raises compelling points in favor of their measure for adversarial robustness. The experiments are thorough, clearly presented, and appear to be easily reproducible as the source code is available and the code style is reasonably clean.
weaknesses: Unfortunately, this paper is, fundamentally, not about benchmarks and datasets. While the authors do phrase parts of the paper in terms of making a benchmark framework, the content of the paper does not back this up.

For example, there are no comparisons to other works relating to benchmarking in graphs, and appear to be no code interfaces that allow other researchers to employ this new measure of adversarial robustness in standard benchmark frameworks. There is no new data collected, nor any discussion of how this new measure of adversarial robustness could make data collection easier.

The authors discuss a "leaderboard" for the best-performing models, but in actuality, this appears to be little more than a static webpage that displays the experimental results from the paper in a pretty format. (The website itself does not appear to work.)

Finally, there is no compelling argument that this framework will stand the test of time, at least as far as other researchers can use this as a standard benchmark. The authors claim that they will maintain the leaderboard. Perhaps the authors will follow through and update the website, but it honestly seems like that would not be worth their time. The paper is not really about benchmarks or leaderboards; it's about new measures for adversarial robustness in graphs. The leaderboard does not need to be continually updated for the paper to achieve its purpose.

To underscore the above points, I will simply point out that the discussion of actual code frameworks and models does not begin until the end of page 5. This raises the strong presumption that these matters are not the highlight of the paper.
correctness: The datasets and models used in this paper appear to be very standard in the literature on graph learning.
clarity: The paper has above-average writing, presentation and style.
relation to prior work: The authors give a reasonably thorough exposition on previous measures and models for attaining adversarial robustness in graph neural networks. They essentially give no discussion of related benchmarking frameworks for graphs.
additional feedback: I want to stress that this is a good paper. If it had been submitted to the NeurIPS regular track, I would have recommended for acceptance, and if it had been submitted to a workshop or conference devoted to graph representation specifically, I would have given strong support for acceptance.

It is unfortunate to reject this paper on the basis that it was submitted to the wrong venue, but I believe accepting this paper would weaken the the Datasets and Benchmarks track. I encourage the authors to submit their work elsewhere, as I'm sure it will find an interested audience.
review_title: Not really about benchmarks or datasets

Meta-review: The paper received mixed reviews, with two reviewers still leaning towards rejecting the paper after taking the authors' response into account. There were two main concerns:

(A) Is the submission in scope for the benchmarks track? In particular, Reviewer 16ob views the role of the benchmark track primarily as a venue for work on datasets and how to conduct benchmarks (frameworks, etc.), rather than on specific benchmarks.

(B) Is the specific setting / threat model in the submission actively studied? Reviewer fGSw points out that adversarial robustness has been studied before in the context of graph learning, but no prior work conforms to the specific threat model in the submission.

Overall this is a difficult decision since for instance Reviewer 16ob agrees that for the specific threat model studied, the paper represents a high quality contribution.

My perspective is as follows: I consider the submission to be in scope for the datasets & benchmarks track since proposing and carefully executing a benchmark for a specific problem is important foundational work in machine learning. However, the specific problem investigated via the benchmark should be relevant and its relationship to other benchmarks should be clear. In the current paper, this is an area for improvement since the specific threat model has not been studied in prior work yet (Reviewer's fGSw point). In addition, the submission does not contain experiments relating the proposed threat model to other threat models already studied in the literature. It would be particularly important to know how the proposed threat model relates to the KDD Cup benchmark the authors mention. For instance, does the ordering of attacks or defenses change between the two benchmarks? Do the high-level conclusions from the two benchmarks agree?

So unfortunately my recommendation is to reject the paper from the first round of the datasets & benchmark track. Overall I find the paper already an interesting contribution, and after taking the reviewers' comments into account, the paper will be a strong submission for the second round of the datasets & benchmarks track or another venue.
Decision: Reject