Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The Authors Present Graph Robustness Benchmark (GRB), a benchmark that aims to provide a standardized evaluation framework for measuring attacks and defenses in adversarial GML on the node classification task. The authors present a benchmark that is:
* Clear and well thought
* Aims to provide a standard and extensible way to examine the performance of various attack and defense methods
* Provides ways to see how the attack and defense methods generalize to different scales and datasets
* Present a unified framework to evaluate these methods, including a clear website leaderboard and documentation
strengths: The work presented by the authors has a clear use in the field of GraphML, and the engineering work the authors have put in to it lowers the amount of effort other researchers need to spend designing similar experiments, freeing up more time to focus on the actually development and advancement of academic work. The authors do a good job of making sure their work is accessible, their website is clear and informative, the data is easily presented and accessible, and the documentation is sufficient to allow a researcher fairly familiar with the field to make use of it without excessive overhead. 
weaknesses: The work does have some limitations that limit it's relevancy and extensibility, for example:
1. The authors only consider the task of node classification. The authors themselves note this limitation and I am hopeful that the authors will extend this work in the future, but at this work can only be used for one task.
2. It is unclear exactly how the authors plan to maintain and extend this work. The authors mention that they'll be updating the leaderboard, which is great, but there are other facets to consider. Given that this is a benchmark for focusing on adversarial attack and defense, this benchmark must surely be updated every so often with the newest relevant attacks to make sure that the evaluations provided are relevant There are many ways the authors could propose to maintain their work reasonably current, but at the time of writing this review, I do not know exactly how the authors plan to keep current which may limit the usefulness of this work in the future.
correctness: For the most part, I don't have many concerns with the approach the authors detailed. However, I am a bit concerned by the splitting methodology in use by the authors. 

I take the authors point that a single random split may not be suitable (as they cite from Shcur et al). However as I recall, the Shcur et al propose that a single spit on it's own can be problematic, and to my understanding while the authors define their own splitting scheme, they only have a single split which sort of goes against the work they are citing. To avoid problems where certain methods work better or worse just due to the spits used and nothing else, shouldn't the authors generate a set of splits and aggregate results over those splits instead of a single split?

My second correctness concern still has to do with the splitting. I'm worried about a case where the fact that the authors use this particular splitting schema (which to my knowledge hasn't been commonly used) results in results that may not generalize to how GML models are used in practice. I understand that the authors choose these schema to try to better evaluate the adversarial methods, but the largest use for a benchmark of this sort is to generate information that can be used to generalize about which attacks are worth worrying about or how to defend from those attacks. By using these splits, I can imagine a scenario where now the training and test distributions are sufficiently different from what other methods use that the information generated from this benchmark ceases to be useful. Do the authors anticipate this to be the case? 
clarity: The figures and the majority of the paper are clear enough, however I encountered some grammatical issues while reading through this work. For example:
Line 80, this is not the proper way to cite something, it should've been something like "In the work of Xu et al [32]" or similar

Line 115, "Facing the challenges" -> "Facing these challenges"

Not an exhaustive list and while these didn't make the work ambiguous, I would still like to see the authors take another pass at editing their work.
relation to prior work: The authors do a really good job of this, they have high level overviews of how this work incorporates previous work in the mai text and a much more thorough explanation for the attacks and defenses that the authors implement is in the supplemental material. 
additional feedback: Overall I am positive about this work, my main concerns are as follows:
1. Currently, the work is limited in scope (only a single task and a few attacks that are being evaluated). How do the authors 
   1. Plan to expand the scope of their work?
   2. Plan to keep their benchmark current and useful with the state of the art attacks?
2. I am a bit concern that due to the authors choice of splits may hinder generalization to real world scenarios and as such may impact how useful this work actually is. Can the authors either provide an argument for why my concern is unlikely to ever materialize or matter, or could they add in some more splits to test their data on to ensure that anything observed in this work generalizes?
review_title: A Clear and Promising Benchmark for Adversarial Attacks in GML

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper contributes the Graph Robustness Benchmark (GRB), measuring the robustness to adversarial examples of graph machine learning models. The benchmark standardizes threat models, datasets, defenses, and attacks, making it easy to run many attacks against many defenses in a unified evaluation scenario.

strengths: The existence of agreed-upon datasets and threat models for evaluating GML models certainly seems useful. Having a _library_ of attacks and defenses seems useful as well.

Turning this into a _benchmark_ is the tricky part (more details below).

weaknesses: The existence of benchmarks in ML is certainly useful, but when benchmarking models in a security context, this fundamentally cannot be done in a fully automated way. An analog would be to compare to software security: we can't measure if software is secure by running all past CVEs on the software and checking if the attacks succeed. Running existing attacks on defenses only gives an upper bound on robustness. Usually, to get tighter bounds on robustness, adaptive attacks are needed (the right thing to do in a security context). The existence of GRB in its current form is dangerous and could mislead researchers: the benchmark does not measure robustness, it simply gives _upper bounds_ for robustness, and even there, the bounds are not necessarily comparable across defenses (the existing attacks may happen to work better on defense A compared to defense B, but it may be the case that against better tuned attacks, defense A outperforms defense B), so it's hard to draw any deep conclusions from benchmarking results. That being said, I don't think it's not possible to have a reasonable benchmark for robustness. One way to do it would be to have a leaderboard where third-party researchers can submit results from adaptive attacks (e.g. like robust-ml.org). Another semi-automated approach would be to follow something like RobustBench (robustbench.github.io): have a suite of attacks that give a rough upper bound for robustness (and even there, having a thorough argument for why the attacks give reasonable bounds for the admitted defense models), and _also_ allow third-party researchers to submit adaptive attacks on models in the leaderboard.

Some more detailed points:

- The paper's definition of black-box attackers is unreasonable. See prior work like https://arxiv.org/abs/1902.06705 and Kerckhoffs's principle. It is not possible/reasonable to model an adversary that has no knowledge of the defense mechanism. (Though it may be reasonable to keep model _parameters_ secret.)
- The leaderboard reports accuracy for each model on each attack. This is fine data to report, but when considering robustness overall, a single number better represents robustness (the attacker runs all the attacks and uses the best result for each data point; under this attack, what is the model accuracy). Again, see prior work arguing that this is the right metric to use. Weighted accuracy of multiple attacks seems like a strange metric.
- For attacks that have parameters (e.g. step size), how are these parameters chosen? Are they tuned per model being attacked? This is another factor that could lead to a misleading leaderboard: when parameter choice makes one defense work better than another.

correctness: I don't think there's anything _incorrect_ said in the paper per se, but the benchmark design has some serious flaws (more details in the weaknesses section).

clarity: Yes.
relation to prior work: The paper would benefit from comparison to prior work on benchmarking robustness in other domains.

additional feedback: A robustness benchmark for graph machine learning certainly seems useful to have! GRB is heading in the right direction, but I think it is critical to address the weaknesses above before such a benchmark is relied upon by the research community.
review_title: A potentially misleading benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper does a unified benchmarking of graph neural networks with a key focus on different attacks vectors, datasets and models. Additionally, the proposed benchmark is modular, thus it supports multiple different types of attack vectors, external datasets, graph development frameworks.
strengths: This paper provides a very rigorous benchmarking of the robustness of graph neural networks. The benchmark is well documented and leaderboards are easily accessible (both in paper and on the webpage). The paper is also well written and easily accessible.
weaknesses: While the proposed framework is extendible to any dataset, I encourage authors to expand the current analysis to multiple datasets in the paper itself. Currently all major results are reported mostly on grb-aminer dataset (from figure 4,5,6,7,10,11, table 1) which natural raises the question: whether the findings are consistent across datasets. 

Similarly the current benchmark remain focused only on node-classification. I encourage authors to at least provide a preliminary sketch or detailed discussion on how they are going to handle the expansion of current benchmark. Being forward looking is essential for a benchmark to be well-established. 

Additionally, I still find the evaluation on normalization and adversarial training to improve the robustness pretty thin. While authors claim their results being state-of-the-art, there are multiple questions left unanswered: 1) is this phenomenon specific to layer normalization or its extendible to other normalization techniques too 2) What will be the performance impact with different variations of adversarial training, etc. I encourage authors to rethink this aspect.



correctness: Yes, the claims made in the submission are correct. 
clarity: Yes, the paper is well written.
relation to prior work: Yes, authors provide adequate references to the related work.
additional feedback: I've already discussed it in an earlier section.
review_title: A unified benchmark for graph neural networks. 

Meta-review: This is a close call, but overall, we feel like there is enough support for the paper from a majority of reviwers.
Decision: Accept