Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper suggests that non-adaptive attacks lead to an overstate on adversarial robustness, and thus the authors recommend using adaptive attacks as a gold-standard. On GNN classifiers, the adversarial structure perturbation is to perturb the adjacency matrix from an evasion attack (after training with fixed model weights) or poisoning attack (before training). Also, adaptive and transferring attacks are compared and the former is stronger. With independency of budget and a baseline of an MLP, the experiments show that their adaptive attacks lower robustness by 40% in average across 7 defenses. The authors provide guidelines to design strong adaptive attacks.
strengths_and_weaknesses: **Strengths:**

1. The paper provided a thorough review and analysis on existing GNN defense methods.

2. The paper provides guidelines to design strong adaptive attacks.

3. There are clear comparisons between non-adaptive and adaptive attacks and between poisoning and evasion attacks. The comparison strongly suggests that adaptive attacks should be a new standard in this area.

4. The experimental design is careful and comprehensive.

**Weakness:**

1. The topic of this paper seems to be overly general. As stated in lines 61-62, the attacks mentioned in the paper only focus on structure perturbations (i.e. by flipping an edge in adjacency matrix $A$). Feature perturbations (i.e. attacks on $X$) are not studied in this paper. But considering the strong conclusion claimed in the paper, these conditions may also need to be evaluated.  In “finding 4” (line 307), the author stated that they cannot confirm a trade-off between accuracy and robustness. As the weakness mentioned above, this trade-off may become perceptible if feature perturbations are introduced. 

 

2. The attack budget is set to 15% of the total number of edges in the dataset which is a little bit tough for existing defense methods. 

 

3. The designed adaptive attack is still highly related to existing works (PGD[47] and Metattack[60]).
questions: See weakness above.
limitations: NA
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper performs a thorough robustness analysis of robust graph neural network models that counteract adversarial attacks. It provides custom adaptive attacks to assess the robustness of robust graph neural networks and outlines the lessons for successfully designing such attacks. Extensive experiments demonstrate that the robustness of existing defense methods is overestimated as they perform badly under the adaptive attacks. Furthermore,  the transferability experiments show that perturbed graphs generated from different GNNs can transfer to other GNNs which can provide a black-box unit test to assess a model’s robustness.


strengths_and_weaknesses: The strengths of this work are as follows. 
1) The research problem is important. The evaluation of robust graph neural networks against adaptive attacks is rarely explored. Such evaluation protocol can help the community reliably assess model robustness. 
2) The observations made in this paper are interesting. For example, the robustness of existing GNN defenses is overestimated and they can be broken under adaptive attacks. 
3) The transferability experiment shows that the proposed method can also offer a black-box attack to assess a model’s robustness. 
4) The paper is relatively well-written and the ideas are clearly clarified.

There are also some concerns that need to be addressed:
1) Compared with the adaptive attack in vision[38], the proposed method seems to requiring the tuning of hyper-parameters, which could be computationally expensive.
2) The authors are encouraged to provide some explanation about why the attacks generated from other GNNs do not transfer well to SVD-GCN?
3) It would be of interest to try the ensemble of the attacks, as what is done in [38]. For example, try the ensemble of the attacks generated from GCN and SVD-GCN, which could form a powerful attack with strong transferability.


questions: Q1 Can you  provide some explanation about why the attacks generated from other GNNs do not transfer well to SVD-GCN?

Q2 Have you tried the ensemble of the attacks, as what is done in [38]. For example, try the ensemble of the attacks generated from GCN and SVD-GCN, which could form a powerful attack with strong transferability.

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work has revealed a serious issue of adversarial defenses on GNN: existing defense GNN models can be easily broken by applying customized adaptive attacks. Specifically, authors first categorize $45$ defense methods in literature, and develop proper methods to adaptively attack $7$ representative defense models. Experimental results indicate that the proposed adaptive attacks lower robustness of prior defense models by $40%$ on average. Moreover, authors also share some important lessons learnt from experiments and provide useful guidelines for evaluating new defense methods, which includes a robustness unit test produced in the experiments.
strengths_and_weaknesses: **Key Strength**

  - The paper is clearly written. All the technical steps are easy to follow.
  - This work addresses a very important problem. Authors also propose a new metric for measuring the quality of attacks/defenses.
  - Authors have conducted extensive experiments to show how adaptive attacks, which include global/local attacks under evasion/poisoning setting, can break several representative defense GNN models.
  - The lessons and guidelines for adaptive attacking are reasonable. The proposed robustness unit test set is useful for evaluating new defense models.
  

**Key Weakness**

  - Authors only use two small datasets in experiments. Considering the scale of real GNN applications (e.g., [1]) for attacking, evaluating attacks/defenses on large graphs (with millions of nodes) is very important, as it may reveal more realistic lessons and guidelines. Note that there are a few scalable defense models such as [2].
  

[1]: Ying et al., "Graph Convolutional Neural Networks for Web-Scale Recommender Systems", KDD'18. \
[2]: Geisler et al., “Robustness of Graph Neural Networks at Scale”, NeurIPS'21.
questions:   - How do authors handle the RAUC score when node features are also attacked (i.e., MLP accuracy will drop)?
  - Some prior works (e.g., SVD-GCN and ProGNN) have shown graph attacks are essentially high-rank attacks. How do adaptive attacks affect the graph spectrum? For instance, does the adaptive attack for SVD-GCN become a low-rank attack, so that it significantly drops SVD-GCN's accuracy? It would be great to see additional insights of adaptive attacks from the spectral perspective.
limitations: Authors mention the potential negative societal impact in a very vague way. Since the topic of this paper is highly related to the security of real GNN applications, a more detailed (e.g., a concrete example/scenario) could better illustrate the negative societal impact.
review_title: null

Meta-review: metareview: The recommendation is based on the reviewers' comments, the area chair's personal evaluation, and the post-rebuttal discussion. 

This paper provides novel insights into the effectiveness of various defenses proposed for graph neural networks. All reviewers find the results convincing and valuable. The authors' rebuttal has successfully addressed the reviewers' concerns. Given the unilateral agreement, I am recommending acceptance

confidence: Certain
Decision: Accept