Adversarial Attacks on Graph Classifiers via Bayesian Optimisation

Abstract: Graph neural networks, a popular class of models effective in a wide range of graph-based learning tasks, have been shown to be vulnerable to adversarial attacks. While the majority of the literature focuses on such vulnerability in node-level classification tasks, little effort has been dedicated to analysing adversarial attacks on graph-level classification, an important problem with numerous real-life applications such as biochemistry and social network analysis. The few existing methods often require unrealistic setups, such as access to internal information of the victim models, or an impractically-large number of queries. We present a novel Bayesian optimisation-based attack method for graph classification models. Our method is black-box, query-efficient and parsimonious with respect to the perturbation applied. We empirically validate the effectiveness and flexibility of the proposed method on a wide range of graph classification tasks involving varying graph properties, constraints and modes of attack. Finally, we analyse common interpretable patterns behind the adversarial samples produced, which may shed further light on the adversarial robustness of graph classification models. 


Reviews and author responses:

 The authors' response has addressed my major concerns. Please make the promised changes accordingly in the updated version. 

The paper proposes GRABNEL, which formulates the adversarial attack on graph classification as a black-box optimisation problem and uses Bayesian optimisation to solve it. Experiments on multiple datasets demonstrate the effectiveness of the proposed method.  Originality: The technical contribution is incremental, which heavily limits the significance of the proposed method. To be specific, Weisfeiler-Lehman (WL) feature extraction methods and Bayesian optimization (BO) are well-known techniques, and the combination of them in this work is straightforward.

Quality: The proposed method is well motivated and is technically sound. Although the attack analysis part provides some insights into the robustness of graph classification models, the last two claims are not well supported by empirical verification.

Clarity: The paper is well organized and articulated in general. 

Significance: The comparison experiments are not convincing. The authors are encouraged to compare the proposed method with more recent SOTA methods, rather than one method that was published in 2018, on more benchmarks, e.g., AIDS, DHFR or Bitcoin. Besides, for the image classification and Fake news detection tasks, it will be more convincing to report the results of other competitive methods.
 One limitation of this work is that the proposed method is only applicable for graph classification models. The authors claim that some progress will be made in future work.

 We thank the reviewers for their insightful feedback. In this general response, we would like to reiterate the key contributions of the paper and respond to some common concerns raised by multiple reviewers.

### Contributions
In this paper, we use a novel Bayesian optimization (BO) to address the problem of query-efficient, perturbation-parsimonious black-box attack on graph classification models, a setup that is important yet under-studied in literature (to our knowledge we are the first in utilising BO in graph adversarial attack, which is also acknowledged by Reviewer yB9K). We consider a range of attack settings (edge perturbation/rewiring, node injection) rarely studied in previous literature, and unveil a number of common and interpretable patterns in adversarial examples.

### Node- vs Graph-level attack
Multiple reviewers have requested discussions with a number of related works focusing on node-level attack. In general, graph-level attacks have only been scarcely studied in the literature and, while there are indeed many node-level attack algorithms, in our view adapting them to the graph classification is at least non-trivial due to the significant difference in settings, and this could itself be an independent and open research question. Here we specifically address some related works mentioned by the reviewers. We will add these discussions to the final version of the paper.

1. Nettack (mentioned by Reviewer FuZ8) constructs a linear surrogate based on two-layer GCN to approximate the change in loss from perturbations. In a semi-supervised node classification setup, this is possible as we can see all the nodes and labels of training nodes at all times (both training and testing take place on the same graph). In our black-box graph classification setup, however, at test time we have no access to training input graphs/ graph labels for the victim model, and it is impossible to train a similar surrogate like Nettack in the first place.

    If, however, we opt not to use a surrogate and instead just use the actual victim model, then the setup becomes white-box. In that case, we argue that the white-box baseline we provide, which edits the edges with largest gradients, already somewhat mimics the behaviour of methods such as Nettack / PGD attack which greedily modify the edges deemed to lead to the largest change in loss.

    Another key contribution of Nettack is introducing various constraints (e.g. power law test) to improve imperceptibility. However, in a graph classification setting, some constraints may not necessarily be relevant: e.g. in datasets such as IMDB featuring very dense graphs, the power law distribution of degrees may not even hold. 

    Nonetheless, we agree that the idea of introducing attack constraints is valuable. In our work, we incorporate them in our experiments, such as the 2-hop and 2-hop + rewire constraints.

2. One method that indeed applies to both graph- and node-level tasks is RL-S2V, the pioneering work in adversarial attack on graph classification, which is also black-box (and only needs a response on whether the attack is sucessful) . However, RL-S2V primarily focuses on the S2V victim model [6], and we compare against it in the Appendix on the dataset considered in the original paper. A related work is ReWatt which suggests rewiring the edges instead of flipping them. We recognize this setup and also conduct experiments under the rewiring setup. We do not directly compare against it as 1) apart from the rewiring setup, it adopts a similar RL-based approach to RL-S2V and 2) no open-source implementation is available to the best of our knowledge.

3. Black-box attacks without surrogates (mentioned by Reviewer yB9K):

-  Ma et al. 2020 (Towards More Practical Adversarial Attacks on Graph Neural Networks, NeurIPS 2020)  is black-box evasion attack for node classification. At least in the current paper, they focus on the problem of selecting a subset of nodes to attack (so naturally not at graph level), it also seems to only attack node features (by adding noise to node features) and not topology (at least according to experiments in the paper). In contrast, our method handles both topology and features (in MNIST-75sp the surrogate model handles both node and edge features). While our present work mainly focuses on topology attack, we may easily adapt GRABNEL for feature-level/hybrid attack. In fact, feature level attack could be an easier task if the perturbation is continuous, as we could possibly rely on existing gradient-based acquisition optimization of BO which is well-suited for continuous domain. 

- Chang et al. 2020 (A Restricted Black-box Adversarial Framework Towards Attacking Graph Embedding Models, AAAI 2020) is black-box evasion attack for node classification. Eq.(5) of the paper is generic and we believe adapting to graph-level attack in this case is more possible.

    However, neither of these works considered graph classification setup, and we believe that the first step is to verify whether their findings still hold in graph classification tasks. If these ideas are indeed applicable to an extent, we note that while both works adopt a very restricted black box setup, the attacker needs some information about the architecture of the victim model, such as the number of layers (see Eq (8) of Chang et al. 2020) and defines the tailored attack loss functions for different architectures (Eq (8) and (12) of Chang et al. 2020). Similarly, Ma et al. 2020 exploits structural bias in common GNNs (as Ma et al. 2020 states in the introduction, there are also efforts to study these biases to develop new models. It is unclear whether performance will be stronger or weaker in these resultant models). On the other hand, our method treats the attack objective as a black-box function without assuming any prior knowledge on the model class; this means our method does not depend on the victim model and should therefore be more generic.

    If, however, we are indeed aware of some  model  information, we believe there could be combinable benefits. For example, the importance scores proposed in Ma et al. 2020 could be potentially used as weights for sampling the nodes so that we bias towards those deemed to be more vulnerable. This prior knowledge can be easily incorporated in our framework to improve attack efficiency.

- Bojchevski and Günnemann 2019 (Adversarial Attacks on Node Embeddings via Graph Poisoning, ICML 2019) is black-box poisoning attack for node classification. Note that poisoning attack occurs on training data before victim model training, but the setting we consider is on evasion setting at inference time after training and without accessing training information. 

In all cases, these works have not explored the node injection/rewiring setup addressed in our work, which is another key novelty of our work.

### Computational Costs

Reviewers raised the common concern on the computational cost of our method. The setup we consider is *sample-efficient* black-box attack on graph classification with minimal amount of perturbation. We particularly emphasise the desideratum of sample efficiency – we aim to find adversarial examples with the minimum number of queries to the victim model. We believe that this is a practical, and difficult, setup that accounts for the prohibitive monetary, logistic and/or opportunity costs of repeatedly querying a (possibly huge and complicated) real-life victim model. With a high query count, the attacker may also run a higher risk of getting detected. 

Given this objective, the cost of the algorithm should *not* only be considered from the viewpoint of computational runtime of the attack algorithm itself alone (unlike several existing works such as [1, 2] where sample efficiency is not emphasised), and this is a primary reason why we use number of queries as the main cost criterion in our paper. It is worth noting that this emphasis on the number of queries over runtime as the main cost metric is also common in adversarial attacks in other data structures emphasising sample efficiency ([3, 4, 5]) and other related domains, like hyperparameter optimization. The common assumption is that the cost of the BO itself is secondary to the cost of querying the objective function (the cost here should not be interpreted as being the computing cost alone, but includes all the potential costs discussed above). Admittedly, the overhead cost of BO can be higher than competing methods, such as those based on random search, because BO maintains a statistical surrogate model and runs an inner acquisition optimisation to actively and carefully select each sample location, trading off slight increase in computation overhead for bigger improvement in sample efficiency. 

With that said, we would also like to provide a more comprehensive analysis of the algorithm runtime:

Each iteration in the main loop of our algorithm can be broadly separated into two parts:

1. Initialization/updates of the surrogates: this step involves the WL feature extraction, and initialization/update of the Bayesian linear regression (BLR) surrogate (the complexity of this step was discussed in Sec 2) with the new data. Note that BLR scales much better than GPs used in related works [3].

2. Acquisition function optimization: this step involves using genetic algorithms to optimize the acquisition function. It is further broken into 2 sub-parts:

    a) genetic algorithm steps, which involve the selection of the population and mutation and crossover operations on the parents. For the manipulations here, we do not have to store the full graphs, but we instead only have to maintain a tuple of the edges that are flipped/rewired.

    b) convert the tuple into full graph objects (we use Deep Graph Library (DGL) for implementation), and call the trained BLR to obtain predicted mean/variance and compute the acquisition function value.

*--- to be continued below ---*


 We thank the Reviewer for their insightful feedback on our work in terms of both its strengths and weaknesses. We are particularly grateful for the comment on discussing possible links with other attack models. In addition to the general response above, we reply to the Reviewer’s comments in detail as follows.

1. >*The black-box setting used in this work falls into the approach with a specific surrogate model. There are several works considering the black-box setting without a surrogate model, such as [1-3]. Though their focuses are on the node classification side, the relationship with this line of research is suggested to be discussed.*

We thank the reviewer for the suggestions and discussions about the works in question will be added in our paper. We refer the reviewer to a discussion on this question in the general response (including the detailed response to [1-3] mentioned).

2. >*The computation analysis of the total attack process should be demonstrated. The authors only provide the complexity of the surrogate model and neglect that of sequential perturbation selection and acquisition function optimization. The complexity analysis is crucial when using large-scale graphs as one strong motivation from authors to choose a proper surrogate model. Further, it would be better if the running time analysis is provided.*

We refer the reviewer to the overall response, which 1) explains why we believe runtime cost is not necessarily the most important metric under the current setup but also 2) provides a runtime analysis. As we have shown, the runtime is moderate for graphs with >5k edges, which are already much larger than typical graphs in graph classification tasks (see the TU dataset for graph classification: https://chrsmrrs.github.io/datasets/docs/datasets/)
As for the specific question on the efficiency of the genetic algorithm, we find the GA to be actually very efficient with the bottleneck being the conversion of the graph representation data structures. We also provided some possible ways for further runtime optimization in the general response.

3. >*For the experimental results, why there is no comparison between RL-S2V and GRABNEL on real-world datasets? Meanwhile, the advantage of GRABNEL over RL-S2V is not strong enough on the ER graphs.*

The authors of RL-S2V generously open sourced their code which we leveraged in our comparisons for the ER graphs dataset attacking an S2V model. Unfortunately, the parts of their code concerning graph classification are highly engineered for the specific victim model (S2V) and ER dataset. In good faith, we spent many hours trying to adapt their code to other settings and datasets, but due to the nature of how the code is written it would have essentially required a complete reimplementation (note that the original RL-S2V paper also only contains ER graphs experiment for the graph classification tasks).  

The performance of GRABNEL is actually stronger than RL-S2V on the ER dataset (Fig 11). Partially since the graphs in question are small, random search, given sufficient budget, performs well (note that the random search figure reported in the original RL-S2V paper corresponds to random search with query budget = 1. This is fair in that setting because RL-S2V is "one-shot" on the test data set and query efficiency is not a main focus for that paper, but it queries the victim model repeatedly on a separate validation set. In our case, the x-axis reflects the total queries to the victim model by each method). 

As for the difference between random and GRABNEL, it is again worth noting that the typical graph sizes in ER are small and the total number of possibilities to edit edges is not huge (also noting additional constraints such as the edges should not change the number of connected components, etc. These constraints were in the RL-S2V paper too), and as such random search will eventually find any adversarial example, if it indeed exists. That said, note that to achieve the similar attack performance GRABNEL is order-of-magnitude faster (x-axis is in log scale in Fig 11).

4. >*If possible, I would suggest exploiting additional hierarchical graph classification models in experiments... other SOTA hierarchical models like DIFFPOOL[4] and G-U-Nets[5] which are also essential parts for graph classification tasks are suggested to be selected as targets.*

We thank the reviewer for suggesting these additional victim models. We focused on GCN and GIN as they are the most popular victim models used in the graph adversarial attacks literature (see a recent survey in [Sun et al. 2020]). While we agree that DiffPool and Graph U-nets can be more powerful for graph classification, unfortunately due to the time and compute constraint, we have not been able to fully evaluate our methods and baselines on these models during the initial response period. However, since the attack is formulated as a black-box, model-agnostic optimisation problem, there is no obvious reason why our method would not work. With that said, we are looking at the implementations of these models, and we look forward to adding results on these additional models in the near future.

References:

Sun et al. 2020: https://arxiv.org/abs/1812.10528

5. >*The limitations and potential negative societal impact are discussed in the Conclusion, though not adequately which could be due to the space limitation. The authors are suggested to further describe both of the limitations and social impact from multiple perspectives more in details, such as the user privacy issue when evaluating on the real-world Twitter dataset.*

Apart from the limitation described in conclusion, from a technical viewpoint a number of things could be done: firstly, in the paper we primarily focus on topological attack. We’d like to note that the surrogate we use handles both continuous and categorical attributes on nodes and/or edges, and thus it would be interesting to consider attacks on attributes and hybrid attacks. Secondly, while the patterns identified in Sec 5 point a way to possibly defend against graph-level adversarial attacks, in the present study we have not concretely developed a defence strategy. A follow-up work could fill this gap in literature.

On societal impact: We realise that there is a potential negative social impact of our work (as for the adversarial attacks literature) that bad actors may use our method to attack real-world systems, such as a fake new detection system on social media platforms, especially if they are built using graph-based machine learning models. However, we emphasise that our paper per se does not study such potentially negative applications. Instead, the main goal of our study is to reveal and raise awareness of potential vulnerability of such graph-based models, which is key to designing more effective defense mechanisms for their safer deployment in real-world scenarios. 

Finally, we emphasise that testing the proposed method on real-world datasets requires care to be taken especially in terms of data privacy issues. We note, however, that the Twitter dataset is publicly available and appropriately anonymised so that individual user privacy is well protected in this case. We thank the reviewer for reminding us of the importance of dataset anonymity and associated T&Cs. We will confirm with the authors of [29] that we have permission to use the data in the way that we have. In the case we cannot, an alternative dataset with similar characteristics exists and is released under the Apache 2.0 license that we can substitute (https://arxiv.org/pdf/2104.12259.pdf).





 We'd like to thank the reviewer for their time and effort. If the reviewer has further concerns that they think could be clarified, we will be more than happy to discuss.

 We thank the reviewer for their time and insightful comments. We are currently running experiments on G-U-Net, and from our preliminary results, GRABNEL remains effective and outperforms baselines such as Sequential Random on the PROTEINS dataset. We will include a full set of results in the final version of the paper.



 We would like to thank the reviewer for their insightful comments. An expanded version of the additional discussions above will be added to our paper.

** I have checked the author feedback **

This paper studies adversarial attacks to graphs and proposes ‚black-box‘ Bayesian optimization (BO) to find harmful perturbations. Moreover, the authors analyze the properties of adversarial examples to increase understanding of the topology of attacked graphs.  Originality:
1) The usage of BO to attack graph classification models is new. Though, BO has been used on black-box attacks before in the image domain. 

2) A new challenge with the application of BO on graphs is discreteness which is handled through the encoding through WL feature extraction. However, it would have been beneficial to investigate in more detail the effect of this feature extraction (e.g. how does the approach perform with other features?)

Quality:
1) The paper only evaluates the approach empirically. However, the evaluation could be more comprehensive. The authors could include more baselines to compare to (e.g. Nettack). The baseline GA is also used as a component of the proposed technique, and the „white-box“-method is difficult to compare to. Also, the Sequential Random in Fig. 12 in the App. F should be evaluated in more detail. In particular, there it is performing on par with the proposed approach. Furthermore, it would be interesting to see how the approach performs 2-hop and 2-hop + rewire experiments compared to the baselines.

2) All empiric results are evaluated over the number of queries. An additional runtime comparison would be interesting as another metric. In general, I feel that it would be beneficial to discuss more carefully the comparison (e.g. the white-box attack is not using any query budget at all; however, requires access to the parameters; is such a comparison fair?). Potentially it would be helpful to include more baselines which require/use the same set-up (this is partially done in App. F; but as discussed above not really in detail).

3) The attack analysis is highly beneficial. However, it could be improved through the evaluation of statistics. Currently, it is unclear how many samples were evaluated. Also, it would be great if the analysis would include different approaches as a comparison to see if there is a difference between what the proposed approach attacks compared to a baseline and what makes it more successful. Moreover, analysis of the patterns of attacks have been performed in other work which should be discussed be the authors (e.g. "Wu H, Wang C, Tyshetskiy Y, Docherty A, Lu K, Zhu L (2019) Adversarial examples for graph data: Deep insights into attack and defense" and "Zugner D, Borchert O, Akbarnejad A, Gunnemann S (2020) Adversarial attacks on graph neural networks: Perturbations and their patterns")




Clarity:
1) The paper has a good structure, is well written, and good to follow.

2) It seems to be counterintuitive why Nettack is not chosen as a baseline when it is discussed the most in related work.

3) Fig. 5 should be clarified: how many edges are on the graph, what dataset it is etc. 

Significance:
1) With a more extensive study on the attack analysis including a discussion of existing works as well as a couple of stronger attack baselines (see point above) this work could add significant insights into the domain of graph classification attacks. 
2) The application of BO on graph classification in combination with the analysis adds the opportunity of using more informed strategies (as briefly highlighted in Appendix G, constraining the search space)

 1) Limitations are not really discussed. Especially the strong baseline sequential random should be discussed in the main paper. With respect to the runtime, an argument could be made that sequential random could be better in some cases. However, the opportunity to use more informed decisions within the framework of the proposed approach would still make it interesting. This would make the limitation discussion sounder. 

2) Clearly, the attack has a potential negative impact, especially on the Twitter environment. But this point is discussed, and new defenses are said to be future work.


 Thank you for your detailed feedback. Please include these points and discussion in the updated version. I am happy to increase my score from 5 to 6.

 We thank the reviewer for their reply. We hope this additional response would clear some of the remaining concerns of the reviewer.

> *The authors argue that the adversarial examples shown in Figure 6 could support the claims in Section 5. But there is no explanation of Figure 6 in the current manuscript.*

We notice that there is a minor omission that Fig 6 is not referred to in the main text which potentially leads to this confusion, but we hope that it is relatively clear that **Sec 5 is an empirical account of Fig 6** (and other examples not in the figure due to page constraint), so Sec 5 *is* the explanation of Fig 6. We will add the missing reference in Sec 5 to Fig 6.

> *Discussions on the baselines*

While we thank the reviewer for providing the new references (which will be discussed in the final version of the paper), we’d respectfully argue that none of the setups of these works mentioned by the reviewer is fully comparable to the black-box evasion attack setup we consider. We argue that our setup is both challenging and highly significant, as in our opinion it resembles the most the capabilities a real-life attacker might have (no access to training data; no access to model parameter/gradients and limited query/perturbation budgets). This is also a setting that has only been scarcely studied in the literature. The works suggested by the reviewer are relevant but often significantly less restrictive, and adapting those to our setting is in our view at least non-trivial, and thus might in itself be an interesting research question that is worth of another independent investigation:


[a] requires more information than our method and is not fully black-box: it trains a surrogate model with access to **all train input and targets** (Sec IV, Paragraph A: Workflow of [a]) and conducts white-box attack on the surrogate model. If the method in [a] is prohibited from accessing training data like GRABNEL, we do not see a clear way to build this surrogate to perform downstream attack tasks.

[b] is Nettack. It has already been discussed in detail in the general response on why it may not be applied verbatim as a baseline, and why the white-box baseline in the paper already somewhat mimics its behaviour.

[c] is a back-door attack method based on poisoning that requires access and back-door trigger modifications to **both training and test inputs**. This is clearly different from evasion attack which attacks only at test time, nor do we see a straightforward way to adapt [c] into an evasion setting.

[d] proposes a method to **detect** adversarial samples. We agree that an interesting future work would be to see whether the detector successfully identifies and defends against adversarial samples produced by our method, and if so, see whether we can incorporate the SGN prediction as a constraint (similar to the 2-hop constraints by only allowing perturbations below the threshold of detection) to evade such a detection. However, since this work proposes a detection instead of an attack method, we do not see how it could straightforwardly be used as a baseline.

As discussed in the general response, the only works that are fully comparable, to the best of our knowledge, are RL-S2V and ReWatt. We already compare against RL-S2V, and we do not compare against ReWatt as 1) no open-sourced implementation is available and 2) the reinforcement learning framework it uses is very similar to RL-S2V with the key novelty being the rewiring operation (on which we also conduct experiment on in our paper). 


 The authors' response has cleared some of my doubts. But the main concerns still remain and thus I will keep my scores.
- My concern on the incremental development remains.
- The authors argue that the adversarial examples shown in Figure 6 could support the claims in Section 5. But there is no explanation of Figure 6 in the current manuscript.
- Some examples of relevant references are given below and could be considered as baseline methods:
[a] Tang, Haoteng, et al. "Adversarial attack on hierarchical graph pooling neural networks." arXiv preprint arXiv:2005.11560 (2020).
[b] Zügner, Daniel, et al. "Adversarial attacks on graph neural networks: Perturbations and their patterns." ACM Transactions on Knowledge Discovery from Data (TKDD) 14.5 (2020): 1-31.
[c] Zhang, Zaixi, et al. "Backdoor attacks to graph neural networks." Proceedings of the 26th ACM Symposium on Access Control Models and Technologies. 2021.
[d] Chen, Jinyin, et al. "Adversarial Detection on Graph Structured Data." Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice. 2020.



 Thank you for the detailed and thoughtful feedback. I have also read the comments from other reviewers as well as the corresponding feedbacks.

Most of my concerns are addressed, especially the connection with other black-box methods and the computational analysis are interesting and helpful. 

On the other hand, it would be better to include the hierarchical models for further evaluations in the future, considering their roles in the graph classification task. Thus, I would like to keep my score.

 Thank you for your thoughtful and detailed response. I now have a better understanding of the paper. However, I currently stand by my original score.

 *--- continued from above ---*

Here we provide some quantitative results of our algorithm running on graphs of different sizes:

Table R1: Runtime analysis of our algorithm in terms of average sec/iter (standard deviation in brackets. Slowest step bolded). Benchmarked on a machine with AMD Ryzen 7 3700X CPU and 32 GB 2666MHz RAM. Victim model = GCN. Results may vary depending on the hardware, system load and hyperparameters.

| \# nodes | \# edges|Step 1 | Step 2a  | Step 2b |
|:---|:---|:---|:---|:---|
| 17| 106  | 0.022 (0.0027) | 0.0344 (0.0002) 	|**0.482 (0.006)**	|
|  72 | 719  | 0.251 (0.003)  | 0.0371 (0.0006) |	**0.458 (0.009)** |
|  1961| 5336   | **1.76 (0.12)** |	0.0555 (0.0007) | 1.52 (0.016)	|
||||||

Note that even for a graph with almost 2k nodes and >5k edges (which is larger than most graphs in the TU dataset for graph classification), the runtime is still manageable on a mainstream desktop-grade PC. In fact, the genetic algorithm itself is efficient, and the much slower step is Step 2b: instead of our algorithm itself being inefficient, this is because of the large overhead in graph representation conversion between the list of changed edges (e.g. $([1,2], [3,4])$, which denotes a perturbed graph with edges $e_{(1, 2)}$ and $e_{(3, 4)}$ flipped from the original graph) and an actual DGL graph object. For better efficiency, it should be possible to reduce the number of such conversions as the perturbed and original graphs differ only at the flipped edges which only make up a very small fraction of all edges. At the very least, since each conversion is independent for other graphs, we can parallelise it for candidates in the population of the genetic algorithm (the current code does this sequentially).

To give a better context, on a shared Intel Xeon Gold server (we unfortunately could not provide very reliable and accurate statistics due to the varying load by other users), each attack on a single graph on the IMDB-M (average ~66 edges per graph) dataset usually takes < 1min. On COLLAB, which on average is much larger (average ~2500 edges per graph), each attack on average takes up to ~1h (up to 20,000 queries and around 4,000 iterations assuming batch size of 5). Note we set the maximum number of queries to be dependent on the sizes of the graph, so larger graphs are also proportionally allocated a higher query budget and hence each run is much longer. Furthermore, the sizes of graphs within a dataset can vary a lot, and runtime also depends a lot on the difficulty of attack (if an adversarial perturbation is easily found the run is terminated early). A further comparison with RL-S2V is that to run the same task on ER-graphs attack with 15-20 nodes, our method takes 30min ~ 1h. RL-S2V, which requires a separate validation set to train policy on, requires ~1.5h with GPU acceleration and ~12h without (we don’t use any GPU acceleration throughout for our method).

### References:

[1] Dai, H., Li, H., Tian, T., Huang, X., Wang, L., Zhu, J. and Song, L., 2018, July. Adversarial attack on graph structured data. In International conference on machine learning (pp. 1115-1124). PMLR.

[2] Ma, Y., Wang, S., Derr, T., Wu, L. and Tang, J., 2019. Attacking graph convolutional networks via rewiring. arXiv preprint arXiv:1906.03750.

[3] Ru, B., Cobb, A., Blaas, A. and Gal, Y., 2019, September. Bayesopt adversarial attack. In International Conference on Learning Representations.

[4] Huang, Z., Huang, Y. and Zhang, T., 2020. CorrAttack: Black-box Adversarial Attack with Structured Search. arXiv preprint arXiv:2010.01250.

[5] Zhang, Z. and Yu, S., 2021. BO-DBA: Query-Efficient Decision-Based Adversarial Attacks via Bayesian Optimization. arXiv preprint arXiv:2106.02732.

 We thank the Reviewer for their feedback on our work. While we feel their comments are at a rather high level, we have tried to respond carefully to the best of our capability. We hope that in light of this the Reviewer would consider increasing their score.

1. >*The technical contribution is incremental, which heavily limits the significance of the proposed method. To be specific, Weisfeiler-Lehman (WL) feature extraction methods and Bayesian optimization (BO) are well-known techniques, and the combination of them in this work is straightforward.*

We acknowledge that WL and BO are known techniques; however, we would like to emphasise that application of *either* of them in graph-level adversarial attack (which itself is an important yet under-studied problem in the literature) is completely new (also pointed out by Reviewer FuZ8). Furthermore, as Reviewer yB9K also pointed out, the flexibility afforded by the proposed method in handling edge rewiring/node injection in addition to edge flipping is also a novel contribution.

2. >*Although the attack analysis part provides some insights into the robustness of graph classification models, the last two claims are not well supported by empirical verification.*

We would like to note that the claims in question (Section 5) *are* qualitative summaries of empirical observations as far as our experiments go. We believe the statements on the three points in Section 5 are empirically supported by the adversarial examples shown in Figure 6. We would appreciate it if the reviewer could elaborate more on why they think these points are not well-supported by empirical verification, and we will address their concern to the best of our capability.

3. >*The comparison experiments are not convincing. The authors are encouraged to compare the proposed method with more recent SOTA methods, rather than one method that was published in 2018, on more benchmarks, e.g., AIDS, DHFR or Bitcoin. Besides, for the image classification and Fake news detection tasks, it will be more convincing to report the results of other competitive methods.*

As discussed above (also noted by Reviewer yB9K), graph-level adversarial attack is an important problem that has rarely been studied by previous works. While the main baseline we compared to was published in 2018, we are not aware of other existing published works focusing on a sufficiently comparable setup (black-box graph-level evasion attack). Having said that, we would be grateful if the reviewer could provide the relevant references we missed out, and we will be happy to carry out further comparisons and add them to our paper.

Image classification entails weighted graphs with both node and edge attributes, and Fake news detection involves node injection. To our knowledge, these setups have not been considered by the already very scarce previous works and that's the reason why no more competitive baselines are provided. Again, if the reviewer is aware of such works, we will be happy to take a look.

On the datasets: while we thank and welcome the reviewer for suggesting additional datasets to conduct experiments on, we would also like to note that TU graph kernel datasets which we use are overwhelmingly the most popular ones used in graph-level machine learning studies. The 3 TU datasets we use also cover graphs of diverse properties, sizes and classification accuracies. For example, PROTEINS is a biochemistry dataset, and it is similar to the DHFR and AIDS datasets mentioned by the reviewer. The Image classification task is also suggested as a new benchmark for graph classification which motivates our use. The Twitter dataset highlights a real world attack scenario (bypassing fake news detection) which we believe is one of the first of its kind in the literature.






 *--- continued from above ---*

10. >*For the case of continuous node features, it seems like the resulting feature vectors can be very high-dimensional (H⋅D⋅N), where D  is the feature dimension. Does this pose a problem of the method for large graphs?*

We acknowledge the possible high dimensionality of the features, and that is a reason why we use the automatic relevance determination (ARD) prior, which encourages sparsity [31] (reference list in paper) to focus on a small subset of features to have non-zero coefficients. Furthermore, we found H <= 1 to work well in practice. 

In the datasets we consider, there are already graphs in the order of 10^3 nodes and up to 10^4 edges. These graphs are already considered to be very large in the graph classification context.

11. >*Remark about the "Beware the low-degree nodes!" point. Since on Twitter the victim model is a GCN, this comes as no surprise: the weight in the neighborhood aggregation of node  is proportional to, i.e., lower-degree neighbors have larger influence in the neighborhood aggregation in GCN. This was also discussed by Zügner et al. (TKDD 2020).*

We agree it is intuitive that the representation of the low-degree node will experience greater change in its aggregation step. However, changes around one node changes the receptive field of nearby nodes in the graph. Changes in a large degree node are likely to cascade to more nodes in the graph than low degree nodes. Since graph classifiers aggregate across all nodes in the readout layer the indirect change of node representations also matter. Therefore for graph classification we would argue this interpretation is less straightforward. We will include a reference to Zügner et al. in this section of the work in the revised version.  




 We thank the Reviewer for their positive feedback on our work, and in particular for the detailed questions about various aspects in the empirical experiments. In addition to the general response above, we reply to the Reviewer’s comments in detail as follows.

1. >*What is the empirical runtime of the attack? Since the method performs a GA for each graph in a dataset, it seems to be quite expensive.*

The reviewer is referred to our overall response, which 1) explains why we believe in this paper the runtime is not necessarily the most important metric and 2) provides a runtime analysis. It is worth mentioning that while it is true that we run GA for each graph, the GA itself is actually very fast as we can use an efficient graph representation. The bottleneck often is due to data structure conversion, which is an engineering problem on which we also give some ideas for further optimization.

Furthermore, while we acknowledge that a possible future direction could be using some transfer learning setup so we don’t have to run GA on each graph, in related works the norm is still running the algorithm separately on each sample (e.g. in [22] (in paper reference list), GP-BO is run on each image). While other related works such as [7] require only running the attack once on the entire test set, they need a separate validation set (that is representative enough of the entire dataset) on which they repeatedly query and train a policy. In a practical setting this is rarely possible and even on benchmarks it is difficult to obtain a small subset of data that is also representative enough, as the graphs can vary significantly even within the same dataset. On the other hand, while we have to run GRABNEL separately on individual samples, we require no additional data for policy training, for example.

2. >*Another main concern is that the authors do not evaluate any defenses. While there may not be many defenses for graph classification attacks, I would expect that defenses for node classification (e.g., [Zhang and Zitnik 2020], [Geisler et al. 2020]) could improve robustness.*

We thank the reviewer for the comment. GNNGUARD [Zhang and Zitnik 2020] is designed to improve robustness of GNNs against poisoning attacks, whereas our method is primarily designed for the evasion setting. The soft medoid layer [Geisler et al. 2020] is shown to improve robustness of models to a variety of adversarial attacks. Furthermore, they show models equipped with soft medoid to have a larger certified radius compared to other trained models. Therefore, the model could be similarly robust to our adversarial attack method. We will add these discussions to the final version of the paper. Given the lack of attack methods in the literature in the context of graph classification, we believe a detailed analysis of defence would be better suitable for a follow-up work as this is a rather under-explored area of research. 

3. >*In many of the result figures, it seems that the proposed approach performs worse than random for a small number of queries. What is the reason for this and how can it be addressed? It would also be interesting to list the "break-even" number of perturbations per dataset evaluated.*

The reason for the behaviour mentioned by the reviewer is that GRABNEL also aims to modify the attacked graphs to differ as little from the original graph as possible (measured by number of edge edits). To achieve so, we are parsimonious w.r.t. the perturbations and we adopt a sequential strategy where an additional edge edit is only added when we fail to find an adversarial example with fewer edge edits (Sec 2, Sequential Perturbation Selection), and GRABNEL will favour a solution with fewer edge edits and often under-uses the budget. In contrast, random search will utilise the entire perturbation budget (which is an easier task, as there are likely to be more adversarial examples in the search space), which explains the early lead.

It is, however, possible for Random to also adopt similar parsimony w.r.t perturbations, and we call this baseline SeqRandom. The reviewer is referred to the appendix and our response to Reviewer FuZ8 for more discussions. It is easy to observe that the “early lead” disappears (e.g. Fig 12).

4. >*How is the second requirement of the surrogate model (probabilistic interpretation of uncertainty) used in the present work? On a related note, what is the performance of the model when using a standard L1- or L2-regularized linear regression surrogate model?*

Uncertainty estimation is key to balancing exploration and exploitation in BO (also allowing the use of “Bayesian” in BO). The surrogate mean will drive *exploitation* towards more promising regions (i.e. higher loss in our case) whereas uncertainty drives *exploration* of regions not seen thus far (which could have even better losses than what we’ve seen so far). Thus, uncertainty quantification is important for us to trade off exploitation and exploration, and is a key reason why BO works.

If we do not have uncertainty estimates by simply maximising the prediction mean, we are essentially doing greedy search by exploiting the surrogate mean only, and we would be sensitive to initialisation and get trapped easily. An example theoretically studying this is https://arxiv.org/pdf/0912.3995.pdf: in Fig 5, it is clear that simply maximising mean leads to a large regret, whereas regret diminishes quickly for mean + uncertainty (EI/UCB in the figure).

To give a small empirical comparison between a classical ridge regressor and Bayesian linear regressor, see the comparison below.

Table R5: Test accuracy of GCN victim model on PROTEINS dataset (clean=71.73%) using different surrogates with $\Delta \leq 0.003N^2$. Ridge regressor underperforms Bayesian linear regression and is slightly better than SeqRandom.

| | Bayesian | Ridge | SeqRandom |
|:---|:---|:---|:---|
| GRABNEL | **26.43** | 30.20  |  32.09|
|||||

5. >*Why does Table 1 show validation and not test accuracy?*

We thank the reviewer for spotting this. The samples on which we report results in Table 1 are not visible to victim models during training, nor did we use these samples to, for example, tune hyperparameters of the victim model. Thus, it should be more appropriately called  “test accuracy”.

6. >*Are the initialization points counted towards the global budget B ? In Algorithm 1 it seems like they are not counted towards B, which would be counterintuitive.*

We thank the reviewer for spotting this minor discrepancy. Initialisation points are indeed counted towards the global budget. We will amend Algo 1 correspondingly.

7. >*Why are there no results for Cheby-GIN in Table 1?*

We use a variety of datasets and models but do not consider all combinations between them. In particular:

We consider GCN/GIN for TU datasets, as these are most commonly used in related works (see a recent survey in  [Sun et al. 2020]).

We consider ChebyGIN on MNIST-75sp, as in the paper that initially proposes MNIST-75sp, ChebyGIN performs the strongest and we directly use the checkpoint provided by the authors.

References:
Sun et al. 2020: https://arxiv.org/abs/1812.10528

8. >*Why does the gradient-based baseline perform much worse than random on PROTEINS?*

While the gradient-based method often performs strongly in white-box attacks, there is no guarantee that it will always perform strongly. There are two main possible reasons: 

1. For general edge flipping problems where we may create or delete edges, we have to compute gradients w.r.t. all possible edges (including those that do not currently exist). Accurate estimation of such high dimensional gradients can be highly difficult. 

2. Gradients only capture local information and they are not necessarily accurate when used to extrapolate function value beyond that neighbourhood. However, relying on gradients to select edge perturbations constitutes such an extrapolation, as edge addition/deletion is binary (e.g. denoting edge absent=0 and edge present=1, when we are selecting an edge to create, we are using the gradients at 0 to extrapolate the loss value at 1. Vice versa if we are deleting an edge).

These two points mean that gradients may not only be inaccurate but sometimes even be misleading, which could account for their poor performance sometimes.

9. >*How to adjust the surrogate model for injected nodes? From the description I understand that injecting a node can lead to a longer vector representation of the graph (i.e., more entries). Therefore, the number of parameters α  and  λ  of the surrogate model do not fit the graph representation any more. How is this resolved?*

In our implementation, the number of injected nodes of each graph  $n^+$ is determined by the size of the graph, which is determined a-priori. When initialising the graph to the feature extractor, we can simply pad $n^+$ zero to the adjacency matrix so there is no inconsistency in vector length. Note that at initialisation, these additional nodes are “isolates” without being connected to the main graph, so they do not affect feature extraction as no message passing occurs. When we subsequently propose candidates for attacks, we wire one or more these isolated nodes to the main graphs.

The reviewer could also take a look at our codes at ./src/attack/bayesopt_attack_inject_nodes.py to see how this is exactly implemented.

*--- to be continued below ---*




 *--- continued from above ---*

6. >*Moreover, analysis of the patterns of attacks have been performed in other work which should be discussed be the authors (e.g. "Wu H, Wang C, Tyshetskiy Y, Docherty A, Lu K, Zhu L (2019) Adversarial examples for graph data: Deep insights into attack and defense" and "Zugner D, Borchert O, Akbarnejad A, Gunnemann S (2020) Adversarial attacks on graph neural networks: Perturbations and their patterns")*

We thank the reviewer for pointing us to these interesting papers. We agree that analysis of the patterns of attacks have been discussed in both works; nevertheless, both focus on node classification and the interpretation of adversarial examples might be different for the case of graph classification (which is the focus of our paper). For example, we found that for graph classification the adversarial examples often target graph-level topological properties such as community structure, and we believe this adds interesting insight to our understanding of patterns of adversarial examples for graph ML models in general. We will add a discussion of this to the revised version of the paper.

7. >*Fig. 5 should be clarified: how many edges are on the graph, what dataset it is etc*

We thank the reviewer for spotting this omission. It is on the MNIST-75sp dataset, as we referred to it in the main text later on.

8. >*Limitations are not really discussed. Especially the strong baseline sequential random should be discussed in the main paper. With respect to the runtime, an argument could be made that sequential random could be better in some cases. However, the opportunity to use more informed decisions within the framework of the proposed approach would still make it interesting. This would make the limitation discussion sounder.*

The issue on comparing against SeqRandom has been addressed in our response to Point 3.

As mentioned above, we believe the decision-making process depends on the setup which determines which algorithm is suitable: in particular, several factors need to be taken into consideration: a) whether there is a query budget (if so, how strict); b) how many edges are we allowed to edit (i.e. perturbation budget); c) the balance between sample efficiency and computational efficiency; d) the size of the graphs in question and e) the capability of the attacker.

An example where Sequential Random could be a good choice is 1) the graph in consideration is relatively small (although in which case the runtime of both SeqRandom and GRABNEL will be negligible), 2) query/perturbation budget is lenient and/or 3) we overwhelmingly care more about computational efficiency over sample efficiency: random enumeration of all possible edges will eventually find good perturbation candidates, and on small graphs this could even take a reasonable amount of time given that random enumeration is very efficient. However, for graphs with larger sizes random enumeration will need to query the model many times, which may not be possible under a stricter query and/or perturbation budget. 

We will add an expanded version of the above discussion in the paper.







 We thank the Reviewer for their appreciation of and insightful comments on our work. We are particularly grateful for the comments on conceptual and empirical comparisons against other attack models and the references on patterns of adversarial attacks. In addition to the general response above, we reply to the Reviewer’s comments in detail as follows. We hope that in light of this the Reviewer would consider increasing their score.

1. >*A new challenge with the application of BO on graphs is discreteness which is handled through the encoding through WL feature extraction. However, it would have been beneficial to investigate in more detail the effect of this feature extraction (e.g. how does the approach perform with other features?)*

Thank you for the suggestion. A possibility would be using the GCN/other graph deep learning model with the last layer replaced by Bayesian linear regression. This feature extractor could learn to extract more relevant features for the specific attack task, but we will have to train both the Bayesian linear regression parameters and the GCN extractor. On the other hand, the WL feature extraction is provably expressive and does not require learning from data, which should be more helpful in our context where we focus on sample efficiency.

2. >*The authors could include more baselines to compare to (e.g. Nettack)... It seems to be counterintuitive why Nettack is not chosen as a baseline when it is discussed the most in related work.*

Thank you for raising this important point. Nettack is discussed in the related work as many of the ideas are relevant, but adapting it to graph-level attack is much less straightforward. We would like to refer the reviewer to our general response for more details on this point.

3. >*Also, the Sequential Random in Fig. 12 in the App. F should be evaluated in more detail. Furthermore, it would be interesting to see how the approach performs 2-hop and 2-hop + rewire experiments compared to the baselines… potentially more baselines on the same setup*

As discussed in our general response, graph-level attack setting has only been scarcely studied in the literature. We have explained in detail the differences between our graph-level attack method and a few existing node-level attack methods and whether a comparison might be possible/fair. Extending node-level attacks to graph classification is in our view an important and open research direction.

Nevertheless, following the Reviewer’s suggestion we add further results and discussions on SeqRandom below. The short answer is that SeqRandom will perform strongly for easier tasks (e.g. high perturbation budget and smaller graphs) but otherwise GRABNEL is significantly better. For a detailed comparison, we refer the reviewer to the additional experiments below which we hope could resolve the Reviewer's concerns.

Table R2: Test accuracy of GCN victim models after attacks by GRABNEL and SeqRandom under default perturbation budget ($\Delta \leq 0.03N^2$). Bracket shows standard deviation when available, and some results are taken from the main paper.

| Dataset | IMDB-M  | PROTEINS  | COLLAB  | MNIST-75sp |
|:---|:---|:---|:---|:---|
| Clean | 50.53(1.4) | 71.73(2.6) | 79.73(2.1)	|92.3 |
|  GRABNEL | **45.23(0.2)**  | 10.82(2.5)   | **35.38(9.3)**  | **0.00**|
|  SeqRandom |  46.44(0.2) | **10.12(1.5)** | 49.80(6.8) | 	**0.00** |
||||||

This already shows that with exception of PROTEINS (which is the easiest task, judging from the ASR) where the two perform similarly within margin of error, GRABNEL outperforms in all other cases. However, looking at the performance *only when the budget has been exhausted* can be misleading: recall that we link the number of queries with the largest amount of perturbation allowed (Sec. 4), and in the paper we set a relatively lenient perturbation budget where we can change up to $\Delta \leq 0.03n^2$ edges. It is worth noting that the more lenient the budget, the easier the attack. With many possible adversarial samples in the search space, the importance of search strategies diminishes and algorithms might perform similarly. However, if we allow fewer edits (making the task more difficult), and take the example on PROTEINS dataset where SeqRandom performs the strongest (we will add a fuller set of results to final version of our paper):

Table R3: Test accuracy of GCN victim model on the PROTEINS dataset (clean=71.73%) after attacks under varying perturbation budget.

| $\Delta$ | $0.03N^2$ | $0.003N^2$ | $0.001N^2$ |
|:---|:---|:---|:---|
| GRABNEL | 10.82 | **26.43**  | **52.59** |
| SeqRandom | **10.12** | 32.09 | 60.51 |
|||||

It is easy to see that the margin of GRABNEL over SeqRandom increases significantly as the task difficulty increases. This is because SeqRandom takes more queries to reach the similar level of ASR of GRABNEL and is less efficient in increasing the attack loss. For a numerical example, please refer to https://anonymous.4open.science/r/GRABNELRebuttal-8F84/attack_loss_comp.pdf for examples which show the attack loss vs number of queries of GRABNEL vs SeqRandom. A line is red if the attack is successful and gray otherwise. We can observe when both are successful, GRABNEL typically requires fewer queries (and thus *fewer perturbations*). Even for the samples that neither managed to successfully attack, GRABNEL pushes the losses closer to 0.

Another concrete example would be the second subfigure of Fig 12, while both GRABNEL and SeqRandom converges eventually to 100% ASR on MNIST-75sp, GRABNEL converges ~2 times faster: on average, in a successfully attacked sample, GRABNEL rewires 1.84 +/- 0.7 edges but SeqRandom rewires 2.54 +/- 1.0 edges, which suggests that SeqRandom needs to modify 38% more edges on average to succeed. Imperceptibility in graph attack is usually measured in terms the number of edge edits, and thus while in some cases the end performance of SeqRandom and GRABNEL can be similar especially when a lenient perturbation budget is given, GRABNEL is both more sample-efficient and *produces less perceptible attacks*. We admit that the second point is not necessarily discussed enough in the current draft, and we will add a detailed discussion in the final version.

On the request of running SeqRandom on a constrained setup, we have added an experiment on PROTEINS below on a more difficult setup (to more clearly see the impact of introducing the additional constraints) and it is clear that GRABNEL outperforms. In the final version of the paper, we will include further results on more settings/datasets.

Table R4: Test accuracy of GCN victim model on the PROTEINS dataset (clean=71.73%) after attacks under various constraints with $\Delta \leq 0.003N^2$.

| |No constraint|2-hop| rewire |
|:---|:---|:---|:---|
| GRABNEL | **26.43** | **28.26**  |  **54.74**|
| SeqRandom | 32.09 | 49.08 | 60.40 |
|||||

4. >*An additional runtime comparison would be interesting as another metric. In general, I feel that it would be beneficial to discuss more carefully the comparison (e.g. the white-box attack is not using any query budget at all; however, requires access to the parameters; is such a comparison fair?).*

 While we agree that the runtime analysis is a relevant metric, we argue in our setup, measuring cost in terms of queries is more appropriate. The reviewer is referred to details of this argument along with a runtime analysis to the overall response above. 

White-box methods are added as an additional comparison to show the effectiveness of our method. However, as the reviewer pointed out, white-box setup is not really possible for practical attacks, which is also a reason why developing black-box algorithms is both interesting and difficult (i.e. without accessing gradients and/or parameters). 

The key ground for fair comparison here is the context and the ability of the attacker: if an attacker is indeed able to access the models, white box methods could be more desirable. However, this is rarely possible in real-life which makes our method more practical. Since our method requires access to much less information of the victim model, we will have to query the model a number of times.

We thank the reviewer for raising this point and will add the discussions above to the final version of the paper.

5. >*The attack analysis is highly beneficial. However, it could be improved through the evaluation of statistics. Currently, it is unclear how many samples were evaluated. Also, it would be great if the analysis would include different approaches as a comparison to see if there is a difference between what the proposed approach attacks compared to a baseline and what makes it more successful*

To arrive at the results we provide in Sec 5, we have considered at least ~100 adversarial samples per dataset (detailed experimental setup is in Appendix, but only a fraction of samples in the test set is originally classified correctly and only a fraction of the correctly classified samples are attacked successfully). However, due to page constraints, we cannot present all of them, so we selected some representative samples in Fig. 6, but the pattern described is not specific to the ones presented in Fig. 6, and we will be happy to include more such examples when we have a chance to update our supplementary materials.

At present, we cannot conclusively say there are significantly different patterns in the adversarial samples produced by different methods. Rather, we find our method to be more *efficient* in finding these *common* patterns in Sec 5.  Nonetheless, we agree that comparing adversarial samples from various approaches is an interesting direction. 

*---- To be continued below ----*


The authors propose an adversarial attack method for graph classification models. Specifically, the method is based on Bayesian optimization and uses a genetic algorithm (GA) as the acquisition function and a Bayesian linear regression surrogate model which provides the fitness scoring for the GA. The approach is greedy in the sense that it selects the best perturbation in a stage and then proceeds to the next one. The authors evaluate their method on a set of standard graph classification datasets ("TU Datasets") as well as on a graph version of MNIST and on a Twitter dataset.   The paper is well-structured and the presentation is clear. The method seems novel, though Bayesian Optimization and graph classification attacks are not my core field of expertise. Overall I am not opposed to publication as the results appear to be stronger than the baselines.

Nevertheless I do have some concerns / questions:
* What is the empirical runtime of the attack? Since the method performs a GA for **each** graph in a dataset, it seems to be quite expensive.
* Another main concern is that the authors do not evaluate any defenses. While there may not be many defenses for graph classification attacks, I would expect that defenses for node classification (e.g., [Zhang and Zitnik 2020], [Geisler et al. 2020]) could improve robustness.
* In many of the result figures, it seems that the proposed approach performs worse than random for a small number of queries. What is the reason for this and how can it be addressed? It would also be interesting to list the "break-even" number of perturbations per dataset evaluated.
* How is the second requirement of the surrogate model (probabilistic interpretation of uncertainty) used in the present work? 
* On a related note, what is the performance of the model when using a standard $L_1$- or $L_2$-regularized linear regression surrogate model?
* Why does Table 1 show validation and not test accuracy?
* Are the initialization points counted towards the global budget $B$? In Algorithm 1 it seems like they are not counted towards $B$, which would be counterintuitive.
* Why are there no results for Cheby-GIN in Table 1?
* Why does the gradient-based baseline perform much worse than random on PROTEINS?
* How to adjust the surrogate model for injected nodes? From the description I understand that injecting a node can lead to a longer vector representation of the graph (i.e., more entries). Therefore, the number of parameters $\alpha$ and $\lambda$ of the surrogate model do not fit the graph representation any more. How is this resolved?
* For the case of continuous node features, it seems like the resulting feature vectors can be very high-dimensional ($H\cdot D \cdot N$), where $D$ is the feature dimension. Does this pose a problem of the method for large graphs?
* Remark about the "Beware the low-degree nodes!" point. Since on Twitter the victim model is a GCN, this comes as no surprise: the weight in the neighborhood aggregation of node $j$ is proportional to $\frac{1}{\sqrt{d_j}}$, i.e., lower-degree neighbors have larger influence in the neighborhood aggregation in GCN. This was also discussed by Zügner et al. (TKDD 2020). The authors do have a short section on limitations. In the case that the empirical runtime is very large, I recommend to list this also as a limitation of the approach.

This paper proposes a query-efficient and parsimonious attack method for graph classification models based on Bayesian optimization. Attacks are performed under the black-box evasion setting by flipping or rewiring edges and injecting nodes. Besides, the authors provide insights between topological properties of perturbed graphs and graph classification models.   Strengths:
- Both the problem of adversarial attacks on graph classification and attack modes of rewiring edges as well as node injection is rarely researched, which shows the novelty of this work.
- The experimental results are strong. Various domains of datasets are used and different attacking strategies are exploited in the experiment. 
- The attack analysis of the relationship between robustness and topological patterns of adversarial samples is interesting and could provide insights for further graph robustness research.

Weakness:
-	The black-box setting used in this work falls into the approach with a specific surrogate model. There are several works considering the black-box setting without a surrogate model, such as [1-3]. Though their focuses are on the node classification side, the relationship with this line of research is suggested to be discussed either. 
-	The computation analysis of the total attack process should be demonstrated. The authors only provide the complexity of the surrogate model and neglect that of sequential perturbation selection and acquisition function optimization. The complexity analysis is crucial when using large-scale graphs as one strong motivation from authors to choose a proper surrogate model. Further, it would be better if the running time analysis is provided.
-	For the experimental results, why there is no comparison between RL-S2V and GRABNEL on real-world datasets? Meanwhile, the advantage of GRABNEL over RL-S2V is not strong enough on the ER graphs.
-	If possible, I would suggest exploiting additional hierarchical graph classification models in experiments. GCN and GIN are strong classic graph classification models but two victims are kind of weak, especially considering GCN is not very powerful for graph classification tasks. Therefore, other SOTA hierarchical models like DIFFPOOL[4] and G-U-Nets[5] which are also essential parts for graph classification tasks are suggested to be selected as targets.

[1] Towards More Practical Adversarial Attacks on Graph Neural Networks, NeurIPS 2020

[2] Adversarial Attacks on Node Embeddings via Graph Poisoning, ICML 2019

[3] A Restricted Black-box Adversarial Framework Towards Attacking Graph Embedding Models, AAAI 2020

[4] Hierarchical graph representation learning with differentiable pooling, NeurIPS 2018

[5] Graph u-nets, ICML 2019
 The limitations and potential negative societal impact are discussed in the Conclusion, though not adequately which could be due to the space limitation.

The authors are suggested to further describe both of the limitations and social impact from multiple perspectives more in details, such as the user privacy issue when evaluating on the real-world Twitter dataset.

Meta-review: This paper provides a new Bayesian Optimization approach to developing adversarial attacks on graphs. While some prior work has used a similar approach for other domains (e.g., images), it seems that the authors are the first to develop a successful attack methods for graphs in this way. The reviewers felt that the experiments were mostly thorough and at least were convinced about the contributions of this paper. Moreover, there was a lot of discussion, but the reviewers seem to have reached consensus with the authors on the major issues. Therefore, I recommend acceptance of this paper. 

Given the lengthy discussion, I would strongly encourage the authors to be thoughtful and thorough in the revision for the next version of the paper. It seems that many of the reviewer concerns can be directly addressed in the paper by improving the writing and exposition. 

Decision: accept