FLAG: Adversarial Data Augmentation for Graph Neural Networks

Abstract: Data augmentation helps neural networks generalize better, but it remains an open question how to effectively augment graph data to enhance the performance of GNNs (Graph Neural Networks). While most existing graph regularizers focus on augmenting graph topological structures by adding/removing edges, we offer a novel direction to augment in the input node feature space for better performance. We propose a simple but effective solution, FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training, and boosts performance at test time. Empirically, FLAG can be easily implemented with a dozen lines of code and is flexible enough to function with any GNN backbone, on a wide variety of large-scale datasets, and in both transductive and inductive settings. Without modifying a model's architecture or training setup, FLAG yields a consistent and salient performance boost across both node and graph classification tasks. Using FLAG, we reach state-of-the-art performance on the large-scale ogbg-molpcba, ogbg-ppa, and ogbg-code datasets.  

Reviews and author responses:

---- Summary

This paper proposes FLAG (Free Large-scale Adversarial Augmentation on Graphs), an adversarial data augmentation technique that can be applied to different GNN models in order to improve their generalization. The proposed technique consists on adding adversarial perturbations to the nodes’ features solving the standard min-max problem for adversarial training. In this setup, a noise vector is added to the input features which tries to maximize the loss by performing gradient ascent, while the classifier is trained to minimize the loss despite the added adversarial noise.
The authors show that their method can easily be added to standard GNN models like GCN, GAT, GraphSAGE and DeeperGCN with minimal changes, and that it improves these models on different tasks.

---- Pros

* The authors evaluate their method on the Open Graph Benchmark (OGB), which is a standard benchmark that facilitates comparison with other methods.
* FLAG is a generic and model agnostic method that can be applied to already established models like GCN, GAT and GraphSAGE.
* Clear explanation of the proposed method with a concise pytorch implementation as an example.
* Good empirical results with different models on different OGB tasks.

---- Cons

* My main concern with this paper is that the proposed augmentation technique is not compared against any other graph augmentation techniques. What is the justification for using this one over other techniques? Can FLAG work with other data augmentation techniques? Showing experiments of FLAG in combination or compared to other techniques would strengthen the experiments section.

---- Questions to the authors

* As I said in my concerns, if this kind of feature data augmentation is orthogonal to structure-based data augmentation that adds or removes edges (or nodes), it would be very interesting to do an experiment showing to which degree it can be used together with other graph augmentation techniques.
*The motivation for the biased perturbation explained in the “Biased perturbation for node classification” paragraph in Section 3 is not very clear. The authors claim that nodes that are far away from a target (labelled) node will have lower effect on the prediction for that node, thus they increase the adversarial step size $\alpha_u$ for the unlabelled nodes. That will increase the amount of noise added to the unlabelled nodes, but why is that beneficial? Also, what is the relation between unlabelled nodes and their distance to the target nodes? Why does it have anything to do with promoting invariance for further-away nodes? 

---- Minor comments

- Although the approach is a bit different, the Adversarial AutoAugment paper [1] seems to be quite related, the authors might want to reference it in their paper. Besides the task domain (images vs graphs) the main difference is that instead of adversarially learning perturbations, in [1] data augmentation policies are adversarially learnt to apply perturbations to the input. 
- Also, there are a couple of contemporary papers on graph data augmentation [2, 3] that the authors could reference too.

---- Justification for score

The paper is well written and the method interesting and well explained, but it lacks comparisons with other graph data augmentation approaches.

------ Post-Rebuttal Update ------

I have read the author's response, thank you for adding more experiments with other graph data augmentation techniques. However, the explanation and justification for the "biased perturbations" still seems a bit weak and under explored in the experiments section, so I've decided to keep my initial score.

---- References

[1] Zhang, Xinyu, et al. "Adversarial autoaugment." arXiv preprint arXiv:1912.11188 (2019).

[2] Jiajun Zhou, Jie Shen, and Qi Xuan. 2020. Data Augmentation for Graph Classification. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM '20). Association for Computing Machinery, New York, NY, USA, 2341–2344. DOI:https://doi.org/10.1145/3340531.3412086

[3] Zhao, T., Liu, Y., Neves, L., Woodford, O., Jiang, M., & Shah, N. (2020). Data Augmentation for Graph Neural Networks. arXiv preprint arXiv:2006.06830.


This paper investigates adversarial feature augmentation for improving the generalizability of graph neural networks. The authors adopt an existing augmentation algorithm and apply on the nodes of each training graph, and use the perturbed graphs for training. The focus of the paper is extensive experimentation in various tasks and settings to illustrate the effectiveness of adversarial augmentation in graph-based tasks. The experiments provide new, non-trivial insights, such as the effect of the number of network layers on the effectiveness of augmentation. The paper is well-written and easy to read.

Nevertheless, there are a few drawbacks:

1. Even though the method is completely adopted from prior work and there are no novelties, the authors claim they propose a new solution for graph data augmentation. It is true that adversarial feature augmentation has not been studied for graph neural nets, but it is a straightforward idea to apply an existing feature augmentation method on graph nodes, which are represented using feature vectors just like other types of data. 

2. Even though the performance improvement is consistent across tasks, it is not significant. Hence, my takeaway from the paper (which is in fact a valuable takeaway) is that adversarial augmentation is not considerably effective for graph neural nets, no matter what dataset and network architecture is used. A truly "free" augmentation is even less effective, as shown in table 2. To be clear, this is not a weakness of this paper. However, the authors are encouraged to acknowledge it and use a more neutral and objective language when describing the performance.

3. Some conclusions are not reliable enough. For instance, the impact of biased perturbation is statistically insignificant in table 2, and those limited numbers are not sufficient to make such a deduction. Moreover, section 6 is not convincing. Just because augmentation hurts the performance of MLP on images but improves on graphs doesn't mean data distribution is the primary determinant of the efficacy of augmentation. It is similarly insufficient that augmentation improves the results on a discrete graph but doesn't improve when noise is added. Moreover, the authors do not elaborate "how" the data distribution affects the augmentation. Just proving that data distribution has an effect is a trivial and unhelpful fact. Furthermore, the authors do not support the claim that model architecture does not affect the efficacy of augmentation. In fact, they prove otherwise by figure 1 (left), where the network depth has a direct effect on the performance gap.

4. Table 5 is confusing, as it does not demonstrate any relevant information. The two bottom rows of both tables are a repetition of results from table 1, and only the first row is new, which only shows that GAT performs worse without BatchNorm or Dropout. The table does not show how FLAG performs without BatchNorm and Dropout, and hence there is no new takeaway from this table that is relevant to the purpose of this paper.

In sum, although the paper provides new and valuable findings, the experiment analysis and conclusions that are made are not strong enough for a purely empirical study. Hence, I recommend improving and resubmitting the paper, either by adding methodological novelty, or by bolstering the analysis sections to make more reliable and useful conclusions. I also encourage the authors to clarify any part that I may have misunderstood, as I am keen to adjust my rating accordingly.

######## Post-Rebuttal Updates:

I appreciate the authors' response, but my main concerns were not addressed. Particularly, I still believe the novelty of this method is extremely limited, as it directly applies an existing method on graph node embeddings. The main novelties are biased perturbation and unbounded attack, which are both very simple modifications, and were not adequately studied in experiments. Although the authors show extensive comparison on various datasets, their comparison does not particularly show the contribution of biased perturbation and unbounded attack. They only show the effect of biased perturbation using a single number in Table 2, which is not convincing. 




We thank the reviewer for the valuable feedbacks and for noting that our paper carries out extensive experiments and provides non-trivial insights. 

1: “The method is completely adopted from prior work.”

We respectfully disagree. As stated in the Section 3, FLAG adopts “Free” method and gradient accumulation for adversary crafting and model weights updating. These two methods are both common practices of adversarial training. Besides, the proposed unbounded attack and biased perturbation are simple but important techniques that work solely for GNNs to simplify hyperparameter search and to promote reasonable extent of invariance with respect to nodes. Therefore we claim that FLAG is a new adversarial augmentation method on graphs.

2: "Even though the performance improvement is consistent across tasks, it is not significant.”

We clarify that, the primary motivation of our work is to show that adversarial augmentations are able to generalize GNNs across varied GNN baselines, prediction tasks (node and graph), graph datasets, etc. We acknowledge the existence of several marginal performance lifts (even drops), but note that such a highly general method can also yield significant improvements like what GAT gets on ogbn-products (2.31%) and what GIN-Virtual obtains on ogbg-ppa (2.08%). Also we agree to use a more neutral tone and we will refine our writing in the newer version of the paper. 

3.1: “section 6 is not convincing”

To clarify, in Section 6 we aim to provide a new perspective to analyze and explain the effect of adversarial augmentation. To our knowledge, our work is the first to discuss and try to explain the different behaviors of adversarial augmentation on different tasks, and our analysis is endorsed by Reviewer1. Intuitively the discrepancy can most possibly come from the following two aspects: model differences (message-passing GNNs vs. feed-forward CNNs (MLPs)), data distribution differences (discrete distribution of graphs vs. continuous distribution of images).

We conjecture that the main cause comes from the latter with the following evidence:
(a). Clean performance of CNNs is harmed by adversarial augmentations on CIFAR-10, CIFAR-100 and ImageNet, but is boosted on MNIST [3], whose pixel values are much closer to discrete distribution.
(b). NLP tasks share the same phenomenon, where word embeddings are represented as discrete one-hot vectors.
(c). Using FLAG to augment MLPs (a kind of feed-forward neural net like CNNs, which adversarial augmentation also has adverse effects on in computer vision), as reported in Section 6, on ogbn-arxiv the test performance goes from 55.50±0.23% to **56.02±0.19%**. To further support the argument, we add another experiment using FLAG to lift MLPs from 61.06±0.08% to **62.41±0.16%** on ogbn-products.
(d). As shown in the simple example, when node features are drawn from continuous normal distribution, FGSM starts to harm the performance of GCN. 

All of the four observations supports our conjecture. We hope our analysis and findings can provide insights and catalyze future works on this important topic.

3.2: "the impact of biased perturbation is statistically insignificant in table 2”

We disagree. Note that the node size of ogbn-products is more than **2.4M**, from which 90% of the nodes are test nodes (2.2M). The lift from biased perturbation is 0.47%, which means **10K+** more nodes are successfully classified and the improvement is non-trivial. It is even more than the amount of the test set of CIFAR-10. We also respectfully point out that this ablation study is endorsed by Reviewer1.

3.3: “the authors do not support the claim that model architecture does not affect the efficacy of augmentation.”

We apologize for the confusion. To clarify, we discuss about the discrepancy between message-passing GNNs vs. feed-forward CNNs (MLPs), rather than specific model architectures (types, layers, widths, etc.). Also we focus on the positive vs. negative effect difference, rather than extent. We will refine our writing in the newer version.

4: "Table 5 is confusing”

We apologize for the confusion. We clarify that it is not intuitive that batchnorm and dropout are supposed to work with adversarial augmentations directly. [1,2] showed that special batch norm and dropout techniques were important to cope with adversarial augmentation. Here in table 5 we show that FLAG can directly work with both techniques to improve performance further without extra designs, which adds to the simplicity and compatibility of our method. The reusing of existing numbers are to make comparisons clearer. We will reorganize the writing in newer version of our paper. 

Please let us know if you have further questions or comments!

[1] Xie, et al. "Adversarial examples improve image recognition. CVPR 2020

[2] Zhu, et al. "Freelb: Enhanced adversarial training for natural language understanding." ICLR 2019

[3] Tsipras, et al. "Robustness May Be at Odds with Accuracy." ICLR 2018

We thank the reviewer for helpful feedbacks and for noting that FLAG is a general method for GNNs.


1.“This work applies FreeLB adversarial training [1] to graph neural networks.”


We wholeheartedly disagree with the reviewer on this viewpoint. “Free” batch-replay and gradient accumulation are both common techniques on adversarial training. To most exploit the power of adversarial augmentation on graphs, biased perturbation and unbounded attack are also adopted. The effect of biased perturbation is shown in the ablation studies, and unbounded attack helps simplify hyperparameter search greatly. Also as shown in the right part of Table 4, on ogbn-products with GAT as backbone, FLAG wins over PGD, Free, and FreeLB by a decent margin.   


2.“The method improves results by a very small percentage even for cases with low baseline accuracy.”


To clarify, we refer the reviewer to our responses of the Question 2 and Question 3.2 to Reviewer3.


3."Adversarial training with small perturbations may be considered as a best practice rather than a novel contribution.”


We disagree. Adversarial augmentation is indeed a best practice to improve the robustness of models against attacks out of security purpose, but there are limited amount of works promoting the generalizing effect of adversarial augmentations which is at odds with the wide belief that it harms model clean accuracy. We hope our work can arouse the interest of graph community in this straightforward and scalable approach to generalizing GNNs.


4.“DeeperGCN+FLAG and GAT+FLAG are currently ranked 5th and 6th on the OGB leaderboard for node property prediction. DeeperGCN+FLAG is currently ranked 2nd on the OGB leaderboard for graph property prediction.”


Although the motivation of our work is to reveal the generalizing ability of adversarial augmentations and propose a method that is widely effective, FLAG still reaches high on the leaderboard.
To rectify, upon submitting to the OpenReview, on the graph classification task of the Open Graph Benchmark Leaderboard, FLAG reaches

| ogbg-molpcba | 1st |
 
| ogbg-ppa | 1st | 

| ogbg-code | 1st | 

| ogbg-molhiv | 2nd | 

Upon submitting to the OpenReview, on the node classification task of the Open Graph Benchmark Leaderboard, FLAG reaches

| ogbn-products | 2nd |

| ogbn-proteins | 2nd |

| ogbn-arxiv | 2nd |

| ogbn-mag (heterogeneous) | 3rd |

Please let us know if you have further questions or comments!

This work applies FreeLB adversarial training [1] to graph neural networks.

Strengths:
There are various best practices which are commonly applied in common task framework competitions
such as data augmentation, adversarial training, and ensembles that improve results by fractions 
or a small number of percentage points. This work falls under such methods, and proposes to 
perform FreeLB adversarial training by computing the loss with respect to perturbations.
The method is general and may be applied to any GNN for a very small improvement.

Weaknesses:
The algorithm (1)[1] is not novel and its application to GNNs is an incremental contribution.
The method improves results by a very small percentage even for cases with low baseline accuracy.
Adversarial training with small perturbations may be considered as a best practice rather than a novel contribution.
DeeperGCN+FLAG and GAT+FLAG are currently ranked 5th and 6th on the OGB leaderboard for node property prediction.
DeeperGCN+FLAG is currently ranked 2nd on the OGB leaderboard for graph property prediction.

[1] FreeLB: Enhanced Adversarial Training for Natural Language Understanding, Zhu et al, 2019.

Due to the many relative improvements of FLAG,
rather than absolute rankings, on the OGB leaderboards, 
I think that the overall contribution is above the acceptance threshold.

1. Node property prediction
a. ogbn-products: +2: DeeperGCN+FLAG, +5: GAT+FLAG, +6: GraphSAGE+FLAG	
b. ogbn-proteins: +1: DeeperGCN+FLAG	
c. ogbn-arxiv: +1: GAT+FLAG, +8: GraphSAGE+FLAG, +5: DeeperGCN+FLAG, +4: GCN+FLAG, +1: MLP+FLAG	
d. ogbn-mag: +1 R-GCN+FLAG	

2. Graph property prediction
a. ogbg-molhiv: +3: DeeperGCN+FLAG, +1: GIN+virtual node+FLAG, +2: GCN+FLAG, +3: GIN+FLAG	
b. ogbg-molpcba: +2: DeeperGCN+VN+FLAG, +2: GIN+virtual node+FLAG, +1: GCN+VN+FLAG, +1: GIN+FLAG, +2: GCN+FLAG	
c. ogbg-ppa: +1: DeeperGCN+FLAG, +1: GIN+virtual node+FLAG, +3: GCN+virtual node+FLAG, +1: GIN+FLAG	
d. ogbg-code: +2: GCN+virtual node+FLAG, +4: GIN+virtual node+FLAG, +4: GIN+FLAG, +2: GCN+FLAG

Hello Yuning,

Thanks for your interest in our paper. Advanced graph augmentation methods will very promisingly boost the performance of Graph Contrastive Learning. We are glad to see future works that incorporate adversarial data augmentation into the Graph Contrastive Learning pipeline. 

We thank the reviewer for constructive feedbacks and valuable comments, and for acknowledging the merits of our approach. 

1.“feature data augmentation is orthogonal to structure-based data augmentation”

We agree, and it adds to the generality of our method. Below we show that our feature-based augmentation is compatible with two widely-used structure-based methods.

Neighbor sampling proposed by [1] is one kind of structure-based data augmentation. During training time, each node randomly samples k neighbors to aggregate information from. The other neighbor nodes and connecting edges are dropped. On the ogbn-products dataset with GraphSAGE as baseline, the experimental results are as below:

| method                                                             |   test acc |

| GraphSAGE                                                      | 78.50 ± 0.14 |

| GraphSAGE + Neighbor Sampling               |  78.70 ± 0.36 |

| GraphSAGE + Neighbor Sampling + FLAG  | 79.36 ± 0.57 |

Virtual Node method [3] augments the graph by adding one synthetic node who connects with all the original nodes in the graph. From Table 3, almost all the numbers support the claim that FLAG works with Virtual Node well to further improve accuracy, the only exception is from GCN on ogbg-molhiv, but note that FLAG fails to improve GCN+virtual node because the existence of virtual node already hurts the performance of the vanilla GCN on this dataset. We highlight one representative group of experiments on ogbg-ppa with GIN as baseline here.

|  method                            |  test acc |

| GIN                                    |  68.92 ±1.00 |

| GIN+virutal node            | 70.37 ±1.07 |

| GIN+virutal node+FLAG | 72.45 ±1.14 |

What’s more, DropEdge [2] is the simple approach to randomly dropping edges in the graph to promote generalization. However our experimental results show that DropEdge fails to further generalize GAT on ogbn-products in the first place, so we exclude it in the discussion here. We thank the reviewer for this valuable comment. We will cover this aspect in our newer version of the paper.

2."That will increase the amount of noise added to the unlabelled nodes, but why is that beneficial?”

It is natural to think that we don’t want the prediction of a node to be highly impacted by another which is far away from it. So when classifying one node, we want the output of the GNN classifier to have higher invariance with the input features of  the nodes that are further away. To realize this idea, our approach is to inject larger noise on the features of unlabeled nodes to make the loss landscape smoother.   

3."Also, what is the relation between unlabelled nodes and their distance to the target nodes? Why does it have anything to do with promoting invariance for further-away nodes?"

When training GNNs for node classification, the most common practice is to compute final objective only with respective to all the labeled nodes. In other words, only the final-layer representations of labeled nodes are fed into the objective function (e.g., cross entropy loss). Our practice of having two different step-sizes is a simplification for our high-level idea (promoting higher invariance for further-away nodes). Labeled nodes are the ones to classify, and the distance between it with itself is 0, so we set smaller step sizes for them; meanwhile the distance between unlabeled node with labeled node is at least 1. For simplicity during implementation we set a universal larger step size for all the unlabeled nodes. We will refine our writing to make it clearer.

4.Reference

We thank the reviewer for mentioning related literature and we will consider referencing them.

Please let us know if you have further questions or comments!

[1] Hamilton, Will, Zhitao Ying, and Jure Leskovec. "Inductive representation learning on large graphs." Advances in neural information processing systems. 2017.

[2] Rong, Yu, et al. "DropEdge: Towards Deep Graph Convolutional Networks on Node Classification." International Conference on Learning Representations. 2019.

[3] Gilmer, Justin, et al. "Neural Message Passing for Quantum Chemistry." ICML. 2017.

We thank the reviewer for insightful feedbacks and for endorsing our analysis and ablation studies. 

1.“For Figure 1, could you provide more analysis on the distribution shift besides the accuracy? “

Yes. Common graph datasets come with discrete node features (i.e., bag-of-words embeddings) and discrete edge connections (symmetric binary adjacency matrix). In our example, when node features are originally discrete, FGSM (a simple one-step adversarial perturbation crafting method) successfully boosts the performance, however when gaussian noises are introduced, the effect goes from positive to negative. For data drawn from discrete distribution (i.e., categorical distribution), it is hard to define its smoothness with respect to the final loss. Our conjecture is that the sparsity of the dataset matters, where graphical data is intrinsically sparse due to the discrete property.  [1] showed that when only a few of training samples are available to train the model, where the sparsity of data is obvious, adversarial data augmentation is helpful in terms of clean accuracy. We believe the question on how and why distribution shift affects the effects of adversarial data augmentation is interesting and worth more attention. 

2.“more insights on how these approaches used in FLAG.”

FLAG leverages the batch-replay technique proposed by “free" to promote efficient adversarial training. Besides that while computing the gradient ascent step for the perturbation, gradients are also accumulated for the model weights. What’ s more biased perturbation and unbounded attack  proposed by us which are targeted at GNNs also help FLAG perform better. While in NLP, the perturbation happens in the embedding space, FLAG injects noise in the input node feature space on the node classification task. All the word embeddings are perturbed with the universal step size, while in FLAG target and training nodes are treated differently. What’ more for the V+L task, the authors added KL loss into the final objective function, which would consume at least twice the GPU memory needed for training, making scalable training more challenging. In FLAG our final objective function always follows the usage of the original baseline, which won’t introduce any extra space overhead and guarantee FLAG’s scalability to larger graphs and deeper nets. Therefore FLAG is a novel adversarial training method for GNNs.  

Please let us know if you have further questions or comments!



[1] Tsipras, Dimitris, et al. "Robustness May Be at Odds with Accuracy." International Conference on Learning Representations. 2018.

We would like to draw your attention that we have related work showing augmentation in graphs can boost robustness while it is incorporated in the contrastive learning framework. We hope to have further discussions and references with you.

Graph Contrastive Learning with Augmentations, NeurIPS 2020.
https://arxiv.org/abs/2010.13902

This paper presented an adversarial augmentation technique for graph neural networks. Particularly, it proposed to inject perturbations to the embedding space of the feature space with the "free" strategy. The results on three datasets show the effect of the proposed method. 

This is the first known work on applying this technique to graph NN. I really like the analysis of why this method works on graph data, which shows that the data distribution shift is important. The ablation studies are also strong, especially on the bias of perturbations, and going deep, etc. For Figure 1, could you provide more analysis on the distribution shift besides the accuracy? 

There are a few concerns about this paper. Similar techniques have been applied to other domains, such as NLP and V+L [1,2]. The "free" strategy is also not new [3]. it is better for the author to provide more insights on how these approaches used in FLAG.

1. FreeLB: Enhanced Adversarial Training for Language Understanding, ICLR 2020
2. Large-Scale Adversarial Training for Vision-and-Language Representation Learning, Neurips 2020
3. Adversarial Training for Free, Neurips 2019

Meta-review: This paper studies the problem of adversarial training for graph neural networks. The proposed method is build on the free training approach, and more specifically FreeLB, with some additional tricks including bias perturbation (for node-classification) and unbounded attacks.  While these additions are potentially useful, there are only limited investigation into their effect.  Putting aside the technical distinctions of the method with prior work, this paper can also be viewed as an empirical study of adversarial training techniques on graph data with various GNN architectures.  It is worth noting that overall the conclusions on "adversarial training" are positive, we do see consistent improvement over a variety of architectures and tasks.   The issues however, is that it is unclear whether these improvements can be similarly achieved using prior technique like FreeLB (oblation is only done on one single task, where biased perturbation is shown to lead to minor improvement).   The paper also provides some results showing the effect of depth of the network as well as different training strategies such as batch norm, dropout with general adversarial training.  These results are interesting to see but do seem to be limited in both scope and depth.  It appears that the authors have two goals in mind, one is to propose FLAG and demonstrate its usefulness, and the other is to provide a better understanding of how adversarial training works for GNNs in general.  Given the limited novelty of FLAG compared to prior methods, the main contribution actually comes from the later part, which unfortunately is somewhat underdeveloped.  


Decision: withdrawn-rejected-submissions