FLAG: Adversarial Data Augmentation for Graph Neural Networks
Authors: Kezhi Kong
Guohao Li
Mucong Ding
Zuxuan Wu
Chen Zhu
Bernard Ghanem
Gavin Taylor
Tom Goldstein
Venue:  (2021)
Keywords: Graph Neural Networks
Data Augmentation
Adversarial Training

Abstract: Data augmentation helps neural networks generalize better, but it remains an open question how to effectively augment graph data to enhance the performance of GNNs (Graph Neural Networks). While most existing graph regularizers focus on augmenting graph topological structures by adding/removing edges, we offer a novel direction to augment in the input node feature space for better performance. We propose a simple but effective solution, FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training, and boosts performance at test time. Empirically, FLAG can be easily implemented with a dozen lines of code and is flexible enough to function with any GNN backbone, on a wide variety of large-scale datasets, and in both transductive and inductive settings. Without modifying a model's architecture or training setup, FLAG yields a consistent and salient performance boost across both node and graph classification tasks. Using FLAG, we reach state-of-the-art performance on the large-scale ogbg-molpcba, ogbg-ppa, and ogbg-code datasets.  

Under review as a conference paper at ICLR 2021
FLAG: ADVERSARIAL DATA AUGMENTATION FOR GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Data augmentation helps neural networks generalize better, but it remains an open question how to effectively augment graph data enhance the performance of GNNs (Graph Neural Networks). While most existing graph regularizers focus on augmenting graph topological structures by adding/removing edges, we of- fer novel direction augment in the input node feature space for better per- formance. We propose a simple but effective solution, FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training, and boosts perfor- mance test time. Empirically, FLAG be easily implemented with dozen lines of code and is flexible enough to function with any GNN backbone, on a wide variety of large-scale datasets, and in both transductive and inductive tings. Without modifying model's architecture or training setup, FLAG yields a consistent and salient performance boost across both node and graph classifica- tion tasks. Using FLAG, we reach wate-of-the--e art performance on the large-scale ogbg-molpcba. ogbg-ppa, and ogbg- datasets.
INTRODUCTION
Graph Neural Networks (GNNs) have emerged as powerful architectures for learning and analyzing graph representations. The Graph Convolutional Network (GCN) (Kipf & Welling 2016) and its variants have been applied wide range of tasks, including visual recognition (Zhao al. 2019
Shen et al.] 2018), meta-learning (Garcia & Brunal 2017). social analysis (Qiu et al. 2018 Li & Goldwasser 2019), and recommender systems (Ying et al. 2018). However, the training of GNNs on large-scale datasets usually suffers from overfitting, and realistic graph datasets often involve a high volume out-of-distribution test nodes (Hu et al. 2020), posing significant challenges for prediction problems.
One promising solution to combat overfitting in deep neural networks is data augmentation (Krizhevsky al. 2012). which is commouplace in computer vision tasks. Data augmentations apply pabee-pres transformations to images, such as translations and reflections. As a result, data augmentation effectively enlarges the training set while incurring negligible computational overhead. However, it remains an open problem how to effectively generalize the notion of data augmentation to GNNs. Transformations on images rely heavily on image structures, and is chal- lenging to design low-cost transformations that preserve semantic meaning for non-visual tasks like natural language processing (Wei & Zou 2019) and graph learning. Generally speaking, graph data for machine learning comes with graph structure (or edge features) and node features. In the lim- ited cases where data augmentation can be done on graphs, generally focuses exclusively on the graph structure by adding/removing edges (Rong et al. 2019). To date, there is no study on how to manipulate graphs in node feature space for enhanced performance.
In the meantime, adversarial data augmentation, which happens in the input feature space, is known to boost neural network robustness and promote resistance to adversarially chosen inputs (Goodfel- low et al. 2014 Madry et al. 2017). Despite the wide belief that adversarial training harms standard generalization and leads to worse accuracy (Tsipras et al. 2018 Balaji et al. 2019), recently a grow- ing amount of attention has been paid to using adversarial perturbations to augment datasets and ultimately alleviate overfitting. For example. /oolpi et al. (2018) showed adversarial data augmenta- tion is data-dependent regularization that could help generalize out-of-distribution samples, and

---

Under review as a conference paper at ICLR 2021
its effectiveness has been verified in domains including computer vision (Xie et al. 2020). , language
understanding (Zhu et al. 2019 Jiang et al. 2019), and visual question answering GGan et al. 2020) Despite the rich literature about adversarial training of GNNs for security purposes (Zigner et al.
2018 Dai et al., 2018 Bojchevski & Günnemann 2019; Zhang & Zitnik 2020). it remains unclear how to effectively and efficiently improve GNN's clean accuracy using adversarial augmentation.
Present work. We propose FLAG, Free Large-scale Adversarial Augmentation on Graphs, to tackle the overfitting problem. While existing literature focuses on modifying graph structures to augment datasets, FLAG works purely in the node feature space by adding gradient-based adversar- ial perturbations the input node features with graph structures unchanged. FLAG leverages "free" methods (Shafahi et al. 2019) to conduct efficient adversarial training so that it is highly scalable on large-scale datasets. We verify the effectiveness of FLAGG on the Open Graph Benchmark (OGB) (Hu et al. 2020), which is collection of large-scale, realistic, and diverse graph datasets for both node and graph property prediction tasks. We conduct extensive experiments across OGB datasets by applying FLAG to prestigious GNN models, which are GCN, GraphSAGE, GAT, and GIN (Kipl) Welling 2016; Hamilton et al. 2017 Veličković et al.) 2017 Xu et al. 2019) and show that FLAG brings consistent and significant improvements. For example, FLAG lifts the test accuracy of GAT on ogbn-product. by an absolute value of 2.31%. %. DeeperGCN (Li al. (020) is an- other strong baseline that achieves top performance on several OGB benchmarks. FLAG enables DeeperGCN to generalize further and reach new state-of-the-art performance on ogbg-moo pcba and ogwg-ppbaggooooooo FLAG is simple (adding just dozen lines of code), general (can be directly ap- plied to any GNN model), versatile (works in both transductive and inductive settings), and efficient (able to bring salient improvement at tractable even no extra cost). Our main contributions are summarized as follows:
We propose adversarial perturbations as a data augmentation in the input node feature space efficiently boost GNN performance. The resulting FLAG framework is scalable and flexible augmentation scheme for GNN, which is easy to implement and applicable to any GNN architecture for both node and graph classification tasks.
We advance the state-of-te of-the--r- margins.
number of large-scale OGB datasets, often by large
We provide a detailed analysis and deep insights on the effects adversarial augmentation has on GNNs.
2 PRELIMINARIES
Graph Neural Networks (GNNs). We denote graph as G(V.E) with initial node features Ty for E V and edge features euv for (u, v) E E. GNNs are built on graph structures learn representation vectors h, for every node v E V and a vector hg for the entire graph G. The k-th iteration of message passing, or the k-th layer of GNN forward computation is:
h (OMBINE(() (n.*-") AGGREGATE (K) ({ (((A(tt)) (( eno) u E N(w))))) (1)
where h is the embedding of node e at the k-th layer, Cuv is the feature vector of the edge between node and v, (N)) node v's neighbor set, and h x.. COMBINE(-) and AGGREGATE()) are functions parameterized by neural networks. To simplify, we view the holistic message passing pipeline as an end-to-end function fo(-) built on graph G:
H = fo(:
where X is the input node feature matrix. After K rounds of message passing we get the final-layer node matrix (K), To obtain the representation of the entire graph hg, the permutation- invvarannt READOUT()) function pools node features from the final iteration K as:
hg = READOUT (f (K) |uEV) VV]).
((n,))
(3)
Additionally from the spectral convolution point of view, the k-th layer of GCN is:
I+ D AD-
D AD 1 S -
(4)
2

---

Under review as a conference paper at ICLR 2021
H( =o (
(5)
where H (K) is the node feature matrix of the k-th layer with H X, O is the trainable weight matrix of layer k, and is the activation function. D and A denote the diagonal degree matrix and adjacency matrix, respectively. Here, we view S as a normalized adjacency matrix with self-loops added.
A Adversarial training. Standard adversarial training seeks to solve the min- max problem as:
min B( max L (fo +))
0
where 'D is the data distribution, y is the label, || is some norm distance metric, c is the perturbation budget, and the objective function. Madry et al. (2017) showed that this saddle- point optimization problem could be reliably tackled by Stochastic Gradient Descent (SGD) for the outer minimization and Projected Gradient Descent (PGD) for the inner maximization. In practice, the typical approximation of the inner maximization under an -nom constraint is as follows,
8,+1 = 11 I. (6, + a sign &V&L (fo(æ + b))),
where perturbation 8 updated iteratively, and []] <e performs projection onto the e-ball in the norm. For maximum robustness, this iterative updating procedure usually loops M times, which makes PGD computationally expensive. While there are M forward and backward steps within the process, 0 gets updated just once using the final 8M.
PROPOSED METHOD: FLAG
Adversarial training is a form of data augmentation. By hunting for and stamping out small pertur- bations that cause the classifier to fail, one may hope that adversarial training should be beneficial to standard accuracy (Goodfellow et al. 2014 Tsipras et al., 2018 Miyato et al., 2018). With an increasing amount of attention paid leverage adversarial training for better clean performance in varied domains (Xie et al.) 2020 Zhu et al. 2019 Gan et al. 2020). we conduct the first study on how to effectively generalize GNNs using ad versariial data augmentation. Here we introduce FLAG, Free Large-scale Adversarial Augmentation on Graphs, to best exploit the power of adversarial aug- mentation. Note that our method differs from other augmentations for graphs in that happens in the input node feature space.
Augmentation for "free". We leverage the "free" adversarial training method (Shafahi al. 2019) to craft adversarial data augmentations. PGD is a strong but inefficient way to solve the inner max- imization of (6). While computing the gradient for the perturbation ö, free training simultaneously computes the model parameter O's gradient. This "free" parameter gradient is then used to compute the ascent step. The authors proposed train on the same minibatch M times in a row to simulate the inner maximization in (6) while compensating by performing M times fewer epochs of train- ing. The resulting algorithm yields accuracy and robustness competitive with standard adversarial training, but with the same runtime as clean training.
Gradient accumulation. When doing "free" adversarial training, the inner/adversarial loop is usu- ally run A times, each time computing both the gradient for ö, and 0,-1- Rather than updating the model parameters in each loop, Zhang et al. (2019) proposed to accumulate the gradients for 0,-1 during the inner loop and applied them all once during the outer/parameter update. The same idea was used by Zhu et al. (201)), who proposed FreeLB to tackle this optimization issue on language understanding tasks. FreeLB accumulated the gradients VoL of model parameters. The gradient accumulation behavior can be approximated optimizing the objective below:
multiple PGD steps to craft adversaries, and meanwhile
M-1
min (
1
Σ
max (fo (a 8c), 6,
(8)
M
where Z, = Bx (at) n BB (e). The gradient accumulation algorithm largely empowers FLAG to further improve GNN with efficient gradient usage for optimization.
3

---

Under review as a conference paper at ICLR 2021
Algorithm FLAG: Free Large-scale Adversarial Augmentation Graphs
Require: Graph G (V.E); input feature matrix X; learning rate т; ascent steps M; ascent step
size o; training epochs N; forward function on graph fa(+) denoted in 2 L(-) as objective
function. We omit the READOUT()) function in (3) for the inductive scenario here.
1: Initialize 0
2: for epoch .....N. do
3: 6g + U a-),aa)
> initialize from uniform distribution
4: go +-
5:
for t = ....M
...MMo do
6:
g: -- 91-1 + M VoL (Ja( +8.-1;9),9)
0 gradient accumulation
7:
98 4- VaL (Jo X 6,-1;9),1)
8:
ō, f ò + a 96/ |
> perturbation gradient ascent
9:
end for
10: + T. 9M
model parameter gradient descent
11: end for
Unbounded attack. Usually on images, the inner maximization is a constrained optimization prob- lem. The largest perturbation can add is bounded by the hyperparameter e, typically 8/255 under the -norm. This c encourages the visual imperceptibilityy of the perturbations, thus making defenses realistic and practical. However, graph node features or language word embeddings do not have such straightforward semantic meanings, which makes the selection of highly heuristic. In light of the positive effect of large perturbations on generalization (Volpi al. 2018), and also to simplify hyperparameter search, FLAG drops the projection step when performing the inner maxi- mization. Note that, although the perturbation is not bounded by an explicit e, it is still implicitly bounded in the furthest distance that ö can reach, i.c. the step size a times the number of ascending steps M.
Biased perturbation for node classification. Conventional conv nets treat each test sample inde- pendently during inference, whereas this is not the case transductive graph learning scenarios. When classifying one target node, messages from the whole k-hop neighborhood are aggregated and combined into its embedding. impact, i.c. higher smoothness, the final decision of the target node, which also be intuitively reflected by the message passing view of GNNs in (T). To promote more invariance for further- away neighbors when doing node classification, we perturb unlabeled nodes with larger step sizes Ou than or for target nodes. We show the effectiveness of this biased perturbation in the ablation study section.
natural to believe that a further neighbor should have lower
The overall augmentation pipeline is presented Algorithm I Note that when doing transductive node classification, we use diverse step sizes on and o. craft adversarial augmentation for target and unlabeled nodes, respectively. In the following sections, we verify FLAG's effectiveness through extensive experiments. In addition, we provide detailed discussions for deep understanding of the effects of adversarial augmentation.
4 EXPERIMENTS
In this section, we demonstrate FLAG's effectiveness through extensive experiments on the Open Graph Benchmark (OGB), which consists of a wide range of challenging large-scale datasets. Shchur al. (2018); Emrca et al. (2019): suffered from problems such as unrealistic and arbitrary data splits, highly limited data sizes, non- rigorous evaluation metrics, and common neglect of cross-validation, etc. In order to empirically conduct experiments OGB (Hu et al., 2020) datasets, which have tackled those major issues and brought more realis- tic challenges to the graph research community. We refer readers to Hu et al. (2020) for detailed information on the OGB datasets.
al. (2020) showed that traditional graph datasets
study FLAG's s effects in fair and reliable manner,
the newly released
Unless otherwise stated, all of the baseline test statistics come from the official OGB leaderboard website, and we conduct all of our experiments using publicly released implementations without touching the original model architecture training setu. We report mean and std values from
4

---

Under review as a conference paper at ICLR 2021
ogbn-produce Test Acc
ogbn-proteins Test ROC-AUC
ogbn-arxiv
Backbone
Test Acc
GCN
72.51
71.24
+FLAG
7..51
72.0410.20
GraphSAGE
78.70=0.36
77.68 +0020
71.49-40.27
+FLAG
79.36 0.57
76.57=00.75
72.19=0.21
GAT
79.45 0.59
73.65++0.11
+FLAG
81.76 0.44
73.71=00.13
DeeperGCN
80.98 0.20
85.80+0.17
71.92-+0.16
+FLAG
81.93 0.31
85.96+0.22
72.14+00.19
Table Node property prediction test performance on www.o- poonnnrrooucssss ogbpsoootinnppp..s..... and ogbn-arxiv datasets. Blank denotes no statistics on the leaderboard.
-goon-rsdducss Test Acc
Backbone
ogbn-mag mag Test Acc
GAT
79.45++0.0
Backbone R-GCN
+FLAG
$0.74
46.78=0.67
+FLAGi
$1.29.10.33
+FLAG
47.37 0.44
+FLAG
81.76
Table 2: Left: Test performance on ogbn-product, s with GAT as baseline. denotes model trained in N/M epochs; f denotes Os, $ denotes O, 20z. Right: Test performance on the heterogeneous OGB node property prediction dataset ogbn-mag.
ten runs with different random seeds. Following common practice on this benchmark, report the test performance associated with the best validation result. We choose the prestigious GCN, baseline models. In addition, we apply FLAG to the recent
GraphSAGE, GAT, and GIN DeeperGCN model to demonstrate effectiveness. Our implementation always uses M = 3 ascent steps for simplicity. Following Goodfellow et al. (2014); Madry al. (2017), use sign(-) for gradient normalization. We leave exhaustive hyperparameter and normalization search for future research. All training hyperparameters and evaluation results can be found in the Appendix.
We summarize the results of node classification in Table [ On ogbn-productss s, GraphSAGE, GAT, and DeeperGCN all receive promising results with FLAG. We adopt neighbor sampling (Hamilton et al. (017)) the mini-batch algorithm for GraphSAGE and GAT to make the experiments scalable. For DeeperGCN, we follow the original setup by Li et al. (2020) to randomly split the graph into clusters. Notably, FLAG yields a 2.31% test accuracy dataset. Because the graph size of ogbn-prot eins is small, all models are trained in full-batch manner. From Table we see that FLAG further enhances the performance of DeeperGCN but harms that of GCN and Graph- SAGE. Considering the dataset's specialty of not having input node features, discussions on the effect of different node feature constructions later. We also do full-batch training on ogbn-arxii arxiv, where FLAG enables GAT and DeeperGCN reach 73.71% and 72.14% accu- racy. Note that the GAT baseline is from the DGL (Wang et al. (2019) implementation, which differs from vanilla GAT with batch norm and label propagation incorporated. We reveal batch norm's 's in- fluence in the discussion. ogbn-man with node features. We use the neighbor sampling mini-batch algorithm train R-GCN and report its results in the right part of Table 2 Surprisingly, FL.AG can also directly bring nontrivial accuracy improvement without special designs for heterogeneous graphs, which demonstrates its versatility.
Node Property Prediction.
lift for GAT, making GAT competitive on the
provide detailed
is heterogeneous network where only "paper" nodes come
Graph Property Prediction. Table 3 summarizes the test scores of GCN, GIN, and DeeperGCN on all four OGB graph property prediction datasets. "Virtual" means the model is augmented with virtual nodes (Li et al. 2017 Gilmer et al.) 2017: Hu et al. 2020). As adversarial perturbations are crafted by gradient ascent, would be unnatural to perturb discrete input node features. Following firstly project discrete node features into the continuous space and then adversarially augment the hidden embeddings. On
Jin & Zhang )211);; Zhu et al. (2019),
FLAG yields notable improvements, but when GCN has already been hurt by virtual nodes, FLAG appears to
5

---

Under review as a conference paper at ICLR 2021
ogbg-molhiv
lgbg-molpcba Test AP
ogbg-9pa
ogbg-code
Backbone
Test ROC-AUC
Test Acc
Test F1
GCN
76.06:10.97
20.20
68.39-10.34
31.6310.18
+FLAG
1.00
21.16=0.17
68.38-00477
32.09:1.11
GCN- Virtull
1..99:11.19
2..24=0.34
68.57=0.61
32.63:10.13
+FLAG
1.54
24.83=0.35
69.44:0.52
33.16*
GIN
75.58+1.40
22.66=0.25
68.92+1.00
31.63+00.20
+FLAG
76.54
23.95 0.44
69.05-0092
32.41
Virtual
77.07+11.44
27.03-0.2
70.37+1.07
32.04 +0.18
+FLAG
77.48-00.9
28.34- 00388
72.45 1.114
32.96
DeeperGCN
78.58
L.0.38
77.12
+FLAG
79.42
77.52*
Table 3: Graph property test performance ogbg-molhiv, ogbg-molpcbaa ogbg-ppa, and ogbg-codk datasets. * denotes state-of-the-an performance on the OGB leaderboard; t denotes the existence of virtual nodes; blank denotes statistics on the leaderhoard.
ogbn products
ogbn products
Backbone
Test Acc
Backbone
Test Acc
GraphSAGE w/ NS
78.70 0.366
GAT
79.45:0.59
+FLAG
79.36-0.55
GAT+PGD
80.96.10.41
GraphSAGE w/ Cluster
78.97_0.3.
GAT+Free
79.42.10.84
+FLAG
78.60
GAT+FreeLBB
81.78
GraphSAGE w/ SAINT
79.08 0.224
GAT+FLAG
81.76
+FLAG
79.60 0.19
Table 4: Left: Test accuracy on ogbn-produee with GraphSAGE trained with diverse mini- batch algorithms. Right: Test performance probuuts with GAT trained with different adversarial augmentations.
exaggerate the harm. Note that the test results on ogbg-mo i all have relatively high variance compared with others, where randomness in the test result is more severe. On ogbg-mo pcba, with FLAGG receives absolute value 1.31% test AP value increase, and DeeperGCN further enhanced to retain its SOTA performance. On ogbg-ppa. FLAG further generalizes DeeperGCN and registers a new state-of-the-at test accuracy of 77.52%. On ogbg code, FLAG boosts GCN- Virtual to state-of-ther art test F1 score of 33.16 Besides node classification, FLAG's strong effects on graph classification prove its high versatility. In most cases, FLAG works well with virtual node augmentation further enhance graph learning.
ABLATION STUDIES AND DISCUSSIONS
Effects of biased perturbation. From the left part of Table we see that there salient increase of accuracy when using larger perturbation on unlabeled nodes, which verifies the effectiveness of biased perturbations.
Comparison with other adversarial training methods. The right part of Table 4 shows GAT's performance with different adversarial augmentations. For PGD and Free, we compute 8 ascent steps for the imerrmaximization while for FreeLB and FLAG we compute 3 steps. FLAG outperforms all other methods by a large margin.
Compatibility with mini-batch methods. Graph mini-batch algorithms are critical to training GNNs on large-scale datasets. We test how different algorithms will work with adversarial data augmentation with GraphSAGE. the backbone. From the left part of Table 4 we that neighbor sampling (Hamilton et al. 2017) and GraphSAINT (Zeng et al. 2019) can all work with FLÁG to further boost performance, while Cluster (Chiang et al.) 2019) suffers accuracy drop.
Compatibility with batch norm. The left part of 5 that batch norm works to generalize GAT, and FLAG works to push the improvement further. In the computer vision domain, Xie et al.

---

Under review as a conference paper at ICLR 2021
ogbn-arxiv Test Acc
ogbn- Test Acc
Backbone
Backbone
GAT w/o BN
73.2910.11
GAT w/o dropout
75.67=0.27
GAT w/ BN GAT w/ BN +FLAG
73.65
GAT w/ dropout
79.45=10.59
73.71 -0.31
GAT w/ dropout +FLAG
81.76 +..45
Table 5: Left: Test Accuracy on the gubnn-product- dataset.
dataset.
Right: Test Accuracy on the
2020) proposed a new batch norm method that makes adversarial training further generalize large- scale CNN models. As there is growing attention on using batch norm on GNNs, it will also be interesting to see how to synergize adversarial augmentation with batch norm in future architectures.
Compatibility with dropout. Dropout is widely used in GNNs. The right part of Table 5 shows that, when trained without dropout, GAT accuracy drops steeply by large margin. What's more, FLAG can further generalize GNN models together with dropout, similar to the phenomenon of image augmentations.
Towards going "free". FLAG introduces tractable extra training overhead. We empirically show that, when we decrease the total training epochs to make it as fast as the standard GNN training pipeline, FLAG still brings significant performance gains. The left part of Table [ablee]showe shows that FLAG with fewer epochs still generalizes the baseline. Empirically, on a single Nvidia RTX 2080Ti, 100- epoch vanilla GAT takes 88 mins, while FLAGG in Table takes 91 mins. We note that heuristics like early stopping and cyclic learning rates can further accelerate the adversarial training process (Wong) al. 2020), so there are abundant opportunities for further research on adversarial augmentation at lower or even no cost.
Towards going deep. Over-smoothing stops GNNs from going deep. FLAG shows its ability to boost both shallow and deep baselines, e.g. GCN and DeeperGCN. In the left part of Figure !I we show FLAG's effects on generalization when a GNN goes progressively deeper. The experiments are conducted on sobnnaariiv with GraphS AGE as the backhone, where consistent improvement is evident.
What if there's no node feature? One natural question can be raised: what if no input node
features are provided? ogbn pnbon-proteing is a dataset without input node features. Hu et al. (2020)
proposed to average incoming edge features to obtain initial node features, while Li et al. (020
used summation and achieved competitive results. Note that GCN and CraphSAGE baselines in Table Tuse the "mean" node features input and suffer accuracy drop with FLAG; DeeperGCN leverages the "sum" " and gets further improved. Interestingly, when DeeperGCN is trained with "mean" node features, it receives high invariance, so that even large magnitude perturbations will not change its result. The diverse behavior of adversarial augmentation implies the importance of node feature construction method selection.
WHERE DOES THE BOOST COME FROM?
It is now widely believed that model robustness appears to be at odds with clean accuracy. Despite the proliferation of literature in using adversarial data augmentation to promote standard perfor- mance, it is still unsettled where the boost or detriment of adversarial training comes from.
Data distribution is the key. We conjecture that the diverse effects of adversarial training in dif- ferent domains stem from differences in the input data distribution rather than model architectures. To ground our claim, we utilize FLAG to augment MLPs (an architecture where adversarial training has adverse effects in the image domain) on ogbn-arx and successfully boost generalization. FLAG directly improves the test accuracy from 55.50 0.23% to 56.02- + 0.1%%. In general, ad- versarial training hurts the clean accuracy in image classification, but Tsipras et al. (2018) showed that CNNs could benefit from adversarial augmentations on MNIST. This is consistent with our guess that model architecture has little to do with the performance using adversarial augmentation. Like one-hot hot word embeddings for language models, input node features usually come from discrete spaces, e.g., bag-of-words binary features in -poduutt....ss We believe that using discrete

---

Under review as a conference paper at ICLR 2021
72.5
GraphSAGE +FLAG
GCN
72.2
80.0
+FGSM
72.0
77.5
Accuracy (%)
71.8
75.0
71.5
Accuracy
71.2
70.0
71.0
67.5
70.8
65.0
0.0
62.5
6.0
12.0
0.0
0.2
0.3
0.4
Number of layer
Std
Figure 1: Left: Test accuracy on ogbn arxiv.. Right: Test accuracy on the Cora dataset.
continuous input features may lead to different adversarial augmentation behavior. We provide simple example on the Cora (Getoor 2005) dataset to illustrate. We choose the classic FGSM to craft adversarial augmentation and GCN as the backbone. By adding Gaussian noise with std simulate node features drawn from a continuous distribution. The result summarized the right part of Figure [ When 8 the discrete distribution of node features persists. At this moment, GCN with adversarial augmentation outperforms the clean model. With increased noise magnitude the features are continuously distributed with large support and FGSM starts harm the clean accuracy, which validates our conjecture.
7 RELATED WORK
Existing graph regularizers mainly focus on augmenting graph structures by modifying edges (Rong al. 2019 Hamilton al. 2017 Chen et al. 2018). We propose effectively augment graph data using adversarial perturbations. On large-scale image classification tasks, Xie et al. 2020 lever- aged adversarial perturbations, along with new batch
methods, to augment data. Zhhu et al. (2019): Jiang et al. 2019) added adversarial perturbations in the embedding space and generalized language models further in the fine-tuning phase. Gan al. (2020) showed that VQA model aceu- racy was further improved by adversarial augmentation. To clarify. FLAG is intrinsically different from the previous graph adversarial training methods (Feng et al. 2019 Deng al. [209 Jin & Zhang 2019). |Fnn et al. (2019) proposed to reinforce local smoothness to make embeddings within communities similar. three methods assigned pseudo-labels to test nodes during training time and utilized virtual adversarial training (Miyato et al. 2018) to make test node predictions similar to their pseudo-labels. This makes them workable for semi-supervised settings, but not for inductive tasks. Besides the original classification loss term, they all introduced KL loss into the final objec- tive functions, which would at least double the GPU memory usage and make training less efficient and less scalable. In contrast, FLAG requires minimal extra space overhead and can directly work in the original training setup.
8 CONCLUSION
We propose FLAG (Free Large-scale Adversarial Augmentation on Graphs), simple, scalable, and general data augmentation method for better GNN generalization. Like widely-used image augmentations, FLAG can be easily incorporated into any GNN training pipeline. FLAG yields consistent improvement over a range of GNN baselines, and reaches state-of-the-art performance on the large-scale experiments, also provide conceptual analysis to validate adversarial augmentation's 'ss different behavior on varied data types. The effects of adversarial augmentation on generalization are still not entirely understood, and we think this is fertile space for future exploration.
opba-ppa, and ogbg-code datasets. Besides extensive
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yogesh Balaji, Tom Goldstein, and Judy Hoffman. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint arXiv: arXiv:1910.08055/ 2019.
Alcksandar Bojchevski and Stephan Günnemann. Adversarial attacks on node embeddings via graph poisoning. In International Conference on Machine Learming, pp. 695-704. PMLR, 2019.
Jie Chen, Tengfei Ma. and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling, arXiv preprint arXiv:1801.10244 2018.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh, Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 257-266, 2019.
Hanjun Dai, Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu, and Le Song. Adversarial attack on graph structured data, arXiv preprint arXiv:1806.0233 2018.
Zhijie Deng, Yinpeng Dong, and Jun Zhu. Batch virtual adversarial training for graph convolutional networks. arXiv preprint arXiv: arXiv:1902.09) 902.09192, 2019.
Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.
Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. arXiv preprint arXiv: 0w1.9992.999999 2019.
Fuli Feng, Xiangnan Hc, Jie Tang, and Tat-Seng Chua. Graph adversarial training: Dynamically regularizing based on graph structure. IEEE Transactions Knowledge and Data Engineering, 2019.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and- and-uagguage representation learning. arXiv preprint arXiv:2006.06 06195, 2020.
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. arXiv:1711.04043. 2017.
arXiv preprint
Lise Getoor. Link-based classification. In Advanced methods for knowledge discovery from complex data, pp. 189-207.800 Springer, 2005.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural 2017.
message passing for quantum chemistry. arXiv preprint
Ian J Goodfellow, Jonathon Shlens. and Christian Szegedy. Explaining and hamnsssing adversarial examples. arXiv preprint arXiv:1412.680 2014.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances neural information processing systems, pp. 1024-1034, 2017.
Weihua Hu, Matthis Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 2020.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. arXiv preprint arXiv: vrXiv:1911.03423 9114971,,337 2019.
Hongwei Jin and Xinhua Zhang. Latent adversarial training of graph convolution networks. In ICML Workshop Learning and Reasoning with Graph-Structured Representations, 2019.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net-
works. arXiv preprint arXiv: 1609.02077, 2016.

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Chang Li and Dan Goldwasser. Encoding social information with graph convolutional networks forpolitical perspective detection news media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2594-2604, 2019.
Guohao Li, Chenxin Xiong. Ali Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gcns, arXiv preprint arXiv:2006.07753 2020.
Junying Li, Deng Cai, and Xiaofei He. Learning graph-level represenattonnnn for drug discovery. arXiv preprint arXiv: 117770907771 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0600 06083, 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, ((8):1979-199)) 2018.
Jiezhong Qiu, Jian Tang, Hao Ma, Yuxiao Dong, Kuansan Wang, and Jie Tang. Deepinf: Social influence prediction with deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2110-2119, 2018.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks node classification. In International Conference on Learning Repre- sentations, 2019.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, pp. 3358-3369, 2019.
Olcksandr Shchur, Maximilian Mumme, Alcksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: 00/18/111..00.. 2018
Yantao Shen, Hongsheng Li, Shuai Yi, Dapeng Chen, and Xiaogang Wang. Person re-identification with deep similarity- graph neural network. In Proceedings of the European conference on computer vision (ECCV), pp. 486-504, 2018
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. arXiv preprint arXiv: 12152, 2018
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint /7 10903, 2017.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in neural information processing systems, pp. 2334-5344, 2018.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis. Jinyang Li, and Zheng Zhang- Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv: / 999909939.. 01515, 2019.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint 190iv:1901.11199 2019.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.00 .03994, 2020.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 819-828, 2020.
10

---

Under review as a conference paper at ICLR 2021
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful graph neural networks? In International Conference on Learning Representations, 2019. URL https:
/oopeneeieww.oo
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974 983, 2018.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Raigopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. arXiv preprint arXiv:1907.0000 2019.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. arXiv preprint 2019.
Xiang Zhang and Marinka Zitnik. Gunguard: Defending graph neural networks against adversarial attacks. arXiv preprint arXiv:2006. 049 2020.
Long Zhao, Xi Peng. Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas. Semantic graph con- volutional networks for 3d human pose regression. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3425-3435, 2019.
Chen Zhu, Yu Cheng, 7he Gan, Siqi Sun, Thomas Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for language understanding. arXiv preprint arXiv: 1909. / 1764, 2019.
Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks neural networks for graph data. In Proceedings the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2847-2856, 2018.

---

Under review as a conference paper at ICLR 2021
Appendix
A FLAG PYTORCH IMPLEMENTATION
#M ascent steps, alpha ascent step size
2 #X denotes input node features, y denotes labels
def flag (model, X, y, optimizer, criterion, M, alpha) model.train() optimizer. (
pert torch. Float.TToot (+X. shape) uniform_(aalphae alpha)
pert requieee_graaeeeeeeeeeeeeereeeee (1
out model (X-pert)
loss = criterion (out, y) /M
for in range (M-1):
13
loss bbckkardd
pert_data = pert detach + alphaxtorch. sign (pert. detach ())
15
pert.data = pert_data.data data
pert.grad
out model (Xtpert)
loss criterion (out, y) /M
loss backward
optimizer.step ()
12

---

Under review as a conference paper at ICLR 2021
B FULL STATISTICS
Here we summarize our main experiment results on both node and graph classification tasks. Hy- perparameters for crafting adversarial augmentations are listed in the table. For other training setups of backbones, we refer readers to the public website of the OGB leaderboard.
Table 6: ogbn-products
Backbone
Test Ace
Val Acc
or
0n/04 04
M
GraphSAGE
78.70
91.70
+FLAG
79.36
92.05=0007
8e-03
2
GAT
0..45
+FLAG
81.76
92.51
5e-03
2
DeeperGCN
8..20
92.38+0.00
+FLAG
81.93 0.31
92.21 +0.37
5e-03
2
3
Table 7: ogbn-proteins
Backbone
Test ROC-AUC
Val ROC-AUC
aq
au/
M
GCN
72.51 1.0.55
79.21
+FLAG
71.71
78.93=0.16
1e-03
3
GraphSAGE
77.68 10.20
83.34
+FLAG
76.57-10.75
82.84-0.17
1e-03
3
DecperGCN
85.80
71.92=0.16
+FLAG
85.96
91.32-0.22
8e-03
1
3
Table 8: ogbn-arxiv
Backbone
Test Ace
Val Ace
ar
0n/04
M
MLP
55.50=0.23
57.65=0.17
+FLAG
56.02 10119
58.17
2e-03
1
3
GCN
71.74
73.00
+FLAG
72.04
73.30=00.10
1c-03
1
3
GraphSAGE
71.49
72.77+0.116
+FLAG
72.19:
73.49
le-03
1
GAT
73.65
75.04
+FLAG
73.71 0..11
74.96+0.10
1e-03
DeeperGCN
1..12
72.62
+FLAG
72.14 -.11
73.11 +0.09
8e-03
1
13

---

Under review as a conference paper at ICLR 2021
Table 9: ogbg-molhiv
Backbone
Test ROC-AUC
Val ROC-AUC
a
M
GCN
76.06=0.97
12.04
+FLAG
76.83=1.02
81.76++0.87
le-02
3
GCN- Virtual
75,99
83.84
+FLAG
75.45
8318
le-03
3
GIN
75.58+
82.32+00.00
+FLAG
76.54 +.14
82.25
5e-03
3
GIN- Virtual
77.07 1..49
84.79:00.68
+FLAG
77.48 10.96
84.38
le-03
3
DeeperGCN
78.58 1..17
84.2710.63
+FLAG
79.42
84.25
1e-02
3
Table 10: mobg-molpcba
Backbone
Test AP
Val AP
a
M
GCN
20.20+10.24
20.39-10.33
+FLAG
21.16
21.50:10.22
8e-03
3
24.24
24.95 #0.42
+FLAG
24.83++00 0.337
25.56++0.40
8e-03
3
GIN
22.66
23.05
+FLAG
23.95+-0.4
24.51+00.42
8e-03
3
GIN- Virtual
27.03--0.23
27.98+0.22
+FLAG
28.34 1.038
29.12-t0.26
8e-03
3
DeeperGCN
27.81:10.33
29.20
+FLAG
28.42* -0043
29.52.1..2
8e-03
3
Table 11: ogbg-ppa
Backbone
Test Acc
Val Acc
M
GCN
68.39
64.97
+FLAG
68.38
64.98:10.45
2e-03
GCN-Virtual
68.57=0.61
65.11
+FLAG
69.44:0.52
66.38
5e-03
GIN
68.92++1.00
65.62++1.07
+FLAG
69.05
64.65-+0.70
8e-03
GiN-Virtual
70.37+1.07
66.78+1.05
+FLAG
72.45 1..15
67 4.089
5e-03
DeeperGCN
77.12
73.13
+FLAG
77.52* 1-0.9
74.84 L0.52
8e-03
Table 12: ogbg-code
Backbone
Test Fl
Val Fl
M
GCN
31.63
29.73.10.14
+FLAG
32.09-0.11
30.16:10.16
8e-03
Virttui
32.63
30.62.10.07
+FLAG
33.16* 1..25
30.99:10116
8c-03
GIN
31.63++0.20
29.81=0.14
+FLAG
32.41:10.40
30.44:10.39
8e-03
GIN-Virtual
32.04 +0.18
30.20
+FLAG
32.96
30.92++0.35
8e-03
14


Reviews:
[{"review_id": "T9ubYQwEiKE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work applies FreeLB adversarial training [1] to graph neural networks.\n\nStrengths:\nThere are various best practices which are commonly applied in common task framework competitions\nsuch as data augmentation, adversarial training, and ensembles that improve results by fractions \nor a small number of percentage points. This work falls under such methods, and proposes to \nperform FreeLB adversarial training by computing the loss with respect to perturbations.\nThe method is general and may be applied to any GNN for a very small improvement.\n\nWeaknesses:\nThe algorithm (1)[1] is not novel and its application to GNNs is an incremental contribution.\nThe method improves results by a very small percentage even for cases with low baseline accuracy.\nAdversarial training with small perturbations may be considered as a best practice rather than a novel contribution.\nDeeperGCN+FLAG and GAT+FLAG are currently ranked 5th and 6th on the OGB leaderboard for node property prediction.\nDeeperGCN+FLAG is currently ranked 2nd on the OGB leaderboard for graph property prediction.\n\n[1] FreeLB: Enhanced Adversarial Training for Natural Language Understanding, Zhu et al, 2019.\n\nDue to the many relative improvements of FLAG,\nrather than absolute rankings, on the OGB leaderboards, \nI think that the overall contribution is above the acceptance threshold.\n\n1. Node property prediction\na. ogbn-products: +2: DeeperGCN+FLAG, +5: GAT+FLAG, +6: GraphSAGE+FLAG\t\nb. ogbn-proteins: +1: DeeperGCN+FLAG\t\nc. ogbn-arxiv: +1: GAT+FLAG, +8: GraphSAGE+FLAG, +5: DeeperGCN+FLAG, +4: GCN+FLAG, +1: MLP+FLAG\t\nd. ogbn-mag: +1 R-GCN+FLAG\t\n\n2. Graph property prediction\na. ogbg-molhiv: +3: DeeperGCN+FLAG, +1: GIN+virtual node+FLAG, +2: GCN+FLAG, +3: GIN+FLAG\t\nb. ogbg-molpcba: +2: DeeperGCN+VN+FLAG, +2: GIN+virtual node+FLAG, +1: GCN+VN+FLAG, +1: GIN+FLAG, +2: GCN+FLAG\t\nc. ogbg-ppa: +1: DeeperGCN+FLAG, +1: GIN+virtual node+FLAG, +3: GCN+virtual node+FLAG, +1: GIN+FLAG\t\nd. ogbg-code: +2: GCN+virtual node+FLAG, +4: GIN+virtual node+FLAG, +4: GIN+FLAG, +2: GCN+FLAG", "extra_scores": {}, "extra_text": {}}, {"review_id": "CJsOFXTopZ1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates adversarial feature augmentation for improving the generalizability of graph neural networks. The authors adopt an existing augmentation algorithm and apply on the nodes of each training graph, and use the perturbed graphs for training. The focus of the paper is extensive experimentation in various tasks and settings to illustrate the effectiveness of adversarial augmentation in graph-based tasks. The experiments provide new, non-trivial insights, such as the effect of the number of network layers on the effectiveness of augmentation. The paper is well-written and easy to read.\n\nNevertheless, there are a few drawbacks:\n\n1. Even though the method is completely adopted from prior work and there are no novelties, the authors claim they propose a new solution for graph data augmentation. It is true that adversarial feature augmentation has not been studied for graph neural nets, but it is a straightforward idea to apply an existing feature augmentation method on graph nodes, which are represented using feature vectors just like other types of data. \n\n2. Even though the performance improvement is consistent across tasks, it is not significant. Hence, my takeaway from the paper (which is in fact a valuable takeaway) is that adversarial augmentation is not considerably effective for graph neural nets, no matter what dataset and network architecture is used. A truly \"free\" augmentation is even less effective, as shown in table 2. To be clear, this is not a weakness of this paper. However, the authors are encouraged to acknowledge it and use a more neutral and objective language when describing the performance.\n\n3. Some conclusions are not reliable enough. For instance, the impact of biased perturbation is statistically insignificant in table 2, and those limited numbers are not sufficient to make such a deduction. Moreover, section 6 is not convincing. Just because augmentation hurts the performance of MLP on images but improves on graphs doesn't mean data distribution is the primary determinant of the efficacy of augmentation. It is similarly insufficient that augmentation improves the results on a discrete graph but doesn't improve when noise is added. Moreover, the authors do not elaborate \"how\" the data distribution affects the augmentation. Just proving that data distribution has an effect is a trivial and unhelpful fact. Furthermore, the authors do not support the claim that model architecture does not affect the efficacy of augmentation. In fact, they prove otherwise by figure 1 (left), where the network depth has a direct effect on the performance gap.\n\n4. Table 5 is confusing, as it does not demonstrate any relevant information. The two bottom rows of both tables are a repetition of results from table 1, and only the first row is new, which only shows that GAT performs worse without BatchNorm or Dropout. The table does not show how FLAG performs without BatchNorm and Dropout, and hence there is no new takeaway from this table that is relevant to the purpose of this paper.\n\nIn sum, although the paper provides new and valuable findings, the experiment analysis and conclusions that are made are not strong enough for a purely empirical study. Hence, I recommend improving and resubmitting the paper, either by adding methodological novelty, or by bolstering the analysis sections to make more reliable and useful conclusions. I also encourage the authors to clarify any part that I may have misunderstood, as I am keen to adjust my rating accordingly.\n\n######## Post-Rebuttal Updates:\n\nI appreciate the authors' response, but my main concerns were not addressed. Particularly, I still believe the novelty of this method is extremely limited, as it directly applies an existing method on graph node embeddings. The main novelties are biased perturbation and unbounded attack, which are both very simple modifications, and were not adequately studied in experiments. Although the authors show extensive comparison on various datasets, their comparison does not particularly show the contribution of biased perturbation and unbounded attack. They only show the effect of biased perturbation using a single number in Table 2, which is not convincing. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AnBRf2GFUqv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presented an adversarial augmentation technique for graph neural networks. Particularly, it proposed to inject perturbations to the embedding space of the feature space with the \"free\" strategy. The results on three datasets show the effect of the proposed method. \n\nThis is the first known work on applying this technique to graph NN. I really like the analysis of why this method works on graph data, which shows that the data distribution shift is important. The ablation studies are also strong, especially on the bias of perturbations, and going deep, etc. For Figure 1, could you provide more analysis on the distribution shift besides the accuracy? \n\nThere are a few concerns about this paper. Similar techniques have been applied to other domains, such as NLP and V+L [1,2]. The \"free\" strategy is also not new [3]. it is better for the author to provide more insights on how these approaches used in FLAG.\n\n1. FreeLB: Enhanced Adversarial Training for Language Understanding, ICLR 2020\n2. Large-Scale Adversarial Training for Vision-and-Language Representation Learning, Neurips 2020\n3. Adversarial Training for Free, Neurips 2019", "extra_scores": {}, "extra_text": {}}, {"review_id": "9690QDAknnr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---- Summary\n\nThis paper proposes FLAG (Free Large-scale Adversarial Augmentation on Graphs), an adversarial data augmentation technique that can be applied to different GNN models in order to improve their generalization. The proposed technique consists on adding adversarial perturbations to the nodes’ features solving the standard min-max problem for adversarial training. In this setup, a noise vector is added to the input features which tries to maximize the loss by performing gradient ascent, while the classifier is trained to minimize the loss despite the added adversarial noise.\nThe authors show that their method can easily be added to standard GNN models like GCN, GAT, GraphSAGE and DeeperGCN with minimal changes, and that it improves these models on different tasks.\n\n---- Pros\n\n* The authors evaluate their method on the Open Graph Benchmark (OGB), which is a standard benchmark that facilitates comparison with other methods.\n* FLAG is a generic and model agnostic method that can be applied to already established models like GCN, GAT and GraphSAGE.\n* Clear explanation of the proposed method with a concise pytorch implementation as an example.\n* Good empirical results with different models on different OGB tasks.\n\n---- Cons\n\n* My main concern with this paper is that the proposed augmentation technique is not compared against any other graph augmentation techniques. What is the justification for using this one over other techniques? Can FLAG work with other data augmentation techniques? Showing experiments of FLAG in combination or compared to other techniques would strengthen the experiments section.\n\n---- Questions to the authors\n\n* As I said in my concerns, if this kind of feature data augmentation is orthogonal to structure-based data augmentation that adds or removes edges (or nodes), it would be very interesting to do an experiment showing to which degree it can be used together with other graph augmentation techniques.\n*The motivation for the biased perturbation explained in the “Biased perturbation for node classification” paragraph in Section 3 is not very clear. The authors claim that nodes that are far away from a target (labelled) node will have lower effect on the prediction for that node, thus they increase the adversarial step size $\\alpha_u$ for the unlabelled nodes. That will increase the amount of noise added to the unlabelled nodes, but why is that beneficial? Also, what is the relation between unlabelled nodes and their distance to the target nodes? Why does it have anything to do with promoting invariance for further-away nodes? \n\n---- Minor comments\n\n- Although the approach is a bit different, the Adversarial AutoAugment paper [1] seems to be quite related, the authors might want to reference it in their paper. Besides the task domain (images vs graphs) the main difference is that instead of adversarially learning perturbations, in [1] data augmentation policies are adversarially learnt to apply perturbations to the input. \n- Also, there are a couple of contemporary papers on graph data augmentation [2, 3] that the authors could reference too.\n\n---- Justification for score\n\nThe paper is well written and the method interesting and well explained, but it lacks comparisons with other graph data augmentation approaches.\n\n------ Post-Rebuttal Update ------\n\nI have read the author's response, thank you for adding more experiments with other graph data augmentation techniques. However, the explanation and justification for the \"biased perturbations\" still seems a bit weak and under explored in the experiments section, so I've decided to keep my initial score.\n\n---- References\n\n[1] Zhang, Xinyu, et al. \"Adversarial autoaugment.\" arXiv preprint arXiv:1912.11188 (2019).\n\n[2] Jiajun Zhou, Jie Shen, and Qi Xuan. 2020. Data Augmentation for Graph Classification. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM '20). Association for Computing Machinery, New York, NY, USA, 2341–2344. DOI:https://doi.org/10.1145/3340531.3412086\n\n[3] Zhao, T., Liu, Y., Neves, L., Woodford, O., Jiang, M., & Shah, N. (2020). Data Augmentation for Graph Neural Networks. arXiv preprint arXiv:2006.06830.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of adversarial training for graph neural networks. The proposed method is build on the free training approach, and more specifically FreeLB, with some additional tricks including bias perturbation (for node-classification) and unbounded attacks.  While these additions are potentially useful, there are only limited investigation into their effect.  Putting aside the technical distinctions of the method with prior work, this paper can also be viewed as an empirical study of adversarial training techniques on graph data with various GNN architectures.  It is worth noting that overall the conclusions on "adversarial training" are positive, we do see consistent improvement over a variety of architectures and tasks.   The issues however, is that it is unclear whether these improvements can be similarly achieved using prior technique like FreeLB (oblation is only done on one single task, where biased perturbation is shown to lead to minor improvement).   The paper also provides some results showing the effect of depth of the network as well as different training strategies such as batch norm, dropout with general adversarial training.  These results are interesting to see but do seem to be limited in both scope and depth.  It appears that the authors have two goals in mind, one is to propose FLAG and demonstrate its usefulness, and the other is to provide a better understanding of how adversarial training works for GNNs in general.  Given the limited novelty of FLAG compared to prior methods, the main contribution actually comes from the later part, which unfortunately is somewhat underdeveloped.  

Decision: Reject