Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work applies FreeLB adversarial training [1] to graph neural networks.

Strengths:
There are various best practices which are commonly applied in common task framework competitions
such as data augmentation, adversarial training, and ensembles that improve results by fractions 
or a small number of percentage points. This work falls under such methods, and proposes to 
perform FreeLB adversarial training by computing the loss with respect to perturbations.
The method is general and may be applied to any GNN for a very small improvement.

Weaknesses:
The algorithm (1)[1] is not novel and its application to GNNs is an incremental contribution.
The method improves results by a very small percentage even for cases with low baseline accuracy.
Adversarial training with small perturbations may be considered as a best practice rather than a novel contribution.
DeeperGCN+FLAG and GAT+FLAG are currently ranked 5th and 6th on the OGB leaderboard for node property prediction.
DeeperGCN+FLAG is currently ranked 2nd on the OGB leaderboard for graph property prediction.

[1] FreeLB: Enhanced Adversarial Training for Natural Language Understanding, Zhu et al, 2019.

Due to the many relative improvements of FLAG,
rather than absolute rankings, on the OGB leaderboards, 
I think that the overall contribution is above the acceptance threshold.

1. Node property prediction
a. ogbn-products: +2: DeeperGCN+FLAG, +5: GAT+FLAG, +6: GraphSAGE+FLAG	
b. ogbn-proteins: +1: DeeperGCN+FLAG	
c. ogbn-arxiv: +1: GAT+FLAG, +8: GraphSAGE+FLAG, +5: DeeperGCN+FLAG, +4: GCN+FLAG, +1: MLP+FLAG	
d. ogbn-mag: +1 R-GCN+FLAG	

2. Graph property prediction
a. ogbg-molhiv: +3: DeeperGCN+FLAG, +1: GIN+virtual node+FLAG, +2: GCN+FLAG, +3: GIN+FLAG	
b. ogbg-molpcba: +2: DeeperGCN+VN+FLAG, +2: GIN+virtual node+FLAG, +1: GCN+VN+FLAG, +1: GIN+FLAG, +2: GCN+FLAG	
c. ogbg-ppa: +1: DeeperGCN+FLAG, +1: GIN+virtual node+FLAG, +3: GCN+virtual node+FLAG, +1: GIN+FLAG	
d. ogbg-code: +2: GCN+virtual node+FLAG, +4: GIN+virtual node+FLAG, +4: GIN+FLAG, +2: GCN+FLAG
review_title: Applying FreeLB adversarial training to GNNs

final_score: rating: 7: Good paper, accept
review_content: This paper presented an adversarial augmentation technique for graph neural networks. Particularly, it proposed to inject perturbations to the embedding space of the feature space with the "free" strategy. The results on three datasets show the effect of the proposed method. 

This is the first known work on applying this technique to graph NN. I really like the analysis of why this method works on graph data, which shows that the data distribution shift is important. The ablation studies are also strong, especially on the bias of perturbations, and going deep, etc. For Figure 1, could you provide more analysis on the distribution shift besides the accuracy? 

There are a few concerns about this paper. Similar techniques have been applied to other domains, such as NLP and V+L [1,2]. The "free" strategy is also not new [3]. it is better for the author to provide more insights on how these approaches used in FLAG.

1. FreeLB: Enhanced Adversarial Training for Language Understanding, ICLR 2020
2. Large-Scale Adversarial Training for Vision-and-Language Representation Learning, Neurips 2020
3. Adversarial Training for Free, Neurips 2019
review_title: Official Review of Paper142

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper investigates adversarial feature augmentation for improving the generalizability of graph neural networks. The authors adopt an existing augmentation algorithm and apply on the nodes of each training graph, and use the perturbed graphs for training. The focus of the paper is extensive experimentation in various tasks and settings to illustrate the effectiveness of adversarial augmentation in graph-based tasks. The experiments provide new, non-trivial insights, such as the effect of the number of network layers on the effectiveness of augmentation. The paper is well-written and easy to read.

Nevertheless, there are a few drawbacks:

1. Even though the method is completely adopted from prior work and there are no novelties, the authors claim they propose a new solution for graph data augmentation. It is true that adversarial feature augmentation has not been studied for graph neural nets, but it is a straightforward idea to apply an existing feature augmentation method on graph nodes, which are represented using feature vectors just like other types of data. 

2. Even though the performance improvement is consistent across tasks, it is not significant. Hence, my takeaway from the paper (which is in fact a valuable takeaway) is that adversarial augmentation is not considerably effective for graph neural nets, no matter what dataset and network architecture is used. A truly "free" augmentation is even less effective, as shown in table 2. To be clear, this is not a weakness of this paper. However, the authors are encouraged to acknowledge it and use a more neutral and objective language when describing the performance.

3. Some conclusions are not reliable enough. For instance, the impact of biased perturbation is statistically insignificant in table 2, and those limited numbers are not sufficient to make such a deduction. Moreover, section 6 is not convincing. Just because augmentation hurts the performance of MLP on images but improves on graphs doesn't mean data distribution is the primary determinant of the efficacy of augmentation. It is similarly insufficient that augmentation improves the results on a discrete graph but doesn't improve when noise is added. Moreover, the authors do not elaborate "how" the data distribution affects the augmentation. Just proving that data distribution has an effect is a trivial and unhelpful fact. Furthermore, the authors do not support the claim that model architecture does not affect the efficacy of augmentation. In fact, they prove otherwise by figure 1 (left), where the network depth has a direct effect on the performance gap.

4. Table 5 is confusing, as it does not demonstrate any relevant information. The two bottom rows of both tables are a repetition of results from table 1, and only the first row is new, which only shows that GAT performs worse without BatchNorm or Dropout. The table does not show how FLAG performs without BatchNorm and Dropout, and hence there is no new takeaway from this table that is relevant to the purpose of this paper.

In sum, although the paper provides new and valuable findings, the experiment analysis and conclusions that are made are not strong enough for a purely empirical study. Hence, I recommend improving and resubmitting the paper, either by adding methodological novelty, or by bolstering the analysis sections to make more reliable and useful conclusions. I also encourage the authors to clarify any part that I may have misunderstood, as I am keen to adjust my rating accordingly.

######## Post-Rebuttal Updates:

I appreciate the authors' response, but my main concerns were not addressed. Particularly, I still believe the novelty of this method is extremely limited, as it directly applies an existing method on graph node embeddings. The main novelties are biased perturbation and unbounded attack, which are both very simple modifications, and were not adequately studied in experiments. Although the authors show extensive comparison on various datasets, their comparison does not particularly show the contribution of biased perturbation and unbounded attack. They only show the effect of biased perturbation using a single number in Table 2, which is not convincing. 



review_title: A good empirical study with several new insights.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ---- Summary

This paper proposes FLAG (Free Large-scale Adversarial Augmentation on Graphs), an adversarial data augmentation technique that can be applied to different GNN models in order to improve their generalization. The proposed technique consists on adding adversarial perturbations to the nodes’ features solving the standard min-max problem for adversarial training. In this setup, a noise vector is added to the input features which tries to maximize the loss by performing gradient ascent, while the classifier is trained to minimize the loss despite the added adversarial noise.
The authors show that their method can easily be added to standard GNN models like GCN, GAT, GraphSAGE and DeeperGCN with minimal changes, and that it improves these models on different tasks.

---- Pros

* The authors evaluate their method on the Open Graph Benchmark (OGB), which is a standard benchmark that facilitates comparison with other methods.
* FLAG is a generic and model agnostic method that can be applied to already established models like GCN, GAT and GraphSAGE.
* Clear explanation of the proposed method with a concise pytorch implementation as an example.
* Good empirical results with different models on different OGB tasks.

---- Cons

* My main concern with this paper is that the proposed augmentation technique is not compared against any other graph augmentation techniques. What is the justification for using this one over other techniques? Can FLAG work with other data augmentation techniques? Showing experiments of FLAG in combination or compared to other techniques would strengthen the experiments section.

---- Questions to the authors

* As I said in my concerns, if this kind of feature data augmentation is orthogonal to structure-based data augmentation that adds or removes edges (or nodes), it would be very interesting to do an experiment showing to which degree it can be used together with other graph augmentation techniques.
*The motivation for the biased perturbation explained in the “Biased perturbation for node classification” paragraph in Section 3 is not very clear. The authors claim that nodes that are far away from a target (labelled) node will have lower effect on the prediction for that node, thus they increase the adversarial step size $\alpha_u$ for the unlabelled nodes. That will increase the amount of noise added to the unlabelled nodes, but why is that beneficial? Also, what is the relation between unlabelled nodes and their distance to the target nodes? Why does it have anything to do with promoting invariance for further-away nodes? 

---- Minor comments

- Although the approach is a bit different, the Adversarial AutoAugment paper [1] seems to be quite related, the authors might want to reference it in their paper. Besides the task domain (images vs graphs) the main difference is that instead of adversarially learning perturbations, in [1] data augmentation policies are adversarially learnt to apply perturbations to the input. 
- Also, there are a couple of contemporary papers on graph data augmentation [2, 3] that the authors could reference too.

---- Justification for score

The paper is well written and the method interesting and well explained, but it lacks comparisons with other graph data augmentation approaches.

------ Post-Rebuttal Update ------

I have read the author's response, thank you for adding more experiments with other graph data augmentation techniques. However, the explanation and justification for the "biased perturbations" still seems a bit weak and under explored in the experiments section, so I've decided to keep my initial score.

---- References

[1] Zhang, Xinyu, et al. "Adversarial autoaugment." arXiv preprint arXiv:1912.11188 (2019).

[2] Jiajun Zhou, Jie Shen, and Qi Xuan. 2020. Data Augmentation for Graph Classification. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM '20). Association for Computing Machinery, New York, NY, USA, 2341–2344. DOI:https://doi.org/10.1145/3340531.3412086

[3] Zhao, T., Liu, Y., Neves, L., Woodford, O., Jiang, M., & Shah, N. (2020). Data Augmentation for Graph Neural Networks. arXiv preprint arXiv:2006.06830.

review_title: Good method and results, but it is not compared with other data augmentation methods

Meta-review: This paper studies the problem of adversarial training for graph neural networks. The proposed method is build on the free training approach, and more specifically FreeLB, with some additional tricks including bias perturbation (for node-classification) and unbounded attacks.  While these additions are potentially useful, there are only limited investigation into their effect.  Putting aside the technical distinctions of the method with prior work, this paper can also be viewed as an empirical study of adversarial training techniques on graph data with various GNN architectures.  It is worth noting that overall the conclusions on "adversarial training" are positive, we do see consistent improvement over a variety of architectures and tasks.   The issues however, is that it is unclear whether these improvements can be similarly achieved using prior technique like FreeLB (oblation is only done on one single task, where biased perturbation is shown to lead to minor improvement).   The paper also provides some results showing the effect of depth of the network as well as different training strategies such as batch norm, dropout with general adversarial training.  These results are interesting to see but do seem to be limited in both scope and depth.  It appears that the authors have two goals in mind, one is to propose FLAG and demonstrate its usefulness, and the other is to provide a better understanding of how adversarial training works for GNNs in general.  Given the limited novelty of FLAG compared to prior methods, the main contribution actually comes from the later part, which unfortunately is somewhat underdeveloped.  

Decision: Reject