Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper designed and tested WT-AWP, a new adversarial weight perturbation approach, on graph neural networks. They demonstrated that by locating flat local minima, our WT-AWP can improve the regularization of GNNs. They carried out comprehensive tests to verify the method. WT-AWP reliably increases GNN performance on a wide range of graph learning tasks, including node classification, graph defense, and graph classification.
main_review: This paper focuses on extending AWP to GNN. This paper analyzes the vanishing-gradient issue existing in AWP and gives a more detailed theoretical proof. The experimental part of this article is comprehensive, and the experimental results also verify the advantages of the proposed method.

However, there still exist some insufficiency and confusing places.

1, What is the term h in equation(3). The statements said h is a monotonously increasing function. Where is it from?

2, The paper utilized comprehensive experiments to show the efficiency when facing attacks, but no analysis and theoretical proof for this advantage. I think the source of the advantage in handling attacks is worth clarification. Especially, in Table 2, some experimental results under attacks are even better than the natural setting.

3, Figure1 and Figure3 are blurred. I can not distinguish face color and border color. It is recommended to use vector graphics.

4, The sentence on Page 5 footnotes "Perturbing only the second layer instead performs similarly." is confusing for me.

summary_of_the_review: The paper is well-organized and has comprehensive experiments to defense its method. However, some statements and equations are not clear enough. The illustrations and figures are not meticulous. I hope the authors can further polish the paper in detail.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a variant of adversarial weight perturbations / sharpness aware minimization for graph (convolutional) neural networks for node and graph classification. In particular, they make two adjustments: “truncating”, i.e., limiting the weight perturbation to specific layers, and weighting the sharpness aware loss with the regular loss during training.
main_review: Strengths:
- The paper is generally clearly structured and written; the toy example visualizations are nice and the algorithm helps understanding.
- An interesting problem/question is addressed: how to effectively use AWP/sharpness-aware minimization for graph neural networks.
- It is also nice to show that the optimum does not change with AWP training.
- Experiments for clean and robust accuracy with several models are conducted. Besides mean, standard deviation is reported which is nice to judge the improvement.
- The approach improves across the board, even though improvement is small at times.
- Analysis in terms of flatness (gradients and visualization) and ablation studies are provided.

Weaknesses:
- In the introduction, the authors state the i.i.d assumption as an important criteria for existing flatness approaches. However, it is not discussed when this becomes relevant. As far as I am aware, the i.i.id assumption does not play an important role in related discussion/papers. Could the authors comment on that? As I understand the paper, the main point is to see how these approaches work in non i.i.d settings such as node classification, but the paper does not really theoretically discuss this issue.
- One paper that could be included in related work as it is quite related:

[a] https://arxiv.org/abs/1609.04836

- Also a discussion of scale-invariance and the criticism of [b] is missing (this should be handled as in Stutz et al., but it would be interesting at least to say why this generalizes to the GCNs considered)

[b] https://arxiv.org/pdf/1703.04933.pdf 

- Notation-wise, the main sections could be improved by making explicit that a per-layer neighborhood B is used from the beginning. Currently, this is made clear in Eq. (4) while Eq. (1) to (3) suggest that all parameters are appended.
- Also, are the GCN baselines equipped with biases in addition to the weights? If so, are biases and weights treated as separate layers (as in Stutz et al.)?
- In terms of contributions, the theoretical contributions (mainly Thm 1) and methodological contributions seem a bit limited. Although I have not seen Thm 1 in other papers (which makes it refreshing to actually see it), I found that the result is very intuitive and the proof is also quite straight-forward. The techniques employed to improve AWP also seem very specific to graph problems where larger \rho are used than for vision problems. Thus, I see the contributions mostly in verifying this approach on graph data.
- Regarding the vanishing gradient problem: I am having difficulties understanding why \rho has to be chosen as large as done throughout the toy example and the experiments. Stutz et al. And Wu et al. Consider very small \rho of 0.5% (=0.005 or lower). Obviously these are much deeper networks and not graph neural networks. I am wondering if the authors could give more details on why a large \rho is needed. Is it because of the 2-layer structure or the architecture differences?
- The two proposed approaches do actually not improve performance on the toy example. While I understand that it is meant for illustration purposes, I believe it is badly chosen. I see that without truncation and weighting the accuracy is very bad, but shouldn’t you show that you can improve over the baseline of 98% and not be stuck at 95%? Did you try optimizing hyper-parameters, or is it a problem where TW-AWP just does not help (which would be interesting!)?
- Regarding 5.2, I am not entirely convinced that the gradient norm is the best indicator of flatness exactly because of the scale-invariance argumentation of Stutz et al.: Can the authors comment on that? I guess that the used GCNs do not use batch normalization (I have not seen BN used for graph neural networks before), but scale-invariance is a problem as described in [b]. Also, Fig. 4 (c) and (d) has some outliers that are not really explained.
- Table 2 is hard to read and very small.
- Some ablation that I find missing: ablation regarding layers. Which layer to skip? Obviously, the networks are two-layer networks, but I would find it very interesting whether it is always the last layer to skip (also in deeper networks) or always the first layer to perturb (also in deeper networks).
summary_of_the_review: I appreciate this paper in terms of applying AWP to graph neural networks and showing how it needs to be adapted to work well. Methodological contributions are small, however, and improvements vary across datasets and cases.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors extend the line of work related to adversarial weight perturbations: first showing that a vanishing gradient issue in standard AWP can hinder training. To remedy this, some natural tweaks are applied to AWP. The authors then focus on using AWP to train graph neural networks, demonstrating a minor boost in both clean and robust accuracy. 

main_review: **Strengths:**
The line of work concerning adversarial weight perturbations is interesting and significant, as it is one of a few locations where the nascent theory of deep networks can provide easy tweaks to training that improve generalization. Identifying a problem with the current approach (even on MLPs) is valuable and significant, and ample evidence is presented to suggest the vanishing gradient problem is actually what is occurring. The proposed solutions are simple and easy to implement. The experimentation is thorough: while the improvements provided by AWP are very minor (less than a percent in some instances), it is yet another easy and cheap trick to ever-so-slightly boost the performance of a network. 

**Weaknesses:**
First, while it is an important insight to notice the vanishing gradient, the proposed remedies (weight truncation and weighted AWP) are natural and not particularly novel. These are likely the first things that one would try to mitigate the observed problems with AWP and don't represent a great stride forward in the field. Similarly, it is not particularly surprising that the benefits of AWP as evidenced in MLPs carry over to GNNs: not enough motivation or evidence is presented to make this seem surprising or unexpected. Third, Theorem 1 is trivial. If the definition of Ltrain is to be the one that only performs a single first-order step, then of course the gradient evaluated at a minimum is going to be zero: the interesting question here is how the true AWP loss (eq2) relates to the standard training loss. This theorem doesn't add add anything to the story. 

**Questions:**
- One of my complaints is about the novelty of contribution re: GNNs vs MLPs. Did I miss this, or is there simply a much more pronounced effect of the gradient-vanishing phenomenon in GNNS than MLPs? 
- How much do the gradient norms (with respect to the weights) actually change when training under AWP vs WT-AWP vs standard training?
summary_of_the_review: This is an interesting line of work, and the pointing out of (and subsequent fixing of) the gradient-vanishing phenomenon in AWP is a valuable contribution. Past this, none of the results are strikingly novel or groundbreaking. I think this is a borderline paper, tending towards rejection, but could be convinced to boost my score slightly if I've misunderstood something.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work studies how AWP improves the GNN, mainly at the generalization aspect. The authors first derive several theoretical results that can be easily derived from the existing works on CNN, e.g., Wu et al., 2020a and Foret et a;., 2021 in section 3.. In section 4, the authors show that if they directly apply large $\rho$ as suggested by (3), they will encounter the gradient vanishing phenomenon during the training. Based on this phenomenon, the authors propose two ways to solve this. One is truncated AWP, the other is weighted AWP. Then, the authors conduct simple experiments using Vanilla GCN, GCN+AWP, GCN+TAWP, GCN+WAWP, where Vanilla perform the best whereas GCN+TAWP, GCN+WAWP have smoother decision boundary. And they decide to combine T+W with AWP (to be frank, I do not know why they do not add experiment result of WT-AWP here.) In section 5, the authors conduct a lot of experiments like clean accuracy, robust accuracy, etc, which follow the existing works.

The contribution as I can see is that the authors are the first ones to conduct extensive experiments using AWP on graph dataset. However, I do not see other contributions listed by the authors, which I will explain later in the main review.
main_review: Strengths: I am satisfied with the numerical experiments in section 5, although they largely follows the routes of existing work. 

Weaknesses: I will point the weaknesses by sections.

Section 3: i) The theoretical results are very incremental based on existing works e.g., Wu et al., 2020a, which can be easily derived or directly used. ii) the AWP algorithm listed here is the same as Wu et al., 2020a (only add a letter A).

Section 4: i) After I read the paper thoroughly twice, I do not see how you assign $\theta^{(awp)}$ and $\theta^{(normal)}$, which is very important. If you have that in your paper, first let me know where I can find it, second move that to section 4 and explain it. It is unacceptable not having it in section 4. ii) Do not have theory or even intuition about T-AWP and W-AWP. In my perspective, how you justify your algorithms is very important, which is much important than the so called "theoretical" results in Section 3. You can even save the space in section 3 for you to defend your algorithms in section 4. I believe what I point out here should be your novelty for this paper. At least, please, add some good intuitions for your algorithms (I will add points if you do that.) iii) Please add the WT-AWG result in Figure 3.

Summary: Good: experiments; Weak: i) too incremental and lack of novelty; ii) do not provides theoretical/intuitive explanation of the proposed algorithms.






summary_of_the_review: This is work i) is quite incremental compared to the existing works, ii) does not explain/justify/describe the algorithms well. I do not think this paper is good enough for ICLR.
review_title: null

Meta-review: This paper considers a variant of adversarial weight perturbations / sharpness aware minimization for graph (convolutional) neural networks for node and graph classification. In particular, they make two adjustments: “truncating”, i.e., limiting the weight perturbation to specific layers, and weighting the sharpness aware loss with the regular loss during training. The reviewers found that the theoretical justifications (characterization of vanishing gradient and understanding of non-iid setting which was added during rebuttal) are interesting, but several reviewers also found the solution/empirical results not convincing enough. I recommend the authors to either shift the focus to the theoretical results or to strengthen the empirical results (and their connections with theory) following the comments of the reviewers.
Decision: Reject