Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper identifies a principal difficulty of training robust GNNs that existing methods ignore: the convoluted mixture distribution between clean and attacked data samples. To tackle the problem, the authors propose GAME, a novel method for learning robust graph representations from three perspectives. Extensive experiments and analyses demonstrate GAME's superiority over state-of-the-art methods.


strength_and_weaknesses: 
Strengths:
+The motivation for activating different experts during adversarial loss by PGD is interesting and efficient in model defense. Additionally, the authors recognize and tackle one fundamental issue (mixture distribution of standard and attacked graphs) in adversarial graph learning, which prevents the traditional GNNs from performing well on both adversarial and clean samples. The motivation of this paper is strong, especially as the number of attack methods increases.

+The developed GAME framework is novel. There are three components (GAME layer, DECOG training, and GRADIV regularization) to address each identified problem (model’s generality, training efficiency, and output diversity) correspondingly. The introduced components are not simply combined, but instead form a complementary chain that serves as the overall framework.

+The authors provide a good catch on why graph adversarial learning is challenging for GNNs, which is important and insightful. I am convinced by these statements, e.g., figure 1 presents a clear distribution difference between standard graphs and attacked graphs. The authors also draw the figure in the experiment section to examine the generality of GAME with respect to the difference, which is interesting.

+Extensive experiments and ablation studies are organized to evaluate the proposed GAME in this paper. For instance, the authors evaluate the model in both the attacking setting with five different attack methods and the standard setting with original graph inputs. The proposed model is further validated on different GRB datasets and shows significant results. The results are impressive. The proposed model exhibits state-of-the-art robust accuracy across a variety of attack evaluations and dataset sizes. The proposed model demonstrates remarkable results and superior performances to other methods, which suggests that GAME can be utilized in a wide variety of real-world applications with respect to both performance and reliability. 


Weaknesses and Questions:
-Some experimental discussions are not clear for those who are unfamiliar with adversarial learning. For example, in the introduction section, “a progressively larger divergence between the two distributions is observed” is stated in Figure 1’s caption. What does this large divergence mean? The authors should provide more information to explain the definition of divergence or make clarification what divergence means in this experiment.

-According to Section 3, the authors state, "In general, DECOG enables the dynamic activation of each expert in GAME and facilitates the computation of more diverse attacked graph adjacency matrix and node features." I'm curious about the GAME model's diversifying effects with the incorporation of MoE. Will the adversarial training with GAME layers contribute more diverse attack samples than standard GNNs? Will the observation in Section 5.5 still be established on a larger dataset?

-Will a bigger number of activated experts produce a more diverse effect? More discussions are required regarding the activated expert number.

clarity,_quality,_novelty_and_reproducibility: The paper is well-organized and the methodology is plainly stated. The proposed model is novel. Code and data sources are available for reproducibility.
summary_of_the_review: By proposing a novel framework comprised of multiple components, this paper addresses three significant difficulties in learning the difference between standard and attacked graphs. The technical contributions are substantial, the motivations are supported by rigorous explanations, and the experimental results are sufficient.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on the adversarial training of robust graph neural networks. The motivation is attractive to me, namely, the divergence of distribution between clean graphs and attacked graphs. This motivation leads to the GAME model, a set of effective all-round designs: leveraging MoE module to construct a more powerful GNN with better capacity, using adversarial training to produce more different attacked samples in the MoE-based GNN, and adding regularization $R_{GraDiv}$ to diversify prediction for more robustness.

strength_and_weaknesses: Strength:

1. The paper has a clear motivation: the authors take a unified investigation for two clean data and attacked in graph adversarial training. Both are shown to shift to different distributions, but their current GNN-based robust methods do not aware. So, the authors become the first to design a GAME model for unification between the two distributions is necessary to gain robustness.

2. The technical approach is solid: The authors first proposed GAME model to achieve diversity and robustness by adversarial training Graph Mixture of Experts, which has never been used in graph adversarial learning. Because MoE generates different adversarial noise by activating different sub-networks to cooperate with PGD, the GAME model can obtain richer adversarial samples in the same training cycle. Considering that MoE makes the number of parameters larger, to avoid redundancy of features, the authors also designed the regularization module to take full advantage of the different outputs of each expert.

3. Experimental results are SOTA: The GAME model yields significantly improved adversarial robustness, outperforming strong previous methods among small, middle, and large graph datasets. Ablation experiments and analytical experiments are convincing: The authors compared MoE, DECOG, and GRADIV regularization's improvements on robust node classification tasks. The results confirm that MoE, DECOG, and GRADIV regularization are three complementary dimensions, and their proper unification strategy is superior to other current alternatives. The authors also use analytical experiments to reveal that mixed distributions are the important reason for the difficulty of graph adversarial training and that the proposed GAME model handles such mixed distributions well.

4. The paper is well-written and easy to follow. The setup section, appendix section, and anonymous codes provide sufficient details for the implementation of this work.


Weakness:

1. I am not fully convinced why GAME model's computational cost can approximate the normal GNN model.  There was no computational complexity analysis or complexity comparisons with baseline models. Please address this concern.

2. Figure 1 illustrates the GNN trained with clean graphs and the GNN trained with attacked graphs have different representation distributions. Then in Figure 4, the authors state that GAME model can unify clean graphs and attacked graphs. Is this figure plotted based on GAME models (e.g., one GAME model trained with clean graphs and one GAME model trained with attacked graphs)? It seems difficult to obtain enough diversity and enough hardness simultaneously. Please provide more exact descriptions to clarify and support your claim.
clarity,_quality,_novelty_and_reproducibility: The proposed method is well-motivated. Also, the writing is clear and good. The authors have provided sufficient information for reproducibility. 

summary_of_the_review: The paper is interesting and well-motivated. The proposed model is novel and the experiments are sufficient. There are some minor concerns that need to be addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for robust graph neural networks. The proposed model contains a modified message-passing layer, PGD-based adversarial training, and an additional loss term encouraging diverse node representations. The authors compare their method on several datasets using the GRB benchmark suite.

strength_and_weaknesses: Strengths:
* Effective adversarial training methods for graph neural networks are rare, so this is an important research area.
* The authors compare to a large number of baselines.
* The authors report strong experimental results.

Weaknesses:
* The authors do not compare to important defense baselines (Soft-Medoid/ Soft-Median [Geisler et al. 2020/2021]) and attacks (PGD attacks [Xu et al. 2018])
* On some datasets there are only n=2 "experts", raising doubts about the usefulness of this contribution.
* The experiment setup is not sufficiently clear (see next section).
clarity,_quality,_novelty_and_reproducibility: **Clarity**: Mostly good, the presentation is clear and easy to follow.

**Quality**: I have some doubts about the significance of the contributions:
* GAME (expert) layer: according to the supplementary material, the number of experts is only 2-3, depending on the dataset -- and the number of sampled experts in the forward pass is often one. If the mixture of experts is indeed useful, I would expect a performance gain when using more experts. When only one expert is sampled in the forward pass, it could happen that only one of the two experts gets sampled all the time, reverting the model back to a vanilla one.
* GraDiv: the additional loss term has complexity $O(N^2)$. This is clearly infeasible for larger datasets, e.g., on the `grb-aminer` dataset this requires roughly 1.6TB of memory when performing full-batch training. There is no mentioning of batched training, so it is unclear how this additional loss term is used.
* DeCoG: What is the difference to standard PGD attacks [Xu et al. 2018]? The fact that only a subset of the GAME model is activated in the forward pass comes from the fact the GAME layer is used, not from the DeCoG contribution itself.
* Using infinity-norm is meaningless on graph adjacency matrices because they are discrete, i.e., either we allow no edge to change (epsilon=0) or all edges can change at the same time (epsilon=1).  

Regarding the experiments:
* Figures 6 and 7 are purely qualitative and do not strongly show an advantage of the proposed method over GCN.
* Figure 4 is confusing. In the top row (GCN) the adversarial and clean representations are much more similar than in the bottom row, contrary to the description in the text. This figure shows an advantage of GCN over the proposed method.
* Similarly in Figure 5, both plots look very similar and it is not clear why the proposed method has an advantage over GCN.
* Table 2: What do the confidence intervals correspond to? Is it standard error/ standard deviation? Over how many training runs? How many different random splits? What is the size of the splits?
* Why are the confidence intervals for "w.o. attack" almost always exactly zero for the baselines?
* Why are the confidence intervals of the proposed method much larger than the baselines'? E.g., on FGSM on grb-aminer, the method has a confidence interval of ±0.69, more than ten times larger than any of the baselines. Does this indicate that the proposed model is less stable?
* Removing GraDiv only has a very small effect on the model performance, and is mostly within the confidence interval of the full model. This raises doubts whether GraDiv actually contributes to the performance. 

**Novelty**: The PGD-based adversarial training is not new ([Xu et al. 2018], [Geisler et al. 2021]), and the authors do not reference these works. The Mixture of Experts model seems novel in this context, but given my concerns above I am not convinced that it substantially increases the performance.

**Reproducibility**: Limited, given the many open questions that remain for me about the experimental procedure (see above).
summary_of_the_review: I have concerns (i) regarding the significance of the individual contributions of the method, (ii) about the lack of important baselines, and (iii) about the experimental setup. Unless these are resolved, I would not recommend publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to defend against evasion attacks in node-level graph learning. The authors introduce a Graph Adversarial Mixture of Experts (GAME) framework for learning robust representation in an adversarial training manner (e.g., PGD). In particular, GAME combines PGD training and the Mixture of Experts mechanism for ensembling adversarial gradients from different sub-models to strengthen the attack on the graph. The experiments further show that this unified design of GAME can be implemented as efficiently as normal GNN adversarial training but offers impressive advantages over prior work while under different graph attack evaluations.
strength_and_weaknesses: Strength:
1. This paper initially discovers an interesting distribution difference between common graph learning and adversarial graph learning. This difference also is one of the main reasons which makes adversarial graph learning become difficult. This discovery is interesting. 

2. The proposed model couples adversarial training with the Mixture of Experts mechanisms. This joint design offers stronger adversarial attack samples for activated experts since the current adversarial noises are generated from other different inactivated experts. The model is relatively new and novel to me. 

3. The authors provide comprehensive thoughts for stronger graph robustness. The investigation of model, optimization and training contributes to a thorough study. Also, the paper provides a deep and meaningful thought on the correlation between MoE and PGD.

4. Comprehensive evaluation across multiple scales of graph robust benchmark datasets are conducted. GAME outperforms many baseline methods in terms of robustness and accuracy. The authors also provide an in-depth analysis of GAME's capacity to mitigate the distribution difference between standard graph learning and adversarial graph learning. The presentation of this paper is overall good. 

Weakness:
1. I tend to think that GAME has the potential for more general use, such as enhancing adversarial training on general machine learning. Since this paper is specifically focusing on graph adversarial training, authors need to clarify why GAME is designed for graph tasks (e.g., the exact benefits of GAME on graph tasks).

2. The authors provide empirical explanations about why the computational cost of the proposed GAME is close to the common GNN while this paper has no mathematical estimations about GAME's computation and the baseline GNN. Authors should provide better complexity analysis for a more convincing claim on GAME's efficiency. 

3. The connection between the $R_{Gradiv}$ and backbone design is not very clear. Although experiments say that the gradiv regularization improves the model's performance, this regularization operation causes extra human-craft efforts in adjusting the hyperparameter.

clarity,_quality,_novelty_and_reproducibility: The motivation is clear. The structure and presentation of GAME in this paper are easy to understand. The proposed method is relatively new and novel. The authors also provide code and data in the appendix for reproducibility. 
summary_of_the_review: This paper proposes to study graph adversarial learning. The proposed GAME method is novel and well-motivated. The extensive experiments practically support the claims in the paper and answer the questions raised with motivation. In summary, this is good and solid work. I tend to accept this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper tackles the problem of adversarial robustness of graph representation learning. The authors hypothesize that one reason for the non-robustness is the mixing of adversarial vs. "normale" data distributions. Based on this insight, they propose an improved learning/training principle. The paper addresses and overall important problem and contains some novel insights. Overall, the majority of reviewers indicated acceptance. However, some serious limitations should be addressed by the authors in the final version of the paper:

- The performed split is extremely simple (60% labeled data). The authors should repeat their experiments with a more realistic scenario as done in other works (e.g. only 10% labeled data)
- The authors should highlight the attacker capabilities (e.g. node injection) more clearly. This is not obvious in the current version.
- The comparison to Soft-Medoid/Soft-Median should be performed since these approaches exactly address the "shift" in data distribution as hypothesized by the authors.

If the authors adequately address these points, the paper should be accepted.
justification_for_why_not_higher_score: Since there are still some limitations (see above), I do not recommend a higher score.
justification_for_why_not_lower_score: A rejection might be justifiable if the recommendations are not addressed.
Decision: Accept: poster