Chasing All-Round Graph Representation Robustness: Model, Training, and Optimization
Authors: Chunhui Zhang
Yijun Tian
Mingxuan Ju
Zheyuan Liu
Yanfang Ye
Nitesh Chawla
Chuxu Zhang
Venue: ICLR 2023 poster (2023)
Keywords: Graph neural networks
Mixture of experts
Graph adversarial learning

Abstract: Graph Neural Networks (GNNs) have achieved state-of-the-art results on a variety of graph learning tasks, however, it has been demonstrated that they are vulnerable to adversarial attacks, raising serious security concerns. A lot of studies have been developed to train GNNs in a noisy environment and increase their robustness against adversarial attacks. However, existing methods have not uncovered a principled difficulty: the convoluted mixture distribution between clean and attacked data samples, which leads to sub-optimal model design and limits their frameworks’ robustness. In this work, we first begin by identifying the root cause of mixture distribution, then, for tackling it, we propose a novel method GAME - Graph Adversarial Mixture of Experts to enlarge the model capacity and enrich the representation diversity of adversarial samples, from three perspectives of model, training, and optimization. Specifically, we first propose a plug-and- play GAME layer that can be easily incorporated into any GNNs and enhance their adversarial learning capabilities. Second, we design a decoupling-based graph adversarial training in which the component of the model used to generate adversarial graphs is separated from the component used to update weights. Third, we introduce a graph diversity regularization that enables the model to learn diverse representation and further improves model performance. Extensive experiments demonstrate the effectiveness and advantages of GAME over the state-of-the-art adversarial training methods across various datasets given different attacks.

Published as a conference paper at ICLR 2023
CHASING ALL-ROUND GRAPH REPRESENTATION Ro- BUSTNESS: MODEL, TRAINING, AND OPTIMIZATION
Chunhui Zhang', Yijun Tian", .Mingxuan Ju², Zheyuan Liu', Yanfang Ye², Nitesh V. Chawla", Chuxu Zhang**
BBrandeis University, (chunhui zhang, zheyuan iu, chuxuzhang htwpruzzeannggraadeeiiii. @brande edu Univerrity of Notre Dame, {yijun jijjun... tian, m mjuu, yye7, nchawla](nd, edu
ABSTRACT
Graph Neural Networks (GNNs) have achieved state-of-the-arr results on a variety of graph learning tasks, however, it has been demonstrated that they are vulnera- ble to adversarial attacks, raising serious security concerns. A lot of studies have been developed train GNNs in a noisy environment and increase their robust- ness against adversarial attacks. However, existing methods have not uncovered a principled difficulty: the convoluted mixture distribution between clean and at- tacked data samples, which leads to sub-optimal model design and limits their frameworks' robustness. In this work, we first begin by identifying the root cause of mixture distribution, then, for tackling it, propose a novel method GAME Graph Adversarial Mixture of Experts to enlarge model capacity and enrich the representation diversity of adversarial samples, from three perspectives of model, training, and optimization. Specifically, we first propose a plug-and houg-and--lay GAME layer that be easily incorporated into any GNNs and enhance their adversar- ial learning capabilities. Second, we design decoupling-basee -based graph adversarial training in which the component of the model used generate adversarial graphs is separated from the component used to update weights. Third, we introduce a graph diversity regularization that enables the model to learn diverse representa- tion and further improves model performance. Extensive experiments demonstrate the effectiveness and advantages of GAME over the state-of-the- art adversarial training methods across various datasets given different attacks.
INTRODUCTION
Graph neural networks (GNNs) have been demonstrated to be effective at learning from graphs. They explore a message- message-passing mechanism to update node representations by iteratively aggregat- ing information from their neighbors, allowing GNNs to achieve ofate-of-the thet-----t-httttt- performance (Kipf & Welling, 2017; Veličković et al., 2018; Hamilton et al., 2017). Many real-world applications based GNNs, such modeling over social networks (Fan et al., 2022; Zhang et al., 2019; Hu et al., 2020), scene graph reasoning (Chen et al., 2020; Zhang et al., 2022), and biological molccules (Jin et al., 2018; Xu et al., 2019; Guo et al., 2022).
Nevertheless, despite their outstanding performance, GNNs are susceptible to perturbations (Zügner al., 2018b; Züggerr& & Günnemann, 2019; Zheng al., 2021; Yue al., 2022), which necessitate techniques to leverage GNN's robustness against adversarial attacks. Attackers can downgrade the performance of GNNs from multiple perspectives, such as adding or removing edges (Geisler al., 2021; Chen et al., 2023), perturbing node properties (Zügner & Günnemann, 2019; Sun et al., 2020; Tian al., 2023), and injecting malicious nodes (Zou et al., 2021; Ju al., 2023). To enhance GNN's robustness, multiple defense methods against graph attacks have been proposed (Jin et al., 2020; Entezari et al., 2020; Zhang & Zitnik, 2020). However, most existing methods have not uncovered the principled difficulty (i.e., the convoluted mixture distribution between clean and attacked data samples), which results in sub-optimal model design, poor robustness, and limited performance. In light of this, we study the robustness of GNNs from more fundamental perspective by discovering the key pattern behind the adversarial attacks that jeopardizes the performance of GNNs.
"oorreponning author

---

Published as a conference paper at ICLR 2023
Input
First BN layer
Last BN layer
0.20
attack clean
variance
1.30
variance
3.2
0.15
1.15
0.10
1:00
0.05
0.85
clean
clean
0.00
-1.0
-0.5
0.5
0.70
-100
-0.5
0
00
0.2 0.6
mean
running mean (b)
running mean
(a)
(c)
Figure I: The distributions of node representations generated by two GNNs trained over clean and adversarial graphs. In (a), these two distributions are extremely similar. In (b) and (c), the model gets deeper, a progressively larger differentiation between the two distributions is observed.
We begin by comparing the statistical differentiation between the latent representations of nodes on the clean graph and the adversarially generated graph, as shown in Figure 1. We observe that the distributions of node representations for clean and adversarial graphs before the message passing are highly similar (i.e., Figure ((a)). However, as the model gets deeper, these two distributions get increasingly distinct, as demonstrated by the progressively larger shift shown from Figure (a) to (c). This demonstrates that adversarial attacks imperil GNN's performance by generating adversarial graphs belonging a distribution different from the clean graph, and the GNN model fails transfer the knowledge learned from the clean graph to the generated adversarial graph.
To address the above challenge, we propose Graph Adversarial Mixture of Experts (GAME), a novel framework that enhances the robustness for GNN by expanding the model capacity and in- creasing the representation diversity for adversarial graphs. Specifically, we design GAME from three perspectives: (i) To strengthen the model capacity, we propose plug-and-play GAME layer to accommodate the adversarial graphs with diverse mixture distributions by dynamically routing mul- tiple assembled expert networks. (iz) From the training perspective, we present decoupling graph adversarial training strategy, namely DECOG, where each expert network is trained by adversarial graphs generated by maximizing the gradient of other experts. DECOG enforees each expert to learn distinct distributions that all other experts under-perform at. (iti) From the optimization perspective, incorporate a graph diversity regularization (GRADIv) to further enhance the diversity of knowl- edge learned via all expert networks such that GAME is capable of handling various adversarial graphs. GAME an all-round robust framework that not only improves GNN's resilience to ad- versarial attacks, but also without too much extra cost compared with normal GNN, since GAME dynamically activates only one subset of the experts participate in the computation. The contri- butions of this paper can be summarized as follows:
To the best of our knowledge, this is the first work to improve GNN's robustness from the per- spective of distribution differentiation. According to our empirical studies, existing GNNs fail to transfer the knowledge learned from one clean graph's distribution to another generated adversar- ial counterpart, which results in vulnerabilities to adversarial attacks.
To solve this challenge, we propose an all-round framework, namely Graph Adversarial Mixture of Experts (GAME), from the perspectives of model design (i.e., GAME layer to bolster the model capacity). training (i.e., DECOG to diversify the adversarial graphs), and optimization (i.e., GRADIV to further diversify the experts' knowledge).
Comprehensive experiments are performed on multiple benchmark datasets across varying scales, demonstrating that the robustness contributed to our proposed all-around GAME. The suggested method beats other common baselines in variety of attack evaluations and natural evaluations, demonstrating that the all-around robust design of GAME handles intricate mixture distribution well and cleverly addresses fundamental difficulty in graph adversarial training.
2 RELATED WORK
Graph Neural Networks. Graph Neural Networks have recently attracted a great deal of interest due to their effectiveness in learning non-Euclidean data and their remarkable performance in a vast array of graph mining tasks (Hamilton et al., 2017; Battaglia et al., 2018; Wu et al., 2020). Graph convolutional network (GCN) is proposed in the early stage of GNN research to apply the concept of convolution from image to graph data (Kipf & Welling, 2017; Gao et al., 2018; Wu et al., 2019a). Instead of simply averaging the features of neighboring nodes, graph attention networks (Veličković al., 201 8; Wang al., 2019) use the attention module to value each neighboring node and learn
2

---

Published as a conference paper at ICLR 2023
more important nodes during message passing. Simultaneously, skip connection is introduced to construct deeper GNNs and learn comprehensive representations in order to overcome the over- smoothing phenomenon (Li et al., 2019; 2021a;b), A concurrent work with ours exploits the sparsity property of the Mixture of Experts (MoE) mechanism to learn fairer representations on graphs (Liu et al., 2023). Unlike prior works focusing on improving model's standard accuracy faimess, to the best of our knowledge, our GAME is the first attempt to improve the robustness of adversarial graphs by introducing the Mixture of Experts mechanism strengthen GNN capacity.
Adversarial Learning on Graphs. It is demonstrated that deep learning models are susceptible inputs with small adversarial perturbations, and several methods are proposed to improve the model's 's robustness (Goodfellow et al., 2015; Kurakin et al., 2017; Xie & Yuille, 2020). Recent re- search has shown that GNNs are susceptible to adversarial attacks without exception (Zheng al., 2021), highlighting the urgent need to improve their robustness. Several methods are proposed for attacking graph data, including inserting or removing connections (Du al., 2017: 70 Chen al., 2018; Waniek et al., 2), perturbing node features (Zügner al., 2018a; Zügner & Günnemann, 2019; Sun et al., 2020), or adding virtual nodes (Wang et al., 2020; Zou et al., 2021). In the meantime, nu- merous defense methods against graph attacks have been developed for learning robust GNNs (Zhu et al., 2019; Feng et al., 2020; Jin et al., 2020) or removing the attacked input during preprocess- ing (Entezari al., 2020; Zhang & Zitnik, 2020). Go beyond prior works, this paper addresses fundamental issue in graph adversarial learning (i.e., overly complex mixture distributions between clean and attacked nodes), subsequently improving the model capacity and representation diversity.
PRELIMINARIES
Mixture of Experts. deep learning. the Mixture of Experts (MoE) constructs big neural networks with a dynamic routing strategy, which facilitates superior model capacity and attains better data parallelism (Shazeer et al., 2017). Given the input æ E Rd, the current expert layer = {E)()))- with n experts, and the gating network P(-) {p:())>= the output of MoE module can be formulated as follows:
y = Σ (,))).(()) ET
(I)
where T represents the set of activated top-k expert indices. In the above equation, the gating module ((x) makes the activated portion of the model the same size as the normal network, hence enabling the efficient training of larger neural network. Specifically, we calculate the gate-value p:())) for i-th expert as follows:
Pi(z)= exp(h(x)))
(2)
xx=(exx((xx))
where (xx) is a linear transformation to compute the logits of experts given input x, and h(x); reflects the i-th value of the obtained logits, which weights the i-th expert in current layer.
Adversarial Training. Given the data distribution DD over inputs x E IRd and their labels y E R a standard classifier (e.g., neural network) Rd - IR with parameter 0 maps an input to labels for classification, utilizing empirical risk minimization ming ((w [(f(x; 0), v), where -(.,)) represents the cross-entropy, loss. Numerous strategies have been proposed to improve neural network adversarial robustness, with adversarial training- waaseng-bbssed methods being the most effective. The majority of cutting-edge adversarial training algorithms optimize hybrid loss consisting of standard classification loss and an adversarial loss term:
Cets L(f(x; f).)). Ladv
max + 6EB(e)
(3)
where Lat denotes the classification loss over standard (or clean) inputs, while Lade is the loss that encourages the model to learn from adversarial data samples, and B(c) = {8 ||0]|cc S <) is the perturbation set. Popular adversarial training methods such as PGD (Madry et al., 2018) employs the same Lats as in Equation 3, but substitute Lado with soft logits-pairing term. Thus, the overall goal of adversarial training is to minimize the following objective:
min E (( [(1 A) + XCadul,
(4)
3

---

Published as a conference paper at ICLR 2023
(2) DeCoG
Activating Different Partial Experts
LGAME
(1) GAME layer
Layers
V aadv
Loyers
Ladv
T
1
1
1
D
+
0
0
Gate
0
1
Gate
cleaan
1
0
0
1
1
Ladv
0
0
Reraviv
(3) GraDiv
maximizing Lade
minimizing GGAME
Adversarial Graph Generation
VxLado
Mixture of Experts Gradient Update
The illustration of the GAME framework: In (1) GAME layer, partial experts (in blue shaded region) are activated to compute adversarial gradient just for adversarial graph construction without weight updates (left part). Then, (2) DECOG decouples expert weights used for adversarial graph generation and model update, where the gate module reactivates other partial experts to fit created adversarial graphs in each GAME layer (right part). Finally, (3) GRADIV regularizes multi- ple experts learn knowledge with more diversity. Note that in the left part, the model maximizes adversarial loss to obtain the adversarial gradient only for graph perturbations. Next, in the right part, the model minimizes overall loss on previously generated adversarial graph for weight updates.
Figure 2:
where À is fixed weight haper-parameter. To balance standard and robust accuracies, the hyperpa- rameter À must be set to a constant value during training for all of these contemporary adversarial learning works.
4 METHODOLOGY
In this section, motivated by the prior analysis of adversarial mixture distribution Section 1, we present novel framework GAME to increase the adversarial robustness of GNNs. Figure/ 2 illus- trates the overall framework of the proposed model. Specifically, first, our model is developed using GAME layers, which increases model capacity on adversarial mixture distribution by introducing the MoF. mechanism. Second, based on the GAME layer, we design a novel DECOG training strategy that augments more varied graphs facilitate GNNs' adversarial training. Third, in order classify adversarial samples from mixture distribution, we suggest graph diversity regularization for learning more distinguishable representations. The details of different parts are described in the following.
4.1 EXPANDING MODEL CAPACITY: GAME LAYER
To improve the learning capacity of GNNs and account for the overly complex mixture distribution, each GNN layer our model includes a GAME layer at the model level. To update the feature of a target node, our model employs GAME layer to transform and aggregate the features of the neighboring nodes. Subsequently, the GAME combines the feature target node and neighboring nodes to formulate node representation in the graph:
(S) where 1.(2) denotes representation of node v at I-th layer; AGGR(.) and COMB(.) represent the neighbor aggregation and combination functions, respectively. In Equation 5, the GAME layer constructed by W {W.(-)}² and P(-), which represent the set of n expert networks and the gate module, respectively. Then, the GAME layer formulated as follows:
h (OMB(") (GAME") (nC--")) AGGR (HAME"((N-D)) V,
Vu Nu
GAMB(h) Σ (h)))))((h).
where T represents the set of activated top-k experts in each GAME layer. In comparison to conven- tional GNNs, the proposed sparse GAME layer is able build wider model with low computational cost, resulting in greater capacity to express mixed distributions, as shown in Figure 2 (1).
4

---

Published as a conference paper at ICLR 2023
4.2 BOOSTING ADVERSARIAL GRAPHS: DECOG
For boosting the diversity of attacked node attributes and attacked adjacency matrices, we present Decoupling-based Graph Adversarial Training Strategy (DECoG). DECOG aims deliver more robust performance than traditional graph adversarial training by transferring adversarial features from all experts to each individual expert. Specifically, given a clean graph G and the node labels y with data distribution D. we first calculate the loss function over both clean graph G and attacked graph 9" between input g's target nodes and their ground truth label y. Then, we learn robust GAME model fo with parameters 0:
min (G,)) (1 (1- C) (fo())) XLed (fo(g")
where (cleam(..)) Cae (+)) indicate the losses (e.g., cross-entropy for node classification) on clean graph and attacked graph, respectively, and À regulates the weight of the adversarial loss. The adver- sarial graph G* is generated by GAME's customized PGD attack: during the learning procedure of PGD, GAME activates partial experts to compute multiple iterations of adversarial gradients. When the computation is finished, the gradients are added to the original graph as the final adversarial input for GNNs to minimize the adversarial learning loss. Therefore. each expert acquires the adversarial features generated by others. From a high-level vantage point, DECOG enables our GAME layer to implicitly transfer the aggregated knowledge of all experts to each individual expert.
To formally describe the aforementioned pipeline, given the initial clean graph G(O), we first ex- tract the initial adjacency matrix ( and node features (() from (()), Then, we dynamically sam- ple the index set of activated experts T using (A X (0) } to determine the GAME model's 's current activated part fo I7o (-), where (0))) (To is subset of the GAME model So with parameters O the current iteration. To maximize Lee, for (-) utilizes {A (() input. The gradients com- puted for ( and (o) are represented as A (fo (g()) and V Vx (forro: (g(O)), v), respectively. Both types of gradients are considered as adversarial noises, which are later incorpo- rated to the current input (A(0), x to derive the perturbed adjacency matrix and node features {A()) X() The procedure can be formulated as:
1(++1) = IIB (,)) (A + a -iign (V AW (forco (g()), y))),
(8)
((-11) =IB (X((0 a- sign( x Cee (for (g z))),
where B(A,e) is the to sphere with radius around clean A, B(X,e) e) is the lx sphere with dius around clean X. IlB(A.e) means the nearest projection to B(A, e). IIB(x,e) means the near- est projection to B(X, c), and a is the step size. With the aim of acquiring the adversarial graph we calculate the gradients T times as in Equation (8) and treat final gradients
g- = as perturbations. Later, 9* is utilized to train different subset of activated experts. In general, DECOG enables the dynamic activation of each expert in GAME and facilitates the computation of more diverse attacked graph adjacency matrices and node features, depicted in Figure 2 (2).
4.3 ENHIANCING CLASSIFIABILITY IN ROBUST REPRESENTATION GRADIV
For empowering GAME in learning more distinguishable representation from the complex mixture distribution, we design Graph Diversity Regularization (GRADIV) from the optimization level. This regularization term penalizes the model to maximize the distance between node embeddings and enforces the model to learn distinct representations. The regularization is formulated as:
N RGRADIV = log
exp(SIM(hs, hh)ss/ k * i,
EK=1 exp(SIM(h,, hw)/T)
where SIM(:,)) calculates the cosine similarity between two node representation, and N denotes the number of nodes. RGRADIV increases the distance between any pairs of learned node embeddings, which offers explicit supervision signal to learn node representations with more variety, and hence improving the model learning capacity. Finally, the overall learning objective function LaAmE is defined as a weighted combination of Celeon Cadu and RORADAY: cee
LGAME = Celean + Cado + RGRADIV.
(10)
Consequently, LGAME enables GAME to learn distinguishable node embeddings and preserve robust- ness to attacked node features, thereby improving the representation quality for downstream task.
5

---

Published as a conference paper at ICLR 2023
5 EXPERIMENT
In this section, we perform comprehensive experiments on the graph robustness benchmarks to demonstrate the effectiveness of our proposed GAME model against adversarial graphs with complex distributions. This section is guided by answers to the following five research questions: RQ-1: Can GAME achieve better robustness compared other SOTAs? RQ-2: How does each component in our all-around framework contribute to the robustness improvement? RQ-3: Can GAME separate the mixed distribution of clean and attacked nodes? RQ-4: Does GAME generate more diversified training adversarial graphs compared to baselines? and RQ-5: Can GAME learn distinguishable node representations?
5.1 EXPERIMENTAL SETUP
Datasets. We utilize Graph Robust Benchmark (Zheng al., 2021) dataset to evaluate our model's performance by graphs with varying scales, including grb-cora (small-scale), grb-citeseer (small- scale), grb-flickr (medium-scale), grb-reddit (large-scale), and grb-aminer (large-scale).
Baseline Methods. We compare GAME with various baseline methods, spanning multiple perspec- tives. For models that specifically focus on robustness, we explore R-GCN (Zhu et al., 2019), GNN- SVD (Entezari et al., 2020), and GNNGuard (Zhang & Zitnik, 2020). In addition, we incoporate general GNN models (i.e., GCN (Kipf & Welling, 2017). GAT (Veličković al., 2018), GIN (Xu et al., 2019), APPNP (Gasteiger et al., 2019a), TAGCN (Du et al., 2017), GraphSAGE (Hamilton al., 2017), SGCN (Wu al., 2019a)) with two generic defense approaches (i.e., layer normaliza- tion (Ba et al., 2016) (LN) and adversarial training (AT) (Madry et al., 2018)).
Attacking Strategies. We explore five effective yet diverse node injection attack methods to imperil the victim GNNs: RND (Zheng et al., 2021), FGSM (Goodfellow et al., 2015; Zheng et al., 2021), PGD (Madry et al., 2018), SPEIT (Zheng et al., 2021), TDGIA (Zou et al., 2021). These node injection methods have been proven to deliver scalable and transferuble attacks (Zheng et al., 2021).
Details of attacking strategies and adversarial training are described in Appendix B. and B.2, re- spectively. Also, include the statistics of datasets in Appendix C. In addition, edge modification attack evaluation based on SoottMcddid/Soft-Matet Mediinn (Geisler et al., 2020; 202.) is in Appendix F.
5.2 OVERALL PERFORMANCE ON GRAPH ROBUST BENCHMARK
To answer RQ-1, we conduct the experiments to evaluate the robustness and report the perfor- mance in Figure 3 (We run 10 times for mean results/standard deviation and the train:valtest split is 0.6.0.1:0.3, Due the page limitation, we include the full results Table 2 with all numerical values in Appendix A). As shown in Figure 7, GAME comprehensively achieves better robust accuracy than other baselines under five distinct attack assessments. For instance, on the small-scale grb-citeseer, the average accuracy of GAME outperforms the second-place R-GCN-AT by 2.96%; on the medium- scale grb-flickr, the average accuracy our GAME outperforms the runner-up GAT-AT by 2.1 %;; on the large-scale grb-aminer, GAME outperforms the second-best GAT by 1.32%. These results demonstrate the outstanding effectiveness of GAME against the distribution differentiation across five GRB graphs with different scales. Besides, under the attack setting, GAME significantly outperforms baselines by a large margin, proving that GAME equipped with multiple expert net- works has stronger learning capability as well higher accuracy on clean graphs. Though more parametrized than other baselines, GAME still enjoys the efficiency like regular GNNs and retains remarkable robustness against adversarial attacks, thanks our proposed dynamic routing strategy that only activates partial experts to approximate the forward and backward of normal GNN.
5.3 CONTRIBUTION OF INDIVIDUAL COMPONENT IN THE ALL-ROUND FRAMEWORK
GAME integrates three different components into a comprehensive graph robust learning framework, and to answer RQ-2, we conduct experiments components in GAME, denoted as (a) w/o GAME layer, (b) w/o DECOG, and (c) w/o GRADIV, as shown in Table 1. We observe that the performance in both adversarial and clean graphs decreases after removing each component, demonstrating the contribution of each design in increasing the model's performance. For (a) w/o GAME layer, removing GAME layer from our framework results
the performance without one of the individual

---

Published as a conference paper at ICLR 2023
Cora TDGIA
CiteSeer TDGIA
Flickr TDGIA 54
88
78
W.O. attack
78
SPEIT W.O. attach
SPEIT W.O. attac
SPEIT
FGSM
FGSM
PGD Reddit TDGIA
PGD
PGD
AMiner
TDGIA 71
97
68
Ous
W.O. attack
SPEIT W.O. attack
65
SPEIT
GAT-AT
GATGu
R GCN+AT
GIN+AT
SGCN-LN
APPNP LN
R R-GON
GraphiSageAAA
GENHLN
GCNGuad
GON+AT
FGSM
RND
FGSM
RND
TAGEN-LN
GrapiSage-L
TAGEN-AT
GraphSaaee
PGD
PGD
Figure 3: Overall assessments of all framework across graph with different scales. We apply TDGIA, SPEIT, RND, PGD-based, and FGSM-based graph attacks to evaluate the robustness of different frameworks. w.o. atlack refers to the performance on clean graphs. For better clarity, we only include strong baselines in each figure.
in GAME being vanilla GCN on the model level. Such removal disables the GRADIV training tech- nique, GRADIV depends on the GAME layer. Removing GAME layer decreases the performance on clean graphs (under w.o. attack) more than that on adversarial graphs. Specifically, under wo attack, this removal causes the model lose 2.58% accuracy, which is higher than the accuracy losses under two representative attacks (0.98% of PGDD-based and 0.98% FDSM-based) on grb-citeseer, respectively. For (b) w/o DECOG, disabling the DECOG training strategy results in the generated adversarial graphs tightly coupled with the activated experts at the current iteration (i.e., the ac- tivated experts adversarial gradient computation are identical to those updated by minimizing the loss). As a result, this removal causes more severe performance deterioration for adversarial graphs than it does for clean graphs. Specifically, under PGD-based and FDSM-based attacks, this removal causes the model grb-citeseer lose 0.56% and 0. 0.75% accuracy respectively, which comparable to the removal of GAME layer. However, the accuracy loss on wo attack only 1.56%, which is significantly less than the accuracy loss caused by GAME layer. It demonstrates that DECOG improves the performance of GNNs against adversarial attacks by generating diverse yet effective adversarial graphs, which is also compatible with the subsequent additional studies in Section 5.5. And for (c) w/o GRADIV, GAME without the use of RoraDIV in Equation 10 leads to scenario where actively diversifying node embeddings is no longer an explicit supervision signal during the optimization process. As result, this removal leads to performance downgrade on both clean graph and ad adversarial graph (0.65% on clean grb-citeseer and 0.42% on FGSM- grb- citeseer, respectively). showing that the effectiveness of GRADIV assisting GAME model to learn distinguishable representations on both clean and adversarial graphs.
5.4 PERFORMANCE AGAINST DISTRIBUTION DIFFERENTIATION
We evaluate the performance of GAME and the vanilla GCN against the attacks from graphs gener- ated through the adversarial distributions that extremely divergent from the distribution of clean graphs, as shown in Figure 4. To answer RQ-3, we visualize the distributions of node representations from GCN and GAME three stages (i.e., input layer (left column) and after first and second mes- sage passing layers (middle and right column respectively).. We observe that for the vanilla GCN, the distribution shift between the node representations in the clean and adversarial graphs still exists,

---

Published as a conference paper at ICLR 2023
Table 1: Ablation studies for GAME on graphs with varying scales. Each row includes the model
variant (mean result + standard deviation) without one of the components in our all-around design.
Methods
TDGIA
SPEIT
RND
PGD bused
FGSM based
w.O. attack
Avg. Acc.
GAME
75.80
75.69
76.96
74.86
0.73
74.85 0.71
77.86
0.22
76.00 0.82
w/o GAME layer
73.96 = 0,60
73.95
+ 0.31
7421
= 0.60
74.01 = 0.52
73.89
0.24
75.28
0.00
74.22
= 0.39
DECoG w/o GRADIV
74.07 0.18
73.98
74.70
74.30
0.82
74.10
0.33
76.30
0.80
74.58
0.51
75.52 + 0.28
76.50
+ 0.91
76.80
+ 0.51
74.42 + 0.64
74.33
+ 0.46
77.21
+ 0.16
75.63
0.49
GAME
$2.40 1.02
50.54
1 0.67
52.63
0.87
52.62 1: 0.14
52.59
1.20
53.83
0.47
$2.44
0.73
grh-Biikk
w/o GAME layer
51.32
1: 0.75
47.43
1: 0.14
51.02 1: 0.42
50.44 1: 0.35
50.60
1.15
52.31
0.22
50.52
0.53
w/o DECOG
51.63 4: 0.58
48.12
1: 0.45
51.52 -: 0.83
51.07
+: 0.83
49.95
1 0.75
52.95
0.89
50.87
0.72
w/o GRADIV
52.27 0.18
50.14 0.19
52.33 + 0.28
52.19 + 0.52
52.10 0.85
53.14 0.99
52.03
0.50
gob pob-aomner
GAME
68.21 = 1.27
69.43 + 1.022
69.03 = 0.72
69.33 = 0.62
69.21
0.69
70.15
0.95
69.22 0.88
w/o GAME layer
67.89
01.27
68.06
67.95
68.34 0.29
0.15
68, 23
0.40
68.11 0.35
w/o DFCOG w/o GRADIV
67.96 + 0.63
68.45 + 0.48
68. 22 + 0.35
68.70 0.18
68.25 0.22
60.26
0.48
68.47 0.39
67.81 0.78
69.32 + 0.26
+ 0.39
09. 12 0.54
69.11 0.41
69. 00 0.50
68, 96 0.48
Input
layer
layer
0 20
1:00
2.4
attack clean
attack clean
attack clean
0.75
GCN
var
0:10
0:50
1.2
0.05
0:25
0.6
0.00
-1.0
-0.5
0.0
0.5
11
1.0
0.00.50
0.25
0.50
0.75
1.00
0.8.
0.6
1.2
1.8
2.4
mean
mean
mean
Input
ayer
layer
1.00
attack ccan
attack clean
0:15-
0.75
Ous
0.10
0.50
0.05
0.25
1.0
-0.5
0.0
0.5
1.0
0.00
0.25
0.50
0.75
1.00
0.0
0.6
1.2
2.4
mean
mean
mean (c)
(a)
(b)
Figure 4: The distributions of node representations on clean and adversarial graphs (upper: adver- sarial trained GCN; lower: GAME model) at the input layer (left), after the first batch normalization (BN) layer (middle), and after the last batch normalization (BN) layer (right). Each point indicates the mean and variance of single channel in the BN layer.
even when the learning model forwards. In addition, the distribution difference between adversarial graphs and clean graphs is small. This phenomenon demonstrates that vanilla GNNs such as GCN suffer from mixture distributions between adversarial graphs and clean graphs, which prevents GCN from learning distinguishable representations. Compared with the vanilla GCN, GAME can success- fully distinguish the node representations of the adversarial graphs from those of the clean graphs. Besides, as node representations pass through deeper layers, GAME still maintains the ability of discrimination, demonstrating GAME'S outstanding effectiveness against adversarial graphs.
5.5 DIVERSITY OF ADVERSARIAL GRAPHIS GENERATED BY GAME
To answer RQ-4, we visualize the distributions of both clean
0.20
0.20
attack dewn
attack
and adversarial graphs gener- ated by GAME and GCN in Fig- ure 5. We observe that the ad- versarial graphs generated from GCN are similarly distributed to the clean graph. On the contrary, GAME generates ad- versarial graphs whose distribu- tion is statistically distinct from that of the clean graphs, further demonstrating the effectiveness
0.15
0.15
clean
0.10
0.10
0.05
0.05
0.00
0.00
-10
-0.5
0.0 mean
0.5
1.0
-10
-0.5
0.0
0.5
10
mean
(a) GCN
(b) Ours
Figure 5: Distributions for clean and adversarial graphs created
by GAME (b) and an adversarially trained GCN (a). For fair com- parisons, we explore the same setting for both models.
8

---

Published as a conference paper at ICLR 2023
20
20
20
15
15
10
10
10
5
-0.0
0.5
-1.0
-1.5
-10
0.0
0.5
10
0.55
0.5
1.-1.00
-0
0.0000
0.5
1.0
1.0
0.5
0.5
1.0-1 10
-05
GCN (standard)
Ours (standard)
GCN (adversarial)
Ours (adversarial)
The loss landscapes of GAME and a vanilla GCN over clean graphs (the 1tt and 2nd figures) and adversarial graphs (the 3rd and 4th figures). Under both settings, visualize the same set of nodes randomly selected from the test set of the grb-cora dataset.
Figure 7:
of DECOG and GRADIV. Diverse experts enable GAME learn distinguishable node represen- tations for robust performance, which significantly mitigates the GNN's training difficulties on graphs with distinct distributions.
5.6 DIVERSITY OF NODE REPRESENTATIONS BY GAME
To answer RQ-5, on grb-cora dataset, visualize the node representations generated by GAMR and a vanilla GCN, shown in Figure 6. We observe that the node representations generated by GCN are generally entangled and intertwined with each other, while those generated by GAME exhibit a very well-clustered representation space with clear inter-cluster difference. This phenomenon demonstrates that GAME can distinguish the com- plex distributions and further
(a) GCN
(b) Ours
Figure 6: t-SNE (Van der Maaten & Hinton, 2008) for representations of nodes in the test set in grb-cora. Nodes colored according to their class labels.
learn distinguishable node representations, thanks to the all-around design of GAME.
5.7 ANALYSES ON OPTIMIZATION LANDSCAPE
To further validate the robustness of GAME, analyze whether our all-round framework GAME reduces the difficulty of learning adversarial graphs by plotting its loss landscape (Li et al., 2018) w.r.t. both input and weight spaces. According to Figure 7, we observe that GAME leads flatter minima optimization landscape than the adversarially trained GCN on both clean graphs and adversarial graphs, indicating that the method advantageously alleviates the learning difficulty on the adversarial graph. our GAME reduces the complexity of learning adversarial graphs, allowing adversarial training model weights to be as simple as a standard environment rather than GCN.
CONCLUSION
In this paper, we first identify the fundamental issue in adversarial graph learning: the mixture dis- tribution between clean and attacked data samples. Motivated by this problem, we propose Graph Adversarial Mixture of Experts (GAME), a novel method improve the model capacity, augment adversarial graphs, and enrich the graph representation diversity. For acquiring these triple im- provements, GAME contains three innovative components, including a plug-and-play GAME layer, decoupling graph adversarial training strategy DECOG, and graph diversity regularization strat- egy GRADIV. GAME outperforms other baselines when evaluated on multiple datasets under dif- ferent attack methods. Additional experimental analysis demonstrates the effectiveness of GAME handling the complex mixture distribution, generating distinct adversarial graphs, and learning distinguishable representations.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
GAME enhances the robustness GNN models against adversarial attacks, and therefore we believe that no ethical issues can be raised by our approach. In general, we should be very careful when applying machine learning models to ensure that there is no negative societal impact.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our experiments, we include the link of the source code in Ap- pendix B.2. addition, the hlyee-parameter and other factors to reproduce our method are also provided in the Appendix B.2.
ACKNOWLEDGMENTS
This work is partially supported by the NSF under grants IIS-2209814, IIS-2203262, IIS-2214376, IIS-2217239, OAC-2218762, CNS-2203261, CNS-2122631, CCMM-2114077 and the NIJ 2018-75- CX-0032. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint
arXiv: (alo:6607.645555 2016.
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Gonzaez- Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1e06.0126.....1 2018.
Aleksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Günnemann. Scaling graph neural networks with approximate pagerank. In KDD, 2020.
Jinyin Chen, Yangyang Wu, Xuanheng Xu, Yixian Chen, Haibin Zheng, and Qi Xuan. Fast gradient attack on network embedding. arXiv preprint vrXiv:1809.0299) 2018.
Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. CVPR, 2020.
Zizhang Chen, Peizhao Li, Hongfu Liu, and Pengyu Hong. Characterizing the influence of graph elements. In ICLR, 2023.
Jian Du, Shanghang Zhang. Guanhang Wu, José MF Moura, and Soummya Kar. Topology adaptive graph convolutional networks. arXiv preprint arXiv:1710.10370 2017.
Negin Entezari, Saba A Al-Sayouri, Amirali Darvishzadeh, and Evangelos E Papalexakis. All you need is low (rank) defending against adversarial attacks on graphs. In WSDM, 2020.
Yujie Fan, Mingxuan Ju, Chuxu Zhang, and Yanfang Ye. Heterogencous temporal graph neural network. SDM. SIAM, 2022.
Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang. Evgeny Kharlamov, and Jie Tang. Graph random neural networks for semi-supervised learning graphs. In NewrlPS, 2020.
Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. Large-scale learnable graph convolutional networks. In KDD, 2018.
Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Combining neural networks with personalized pagerank for classification graphs. In ICLRR 2019a.
10

---

Published as a conference paper at ICLR 2023
Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. NeurIPS, 2019b.
Simon Geisler, Daniel Zügner, and Stephan Günnemann. Reliable graph neural networks via robust aggregation. In NewrIPS, 2020.
Simon Geisler, Tobias Schmidt, Hakan Sirin, Daniel Zügner, Aleksandar Bojchevski, and Stephan Günnemann. Robustness of graph neural networks at scale. NeurIPS, 2021.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples. In ICLR, 2015.
Zhichun Guo, Bozhao Nan, Yijun Tian, Olaf Wiest, Chuxu Zhang, and Nitesh V Chawla. Graph- based molecular representation learning. arXiv preprint arXiv:2207.04880 2022.
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurlPS, 2017.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning graphs. In NeurIPS, 2020.
Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. KDD, 2020.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In ICML, 2018
Mingxuan Ju, Yujie Fan, Chuxu Zhang, and Yanfang Ye. Let graph be the go board: Gradient-fri free node injection attack for graph neural networks via reinforcement learning. In AAAI, 2023.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In ICLR Workshop, 2017.
Guohao Li, Matthias Müller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns? In ICCV, 2019.
Guohao Li, Matthias Müller, Guocheng Qian, Itzel Carolina Delgadillo Perez, Abdulellah Abual- shour, Ali Kassem Thabet, and Bernard Ghanem. Deepgens: Making gens go deep cnns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021a.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. ICML, 2021b.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. NeurlPS, 2018
Zheyuan Liu, Chunhui Zhang, Yijun Tian, Erchi Zhang, Chao Huang, Yanfang Ye, and Chuxu Zhang. Fair graph representation learning via diverse mixture-of-experts. In WWW, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018.
Zheng Qinkai, Fei Yixiao, Li Yanhao, Hu Minhao Liu Qingmin, and Sun Qibo. Kdd cup 2020 ml track adversarial attacks and defense on academic graph 1st place solution. 2020. URL https: : / / thub. com/ Stt./nnaniiiie ianinll O wwDp__U______ 2020 10200_M.TTTr__e wtw.MMrrrrk__PPP..
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated ofxture-of-experts layer. In ICL.R, 2017.
11

---

Published as a conference paper at ICLR 2023
Yiwei Sun, Suhang Wang, Xianfeng Tang, Tsung- Yu Hsieh, and Vasant Honavar. Adversarial attacks on graph neural networks via node injections: A hierarchical reinforcement learning approach. In WWW, 2020.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh Chawla. Learning MLPs on graphs: A unified view of elfectiveness, robustness, and efficiency. In ICLR, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua
Bengio. Graph attention networks. In ICLR, 2018.
Jihong Wang, Minnan Luo, Fnu Suya, Jundong Li, Zijiang Yang, and Qinghua Zheng. Scalable attack on graph data by injecting vicious nodes. Data Mining and Knowledge Discovery, 34(5):
1363-1389, 2020.
Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. Kgat: Knowledge graph attention network for recommendation. In KDD, 2019.
Marcin Waniek, Tomasz Michalak, Michael J Wooldridge, and Talal Rahwan. Hiding individuals and communities in social network. Nature Human Behaviour, 2(2):139-147, 2018.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simpli- fying graph convolutional networks. In ICML., 2019a.
Huijun Wu, Chen Wang, Yuriy Tyshetskiy, Andrew Docherty, Kai Lu, and Liming Zhu. Adversarial examples for graph data: Deep insights into attack and defense. In IJCAI, 2019b.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. ICLR, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? In ICLR, 2019.
Han Yue, Chunhui Zhang, Chuxu Zhang, and Hongfu Liu. Label-invariant augmentation for semi- supervised graph classification. In NeurlPS, 2022.
Chunhui Zhang, Chao Huang, Youhuan Li, Xiangliang Zhang, Yanfang Ye, and Chuxu Zhang. Look twice as much as you say: Scene graph contrastive learning for self-supervised image caption generation. In CIKM, 2022.
Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heteroge- neous graph neural network. In KDD, 2019.
Xiang Zhang and Marinka Zitnik. Gnnguard: Defending graph neural networks against adversarial attacks. In NeurIPS, 2020.
Qinkai Zheng, Xu Zou, Yuxiao Dong, Yukuo Cen, Da Yin, Jiarong Xu, Yang Yang, and Jie Tang- Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learn- ing. In NewrIPS on Datasets and Benchmarks, 2021.
Dingyuan Zhu, Ziwei Zhang, Peng Cui, and Wenwu Zhu. Robust graph convolutional networks against adversarial attacks. In KDD, 2019.
Xu Zou, Qinkai Zheng, Yuxiao Dong, Xinyu Guan, Evgeny Kharlamov, Jialiang Lu, and Jie Tang. Tdgia: Effective injection attacks on graph neural networks. In KDD, 2021.
Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks on neural networks for graph data. In KDD), 201 8a.
Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks on neural networks for graph data. In KDD, 2018b.
Daniel Zügner and Stephan Günnemann. Adversarial attacks on graph neural networks via meta learning. In ICLR, 2019.
12

---

Published as a conference paper at ICLR 2023
A FULL RESULTS OF PERFORMANCE COMPARISON
We conduct extensive experiments all five datasets in Table 2. Here we display the results of graph injection scenario with Top-5 attacks v.s. Top-10 defenses plus our GAME model. Since we have chosen strong defense methods baselines, it is generally hard for attacks to be all effectives. The best performance is bolded and the runner-up is underlined.
Table 2: Main results on Graph Robust Benchmark datasets (i.e., grb-cora, grb-citeseer, grb-flickr, grb-reddit and grb-aminer). This table is a comprehensive surrl to Figure 3. Partial results are cited from GRB (Zheng et al., 2021).
Methods
SPEIT
TDGIA
FGSM-based
PGD-based
RND
w.O. attack
Avg. Acc.
GAME
86.10:10.81
86.42::0.61
86.53-0.72
87.68:1062
87.04:10.58
88,00=10.73
86.96=0.68
R-GCN.a
85,21=0.41
84.4310.27
85.60=0.38
85.01
85.36:10.4]
86.07=0.000
85.28:0.15
GAT
85.35:10.119
84.55-00.50
85.43-0.34
85,33-1.72
84.9510..58
85.57-10.00
85,20-10211
SGCNaN
75.65++0.87
77.39+1.05
81.90-0.90
81.53++..27
82.85+1.299
85.20++0000
80.75+0.41
R-GCN
79.85++0.48
74.58+1.76
76.77-0.74
76.62400.0
79.53++0.74
84.83++0.00
78.69++.27
TAGCN44
71.73++1.14
79.67=1.53
83.22-0.60
83,00=0.61
84.22+00.58
85.07=0..00
81.15=0.42
GIN
64.75::0.62
1..14=1.80
78.58-0.51
78.26:10.88
76.75
77.2411.00
75.29=0.49
4PPNPas
73.11=:0.76
68.1622.10
66.49=0.68
67.09:11.18
68.93+40.2
82.84=10.00
71.10=0040
GIN
63.05:11.37
7..51
71.30=0944
71.6410.78
74.11::0.71
74.88:10.00
70.91=:043
GATGuard
65.67=10.00
65.67=0.00
65.67=0.00
65.67=0..00
65.671:0.00
65.67:0..00
65.67=00.00
GCNaN
59,45:10.64
72.58:22.71
78.21-0.68
77 1.16
81.34:10.00
83,58:10.00
75.46:10.49
GAME
7.269+1.24
75.80
74.85 0.71
74.86+
76.96 1.04
77.86+
76.00-0.82
R-GCN.<r
71.30::0.64
7.7750078
73.63=0.50
03.58=00.6
72.95+0.68
75.03=+0.00
73.04:00117
GAT.ar
73,85+0.22
73.89+0.40
73.86-0.28
03.84
74.00+0.45
73.88+00.00
73.89+0.11
SGCN.IN
53.64:11.43
59.7622.10
58.05-0.88
58.31
75.79
75.44
60.16:20.55
R-GCN
59.32:13.65
1..69:1.77
14.07-143
14.90
60.4310.71
73.16:10.09
59.93-0.57
TAGCNA
51.09:6.38
46.8714.99
13.
64.25
73.06110.58
73.56110.00
62.04+1..00
GINaN
52.82-40093
52.16++2.47
61.36-1.03
60
63.66-L0.43
66.04-10.00
59 99.944-0.33
SAGE
71.62:+1.39
72.25:0062
72.40-0.0
72.38:1..54
72.70:0.44
03.35:10.00
72.45:0.117
GIN..r
44.71+99.2
62.540.8 0.81
63.21+1.19
65.24+0.51
0..02+00.0
60.42+1.30
GATGuard
67.08:10.000
67.00 67.08=0.00
67.08=0.00
67.08=0.00
67.08
67.08=10.00
67.08=#0.00
GCNGuard
64,54:+0.13
64.58+0000
6..56-0.04
64.46++0.16
64.55++0.16
6.05
64.54++0.04
GAME
50.54:10.67
52.40=1002
52.59-1.20
52.62-10.14
52.63-1,877
53.83:10.47
02.44:10.73
R-GCN.a
47.14:10.13
98.07=0.05
50.22-0.11
50.20-10.88
49,3244.10
02.28810.06
49,
GAT
46.39110.13
512510.07
50.62-0.12
50
51.0910.08
5..66:00000
50 20.00303
SAGEaN
47.62:10.00
47.13+0.07
47.56-0.07
47.58=1.08
47.47110.77
47.47-10.00
47.47-10.03
gr Be
SAGE
48.17-+0.14
49.52+0.05
43.09-0.15
43.10+0.14
49.18+00.77
50.52++0000
47.26+0.04
GCN.NT
43.52-40.07
47.54+0.06
48.58-0.09
48.5840.09
51.00++0.09
51.53-10.00
0.02
GIN-IN
46.18-10.09
49.05=00055
46.94=0.10
46.94=0.07
48.58++0.06
48.31=00.00
47.67=0.03
SAGE
43.66:100.07
50.6310.08
43.32-0.14
43.33-10.19
49.00110.08
50.79-10.00
46.7910.06
GIN.
43.73:10.05
43.77=0.01
45.24=0.10
45.28-0.11
44 44.36:00.03
45.23:10.00
44.60=0.03
GAT
49.91-10117
5..46=0.09
42.90-0244
42.7810.27
50.8410.20
50.12-10.00
48.1710.09
APPNPAN
42.30:10.01
42.30=0.00
44.14=0.07
44.1620.099
42.3110.001
44.23-10.00
43.24-1.001
GAME
96.45:10.20
96.03=10.45
96.22-0.28
96.6910.38
95.73100.7
96.78110.32
96. -001
GIN-x
96.31+00.22
95.92+
9648-0.02
96.88+0.01
9..60-0.01
06.68+0000
06.41:10.5
TAGCNA4
96.31+00.02
95.89=0.01
95.91=0011
95.90=00011
96.30-0.002
96.37+00.00
96.1 11000
TAGCN-<7
95.77+00.02
93.12+0.01
96.74-0011
96.74+0.01
96.54+00.01
96.77+0.00
95.14++001
GAT
93.76:10.02
93.7320.01
93.91-0.02
93.91
93.8410.02
94.15:0.000
93 93.88:10.01
R-GCN
93.59110.01
93.08:1001
93.56-0011
93.57-0.01
93.23440.2
93.78.10.00
93.47-10.01
TAGCN
93.7610.03
91.7710011
95.24-0.03
95.2410.02
94.8810.03
95.3910.00
94.38-10.01
GCNaN
95,57-10.01
91.0910.00
95.23-0.02
95.22-10.01
95.63-10.01
95.68-10.00
94,74-10.01
SAGE
90.21110.02
90.1620.01
90.54-0.22
90.5410.02
90.37110:3
90.48:10.00
90.38-10.01
SAGE
92.96-+0.04
85.98+0.03
93.75 0.02
93.78+0.03
94.59+00.2
95.16++0.00
9..70++0001
SGCNAN
87.02+00.03
86.61
88.99-0.03
80.03
87.72+00.3
90.15:+0.00
88.25=00011
GAME
69.43::11.0
1..21
69.21-0.69
09633
69.03-10.72
70.15:10.95
69.22:10.88
GAT
68.04:10.03
67.69=0.03
68.01-0.02
68.00=00.22
67.72+00.04
67.93-10.00
67. 67.90=+0.01
R-
64.05-10.04
63.62-0.32
0.041=0001
10.02
64.98
65.76+10.00
64.8740.05
SGCNAN
64.84-10.04
62.20=0015
0.03
65.5411004
65.31110.04
66.68-10.0
65.02-40.03
R-GCN
64.06:10004
61.9910.22
65.05-0.03
65.05:10.44
64.4511.004
65.85.00.00
64,41-10.04
GCNNN
65.51++0.02
60.38+146
66.22-0.02
66.2240.02
66.17-00.02
66020+0000
65.12+0..25
GATaN
64.02-10.04
59.69=11.7
6..49-0.04
66.50-00.6
67.54.10.04
68.4--0.0
65.45-10.26
GIN-IN
63.11+-0.02
59.59=0.42
64.63-0.44
64.65=0.044
64.36+00066
65.59++000
63.65++0.07
TAGCNA1
62.59-10.04
59.06211.75
64.82-0.04
64.82200.33
64.3310.03
64.91:10.00
63 63.42-1.229
TAGCN-<r
63.77+10.06
57.24=5.04
66.32-0.02
66.34+0.03
66.42+4.03
67.08++0.00
64.53-+0.44
GAT
63.5811.06
56.6316.75
66.14=0.04
66.1510.06
66.23310.04
68.02-10.00
64.46.11.13
13

---

Published as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
B. REPRODUCIBILITY SETTINGS OF ATTACK METHODS
We evaluate the methods using five most common graph attack methods provide by GRB bench- mark (Zheng et al., 2021), including random, fast gradient sign method, projected gradient descent, SPEIT, and topological defective graph injection attack. The details these five attack methods are as follows:
RND (Random): method that injects random noises generated by Gaussian distribution (Zügner et al., 2018a).
FGSM (Fast Gradient Sign Method); a method that linearizes the loss function around the cur- rent value of parameters to get an optimal max-norn constrained perturbation (Goodfellow et al., 2015).
PGD (Projected Gradient Descent): first-order adversary method that generates strongest assault using local first-order information about the network (Madry et al., 2018).
SPEIT: winning solution of KDD-CUP 2020 Graph Adversarial Attack & Defense competition, which is a global black-box graph injection attack with adversarial adjacent matrix and feature gradient attacks (Qinkai et al., 2020).
TDGIA (Topological Defective Graph Injection Attack): a powerful graph injection attack that injects malicious nodes progressively around topologically vulnerable nodes in the graph (Zou et al., 2021).
Following the prior work (Zheng al., 2021), using vanilla GCN as the surrogate model brings more transferable and better black-box node injection attack effects than other GNN models. There- fore, we choose GCN the surrogate model for all attacks in our experiments.
B.2 HYPER PARAMETERS AND ADVERSARIAL TRAINING DETAILS
The hyper-parameters of GAME shown in Table 3. The Hyper-parameters for adversarial training used in DECoG are included in Table 4. We then show the the adversarial training (AT) procedure of GAME as follows:
Step Y Initialization: The warm-up step. The training graph is utilized to train GAME model for few iterations.
The PGD attack is employed to inject the malicious nodes and edges that assault the training nodes by message passing and create an attacked graph.
Step 2 PGD attack:
Step @ Update gradients: The model parameters are updated based on the gradients that are calcu- lated by training on the attacked graph and minimizing the node classification loss.
This adversarial training procedure is repeated until the training loss converges. Finally, we can obtain GAME model with better learning capability and outstanding robustness. According to Table 2, we intriguingly discover that GAME with PGD can also defend against other types of attacks, demonstrating the superior generality and applicability of GAME.
Step + Repetition:
Notably, we discover that the GAME layer's favorable attribute, i.e., sparsely activating partial ex- perts, can be employed to design DECOG training strategy, which augments more diverse graphs. Specifically, in the Step @ and Step @, we activate different portions of experts in GAME model, de- spite the fact that both steps are performed in the same epoch. The code is provided this anonymous link In addition, the suggested GAME is scalable to graphs of varying sizes. We mini-batch training via neighborhood sampling with batchsize to train GAME on grb-reddit and grb-aminer datasets. In the future, will transfer the framework of GAME other GNN models.
14

---

Published as a conference paper at ICLR 2023
Table 3: Hyper-parametesn of GAME for grb-cora, grb-citeseer, grb-flickr, grb-reddit and grb- aminer datasets. The and k represent the number of total experts and activated experts in each layer, respectively. Note that during generating adversarial samples, we activate all experts. Noisy rate controls the randomness when the gate module activates the partial experts during minimizing the loss.
Model
Datascts
π
k
Hidden sizes
LR
Dropout
Optimizer
Noisy rate
grb-cora
2
64, 64,
0.01
0.5
Adam
lc-2
grb-citeseer
2
64, 64, 64
0.01
0.5
Adam
le-1
GAME
grb-flickr
3
128, 128, 128
0.01
0.5
Adam
le-1
grb-reddit
3
128, 128, 128
0.01
Adam
le-2
grb-aminer
3
2
64, 64, 64
0.01
0.5
Adam
le-1
adversarial training in DECOG for grb-cora, grb-citeseer, grb-flickr, grb-reddit and grb-aminer datasets. Noisy rate controls the randomness when the gate module maximizes the loss and activates the partial experts. Nodes represents the number of injected nodes, and Edges indicates the number of added edges.
Table 4:
Model
Datasets
Attack
Step size
Iter.
Nodes
Edges
Feature range
Noisy rate
grb-cora
PGD
0.01
10
20)
20
[-0.94, 0.94]
le-1
grb-citeseer
PGD
0.01
10
30
20
[-0.96, 0.891
le-1
GAME
grb-flickr
PGD
0.01
10
200
100
[-0.47, 0.99]
2c-1
grb-reddit
PGD
0.01
10
500)
200
[-0.98, 0.991
le-1
grb-aminer
PGD
0.01
500
100
[-0.93, 0.93]
2e-1
Table 5: Statistics of five GRB datasets covering from small- to large-scale graphs.
Feat. Range (original)
Feat. Range
Datasct
Scale
#Nodes
#Edges
#Fcat.
#Classes
(normalized)
grb-cora grb-citeseer
Small
2,680
5,148
302
7
[-2.30, 2.40]
[-0.94, 0,94]
Small
3,191
4,172
768
6
[-4.55, 1.67]
[-0.96, 0.89]
grb-flickr
Medium
89,250
449,878
500
7
[-0.90, 269.96]
[-0.47, 1.00]
grb-redes rebreddit
Large
232,965
11,606,919
602
41
1-28.19, 19, 120.96) 96]
[-0.98, 0.99]
grb-aminer
Large
659,574
2,878,577
100
18
[-1.74, 1.62]
[-0.93, 0.93]
C STATISTICS OF GRAPH ROBUST BENCHMARK DATASETS
We evaluate our proposed GAME as well as adversarial learning framework on five real-world GRB datasets (Zheng et al., 2021), spanning from small- to large-scales. The data statistics are displayed Table 5. To utilize grb-cora, grb-citeseer, grb-flickr, grb-reddit, grb-aminer, we apply the tool provided by Graph Robustness Benchmark
D GRADIV'S EFFECTIVENESS ON DIVERSITY OF NODE REPRESENTATIONS
To discuss GRADIV'S impact on nodes' representation diversity, on grb-cora dataset, we visualize the node representations generated by GAME without GRADIV and GAME, shown in Figure 8. We observe that the node representations generated by GAME without GRADIV are more entangled and intertwined with each other then GAME. This phenomenon demonstrates that GRADIV able to learn more distinguishable node representations.
E ADVERSARIAL GRAPHS GENERATED BY GAME ON GRB-FLICKR
We observe that the adversarial graphs generated from GCN are similarly distributed to the clean graph in Figure (i.e..the red and black dots in the lower left corner of the GCN figure overlap signif- icantly, but in the GAME figure the overlap is minor). On the contrary, GAME generates adversarial
1 ctwll...coo///o 2oom//amm2ccode 'https: : //t Phttps://oe thub. com/ ttt.m/grbdbrbbo
15

---

Published as a conference paper at ICLR 2023
(a) Ours w.o. GraDiv
(b) Ours
Figure t-SNE for node features in grb-cora.
0.100
0.100
attack clean
attack
0.075
0.075
clean
var
var
0.050
0.050
0.025
0.025
0.000
0.000
1.0
-0.5
0.0 mean
0.5
1.0
1.0
0.5
0.0 mean
0.5
1.0
(a) GCN
(b) Ours
Figure 9: Distributions for clean and adversarial graphs created by GAME (b) and adversarially trained GCN (a). For fair comparisons, we explore the same setting for both models on grb-flickr.
graphs whose distribution is statistically distinct from that of the clean graphs, further demonstrating the effectiveness of DECOG and GRADIV. Diverse experts enable GAME to learn distinguishable node representations for robust performance, which significantly mitigates the GNN's training diffi- culties on the graphs with distinct distributions.
F EVALUATION ON EDGE MODIFICATION ATTACK
We apply the proposed GAME framework into Hii--eedoid/Sff-MMedia GDCs (Geisler et al., 2020; 2021). ). Note that the edge Modification attack (Geisler et al., 2020; 2021) are also crucial for assess- ing GNN's robustness. Therefore, we present experimental results in Table 6, which are based on three random splits for rigorous comparisons. We observe that GAME enhances their accuracy.
Table 6: The robust accuracy of Soft-Median GDC and Moft-Medoid GDC without or with our GAME framework the Cora dataset with the global attacks (GR-BCD & PR-BCD, proposed by Soft-Medoid GDC (Geisler et al., 2021). We set the number of experts to 10 and the hidden units of each expert 32. We them on three random splits and report the mean and standard error results.
0.1)
GR-BCD (Geisler al., 2021)
PR-BCD (Geisler et al., 2021)
GCN (Kipf & Welling, 2017)
0.622 0.003
0.645 : 0.002
GDC (Gasteiger al., 2019b)
0.677 0.005
0.674 # 0.004
PPRGo (Bojchevski et al., 2020)
0.726 0.002
0,700 + 0.002
SVD GCN (Entezari et al., 2020)
0.755 + 0.006
0.724 + 0.006
Jaccard GCN (Wu et al., 2019b)
0,664 + 0.001
0.667 + 0.003
RGCN (Zhu et al., 2019)
0.665 0.005
0.664 + 0.004
Soft-Median GDC (Geisler al., 2020)
0.765 0.001
0.752 0.002
Soft-Medoid GDC (Geisler al., 2021)
0.775 0.003
0.761 + 0.003
Soft-Median GDC (+GAME)
0.772 0.005
0.759 + 0.005
Soft-Medoid GDC (+GAMB)
0.780 + 0.007
0.772 + 0.006
16


Reviews:
[{"review_id": "REE9OGIPiVA", "reviewer": "Reviewer_SpxS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method for robust graph neural networks. The proposed model contains a modified message-passing layer, PGD-based adversarial training, and an additional loss term encouraging diverse node representations. The authors compare their method on several datasets using the GRB benchmark suite.\n", "summary_of_the_review": "I have concerns (i) regarding the significance of the individual contributions of the method, (ii) about the lack of important baselines, and (iii) about the experimental setup. Unless these are resolved, I would not recommend publication. \n", "strength_and_weaknesses": "Strengths:\n* Effective adversarial training methods for graph neural networks are rare, so this is an important research area.\n* The authors compare to a large number of baselines.\n* The authors report strong experimental results.\n\nWeaknesses:\n* The authors do not compare to important defense baselines (Soft-Medoid/ Soft-Median [Geisler et al. 2020/2021]) and attacks (PGD attacks [Xu et al. 2018])\n* On some datasets there are only n=2 \"experts\", raising doubts about the usefulness of this contribution.\n* The experiment setup is not sufficiently clear (see next section).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NgA40ifpNA", "reviewer": "Reviewer_QLnJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper identifies a principal difficulty of training robust GNNs that existing methods ignore: the convoluted mixture distribution between clean and attacked data samples. To tackle the problem, the authors propose GAME, a novel method for learning robust graph representations from three perspectives. Extensive experiments and analyses demonstrate GAME's superiority over state-of-the-art methods.\n\n", "summary_of_the_review": "By proposing a novel framework comprised of multiple components, this paper addresses three significant difficulties in learning the difference between standard and attacked graphs. The technical contributions are substantial, the motivations are supported by rigorous explanations, and the experimental results are sufficient. ", "strength_and_weaknesses": "\nStrengths:\n+The motivation for activating different experts during adversarial loss by PGD is interesting and efficient in model defense. Additionally, the authors recognize and tackle one fundamental issue (mixture distribution of standard and attacked graphs) in adversarial graph learning, which prevents the traditional GNNs from performing well on both adversarial and clean samples. The motivation of this paper is strong, especially as the number of attack methods increases.\n\n+The developed GAME framework is novel. There are three components (GAME layer, DECOG training, and GRADIV regularization) to address each identified problem (model’s generality, training efficiency, and output diversity) correspondingly. The introduced components are not simply combined, but instead form a complementary chain that serves as the overall framework.\n\n+The authors provide a good catch on why graph adversarial learning is challenging for GNNs, which is important and insightful. I am convinced by these statements, e.g., figure 1 presents a clear distribution difference between standard graphs and attacked graphs. The authors also draw the figure in the experiment section to examine the generality of GAME with respect to the difference, which is interesting.\n\n+Extensive experiments and ablation studies are organized to evaluate the proposed GAME in this paper. For instance, the authors evaluate the model in both the attacking setting with five different attack methods and the standard setting with original graph inputs. The proposed model is further validated on different GRB datasets and shows significant results. The results are impressive. The proposed model exhibits state-of-the-art robust accuracy across a variety of attack evaluations and dataset sizes. The proposed model demonstrates remarkable results and superior performances to other methods, which suggests that GAME can be utilized in a wide variety of real-world applications with respect to both performance and reliability. \n\n\nWeaknesses and Questions:\n-Some experimental discussions are not clear for those who are unfamiliar with adversarial learning. For example, in the introduction section, “a progressively larger divergence between the two distributions is observed” is stated in Figure 1’s caption. What does this large divergence mean? The authors should provide more information to explain the definition of divergence or make clarification what divergence means in this experiment.\n\n-According to Section 3, the authors state, \"In general, DECOG enables the dynamic activation of each expert in GAME and facilitates the computation of more diverse attacked graph adjacency matrix and node features.\" I'm curious about the GAME model's diversifying effects with the incorporation of MoE. Will the adversarial training with GAME layers contribute more diverse attack samples than standard GNNs? Will the observation in Section 5.5 still be established on a larger dataset?\n\n-Will a bigger number of activated experts produce a more diverse effect? More discussions are required regarding the activated expert number.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Gy7OMWg4ai", "reviewer": "Reviewer_iMjk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the adversarial training of robust graph neural networks. The motivation is attractive to me, namely, the divergence of distribution between clean graphs and attacked graphs. This motivation leads to the GAME model, a set of effective all-round designs: leveraging MoE module to construct a more powerful GNN with better capacity, using adversarial training to produce more different attacked samples in the MoE-based GNN, and adding regularization $R_{GraDiv}$ to diversify prediction for more robustness.\n", "summary_of_the_review": "The paper is interesting and well-motivated. The proposed model is novel and the experiments are sufficient. There are some minor concerns that need to be addressed. \n", "strength_and_weaknesses": "Strength:\n\n1. The paper has a clear motivation: the authors take a unified investigation for two clean data and attacked in graph adversarial training. Both are shown to shift to different distributions, but their current GNN-based robust methods do not aware. So, the authors become the first to design a GAME model for unification between the two distributions is necessary to gain robustness.\n\n2. The technical approach is solid: The authors first proposed GAME model to achieve diversity and robustness by adversarial training Graph Mixture of Experts, which has never been used in graph adversarial learning. Because MoE generates different adversarial noise by activating different sub-networks to cooperate with PGD, the GAME model can obtain richer adversarial samples in the same training cycle. Considering that MoE makes the number of parameters larger, to avoid redundancy of features, the authors also designed the regularization module to take full advantage of the different outputs of each expert.\n\n3. Experimental results are SOTA: The GAME model yields significantly improved adversarial robustness, outperforming strong previous methods among small, middle, and large graph datasets. Ablation experiments and analytical experiments are convincing: The authors compared MoE, DECOG, and GRADIV regularization's improvements on robust node classification tasks. The results confirm that MoE, DECOG, and GRADIV regularization are three complementary dimensions, and their proper unification strategy is superior to other current alternatives. The authors also use analytical experiments to reveal that mixed distributions are the important reason for the difficulty of graph adversarial training and that the proposed GAME model handles such mixed distributions well.\n\n4. The paper is well-written and easy to follow. The setup section, appendix section, and anonymous codes provide sufficient details for the implementation of this work.\n\n\nWeakness:\n\n1. I am not fully convinced why GAME model's computational cost can approximate the normal GNN model.  There was no computational complexity analysis or complexity comparisons with baseline models. Please address this concern.\n\n2. Figure 1 illustrates the GNN trained with clean graphs and the GNN trained with attacked graphs have different representation distributions. Then in Figure 4, the authors state that GAME model can unify clean graphs and attacked graphs. Is this figure plotted based on GAME models (e.g., one GAME model trained with clean graphs and one GAME model trained with attacked graphs)? It seems difficult to obtain enough diversity and enough hardness simultaneously. Please provide more exact descriptions to clarify and support your claim.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CaYDpgh-WIl", "reviewer": "Reviewer_jUru", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to defend against evasion attacks in node-level graph learning. The authors introduce a Graph Adversarial Mixture of Experts (GAME) framework for learning robust representation in an adversarial training manner (e.g., PGD). In particular, GAME combines PGD training and the Mixture of Experts mechanism for ensembling adversarial gradients from different sub-models to strengthen the attack on the graph. The experiments further show that this unified design of GAME can be implemented as efficiently as normal GNN adversarial training but offers impressive advantages over prior work while under different graph attack evaluations.", "summary_of_the_review": "This paper proposes to study graph adversarial learning. The proposed GAME method is novel and well-motivated. The extensive experiments practically support the claims in the paper and answer the questions raised with motivation. In summary, this is good and solid work. I tend to accept this paper.\n\n", "strength_and_weaknesses": "Strength:\n1. This paper initially discovers an interesting distribution difference between common graph learning and adversarial graph learning. This difference also is one of the main reasons which makes adversarial graph learning become difficult. This discovery is interesting. \n\n2. The proposed model couples adversarial training with the Mixture of Experts mechanisms. This joint design offers stronger adversarial attack samples for activated experts since the current adversarial noises are generated from other different inactivated experts. The model is relatively new and novel to me. \n\n3. The authors provide comprehensive thoughts for stronger graph robustness. The investigation of model, optimization and training contributes to a thorough study. Also, the paper provides a deep and meaningful thought on the correlation between MoE and PGD.\n\n4. Comprehensive evaluation across multiple scales of graph robust benchmark datasets are conducted. GAME outperforms many baseline methods in terms of robustness and accuracy. The authors also provide an in-depth analysis of GAME's capacity to mitigate the distribution difference between standard graph learning and adversarial graph learning. The presentation of this paper is overall good. \n\nWeakness:\n1. I tend to think that GAME has the potential for more general use, such as enhancing adversarial training on general machine learning. Since this paper is specifically focusing on graph adversarial training, authors need to clarify why GAME is designed for graph tasks (e.g., the exact benefits of GAME on graph tasks).\n\n2. The authors provide empirical explanations about why the computational cost of the proposed GAME is close to the common GNN while this paper has no mathematical estimations about GAME's computation and the baseline GNN. Authors should provide better complexity analysis for a more convincing claim on GAME's efficiency. \n\n3. The connection between the $R_{Gradiv}$ and backbone design is not very clear. Although experiments say that the gradiv regularization improves the model's performance, this regularization operation causes extra human-craft efforts in adjusting the hyperparameter.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster