Graph Neural Networks with Adaptive Residual
Authors: Xiaorui Liu
Jiayuan Ding
Wei Jin
Han Xu
Yao Ma
Zitao Liu
Jiliang Tang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Neural Networks
Adaptive Residual
Abnormal Features

Abstract: Graph neural networks (GNNs) have shown the power in graph representation learning for numerous tasks. In this work, we discover an interesting phenomenon that although residual connections in the message passing of GNNs help improve the performance, they immensely amplify GNNs' vulnerability against abnormal node features. This is undesirable because in real-world applications, node features in graphs could often be abnormal such as being naturally noisy or adversarially manipulated. We analyze possible reasons to understand this phenomenon and aim to design GNNs with stronger resilience to abnormal features. Our understandings motivate us to propose and derive a simple, efficient, interpretable, and adaptive message passing scheme, leading to a novel GNN with Adaptive Residual, AirGNN. Extensive experiments under various abnormal feature scenarios demonstrate the effectiveness of the proposed algorithm. 

Graph Neural Networks with Adaptive Residual
Xiaorui Liu'
Jiayuan Ding' dingj ia5@msu. edu
Wei Jin 28msu. jinwei28msu.
xiaorui Casuuummmuue.emmm edu
j jinwei
edu
Han Xu' xuhan1 @ms11mmmu...m edu
Yao Ma² yao www.mmaaiiiiee...e
Zitao Liu
Jiliang Tang' tangj ili @msu. edu
liuzitao@100tal.
Michigan State University, East Lansing, MI, USA New Jersey Institute of Technology, Newark, NJ, USA STAL Education Group, Beijing, China
Abstract
Graph neural networks (GNNs) have shown the power in graph representation learning for numerous tasks. In this work, we discover an interesting phenomenon that although residual connections in the message passing of GNNs help improve the performance, they immensely amplify GNNs' vulnerability against abnormal node features. This is undesirable because in real-world applications, node features in graphs could often be abnormal such as being naturally noisy adversarially manipulated. We analyze possible reasons to understand this phenomenon and aim to design GNNs with stronger resilience to abnormal features. Our understandings motivate us to propose and derive a simple, efficient, interpretable, and adaptive message passing scheme, leading novel GNN with Adaptive residual, AirGNN Extensive experiments under various abnormal feature scenarios demonstrate the effectiveness of the proposed algorithm.
Introduction
Recent years have witnessed the great success of graph neural networks (GNNs) in representation learning for graph structure data [1] Essentially, GNNs generalize deep neural networks (DNNs) from regular grids, such as image, video and text, to irregular data such as social, energy, transportation, citation, and biological networks. Such data can be naturally represented as graphs with nodes and edges. The key building block for such generalization the neural message passing framework 2
k-1) UPDATE (k) (x!!) (M))
(1)
where Xa E Rd denotes the feature vector of node u in the k-th iteration of message passing, and m (k)
MN(u is the message aggregated from u's neighborhood (u). The specific design of message
passing scheme can be motivated from spectral domain 3 4 spatial domain 5
7
It usually
linearly smooths the features a local neighborhood on the graph.
GNNs have achieved superior performance in large number of benchmark datasets 8 where the node features are assumed to complete and informative. However, in real-world applications, some node features could be abnormal from various aspects. For instance, in social networks, new users might not have complete profile before they make connections with others, leading to missing user features. In transportation networks, node features can be noisy since there exist certain
TTe implementation is available alttppe//ggthaaa.c/////////////
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

uncertainty and dynamics in the observation of the traffic information. What is worse, node features can be adversarially chosen by the attacker to maliciously manipulate the prediction made by GNNs. Therefore, it is greatly desired to design GNN models with stronger resilience to abnormal node features.
In this work, we first perform empirical investigations on how representative GNN models behave on graphs with abnormal features. Specifically, based upon standard benchmark datasets, we simulate the abnormal features by replacing the features of randomly selected nodes with random Gaussian noise. Then the performance of node classification on abnormal features and normal features are examined separately. From our preliminary study in Section 2 we reveal two interesting observations: (1) Feature aggregation can boost the resilience abnormal features, but too many aggregations could hurt the performance on both normal and abnormal features; and (2) Residual connection helps GNNs benefit from more layers for normal features, while making GNNs more fragile to abnormal features. We then provide possible explanations to understand these observed phenomena from the perspective of graph Laplacian smoothing. Our analyses imply that there might exist an intrinsic tension between feature aggregation and residual connection, which results in performance tradeoff' between normal features and abnormal features.
Motivated by these findings and understandings, we aim to design new GNNs with stronger resilience abnormal features while largely maintaining the performance on normal features. Our contributions can be summarized as follows:
We discover an intrinsic tension between feature aggregation and residual connection in GNNs, and the corresponding performance tradeoff between abnormal and normal features. We also analyze possible reasons to explain and understand these findings.
We propose a simple, efficient, principled and adaptive message passing scheme, which leads to a novel GNN model with adaptive residual, named as AirGNN.
Extensive experiments under various abnormal feature scenarios demonstrate the superiority of the proposed algorithm. The ablation study demonstrates how the adaptive residuals mitigate the impact of abnormal features.
2 Preliminary
Before introducing the preliminary study, we first define the notations used throughout the paper.
Notations. We use bold upper-case letters such as X to denote matrices. Given a matrix X E Rxx we use X; denote its i-th row and X.j to denote its element i-th row and j-th column. The
Frobenius norm and to norm of matrix X are defined |X|]|
F Viig
X2 and |X|
Σ. IX 112 = E. VI, E, X respectively. We define | = Tmax (X) where Tmax (X) is the largest singular value of X.
Let G = (V.E} be a graph with the node set V = (ei...., Un} } and the undirected edge set E = fei....., }. We use Nw;)) to denote the neighboring nodes of node u, including U; itself. Suppose that each node is associated with d-dimensional feature vector, and the features for all nodes denoted as X Raxd The graph structure G can be represented as an adjacent matrix E RRxD, where Aij = when there exists an edge between nodes U; and 0j. and Ai O otherwise. The graph Laplacian matrix is defined L D A, where D is the diagonal degree matrix. Let us denote the commonly used feature aggregation matrix in GNNs [3] as A = D AD where Å = A + I is the adjacent matrix with self-loop and its degree matrix is D. The corresponding Laplacian matrix is defined as I-I-À.
In this work, focus the setting where subset of nodes in the graph contain abnormal features, while the remaining nodes have normal features. In the remaining of this paper, we use abnormal/normal
features featurr
denote nodes with abnormal/normal features, for simplicity.
2.1 Preliminary Study
Experimental setup. To investigate how GNNs behave on abnormal and normal node features, design semi-supervised node classification experiments on three common datascts (i.c., Cora,

---

CiteSeer and PubMed), following the data splits in the work [.. Moreover, we simulate the abnormal features by assigning 10% of the nodes with random features sampled from standard Gaussian distribution. The experiments are performed on representative GNN models covering coupled and decoupled architectures, including GCN R GCNII (91, APPNP IQ] and their variants with or without residual connections in feature aggregations, denoted as w/Res and wo/Res. All methods follow the hyperparameter settings in their original papers. We examine how these models perform when the number of layers increases. Note that for the decoupled architectures such as APPNP, we fix the 2-layer MLP and increase the number of propagation layers. While for the coupled architectures such GCN and GCNII, we increase the number of feature transformation and propagation layers simultaneously. We report the average performance over 10 times of random selection of the noise node sets. The node classification accuracy (mean and standard variance) nodes with abnormal separately.
and normal features is illustrated in Figure T and Figure. 2
0.11
0..5
3.40
0.65
APPNP wiles
GCNI whos
GCN wiNes
0.69
wullis
0.30
GChil workes
9.35
GCN woftes
0.55
0.59
0.25
0.34
0.45
0.48
0.35
0.39
0.25
0.21
0.35
0.38
Sumberrat of tayere
Number of layaer
Number of layens
(a) APPNP
(b) GCNII
(c) GCN
Figure I: Node classification accuracy on abnormal nodes (Cora)
6.7
6.6
0.5
0.4
APPNP wiltss
CCNI WATRe OCHI wathes
GIN wiles
APPNP worfees
GCN wo.fus
0.5
32
34
14
0.2
3
10
12
34
16
Number layers
Number lapers
Number layera
(a) APPNP
(b) GCNII
(c) GCN
Figure 2: Node classification accuracy on normal nodes (Cora)
Observations. From Figure I and Figure 2 we can make the following observations: (1) Without residual connection, more layers (e.g.. > for GCN and GCNII, > 10 for APPNP) hurt the accuracy on nodes with normal features. However, more layers boost the accuracy nodes with abnormal features significantly, before finally starting to decrease; (2) With residual connection, the accuracy on nodes with normal features keeps increasing with more layerF] However, the accuracy on nodes with abnormal features only increases marginally when stacking more layers, and then starts to decrease. While we only present the experiments on Cora, we defer the results on other datasets to Appendix C which provide similar observations. To conclude, we can summarize these observations into two major findings:
Finding I: Feature aggregation can boost the resilience to abnormal features, but too many aggregations could hurt the performance on both normal and abnormal nodes;
Finding II: Residual connection helps GNNs benefit from more layers for nodes with normal features, while making GNNs more fragile to abnormal features.
'GCN w/Res is an exception because its residual is not appropriate, which is consistent with the experiments in the work (3).
3

---

2.2 Understandings
In this subsection, we provide the understanding and explanation for aforementioned findings, from the perspective of graph Laplacian smoothing.
Understanding Finding I: Feature aggregation Laplacian smoothing
The message passing in GCN [3, GCNII wo/ residual and APPNP wo/ residual (as well as many popular GNN models), follows the feature aggregation
X AX
where Xn and Xout represent the features before and after message passing layer, respectively. It can be interpreted one gradient descent step for the Laplacian smoothing problem [I]]
arg min (i( = J x (I Á)x XERaXA
=
E (NAA)))E
vd.
X
+
Vd,
X,
(3)
where d, is the node degree of node oz Eq. (2) can be derived from X Xin -I- AXin.
with the initialization X Xn and stepsize 7 The Laplacian smoothing problem penalizes the feature difference between neighboring nodes. To reduce this penalty, the feature aggregation in Eq. 2 smooths the node features by taking the average of local neighbors, and thus can considered as low-pass filter which gradually filters out high-frequency signals 12 [3] Therefore, it increases the resilience to abnormal features which are likely be high-frequency signals. In other words, the local neighboring nodes help to correct the abnormal features. Unfortunately, if applied too many times, these low-pass filters could overly smooth the features weell-known as oversmoothing [4 [5] such that nodes are not distinguishable enough, providing an explanation to the degraded performance on both abnormal and normal features when stacking too many layers.
Understanding Finding II: Residual connection maintains feature proximity
To adjust the feature smoothness for better performance, APPNP [0] utilizes residual connections in message passing as follows
x = =1-1=(1-m (1- a)xa + aXin
(4)
where X
Xin. It can be considered as an iterative solution for the regularized Laplacian smoothing
problem
arg min L2(X)= 2(1 - ax) XeRaxd
X x 11 + 1 x (I- A)x).
(5)
with initialization X X and stepsize & = a due
x (1 a) (1-- (x* - - = (1- +aXi-
GCNII (9] adopts similar message passing but further combines feature transformation layer in each message passing step, which leads to coupled architecture, as contrast to the decoupled architecture of APPNP. The residual connection naturally arises when regularizing the proximity between input and output features, as showed in the first term of (X). Such proximity can help avoid the trivial solution for the problem in Eq. (3). i.c., totally oversmoothed features only depending on node degrees, and consequently mitigates the oversmoothing issue. More intuitively, residual connections in GNNs provide direct information flows between layers that can preserve some necessary high-frequency signals for better discrimination between classes. More layers with residual provide more accurate solution to Eq. (5). which explains the performance gain from deeper GNNs. Unfortunately, these residual connections also undesirably carry on abnormal features which are detrimental, leading to the inferior performance on abnormal features.
3 The Proposed Framework
In this section, we first motivate the proposed adaptive message passing scheme (AMP) with further discussions on our preliminary study. We then introduce more details about AMP, its interpretations, convergence guarantee and computation complexity, as well as the model architecture of AirGNN.

---

3.1 Design Motivation
Our preliminary study in Section reveals an intrinsic tension between feature aggregation and residual connection: (1) feature aggregation helps smooth out abnormal features, while it could cause inappropriate smoothing for normal features; (2) residual connection is essential for adjusting the feature smoothness, but it could be detrimental for abnormal features. Although this conflict can be partially mitigated by adjusting the residual connection such as the residual weight a in GCNII and APPNP [10], such global adjustment cannot be adaptive to subset of the nodes, e.g., the nodes with abnormal features. This is crucial because in practice we often encounter the scenario where only a subset of nodes contain abnormal features. Therefore, how to reconcile this dilemma still desires dedicated efforts. We then naturally ask question: Can we design a better message passing scheme with node-wise adaptive feature aggregation and residual connection?
The motivation of the proposed idea builds upon the following intuition: maintain the proximity between input and output features as in Eq.
while it is important to (5). it could be over aggressive to
penalize their deviations by the square of Frobenius norm, i.e., X in = - ||; - (X 1n):ll
The fact that this penalty does not tolerate large deviations weakens the capability to remove abnormal features through Laplacian smoothing. This motivates us consider an alternative proximity penalty
X Xm
Σ
IX: (X2.)
(6)
which instead penalizes the deviations by the & norm of row-wise le norms, namely l21 norm. The lei norm promotes row sparsity in X - Xin and it also allows large deviations because the penalty on large values is less aggressive, lcading to the potential removal of abnormal features. Therefore, propose the following Laplacian smoothing problem regularized by f21 norm proximity control:
arg min X|X - 21 + tr (I -)))).
XeRuxd
where À E [0, xo) is a parameter to adjust the balance between proximity and Laplacian smoothing. In order easy the tuning of A, we made modification of Eq. 7
arg min
((X) = A||X - Xinl/21 + A)tr(X T (I ()x),,
(8)
Xf
where À E [0, 1]
controls the balance.
3.2 Adaptive Message Passing
Y
X
Aggregution
[.S.}
Residual
Xk+1
X
Figure 3: Diagram of Adaptive Message Passing
C(X) is composite objective with non-smooth and smooth components. We optimize by proximal
gradient descent [16] and obtain the following iterations as the adaptive message passing (AMP):
Y x 27(1 A) A (1
2- x)) + -1)Ax)
x k+1 arg
argmin min {NlKX Xin 2| + x
1
X Y 1
(10)
2
where X rewritten
Xin and 7 is the stepsize to be specified later. Let Z = X Xin- and Eq. 10 can be
1
z arg min
A +
1 (Y Xin) |1
z
2 x,))
= =prox -ll- (Y*
(11)
X k+1 = X + ++1
(12)
5

---

The i-th row of the proximal operator in Eq. (11 can be computed analytically
X max
Y>
(prox,A H1a1 (X));
X la yd, 0) max(1
0)
X,
(13)
||x:12
Note that the proximal operator returns 0 if the input vector is Substituting X in Eq. 13 with
Y X and combining Eq. (TT) and Eq. (12). then Eq. (12) becomes
x (x + (Y (Xi))) = (1 - (Xin)) + B.Y. eie [n],
(14)
where B = max(1 Y
7>
X..) ll 0). To summarize, the proposed adaptive message passing
(AMP) scheme is showed in Figure 4 and diagram is showed in Figure In detail, AMP works as follows:
The first step takes feature aggregation within the local neighbors with self-loop weighted by 1 - 2-(1 - A);
The second step computes a weight B, [0, 1] for each node v depending on the local deviation Y lx22:
The final step takes linear combination of input features X, and the aggregated features Y where the node-wise residual is adaptively weighted by 1 - 3g for each node
Y (1-2x(1 - X)) + +21(1- A)AX*
7t
B max(1 Y (Xin)ill²"
0) vi [n]
x - (X,)( + 3, Y
Vi E [r]]
Figure 4: Adaptive Message Passing (AMP)
The convergence guarantee of AMP and parameter setting for the stepsize 7 are illustrated in Theorem and proved in Appendix A
According to Theorem 1. if we set y 4(1-} or = 2[-x]*
then the first step of AMP can be simplified as Y x + Ax and Y = A respectively. The choice of stepsize will only impact the convergence speed but not the ultimate effect of AMP when it convergences to the fixed point solution. We also discuss the computation complexity per iteration of AMP Remark
Theorem 1 (Convergence of AMP). Under the stepsize setting " < "1-Ai||L[==" the proposed adaptive message passing scheme (AMP) in Eq. (9) and Eq. (I0) converges to the optimal solution of the problem defined in Eq. (8). In practice, it is sufficient to choose any Y < शा| since ||L| S Moreover, if the connected components of the graph G are not bipartite graphs, it is sufficient to [[l| 2.
choose 7 2[1 since
Remark (Computation complexity). AMP is as efficient as simple feature aggregation Xow A because the additional computation cost from the second and third steps in Figure 4 is in the order O(n)), where n is the number of nodes and is the feature dimension. This is negligible compared with the computation cost ((d)) in feature aggregation, where is the number of edges, due to the fact that usually there are many more edges than nodes in real-world graphs, i.e., m > n.
3.3 Interpretation of AMP
Interestingly, the proposed AMP has simple and intuitive interpretation as adaptive residual connec- tion, which aligns well with our design motivation:
If the feature of node i.c., (Xi),, . is significantly inconsistent with its local neighbors, i.c., the aggregated feature Y then the local deviation || Y (Xin) 2ll2 will be large, which leads to a 3, close to 1. Therefore, the final step will assign small weight to the residual, i.e., (1 (X) and the aggregated feature Y will dominate.

---

On the contrary, if (Xma); / is already consistent with its local neighbors, I small, which leads to B. close Thus, the residual will dominate, which reasonable since there is less need to aggregate features in this case.
(Xm))lla will be
To summarize, the local deviation |Y (Xm))lla provides a natural transition from 3; + 1 to 3; - 0, and the transition can be modulated by À which can be either learned or tuned hyperparameter through cross vooss-validttion. This transition provides an node-wise adaptive residual connection for the message passing scheme.
Adaptivity for abnormal normal features. According to the homophily assumption on graph structure data [17 19 3 the feature representations of normal features should be more consistent with local neighbors than abnormal features. As result, AMP will assign more residual (i.e., smaller 9) to normal features but less residual (i.e., larger 3) to abnormal features, providing a customized tradeoff between feature aggregation and residual connection. Consequently, can promote both the resilience to abnormal features and the performance on normal features. Above discussion also implies clear physical meaning for 3 in AMP, and we formally define it as the adaptive score.
Definition 1 (Adaptive score). The variables {3 ,,n in the adaptive message passing scheme (AMP) are defined as the adaptive scores for nodes {U1, ... Un respectively in graph G. In particular, the larger Br is, the more likely the feature of node v, is abnormal.
Remark 2 (Nonlinear smoother). Different from most existing message passing scheme which are linear smoothers, AMP is nonlinear smoother because the weights {B,} are computed from Y and Xin. This nonlinearity is the key to achieve adaptive residual connection for different nodes.
3.4 The Model Architecture
The proposed adaptive message passing (AMP) can be used as a building block in many GNN models to improve the resilience to abnormal node features. In this work, we choose the the decoupled architectures as APPNP [10) and DAGNN (20), and propose the Adaptive residual GNN (AirGNN):
Xin ho (Xie));
(15)
Y AMP (Xin; K,
(16)
ho(-) is any machine learning model parameterized by learnable parameters 6, such multilayer perceptrons (MLPs). X E Raxd denotes the initial node features. The model he(+) will first transform the initial node features as X = hg (Xaaa) AMP takes ha (Xtaa) as input, and performs K steps of AMP with the hyperparameter A. Similar to the majority of cxisting GNN models, the training objective is the cross-entropy classification loss on the labeled nodes, and the whole model is trained in an end-to-end way. Note that AirGNN is very efficient explained in Remark [ and it only requires two hyperparameters K and X without introducing additional parameters to learn, which could reduce the risk of overfitting.
4 Experiment
In this section, we aim to verify the effective of the proposed adaptive message passing scheme (AMP) and the AirGNN model through the semi-supervised node classification tasks. Specifically, we try to answer the following questions: (1) How does AirGNN perform on abnormal and normal and (2) How does AirGNN work by adjusting the adaptive residual?
features? (Section 4.2 and 4.3
(Section 4.4)
4.1 Experimental Settings
Datasets and baselines. We conduct experiments on 8 real-world datasets including three citation graphs, i.e.. Cora, Citeseer, Pubmed (2T) two co-authorship graphs, i.c., Coauthor CS and Coauthor Physics [22] two co- co-purchase graphs, i.e., Amazon Computers and Amazon Photo 222. and one OGB dataset, i.e., ogbn-arxiv [23]. Due the space limit, we only present the results on Cora, Citeseer, and Pubmed in this section, but defer the results on other datasets to Appendix D.I More details about the data statistics and data splits are summarized Appendix [ The proposed AirGNN is compared with representative GNNs, including GCN 3. We defer the comparison with the variants of APPNP and Robust GCN respectively.
GAT 6), APPNP [TQ and GCNII 9
24 to Appendix
D33 and D.4
7

---

Parameter settings. For all baselines, we follow the best hyperparameter settings in their original papers, Additionally, we tune best residual weight a for APPNP and GCNII in the range [o. 1]. For AirGNN, we use a two-layer MIP as the base model ha(-): following APPNP, We fix the learning rate 0.01, dropout 0.8, and weight decay 0.0005. Moreover, we set 7
2(I-X) as suggested by
Theorem We choose K = 10 and tune À in the range [0. 1]. Adam optimizer 25 is used in all experiments. We run all experiments by 10 times, and report the mean and variance.
Evaluation setting. We assess the performance of all models under two types of abnormal feature scenarios, including noisy features and adversarial features. The abnormal features are injected to randomly selected test nodes after model training. By default, all hyperparameters are tuned according the performance on validation sets when the dataset clean. If tuning the hyperparameter of AirGNN according to the validation sets after injecting abnormal features, the performance will be even better, as discussed in Appendix D.2 The performance on clean data are showed in Appendix t. demonstrate that AirGNN doesn' need to sacrifice accuracy for better robustness against abnormal features.
4.2 Performance Comparison with Noisy Features
In this subsection, consider the abnormal features in the noisy feature scenario. Specifically, simulate the noisy features by assigning subset of the nodes with random features sampled from a multivariate standard Gaussian distribution. Note that the selection of noise subsets has apparent impact on the performance since some nodes are less vulnerable to abnormal features while others are more vulnerable. To reduce such variance, we report the average performance over 10 times of random selection of the noise node sets, similar to the settings in the preliminary study in Section 2 We report the node classification test accuracy on abnormal (noisy) features and normal features in Figure 5 and Figure
6. separately, under varying noisy ratio. From these figures, we can observe:
Figure 5 shows that AirGNN significantly outperforms all baselines on all datasets in terms of the performance noisy nodes. This verifies that AMP is able improve the resilience to noisy features, aligning well with the design motivation.
that AirGNN promotes the performance on normal nodes when abnormal nodes exist. This is because AMP can remove some abnormal features which detrimental normal nodes.
GAT
0.5
CCM
0.9
AYONM
Accunacy
64
0.3
CAT
6.3
GAT
SCh
60)
GCHI
G0H
61
25
Ratio of Naby Nodes 1%6
Rellin of Noney Nodee 1961
Babio of Naaiy Nodes (S)
Cora
(h) CiteSeer
(c) PuhMed
Figure 5: Node classification accuracy on abnormal (noisy) nodes
4.3 Performance Comparison with Adversarial Features
In this subsection, consider the abnormal feature scenario when the node features are maliciously attacked by the attacker to manipulate the prediction of GNNs. We use the Nettack (26) implemented in DeepRobus (27], PyTorch library for adversarial attacks and defenses, to generate the adversarial features. We randomly choose 40 test nodes as the targeted nodes, and assess the performance under 50,80), 80). where the perturbation numbers denote the number of feature dimensions that can be manipulated. The node classification accuracy on these attacked nodes are showed Figure From these figures, we can make the following observations:
increasing perturbation budgets {0, 5, 10, 20, 50,
8

---

3.9
5.8
Accurac
Accuracy
Accuracy
GAT GEN -X- GCNI
3.4
CCN
GCHI
--- GCN1
APT7P
APPNP
AHCNS
-8. ArONN
10
0.4
10
20
30
0.8
10
15
20
25
30
Ratio Noisy Nodel
Ratio of Naley Nodes (86
Ratio of Noiey Nodas (%)
(a) Cora
(b) CiteSeer
(c) PubMed
Figure 6: Node classification accuracy on normal nodes
AirGNN is significantly more robust against adversarially attacked features than all baselines. MLP is the most vulnerable model, which demonstrates the usefulness of graph structure information in combating against abnormal node features.
The advantages of AirGNN over the baselines become much stronger with larger perturbation budgets. This suggests that AMP can significantly improve the resilience to abnormal features.
0.7
6.7
Accuraty
Accursyy
0.4
3.5
6.5
22
50
80
6.2
Pettur bution Number
Pertarbation Number
Purtorbation Number
Cora
(b) CiteSeer
(c) PubMed
Figure 7: Node classification accuracy on adversarial nodes
4.4 Adaptive Residual for Abnormal & Normal Nodes
To further understand and verify how AMP and AirGNN work, we investigate the adaptive score ;3; for each node ve. Specifically, the average adaptive scores for abnormal nodes and normal nodes in the last layer of AMP are computed separately. the noisy feature scenario, we fix ratio of noisy nodes as 10%. In the adversarial feature scenario, we choose 40 target nodes and fix the perturbation number as 80. The results in noisy and adversarial feature scenarios are showed in Table [ and Table 2 respectively. From these tables, we can observe:
On the one hand, it can be clearly observed that in both scenarios, the average adaptive scores for abnormal nodes are significantly higher than those for normal nodes. Therefore, it verifies our intuition that large adaptive scores are strongly related to abnormal features.
On the other hand, it also implies that the residual weights (i.c., - B,) for abnormal nodes are much lower than those of normal nodes. This perfectly aligns with our motivation remove abnormal features by reducing their residual connections.
The study on adaptive scores verifies how the adaptive residuals in AMP and AirGNN work as designed. It corroborates that AirGNN not only tremendously boosts the resilience to abnormal features but also provides interpretable information for anomaly detection that will useful many security-critical scenarios since the adaptive score serves good indicator of abnormal nodes. Morever, expected that APPNP without residual will perform well on abnormal nodes but it will sacrifice the performance on normal nodes. We provide detailed comparison with APPNP w/Res and APPNP wo/Res in Appendix DD3 to show the advantages of adaptive residual of AirGNN.

---

Table 1: Average adaptive score (3) and residual weight (1 8) in the noisy feature scenario.
Measure
Cora
CiteSeer
PubMed
Average adaptive score for abnormal nodes
0.998 + 0.000
0.988 + 0.000
0.996 + 0.000
Average adaptive score for normal nodes
0.924 0.002
0.807 + 0.005
0.869 + 0.006
Average residual weight for abnormal nodes
(1.022 + 0.000
0.012 + 0.000
0.004 + 0.000
Average residual weight for normal nodes
0.076 + 0.002
0.193 +: 0.005
0.131 + 0.006
Table 2:
Average adaptive score (3) and residual weight (1 - 3) in the adversarial feature scenario.
Measure
Cora
CiteSeer
PubMed
Average adaptive score for abnormal nodes
0.987 0.000
0.930
0.007
0.959
0.005
Average adaptive score for normal nodes
0.922 0.004
0.689 0.024
0.826 0.016
Average residual weight for abnormal nodes
0,013 0.000
0.070 +: 0.007
0.041 + 0.005
Average residual weight for normal nodes
0.078 0.004
0.311 0.024
0.174 + 0.016
5
Related Work
GNNs generalize convolutional neural networks (CNN) to graph structure data through the message passing framework 2 77. The design of message passing and GNN architectures are majorly
motivated in spectral domain 3 and spatial domain 5 6 2 Recent works have shown that
the message passing in GNNs can be regarded as low-pass graph filters [12 [3] More generally,
has been proven that message passing many GNNs can be uniformly derived from graph
signal denoising [T] 28, 29 30 Classic GNNs such as GCN [3] and GAT 66] achieve their best
performance with shallow models, but their performance degrades when stacking more layers, which can be partially explained through oversmoothing analyses [5]. Recent works propose to use residual connections or skip connections to mitigate the oversmoothing issues, and they demonstrate the potential benefits from more feature aggregations. Examples include but not limited to DeepGCNs [11, JKNet [32), GCNII 91, APPNP [10] and DeeperGNN [20). These models use global residual connection that can not be adaptive for each node, which significantly differ from the proposed AirGNN.
Recently, there are growing interests in reducing GNNs' vulnerability to the graph structure noise, such as Robust GCN 24, GCN-SVD 33 Pro-GNN Ea IDGL (35). ElasticGNN (36). etc. Please refer to the comprehensive surveys 37 38 for more details. However, how to design GNNs with strong resilience to abnormal node features remains be developed. To the best of our knowledge, AirGNN is the first GNN model that is intrinsically robust to many types of abnormal node features by design. It improves the performance in various kinds of abnormal scenarios without needing to sacrifice clean accuracy in normal settings.
6 Conclusion
In this work, we discover an intrinsic tension between feature aggregation and residual connection in the message passing scheme of GNNs, as well as the corresponding performance tradeoff between nodes with abnormal and normal features. We analyze possible reasons to explain these findings from the perspective of graph Laplacian smoothing. Our understandings further motivate us to propose a simple, efficient, interpretable and adaptive message passing scheme as well as a new GNN model with adaptive residual, named AirGNN. AirGNN provides a node-wise adaptive transition between feature aggregation and residual connection, and the significant advantages of AirGNN are demonstrated through extensive experiments.
Acknowledgments and Disclosure of Funding
This research supported by the National Science Foundation (NSF) under grant numbers ISS171474,, CNS1815636, IIS IIS 19079077 DRL2025244, I1S1928278, IIS1955285,
10

---

I0$2107215, IOS2035472 and Army Research Office (ARO) under grant number W911NF-21- 1-0198.
Societal Impact and Limitations
The methodology proposed in this paper might have signifieant positive societal impact since it reduces machine learning models' vulnerability to abnormal datasets that generally exist in real-world applications, especially many recurity-criticn not have the prior knowledge about if the graph is clean, has noisy features or adversarial features. Therefore, algorithms like the proposed AirGNN that can work under both the clean and various abnormal feature settings are appealing. While we are unaware of any potential negative society impact, we point out two limitations of this work: (1) this paper focuses on abnormal node features and has not evaluated the performance of the proposed method when the dataset contains both abnormal node features and edges: (2) unclear how performs on heterophilic graphs. It will be interesting to investigate these problems in future works.
scenarios. In practice. for given graph, we do
References
[1] Yao Ma and Jiliang Tang. Deep Learning Graphs. Cambridge University Press, 2020.
[2] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning. pages 1263-1272 PMLR, 2017.
[3] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint
2016.
[4] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3844-3852, 2016
[5] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. arXiv preprint 1eXXiv:706.0220 2017.
[6] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 2017.
[7] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions neural networks, 20(1):61-80, 2008.
[8] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 2020.
[9] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Hal Daumé III and Aarti Singh, editors, Proceedings the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1725-1735. PMLR, 13-18 Jul 2020.
[10] Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv: (afi::8810000 8100059977 2018.
[11] Yao Ma. Xiaorui Liu, Tong Zhao, Yozen Liu. Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. Proceedings of the 30th ACM International Conference on Information and Knowledge Management, 2021.
[12] Hoang Nt and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv: /905.0955000 2019.
[13] Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing gnns. International Conference Learning Representations,
2019 2019.

---

[14] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi- supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 32, 2018.
[15] Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference on Learning Representations, 2020.
[16] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[17] Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of feather: Homophily in social networks. Annual review of sociology, 27(1):415 444, 2001.
[18] Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation.
[19] Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. 2004.
[20] Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, 2020.
[21] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93 1339999-3,229 2008.
[22] Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.0586 2018.
[23] Marinka Zitnik Yuxiao Dong Hongyu Ren Bowen Liu Michele Catasta Jure Leskovec Wei- hua Hu, Matthias Fey. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint 20xv:2005.00687, 2020.
[24] Dingyuan Zhu, Ziwei Zhang, Peng Cui, and Wenwu Zhu, Robust graph convolutional networks against adversarial attacks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1307, 2019.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[26] Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks on neural networks for graph data. In KDD. ACM, 2018.
[27] Yaxin Li, Wei Jin, Han Xu, and Jiliang Tang. Deeprobust: A pytorch library for adversarial attacks and defenses, 2020.
[28] Xuran Pan, Song Shiji. and Huang Gao. A unified framework for convolution-base graph neural networks.
2020.
[29] Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework, 2021.
[30] Siheng Chen, Yonina C. Eldar, and Lingxiao Zhao. Graph unrolling networks: Interpretable neural networks for graph signal denoising, 2020.
[31] Guohao Li, Matthias Müller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep as cnns? In The IEEE International Conference on Computer Vision (ICCV), 2019.
[32] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pages 5453-5462. PMLR, 10-15 Jul 2018
12

---

[33] Negin Entezari, Saba A Al-Sayouri, Amirali Darvishzadeh, and Evangelos E Papalexakis. All you need is low (rank) defending against adversarial attacks on graphs. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 169-177, 2020.
[34] Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 66-74, 2020.
[35] Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances in Neural Information Processing Systems, 33, 2020.
[36] Xiaorui Liu, Wei Jin, Yao Ma, Yaxin Li, Hua Liu, Yiqi Wang, Ming Yan, and Jiliang Tang. Elastic graph neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1837-6849, PMLR, 18-24 Jul 2021.
[37] Wei Jin, Yaxing Li, Han Xu, Yiqi Wang, Shuiwang Ji, Charu Aggarwal, and Jiliang Tang. Adversarial attacks and defenses on graphs. ACM SIGKDD Explorations Newsletter, 22(2):19-
34, 2021.
[38] Yanqiao Zhu, Weizhi Xu, Jinghao Zhang, Qiang Liu, Shu Wu, and Liang Wang. Deep graph structure learning for robust representations: A survey. arXiv preprint arXiv:2
2021.
[39] Laurent Condat. A primal-dual splitting method for convex optimization involving lipschitzian, proximable and linear composite terms. Journal of optimization theory and applications. 158())460-479, 2013.
[40] Fan RK Chung and Fan Chung Graham. Spectral graph theory. Number 92. American Mathematical Soc., 1997.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "y5OPDpU_vLO", "reviewer": "Reviewer_tp11", "summary": "This paper discusses how message passing and feature aggregation are affected by abnormal node features, and designs a message passing module called Adaptive Message Passing (AMP) with adaptive residual connection and feature aggregation. In this way, the module can adaptively deal with the node that is inconsistent with the local feature during the message passing. The paper proposes the AirGNN that is built up with AMP as the message-passing layer in GNN, the experiment result shows that the AirGNN gets a more robust result on both the abnormal node and normal node classification comparing with the baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper gives a solution for dealing with the abnormal node feature in the graph. The authors first give an observation of how the residual connection and number of layers affect the models' performance when the layer grows. Then the author gives the explanation and their solutions to balance the feature aggregation and residual connections affect during the message passing. The experiment result shows their advantages on various datasets comparing with the baselines.\n\nThe paper writes in good order, the discussion at the beginning of the paper helps a lot for me to understand the motivation of the paper. The authors give a straightforward solution by designing an adaptive module to control the feature aggregation and residual connection to deal with the abnormal feature in the graph. I am not sure if it is a novel method but seems reasonable to me. \n\nFor the experiment part, the authors compared their method with GCN, GAT, APPNP, and GCNII and evaluate the performance on both normal and abnormal features. From the plot, the AirGNN consistently outperforms other baselines when an abnormal feature exists and increases. Overall, the experiment shows the model's advantage properly.\n\nMy questions and concerns are the follows:\n\n1. Authors use a single noise setting in the whole paper, I think for this setting, the noise part is an important thing to understand, however, authors do not even write down the detailed noise setting ( what is the mean and var of the noise distribution). I am expecting to see a careful ablation study towards the noise to see if the method is robust to detect variations of the noise feature.\n\n2. I see from the appendix that for other datasets like Coauthor Physics, Amazon photo, obg-arxiv, the AirGNN is pretty close or even worse than other methods like APPNP on the normal features. Do you have any explanation about this? I not confident for the statement that the model can deal with the real-world Graph with abnormal features. Combining with the first suggestions, it is necessary to see if the proposed model performs well when the noise setting changes.\n\n3. I am not an expert in this area, but it seems that from the related works and the baseline, the authors do not mention some competing ''Robust GNN'' baselines.  I believe there should have some adversarial GNN paper that studies the noise feature during the training process. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not give a section that discusses the limitations and potential negative societal impact. However, since the author stresses that the abnormal node feature exists a lot in the real-world dataset, I may expect the author has a brief discussion on this aspect. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "caxUgK7ZDkN", "reviewer": "Reviewer_TraF", "summary": "\nThis paper observed interesting phenomenons: feature aggregation helps smooth out abnormal features but would lead to over-smoothing for normal features, residual link helps adjust feature smoothness for normal features but may hurt the performance when we have abnormal features. Therefore, it points out that we need to tradeoff between normal and abnormal features while designing GNNs. Then, based on the observation, this work proposed the AMP method and the AirGNN model, which are shown by experiments to be effective under various abnormal feature scenarios.\n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper provides very interesting observations and finds the tradeoff problem between normal and abnormal feature scenarios, the observations and the understandings (in section 2.2) make sense to me, and the results show in figure 5, figure 12(a)-16(a), are impressive.\n\nHowever, I have the following questions/concerns:\n\n1. What is the training objective function? Is it just Eq.(7) or Eq.(7) +classification loss?\n\n2. Cora, Citeseer, and Pubmed are very classic datasets but they are not enough. It would be better if the authors can provide experimental results on other datasets in the main paper, instead of providing them in the appendix.\n\n3. The author claimed that, in real-world applications, node features in graphs could often be abnormal such as being naturally noisy, then could you please provide some experimental results on some abnormal real-world datasets? For now, it seems to me that the authors are manually making the features in the rea-world dataset abnormal by replacing the raw features with Gaussian-generated features, so it is more like a synthetic scenario instead of a real-world scenario.\n\n4. For figure 6, since they are on normal features, then why would the x-axis be \"Ratio of Noisy Nodes(%)\"? Why would the \"Ratio of Noisy Nodes\" influence the performance of GNNs with normal features? According to my understanding, noisy feature = abnormal features, am I misunderstanding?\n\n5. I am curious, now you're using one Gaussian distribution to generate noisy features, what if the noise is generated in a more complicated way? (for example, what if different node class has different Gaussian distribution generation?)\n\n6. I think the writing is not clear enough for me. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nThe limitation of this work is not discussed in the paper.\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hrs"}, "extra_text": {}}, {"review_id": "UJNiU_Ge_v", "reviewer": "Reviewer_Sbcq", "summary": "This paper targets to design GNNs with stronger resilience to abnormal node features. Empirical examination is firstly conducted by replacing the features of randomly selected nodes with random Gaussian noise. The comparison between models with and without the residual connections show the helpfulness of residual connections on learning with abnormal features. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the designed model is simply reasonable, the missing of key related work in discussion and in evaluation comparison makes it hard to know how the proposed solution is effective.  The related work discussion is short and missing the closely related work that also addresses abnormal or noisy feature problems in GNN. The baselines include only the representative GNNs, not taking into account the related work of other robust GNN models. \nFor example:\nRobust Graph Convolutional Networks Against Adversarial Attacks  in KDD 2019\nGNNGuard: Defending Graph Neural Networks against Adversarial Attacks in NeurIPS 2020\nAnd many other GNN defense models listed in Table 3 in paper entitled “Adversarial Attack and Defense on Graph Data: A Survey”.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "-i9idSZopSH", "reviewer": "Reviewer_AKM9", "summary": "This paper examines the behavior of GNNs when node features are perturbed during test time.  AirGNN is proposed based on the observations from the standard GNNs. AirGNN adaptively computes the weight of the initial embedding and aggregated embeddings and is robust to random/adversarial noise keeping the standard classification performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengthes\n\n* The proposed method is well motivated (Sections 2.1, 2.2, and 3.1). These sections argue that deep GNNs without residual connections are too smooth, whereas GNNs with residual connections can rely on noisy features. The proposed method tries to select the better scenario adaptively. This is reasonable.\n* Section 3.3 provides interpretations of the proposed method. These explanations are intuitive and helpful for understanding.\n\n# Weaknesses\n\n* Many robust GNNs have been proposed [A, B]. These methods are not discussed or compared with the proposed method. I would like to see qualitative discussions and quantitative comparisons with them.\n* The proposed method is a variant of APPNP. The proposed method adopts the L21 norm instead of the Frobenius norm. It is important to make more exhaustive comparisons with APPNP. See also the questions below.\n\n# Questions\n\n* Figure 1 (a) says APPNP wo/Res achieves accuracy 0.6 on abnormal features, but Figure 5 (a) says APPNP's accuracy is around 0.2. Why does this happen? Is APPNP w/Res used in Figure 5 (a)? (as a result of hyperparameter tuning?) I would like to see the performance of APPNP wo/Res in all settings. As far as I understand, AirGNN should perform worse on abnormal features than APPNP w/Res while it should perform better on normal features. This should be clarified in the experiments.\n* See also Limitations And Societal Impact below.\n\n## Reference\n\n* [A] Robust Graph Convolutional Networks Against Adversarial Attacks. KDD 2019.\n* [B] Adversarial attack and defense on graph data: A survey. https://arxiv.org/abs/1812.10528", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No limitations are discussed in the manuscript. It is important to discuss the cons against APPNP as the proposed method is a variant of it. What information is lost using the L21 norm instead of L2? As the proposed method is related to adversarial attacks, the societal impact can also be discussed. Is the proposed method reliable enough to deploy in the real world?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discovers the phenomenon that residual connections can amplify GNNs' vulnerability against abnormal node features. It analyzes possible reasons and based on the understanding designs an effective message passing scheme to help solve the issue. 

The observed phenomenon is interesting and important for related applications, and the study and design are reasonable with impressive results. The good response from the authors has clarified several concerns of the reviewers (better presentation, more thorough experiments, comparison with related work etc). The clarification in the response should be incorporated into the final version if accepted.  
Decision: Accept (Poster)