Title: Robust Graph Neural Networks via Unbiased Aggregation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Adversarial Robustness

Graph Neural Networks

Robust Estimation
Abstract: The adversarial robustness of Graph Neural Networks (GNNs) has been questioned due to the false sense of security uncovered by strong adaptive attacks despite the existence of numerous defenses.
In this work, we delve into the robustness analysis of representative robust GNNs and provide a unified robust estimation point of view to
understand their robustness and limitations.
Our novel analysis of estimation bias motivates the design of a 
robust and unbiased graph signal estimator. 
We then develop an efficient Quasi-Newton Iterative Reweighted Least Squares algorithm to solve the estimation problem, which is unfolded as robust unbiased aggregation layers in GNNs with theoretical guarantees.
Our comprehensive experiments confirm the strong robustness of our proposed model under various scenarios, and the ablation study provides a deep understanding of its advantages.

Robust Graph Neural Networks via Unbiased Aggregation

Abstract

The adversarial robustness of Graph Neural Networks (GNNs) has been questioned due to the false sense of security uncovered by strong adaptive attacks despite the existence of numerous defenses. In this work, we delve into the robustness analysis of representative robust GNNs and provide a unified robust estimation point of view to understand their robustness and limitations. Our novel analysis of estimation bias motivates the design of a robust and unbiased graph signal estimator. We then develop an efficient Quasi-Newton Iterative Reweighted Least Squares algorithm to solve the estimation problem, which is unfolded as robust unbiased aggregation layers in GNNs with theoretical guarantees. Our comprehensive experiments confirm the strong robustness of our proposed model under various scenarios, and the ablation study provides a deep understanding of its advantages. Our code is available at https://github.com/chris-hzc/RUNG.

1 Introduction

Graph neural networks (GNNs) have gained tremendous popularity in recent years due to their ability to capture topological relationships in graph-structured data [1]. However, most GNNs are vulnerable to adversarial attacks, which can lead to a substantial decline in predictive performance [2, 3, 4]. Despite the numerous defense strategies proposed to robustify GNNs, a recent study has revealed that most of these defenses are not as robust as initially claimed [5]. Specifically, under adaptive attacks, they easily underperform the multi-layer perceptrons (MLPs) which do not utilize the graph topology information at all [5]. Therefore, it is imperative to thoroughly investigate the limitations of existing defenses and develop innovative robust GNNs to securely harness the topology information in graphs.

Existing defenses attempt to bolster the resilience of GNNs using diverse approaches. For instance, Jaccard-GCN [6] and SVD-GCN [7] aim to denoise the graph by removing potential adversarial edges during the pre-processing procedure, while ProGNN [3] learns the clean graph structure during the training process. GRAND [8] and robust training [9, 10] also improve the training procedure through data augmentation. GNNGuard [2] and RGCN [11] reinforce their GNN architectures by heuristically reweighting edges in the graph. Additionally, there emerge some ODEs-inspired architectures including the GraphCON [12] and HANG [13] that demonstrate decent robustness. Although most of these defenses exhibit decent robustness against transfer attacks, i.e., the attack is generated through surrogate models, they encounter catastrophic performance drops when confronted with adaptive adversarial attacks that directly attack the victim model [5].

38th Conference on Neural Information Processing Systems (NeurIPS 2024).

Concerned by the false sense of security, we provide a comprehensive study on existing defenses under adaptive attacks. Our preliminary study in Section 2 indicates that SoftMedian [4], TWIRLS [14], and ElasticGNN [15] exhibit closely aligned performance and notably outperform other defenses despite their apparent architectural differences. However, as attack budgets increase, these defenses still experience a severe performance decrease and underperform the graph-agnostic MLPs. These observations are intriguing, but the underlying reasons are still unclear.

To unravel the aligned robustness and performance degradation of SoftMedian, TWIRLS, and ElasticGNN, we delve into their theoretical understanding and unveil their inherent connections and limitations in the underlying principles. Specifically, their improved robustness can be understood from a unified view of ℓ 1 -based robust graph smoothing. Moreover, we unearth the problematic estimation bias of ℓ 1 -based graph smoothing that allows the adversarial impact to accumulate as the attack budget escalates, which provides a plausible explanation of their declining robustness. Motivated by these understandings, we propose a robust and unbiased graph signal estimator to reduce the estimation bias in GNNs. We design an efficient Quasi-Newton IRLS algorithm that unrolls as robust unbiased aggregation layers to safeguard GNNs against adversarial attacks. Our contributions can be summarized as follows:

We provide a unified view of ℓ 1 -based robust graph signal smoothing to justify the improved and closely aligned robustness of representative robust GNNs. Moreover, we reveal their estimation bias, which explains their severe performance degradation as the attack budgets increase.

We propose a robust and unbiased graph signal estimator to mitigate the estimation bias in ℓ 1 -based graph signal smoothing and design an efficient Quasi-Newton IRLS algorithm to solve the non-smooth and non-convex estimation problem with theoretical guarantees.

The proposed algorithm can be readily unfolded as feature aggregation building blocks in GNNs, which not only provides clear interpretability but also covers many classic GNNs as special cases.

Comprehensive experiments demonstrate that our proposed GNN significantly improves the robustness while maintaining clean accuracy. We also provide comprehensive ablation studies to validate its working mechanism.

2 Estimation Bias Analysis of Robust GNNs

In Section 2.1, we conduct a preliminary study to evaluate the robustness of several representative robust GNNs. In Section 2.2, we establish a unified view as ℓ 1 -based models to uncover the inherent connections of three well-performing GNNs, including SoftMedian, TWIRLS and ElasticGNN. In Section 2.3, we leverage the bias of ℓ 1 -based estimation to explain the catastrophic performance degradation in the preliminary experiments.

Notation. Let G = {V , E} be a graph with node set V = { v 1 , . . . , v n } and edge set E = { e 1 , . . . , e m } . The adjacency matrix of G is denoted as A ∈ { 0 , 1 } n × n and the graph Laplacian matrix is L = D -A . D = diag ( d 1 , . . . , d n ) is the degree matrix where d i = |N ( i ) | and N ( i ) is the neighborhood set of v i . The node feature matrix is denoted as F = [ f 1 , . . . , f n ] ⊤ ∈ R n × d , and f (0) ( F (0) ) denotes the node feature vector (matrix) before graph smoothing in decoupled GNN models. Let ∆ ∈ {-1 , 0 , 1 } m × n be the incidence matrix whose l -th row denotes the l -th edge e l = ( i, j ) such that ∆ li = -1 , ∆ lj = 1 , ∆ lk = 0 ∀ k / ∈ { i, j } . ˜ ∆ is its normalized version : ˜ ∆ lj = ∆ lj / √ d j . For a vector x ∈ R d , we use ℓ 1 -based gragh smoothing penalty to denote either ∥ x ∥ 1 = ∑ i | x i | or ∥ x ∥ 2 = √∑ i x 2 i . Note that we use ℓ 2 -based gragh smoothing penalty to denote ∥ x ∥ 2 2 = ∑ i x 2 i .

2.1 Robustness Analysis

To test the robustness of existing GNNs without the false sense of security, we perform a preliminary evaluation of existing robust GNNs against adaptive attacks. We choose various baselines including the undefended MLP, GCN [16], some of the most representative defenses in [5], and two additional robust models TWIRLS [14] and ElasticGNN [15]. We execute adaptive local evasion topological attacks and test the node classification accuracy on the Cora ML and Citeseer datasets. The detailed settings follow Section 4.1. From Figure 1, it can be observed that:

MLP

GNNGuard

GRAND

一☆一

RGCN

GCN

SoftMedian

TWIRLS

一☆

ElasticGNN

Citeseer

80

60

2

40

Accur

20

0

020

50

100

150

200

020

50

100

150

200

Budget (%)

Budget (%)

Among all the selected robust GNNs, only SoftMedian, TWIRLS, and ElasticGNN exhibit notable and closely aligned improvements in robustness whereas other GNNs do not show obvious improvement over undefended GCN.

SoftMedian, TWIRLS, and ElasticGNN encounter a similar catastrophic performance degradation as the attack budget scales up. Their accuracy easily drops below that of the graph-unware MLP, indicating their failure in safely exploiting the topology of the data.

2.2 A Unified View of Robust Estimation

Our preliminary study provides intriguing observations in Section 2.1, but the underlying reasons behind these phenomena remain obscure. This motivates us to delve into their theoretical understanding and explanation. In this section, we will compare the architectures of three well-performing GNNs, aiming to reveal their intrinsic connections.

SoftMedian [4] substitutes the GCN aggregation for enhanced robustness with the dimension-wise median m i ∈ R d for all neighbors of each node i ∈ V . However, computing the median involves operations like ranking and selection, which is not compatible with the back-propagation training of GNNs. Therefore, the median is approximated as a differentiable weighted sum ˜ m i = 1 Z ∑ j ∈N ( i ) w ( f j , m i ) f j , ∀ i ∈ V , where m i is the exact non-differentiable dimension-wise median, f j is the feature vector of the j -th neighbor, w ( x , y ) = e -β ∥ x -y ∥ 2 , and Z = ∑ k w ( f k , m k ) is a normalization factor. In this way, the aggregation assigns the largest weights to the neighbors closest to the actual median.

TWIRLS [14] utilizes the iteratively reweighted least squares (IRLS) algorithm to optimize the objective with parameter λ , and ρ ( y ) = y is the default:

ElasticGNN [15] proposes the elastic message passing which unfolds the proximal alternating predictor-corrector (PAPC) algorithm to minimize the objective with parameter λ { 1 , 2 } :

A Unified View of Robust Estimation. While these three approaches have seemingly different architectures, we provide a unified view of robust estimation to illuminate their inherent connections. First, the objective of TWIRLS in Eq. (1) can be considered as a particular case of ElasticGNN

with λ 2 = 0 and p = 2 when neglecting the difference in the node degree normalization. However, TWIRLS and ElasticGNN leverage different optimization solvers, i.e., IRLS and PAPC, which leads to vastly different GNN layers. Second, SoftMedian approximates the computation of medians in a soft way of weighted sums, which can be regarded as approximately solving the dimension-wise median estimation problem [17]: arg min f i ∑ j ∈N ( i ) ∥ f i -f j ∥ 1 . Therefore, SoftMedian can be regarded as the ElasticGNN with λ 2 = 0 and p = 1 . We also note that the SoftMedoid [18] approach also resembles ElasticGNN with λ 2 = 0 and p = 2 , and the Total Variation GNN [19] also utilizes an ℓ 1 estimator in spectral clustering.

The above analyses suggest that SoftMedian, TWIRLS, and ElasticGNN share the same underlying design principle of ℓ 1 -based robust graph signal estimation, i.e. a similar graph smoothing objective with edge difference penalties ∥ f i -f j ∥ 1 or ∥ f i -f j ∥ 2 . However, they adopt different approximation solutions that result in distinct architecture designs. This unified view of robust estimation clearly explains their closely aligned performance. Besides, the superiority ℓ 1 -based models over the ℓ 2 -based models such as GCN [16], whose graph smoothing objective is essentially ∑ ( i,j ) ∈E ∥ f i / √ d i -f j / √ d j ∥ 2 2 [20], can be explained since ℓ 1 -based graph smoothing mitigates the impact of the outliers [15].

2.3 Bias Analysis and Performance Degradation

The unified view of ℓ 1 -based graph smoothing we established in Section 2.2 not only explains their aligned robustness improvement but also provides a perspective to understand their failure as attack budgets scale up through an estimation bias analysis.

Cleansample

Truemean

11 12

1I 12

Ours

10

10

10

1

1

1

hension

ion

sion

5

dimensi

5

hensi

5

dime

dim

2

2

2

ature

ire

0

tur

0

tur

0

ea

F

-2

15% outliers

-2

30%outliers

F

e

-2

45%outliers

-4-2

0

2

5

10

-4

-2

0

2

5

10

-4

-2

0

2

5

10

Featuredimension2

Featuredimension2

Featuredimension2

Bias of ℓ 1 -based Estimation. In the literature of high-dimensional statistics, it has been well understood that the ℓ 1 regularization will induce an estimation bias. In the context of denoising [21] or variable selection [22], small coefficients β are undesirable. To exclude small β in the estimation, a soft-thresholding operator can be derived as S λ ( β ) = sign ( β ) max( | β | -λ, 0) . As a result, large β are also shrunk by a constant, so the ℓ 1 estimation is biased towards zero.

A similar bias effect also occurs in graph signal estimation in the presence of adversarial attacks.

For example, in TWIRLS (Eq. (1)), after the graph aggregation ˜ f ( k +1) i = ∑ j ∈N ( i ) w ij ˜ f ( k ) j where w ij = ∥ ˜ f i -˜ f j ∥ -1 2 , the edge difference ˜ f i -˜ f j will shrink towards zero. Consequently, every adversarial edge the attacker adds will induce a bias that can be accumulated and amplified when the attack budget scales up.

Numerical Simulation. To provide a more intuitive illustration of the estimation bias of ℓ 1 -based models, we simulate a mean estimation problem on synthetic data since most message passing schemes in GNNs essentially estimate the mean of neighboring node features. The results in Figure 2

shows that ℓ 1 -based estimator is more resistant than ℓ 2 -based estimator. However, as the ratio of outliers escalates, the ℓ 1 -based estimator encounters a greater shift from the true mean due to the accumulated bias caused by outliers. This observation explains why ℓ 1 -based graph smoothing models suffer from catastrophic degradation under large attack budgets. The detailed simulation settings and results are available in Appendix A.

3 Robust GNNs with Unbiased Aggregation

In this section, we first design a robust unbiased estimator to reduce the bias in graph signal estimation in Section 3.1 and propose an efficient second-order IRLS algorithm to compute the robust estimator with theoretical convergence guarantees in Section 3.2. Finally, we unroll the proposed algorithm as the robust unbiased feature aggregation layers in GNNs in Section 3.3.

3.1 Robust and Unbiased Graph Signal Estimator

Our study and analysis in Section 2 have shown that while ℓ 1 -based methods outperform ℓ 2 -based methods in robustness, they still suffer from the accumulated estimation bias, leading to severe performance degradation under large perturbation budgets. This motivates us to design a robust and unbiased graph signal estimator that derives unbiased robust aggregation for GNNs with stronger resilience to attacks.

Theoretically, the estimation bias in Lasso regression has been discovered and analyzed in highdimensional statistics [23]. Statisticians have proposed adaptive Lasso [23] and many non-convex penalties such as Smoothly Clipped Absolute Deviation (SCAD) [24] and Minimax Concave Penalty (MCP) [25] to alleviate this bias. Motivated by these advancements, we propose a Robust and Unbiased Graph signal Estimator (RUGE) as follows:

where ρ γ ( y ) denotes the function that penalizes the feature differences on edges by MCP:

As shown in Figure 3, MCP closely approximates the ℓ 1 norm when y is small since the quadratic term y 2 2 γ is negligible, and it becomes a constant value when y is large. This transition can be adjusted by the thresholding parameter γ . When γ approaches infinity, the penalty ρ γ ( y ) reduces to the ℓ 1 norm. Conversely, when γ is very small, the 'valley' of ρ γ near zero is exceptionally sharp, so ρ γ ( y ) approaches the ℓ 0 norm and becomes a constant for a slightly larger y . This enables RUGE to suppress smoothing on edges whose node differences exceeding the threshold γ . This not only mitigates the estimation bias against outliers but also maintains the estimation accuracy in the absence of outliers. The simulation in Figure 2 verifies that our proposed estimator ( η ( x ) := ρ γ ( ∥ x ∥ 2 ) ) can recover the true mean despite the increasing outlier ratio when the outlier ratio is below the theoretical optimal breakdown point.

3.2 Quasi-Newton IRLS

Despite the advantages discussed above, the proposed RUGE in Eq. (3) is non-smooth and non-convex, which results in challenges for deriving efficient numerical solutions that can be readily unfolded as neural network layers. In the literature, researchers have developed optimization algorithms for MCP-related problems, such as the Alternating Direction Multiplier Method (ADMM) and Newtontype algorithms [24, 25, 26]. However, due to their excessive computation and memory requirements as well as the incompatibility with back-propagation training, these algorithms are not well-suited

for the construction of feature aggregation layers in GNNs. To solve these challenges, we propose an efficient Quasi-Newton Iteratively Reweighted Least Squares algorithm (QN-IRLS) to solve the estimation problem in Eq. (3).

IRLS. Before stepping into our QN-IRLS, we first introduce the main idea of iteratively reweighted least squares (IRLS) [27] and analyze its weakness in convergence. IRLS aims to circumvent the non-smooth H ( F ) in Eq. (3) by computing its quadratic upper bound ˆ H based on F ( k ) in each iteration k and optimizing ˆ H ( F ) :

̸

where W ( k ) ij = 1 i = j dρ γ ( y ij ) dy 2 ij ∣ ∣ y ij = y ( k ) ij 3 , where y ( k ) ij = ∥ ∥ f ( k ) i / √ d i -f ( k ) j / √ d j ∥ ∥ 2 and ρ γ ( · ) is the MCP function. For the detailed proof of the upper bound, please refer to Lemma 1 in Appendix B. Then, each iterative step of IRLS can be formulated as the first-order gradient descent for ˆ H ( F ) :

where η is the step size, ˆ Q ( k ) = 2( diag ( q ( k ) )+ λ I ) , and q ( k ) m = ∑ j W ( k ) mj A mj /d m . Its convergence condition is given in Theorem 1, with a proof in Appendix B.

Theorem 1. If F ( k ) follows the update rule in Eq. (6) where ρ defining W satisfies that dρ ( y ) dy 2 is non-decreasing ∀ y ∈ (0 , ∞ ) , then a sufficient condition for H ( F ( k +1) ) ≤ H ( F ( k ) ) is that the step size η satisfies 0 < η ≤ ∥ diag ( q ( k ) ) -W ( k ) ⊙ ˜ A + λ I ∥ -1 2 .

Quasi-Newton IRLS. Theorem 1 suggests the difficulty in the proper selection of stepsize for (first-order) IRLS due to its non-trivial dependency on the graph ( ˜ A ) and the dynamic terms ( q ( k ) and W ( k ) ) 4 . The dilemma is that a small stepsize will lead to slow convergence but a large step easily causes divergence and instability as verified by our experiments in Section 4.3 (Figrue 5), which reveals its critical shortcoming for the construction of GNN layers.

To overcome this limitation, we aim to propose a second-order Newton method, F ( k +1) = F ( k ) -( ∇ 2 ˆ H ( F ( k ) )) -1 ∇ ˆ H ( F ( k ) ) , to achieve faster convergence and stepsize-free hyperparameter tuning by better capturing the geometry of the optimization landscape. However, obtaining the analytic expression for the inverse Hessian matrix ( ∇ 2 ˆ H ( F ( k ) )) -1 ∈ R n × n is intractable, and the numerical solution requires expensive computation for large graphs. Therefore, we propose a novel Quasi-Newton IRLS algorithm (QN-IRLS) that approximates the Hessian matrix ∇ 2 ˆ H ( F ( k ) ) = 2( diag ( q ( k ) ) -W ( k ) ⊙ ˜ A + λ I ) by the diagonal matrix ˆ Q ( k ) = 2( diag ( q ( k ) ) + λ I ) such that the inverse is trivial. The proposed QN-IRLS works as follows:

where ( ˆ Q ( k ) ) -1 automatically adjusts the per-coordinate stepsize according to the local geometry of the optimization landscape, q ( k ) and W ( k ) are defined as in Eq. (5) and (6). In this way, QN-IRLS provides faster convergence without needing to select a stepsize. The convergence is guaranteed by Theorem 2 with the proof in Appendix B.

Theorem 2. If F ( k +1) follows update rule in Eq. (7) , where ρ satisfies that dρ ( y ) dy 2 is non-decreasing ∀ y ∈ (0 , ∞ ) , it is guaranteed that H ( F ( k +1) ) ≤ H ( F ( k ) ) .

3.3 GNN with Robust Unbiased Aggregation

The proposed QN-IRLS provides an efficient algorithm to optimize the RUGEin Eq. (3) with a theoretical convergence guarantee. Instantiated with ρ = ρ γ , each iteration in QN-IRLS in Eq. (7) can be used as one layer in GNNs, which yields the Robust Unbiased Aggregation (RUNG):

̸

Interpretability. The proposed RUNG can be interpreted intuitively with edge reweighting. In Eq. (8), the normalized adjacency matrix ˜ A is reweighted by W ( k ) , where W ( k ) ij = dρ ( y ) dy 2 | y = y ( k ) . It ij

is shown in Figure 4 that W ij becomes zero for any edge e k = ( i, j ) with a node difference y ( k ) ij ≥ γ , thus pruning suspicious edges. This implies RUNG's strong robustness under large-budget adversarial attacks. With the inclusion of the skip connection F (0) , diag ( q ( k ) ) + λ I can be seen as a normalizer of the layer output.

Relations with Existing GNNs. RUNG can adopt different ρ that Theorem 2 allows, thus covering many classic GNNs as special cases. When ρ ( y ) = y 2 , RUNG in Eq. (8) exactly reduces to APPNP [28] ( F ( k +1) = 1 1+ λ ˜ AF ( k ) + λ 1+ λ F (0) ) and GCN ( F ( k +1) = ˜ AF ( k ) ) if chosing λ = 0 . When ρ ( y ) = y , the objective of RUGE is equivalent to ElasticGNN with p = 2 , which is analogous to SoftMedian and TWIRLS due to their inherent connections as ℓ 1 -based graph smoothing.

Complexity analysis. RUNGisscalable with time complexity of O ( k ( m + n ) d ) and space complexity O ( m + nd ) , where m is the number of edges, d is the number of features, n is the number of nodes, and k is the number of GNN layers. Therefore, the complexity of our RUNG is comparable to normal GCN (with a constant difference) and it is feasible to implement. The detailed discussions about computation efficiency can be found in Appendix C.

4 Experiment

In this section, we perform comprehensive experiments to validate the robustness of the proposed RUNG. Besides, ablation studies show its convergence and defense mechanism.

4.1 Experiment Setting

Datasets. We test our RUNG with the node classification task on two widely used real-world citation networks, Cora ML and Citeseer [29], as well as a large-scale networks Ogbn-Arxiv [30]. We adopt the data split of 10% training, 10% validation, and 80% testing, and report the classification accuracy of the attacked nodes following [5]. Each experiment is averaged over 5 different random splits.

Baselines. To evaluate the performance of RUNG, we compare it to ℓ 2 other representative baselines. Among them, MLP, GCN [16], APPNP [28], and GAT [31] are undefended vanilla models. GNNGuard [2], RGCN [11], GRAND [8], ProGNN [3], Jaccard-GCN [6], SVD-GCN [7], EvenNet [32], HANG [13], NoisyGNN [33], and GARNET [34] are representative robust GNNs. Besides, SoftMedian and TWIRLS are representative methods with ℓ 1 -based graph smoothing 5 . We also evaluate a variant of TWIRLS with thresholding attention (TWIRLS-T). For RUNG, we test two variants: default RUNG (Eq. (8)) and RUNGℓ 1 with ℓ 1 penalty ( ρ ( y ) = y ).

Hyperparameters. The model hyperparameters including learning rate, weight decay, and dropout rate are tuned as in [5]. Other hyperparameters follow the settings in the original papers. RUNG uses an MLP connected to 10 graph aggregation layers following the decoupled GNN architecture of APPNP. ˆ λ = 1 1+ λ is tuned in {0.7, 0.8, 0.9}, and γ tuned in { 0 . 5 , 1 , 2 , 3 , 5 } . We chose the

hyperparameter setting that yields the best robustness without a notable impact (smaller than 1% ) on the clean accuracy following [35].

Attack setting. We use the PGD attack [36] to execute the adaptive evasion and poisoning topology attack since it delivers the strongest attack in most settings [5]. The adaptive attack setting is provided in Appendix F. The adversarial attacks aim to misclassify specific target nodes ( local attack ) or the entire set of test nodes ( global attack ). To avoid a false sense of robustness, our adaptive attacks directly target the victim model instead of the surrogate model. Additionally, we include the transfer attacks with a 2-layer GCN as the surrogate model. We also include graph injection attack following the setting in TDGIA [37].

4.2 Adversarial Robustness

QN-IRLS

IRLS, n = 0.5

IRLS, n = 0.1

IRLS, n = 0.6

2.0

1.5

1e4

H

1.0

0.5

0

5

10

15

20

Layernumber

1e7

8

22

4

6

?

e1

RUNG

l2

M

W

2

0

10

20

30

40

Budget/%

'n'

function

0.6

0.4

RUNG

1

l2

Density

0.2

0.0

0

2

4

6

8

10

Featuredifference

Here we evaluate the the performance of RUNG against the baselines under different settings. The results of local and global adaptive attacks on Cora ML are presented in Table 1 and Table 2, while those on Citeseer are presented in Table 3 and Table 4 in Appendix E due to space limits. We summarize the following analysis from Cora ML, noting that the same observations apply to Citeseer.

Under adaptive attacks, many existing defenses are not significantly more robust than undefended models. The ℓ 1 -based models such as TWIRLS, SoftMedian, and RUNGℓ 1 demonstrate considerable and closely aligned robustness under both local and global attacks, which supports our unified ℓ 1 -based robust view analysis in Section 2.2.

RUNGexhibits significant improvements over all baselines across various budgets under both global and local attacks. Local attacks are stronger than global attacks since local attacks concentrate on targeted nodes. The robustness improvement of RUNG appears to be more remarkable in local attacks.

When there is no attack, RUNG largely preserves an excellent clean performance. RUNG also achieves state-of-the-art performance under small attack budgets.

4.3 Ablation study

Convergence. To verify the advantage of our QN-IRLS method in Eq (7) over the first-order IRLS in Eq (6), we show the objective H on each layer in Figure 5. It can be observed that our stepsize-free QN-IRLS demonstrates the best convergence as discussed in Section 3.

Estimation bias. The bias effect in ℓ 1 -based GNNs and the unbiasedness of our RUNG can be empirically verified. We quantify the bias with ∑ i ∈V ∥ f i -f ⋆ i ∥ 2 2 , where f ⋆ i and f i denote the aggregated feature on the clean graph and attacked graph, respectively. As shown in Figure 6, when the budget scales up, ℓ 1 GNNs exhibit a notable bias, whereas RUNG has nearly zero bias. We provide comprehensive discussion of unbiasedness of RUNG in Appendix D.

Defense Mechanism To further investigate how our defense takes effect, we analyze the edges added under adaptive attacks. The distribution of the node feature differences ∥ ∥ f i / √ d i -f j / √ d j ∥ ∥ 2 of attacked edges is shown in Figure 7 for different graph signal estimators. It can be observed that our RUNG forces the attacker to focus on the edges with a small feature difference, indicating that our RUNG can improve robustness by down-weighting or pruning some malicious edges that connect distinct nodes. Therefore, the attacks become less influential, which explains why RUNG demonstrates outstanding robustness.

Transfer Attacks. In addition to the adaptive attack, we also conduct a set of transfer attacks that take every baseline GNN as the surrogate model to comprehensively test the robustness of RUNG, following the unit test attack protocol proposed in [5]. We summarize the results on Cora ML and Citeseer in Figure 9 and Figure 10 in Appendix E due to space limits. All transfer attacks are weaker than the adaptive attack in Section 4.2, indicating the necessity to evaluate the strongest adaptive attack to avoid the false sense of security emphasized in this paper. Note that the attack transferred from RUNG model is slightly weaker than the adaptive attack since the surrogate and victim RUNG models have different model parameters in the transfer attack setting.

Hyperparameters. Due to the space limit, we provide the additional ablation studies on the hyperparameters (including γ and λ in MCP as well as the number of layers) of RUNG in Appendix G.

The results offer an overview strategy for the choice of optimal hyperparameters. We can observe that γ in MCP has a significant impact on the performance of RUNG. Specifically, a larger γ makes RUNG closer to an ℓ 1 -based model, while a smaller γ encourages more edges to be pruned. This pruning helps RUNG to remove more malicious edges and improve robustness, although a small γ may slightly reduce clean performance.

Robustness under various scenarios. Besides the evaluation under strong adaptive attacks, we also validate the consistent effectiveness of our proposed RUNG under various scenarios, including Transfer attacks (Appendix E.2), Poisoning attacks (Appendix E.3), Large scale datasets (Appendix E.4), Adversarial training (Appendix E.5), Graph injection attacks (Appendix E.6).

5 Related Work

To the best of our knowledge, although there are works unifying existing GNNs from a graph signal denoising perspective [20], no work has been dedicated to uniformly understand the robustness and limitations of robust GNNs such as SoftMedian [4], SoftMedoid [18], TWIRLS [14], ElasticGNN [15], and TVGNN [19] from the ℓ 1 robust statistics and bias analysis perspectives. To mitigate the estimation bias, MCP penalty is promising since it is well known for its near unbiasedness property [25] and has been applied to the graph trend filtering problem [26] to promote piecewise signal modeling, but their robustness is unexplored. Nevertheless, other robust GNNs have utilized alternative penalties that might alleviate the bias effect. For example, GNNGuard [2] prunes the edges whose cosine similarity is too small. Another example is that TWIRLS [14] with a thresholding penalty can also exclude edges using graph attention. However, the design of their edge weighting or graph attention is heuristic-based and exhibits suboptimal performance compared to the RUNG proposed in this work.

6 Conclusion & Limitation

In this work, we propose a unified view of ℓ 1 robust graph smoothing to uniformly understand the robustness and limitations of representative robust GNNs. The established view not only justifies their improved and closely aligned robustness but also explains their severe performance degradation under large attack budgets by a novel estimation bias analysis. To mitigate the estimation bias, we propose a robust and unbiased graph signal estimator. To solve this non-trivial estimation problem, we design a novel and efficient Quasi-Newton IRLS algorithm that can better capture the landscape of the optimization problem and converge stably with a theoretical guarantee. This algorithm can be unfolded and used as a building block for constructing robust GNNs with Robust Unbiased Aggregation (RUNG). As verified by our experiments, RUNG provides the best performance under strong adaptive attacks among all the baselines. Furthermore, RUNG also covers many classic GNNs as special cases. Most importantly, this work provides a deeper understanding of existing approaches and reveals a principled direction for designing robust GNNs.

Regarding the limitations, first, the improvement of RUNG is more significant under large budgets compared to the robust baselines. Second, we primarily include experiments on homophilic graphs in the main paper, but we can generalize the proposed robust aggregation to heterophilic graphs in future work. Third, although our Quasi-Newton IRLS algorithm has exhibited excellent convergence compared to the vanilla IRLS, the efficiency of RUNG could be further improved.

Acknowledgment

Zhichao Hou and Dr. Xiaorui Liu are supported by the National Science Foundation (NSF) National AI Research Resource Pilot Award, Amazon Research Award, NCSU Data Science Academy Seed Grant Award, and NCSU Faculty Research and Professional Development Award.

References

Yao Ma and Jiliang Tang. Deep learning on graphs . Cambridge University Press, 2021.

Xiang Zhang and Marinka Zitnik. Gnnguard: Defending graph neural networks against adversarial attacks. Advances in neural information processing systems , 33:9263-9275, 2020.

Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 66-74, 2020.

Simon Geisler, Tobias Schmidt, Hakan ¸ Sirin, Daniel Zügner, Aleksandar Bojchevski, and Stephan Günnemann. Robustness of graph neural networks at scale. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 7637-7649. Curran Associates, Inc., 2021.

Felix Mujkanovic, Simon Geisler, Stephan Günnemann, and Aleksandar Bojchevski. Are defenses for graph neural networks robust? In Neural Information Processing Systems, NeurIPS , 2022.

Huijun Wu, Chen Wang, Yuriy Tyshetskiy, Andrew Docherty, Kai Lu, and Liming Zhu. Adversarial examples on graph data: Deep insights into attack and defense. arXiv preprint arXiv:1903.01610 , 2019.

Negin Entezari, Saba A Al-Sayouri, Amirali Darvishzadeh, and Evangelos E Papalexakis. All you need is low (rank) defending against adversarial attacks on graphs. In Proceedings of the 13th International Conference on Web Search and Data Mining , pages 169-177, 2020.

Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jie Tang. Graph random neural networks for semi-supervised learning on graphs. Advances in neural information processing systems , 33:22092-22103, 2020.

Zhijie Deng, Yinpeng Dong, and Jun Zhu. Batch virtual adversarial training for graph convolutional networks. AI Open , 2023.

Jinyin Chen, Xiang Lin, Hui Xiong, Yangyang Wu, Haibin Zheng, and Qi Xuan. Smoothing adversarial training for gnn. IEEE Transactions on Computational Social Systems , 8(3):618-629, 2020.

Dingyuan Zhu, Ziwei Zhang, Peng Cui, and Wenwu Zhu. Robust graph convolutional networks against adversarial attacks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1399-1407, 2019.

T Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bronstein. Graph-coupled oscillator networks. In International Conference on Machine Learning , pages 18888-18909. PMLR, 2022.

Kai Zhao, Qiyu Kang, Yang Song, Rui She, Sijie Wang, and Wee Peng Tay. Adversarial robustness in graph neural networks: A hamiltonian approach. Advances in Neural Information Processing Systems , 36, 2024.

Yongyi Yang, Tang Liu, Yangkun Wang, Jinjing Zhou, Quan Gan, Zhewei Wei, Zheng Zhang, Zengfeng Huang, and David Wipf. Graph neural networks inspired by classical iterative algorithms. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11773-11783. PMLR, 18-24 Jul 2021.

Xiaorui Liu, Wei Jin, Yao Ma, Yaxin Li, Hua Liu, Yiqi Wang, Ming Yan, and Jiliang Tang. Elastic graph neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6837-6849. PMLR, 18-24 Jul 2021.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Peter J Huber. Robust statistics , volume 523. John Wiley & Sons, 2004.

Simon Geisler, Daniel Zügner, and Stephan Günnemann. Reliable graph neural networks via robust aggregation. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

Jonas Berg Hansen and Filippo Maria Bianchi. Total variation graph neural networks. In Proceedings of the 40th international conference on Machine learning . ACM, 2023.

Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , CIKM '21, page 1202-1211, New York, NY, USA, 2021. Association for Computing Machinery.

David L Donoho. De-noising by soft-thresholding. IEEE transactions on information theory , 41(3):613-627, 1995.

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological) , 58(1):267-288, 1996.

Hui Zou. The adaptive lasso and its oracle properties. Journal of the American Statistical Association , 101(476):1418-1429, 2006.

Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association , 96(456):1348-1360, 2001.

Cun-Hui Zhang. Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics , 38(2):894 - 942, 2010.

Rohan Varma, Harlin Lee, Jelena Kovaˇ cevi´ c, and Yuejie Chi. Vector-valued graph trend filtering with non-convex penalties. IEEE transactions on signal and information processing over networks , 6:48-62, 2019.

Paul W Holland and Roy E Welsch. Robust regression using iteratively reweighted least-squares. Communications in Statistics-theory and Methods , 6(9):813-827, 1977.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph attention networks. stat , 1050(20):10-48550, 2017.

Runlin Lei, Zhen Wang, Yaliang Li, Bolin Ding, and Zhewei Wei. Evennet: Ignoring odd-hop neighbors improves robustness of graph neural networks. Advances in Neural Information Processing Systems , 35:4694-4706, 2022.

Sofiane Ennadir, Yassine Abbahaddou, Johannes F Lutzeyer, Michalis Vazirgiannis, and Henrik Boström. A simple and yet fairly effective defense for graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 21063-21071, 2024.

Chenhui Deng, Xiuyu Li, Zhuo Feng, and Zhiru Zhang. Garnet: Reduced-rank topology learning for robust and scalable graph neural networks. In Learning on Graphs Conference , pages 3-1. PMLR, 2022.

Aleksandar Bojchevski and Stephan Günnemann. Certifiable robustness to graph perturbations. In Neural Information Processing Systems , 2019.

Kaidi Xu, Hongge Chen, Sijia Liu, Pin-Yu Chen, Tsui-Wei Weng, Mingyi Hong, and Xue Lin. Topology attack and defense for graph neural networks: an optimization perspective. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 3961-3967, 2019.

Xu Zou, Qinkai Zheng, Yuxiao Dong, Xinyu Guan, Evgeny Kharlamov, Jialiang Lu, and Jie Tang. Tdgia: Effective injection attacks on graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2461-2471, 2021.

Emmanuel J. Candès, Michael B. Wakin, and Stephen P. Boyd. Enhancing sparsity by reweighted l 1 minimization. Journal of Fourier Analysis and Applications , 14(5-6):877-905, Dec 2008. Funding by NSF.

Amir Beck and Shoham Sabach. Weiszfeld's method: Old and new results. Journal of Optimization Theory and Applications , 164(1):1-40, 2015.

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology , 58(1):267-288, 1996.

Cun-Hui Zhang. Nearly unbiased variable selection under minimax concave penalty. 2010.

Simon Geisler, Tobias Schmidt, Hakan ¸ Sirin, Daniel Zügner, Aleksandar Bojchevski, and Stephan Günnemann. Robustness of graph neural networks at scale. Advances in Neural Information Processing Systems , 34:7637-7649, 2021.

A Bias Accumulation of ℓ 1 Models

A.1 Details of the Numerical Simulation Settings

To provide a more intuitive illustration of the estimation biases of different models, we simulate a mean estimation problem on synthetic data since most message passing schemes in GNNs essentially estimate the mean of neighboring node features. In the context of mean estimation, the bias is measured as the distances between different mean estimators and the true mean. We firstly generated clean samples { x i } n i =1 (blue dots) and the outlier samples { x i } n + m i = n +1 (red dots) from 2-dimensional Gaussian distributions, N ((0 , 0) , 1) and N ((8 , 8) , 0 . 5) , respectively. We calculate the mean of clean samples 1 n ∑ n i =1 x i as the ground truth of the mean estimator. Then we estimate the mean of all the samples by solving arg min z ∑ n + m i =1 η ( z -x i ) using the Weiszfeld method [38, 39], where η ( · ) can take different norms such as ℓ 2 norm ∥ · ∥ 2 2 and ℓ 1 norm ∥ · ∥ 2 .

The mean estimators are formulated as minimization operators

where n is the number of clean samples and m is the number of adversarial samples.

ℓ 1 estimator. The ℓ 1 estimator ( η ( y ) := ∥ y ∥ 2 ), essentially is the geometric median. We adopted the Weiszfeld method to iteratively reweight z to minimize the objective, following

where w ( k ) i = 1 ∥ z ( k ) -x i ∥ 2 . This can be seen as a gradient descent step of z ( k +1) = z ( k ) -α ∇ z ∑ i ∥ z -x i ∥ 2 = z ( k +1) -α ∑ i z ( k ) -x i ∥ z ( k ) -x i ∥ 2 . Taking α = 1 ∑ i w ( k ) i instantly yields Eq. (10).

MCP-based estimator. We therefore adopt a similar approach for the MCP-based estimator ('Ours' in Fig. Figure 2), where η ( y ) := ρ γ ( y ) :

Denoting max(0 , ∥ z ( k ) -x i ∥ -1 2 -1 γ ) as w i , and then α = 1 ∑ i w i yields a similar reweighting iteration z ( k +1) = ∑ i w ( k ) i x i ∑ i w ( k ) i .

ℓ 2 estimator. It is worth noting that the same technique can be applied to the ℓ 2 estimator with ρ ( z ) := ∥ z ∥ 2 2 . The iteration becomes

and α = 1 n + m yields z ( k +1) = 1 n + m ∑ i x i , which gives the mean of all samples in one single iteration.

Similarities between this mean estimation scenario and our QN-IRLS in graph smoothing can be observed, both of which involve iterative reweighting to estimate similar objectives. The approximated Hessian in our QN-IRLS resembles the Weiszfeld method, canceling the z ( k ) by tuning the stepsize.

In Figure 2, we visualize the generated clean samples and outliers, as well as the ground truth means and the mean estimators with η ( · ) = ∥ · ∥ 2 2 or ∥ · ∥ 2 under different outlier ratios (15%,

30%, 45%). The results show that the ℓ 2 -based estimator deviates far from the true mean, while the ℓ 1 -based estimator is more resistant to outliers, which explains why ℓ 1 -based methods exhibit stronger robustness. However, as the ratio of outliers escalates, the ℓ 1 -based estimator encounters a greater shift from the true mean due to the accumulated bias caused by outliers. This observation explains why ℓ 1 -based graph smoothing models suffer from catastrophic degradation under large attack budgets. Our estimator keeps much closer to the ground truth than other estimators with the existence of outliers.

A.2 Additional Simulation Results and Discussions

Here, we complement Figure 2 with the settings of higher attack budgets. As the outlier ratio exceeds the breakdown point 50% , we observe that our MCP-based mean estimator can correctly recover the majority of the samples, i.e. converge to the center of 'outliers'.

Ours,trajectory

Clean sample

Ours, end

Ours,initial

10

Feature dimension 2

5

2

Q

X

-2

10%outliers

25%outliers

40% outliers

-4

10

Feature dimension 2

X

5

2

60%outliers

75%outliers

90% outliers

-4

-20

2

5

10

-4-20

2

5

10

-4-20

2

5

10

Feature dimension 1

Feature dimension 1

Feature dimension1

B Convergence Analysis

To begin with, we will provide an overview of our proof, followed by a detailed presentation of the formal proof for the convergence analysis.

Overview of proof. First, for both IRLS and QN-IRLS, we construct, for F ( k ) in every iteration k , a quadratic upper bound ˆ H that satisfies ˆ H + C ≥ H where the equality is reached at F ( k ) . Then we can minimize ˆ H to guarantee the iterative descent of H since H ( F ( k +1) ) ≤ ˆ H ( F ( k +1) ) + C ≤ ˆ H ( F ( k ) ) + C = H ( F ( k ) ) .

To find the F ( k +1) such that ˆ H ( F ( k +1) ) ≤ ˆ H ( F ( k ) ) , IRLS simply adopts the plain gradient descent F ( k +1) = F ( k ) -η ∇ ˆ H ( F ( k ) ) whose convergence condition can be analyzed with the β -smoothness of the quadratic ˆ H (Theorem 1). To address the problems of IRLS as motivated in Section 3.2, our Quasi-Newton IRLS utilizes the diagonal approximate Hessian ˆ Q to scale the update step size in different dimensions respectively as F ( k +1) = F ( k ) -ˆ Q -1 ∇ ˆ H ( F ( k ) ) . Thereafter, by bounding the Hessian with 2 ˆ Q , the descent condition of ˆ H is simplified (Theorem 2).

Lemma 1. For any ρ ( y ) satisfying dρ ( y ) dy 2 is non-increasing, denote y ij := ∥ ∥ ∥ ∥ f i √ d i -f j √ d j ∥ ∥ ∥ ∥ 2 , then

̸

H ( F ) = ∑ ( i,j ) ∈E ,i = j ρ ( y ij ) + λ ∑ i ∈V ∥ f i -f (0) i ∥ 2 2 has the following upper bound:

̸

where W ( k ) ij = ∂ρ ( y ) ∂y 2 ∣ ∣ y = y ( k ) ij and y ( k ) ij = ∥ ∥ ∥ ∥ f ( k ) i √ d i -f ( k ) j √ d j ∥ ∥ ∥ ∥ 2 and C = H ( F ( k ) ) -ˆ H ( F ( k ) ) is a constant. The equality in Eq. (15) is achieved when F = F ( k ) .

Proof. Let v = y 2 and define ψ ( v ) := ρ ( y ) = ρ ( √ v ) . Then ψ is concave since dψ ( v ) dv = dρ ( y ) dy 2 is non-increasing. According to the concavity property, we have ψ ( v ) ≤ ψ ( v 0 ) + ψ ′ ( ν ) ∣ ∣ ν = v 0 ( v -v 0 ) . Substitute v = y 2 , v 0 = y 2 0 , we obtain:

where the inequality is reached when y = y 0 . Next, substitute y = y ij and y 0 = y ( k ) ij , we can get ρ ( y ij ) ≤ W ( k ) ij y 2 ij + C ( y ( k ) ij ) which takes the equality at y ij = y ( k ) ij . Finally, by summing up both sides and add a regularization term, we can prove the Eq. (15).

Remark 1. It can be seen that the definition of ˆ H depends on F ( k ) , which ensures that the bound is tight when F = F ( k ) . This tight bound condition is essential in the majorization-minimization algorithm as seen in Theorem 1.

̸

Lemma 2. For ˆ H = ∑ ( i,j ) ∈E ,i = j W ij y 2 ij + λ ∑ i ∈V ∥ f i -f (0) i ∥ 2 2 , the gradient and Hessian w.r.t. F 6 satisfy

and

where q m = ∑ j W mj A mj /d m and ˜ A ij = A ij √ d i d j is the symmetrically normalized adjacency matrix.

̸

Proof. Follow A = A ⊤ and define y 2 ij := ∥ ∥ ∥ ∥ f i √ d i -f j √ d j ∥ ∥ ∥ ∥ 2 2 , then the first-order gradient of ∑ ( i,j ) ∈E ,i = j W ij y 2 ij will be

̸

̸

and the second-order hessian will be:

̸

̸

̸

To minimize ˆ H , the gradient descent update rule takes the form of Eq. (6). One may assume that when η is chosen to be small enough, ˆ H ( F ( k +1) ) ≤ ˆ H ( F ( k ) ) . For a formal justification, we have Theorem 1 to determine the convergence condition of η .

Theorem 1. If F ( k ) follows the update rule in Eq. (6) , where the ρ satisfies that dρ ( y ) dy 2 is nondecreasing for y ∈ (0 , ∞ ) , then a sufficient condition for H ( F ( k +1) ) ≤ H ( F ( k ) ) is that the step size η satisfies 0 < η ≤ ∥ diag ( q ( k ) ) -W ( k ) ⊙ ˜ A + λ I ∥ -1 2 .

Proof. The descent of ˆ H ( F ) can ensure the descent of H ( F ) since H ( F ( k +1) ) ≤ ˆ H ( F ( k +1) ) ≤ ˆ H ( F ( k ) ) = H ( F ( k ) ) . Therefore, we only need to prove ˆ H ( F ( k +1) ) ≤ ˆ H ( F ( k ) ) .

Noting that ˆ H is a quadratic function and F ( k +1) -F ( k ) = -η ∇ ˆ H ( F ( k ) ) , then ˆ H ( F ( k +1) ) and ˆ H ( F ( k ) ) can be connected using Taylor expansion 6 , where ∇ ˆ H and ∇ 2 ˆ H is given in Lemma 2:

Insert ∇ 2 ˆ H ( F ( k ) ) = 2( diag ( q ) -W ( k ) ⊙ ˜ A + λ I ) from Lemma 2 into the above equation and we can find a sufficient condition for ˆ H ( F ( k +1) ) ≤ ˆ H ( F ( k ) ) to be

or

Now we prove that when taking the Quasi-Newton-IRLS step as in Eq. (7), the objective ˆ H is guaranteed to descend. Since the features in different dimensions are irrelevant, we simplify our notations as if feature dimension was 1 . One may easily recover the general scenario by taking the trace.

̸

Lemma3. 2 ˆ Q -∇ 2 ˆ H ( y ) is positive semi-definite, where ˆ H = ∑ ( i,j ) ∈E ,i = j W ij y 2 ij + λ ∑ i ∈V ∥ f i -f (0) i ∥ 2 2 , ˆ Q = 2( diag ( q ) + λ I ) , and q m = ∑ j W mj A mj /d m .

Proof. In Lemma 2, we have ∇ 2 ˆ H ( y ) = 2( diag ( q ) + λ I -W ⊙ ˜ A ) , then

Recall how we derived Eq. (27) from Eq. (15), where we proved that

̸

which holds for all F . Similarly, the equation still holds after flipping the sign before f j / √ d j and W ⊙ ˜ A . We then have this inequality: ∀ F , ∀ λ ≥ 0

̸

Thus, ( diag ( q ) + W ⊙ ˜ A + λ I ) ⪰ 0 , and thus 2 ˆ Q -∇ 2 ˆ H ( y ) ⪰ 0 .

Using Lemma 1 and Lemma 3 we can prove Theorem 2. Note that we continue to assume #feature = 1 for simplicity but without loss of generality 6 .

Theorem 2. If F ( k +1) follows update rule in Eq. (7) , where ρ satisfies that dρ ( y ) dy 2 is non-decreasing ∀ y ∈ (0 , ∞ ) , it is guaranteed that H ( F ( k +1) ) ≤ H ( F ( k ) ) .

Proof. Following the discussions in Theorem 1, we only need to prove ˆ H ( F ( k +1) ) ≤ ˆ H ( F ( k ) ) .For the quadratic ˆ H , we have:

We can define Q ( y ) = 2 ˆ Q ( y ) in Lemma 3 such that Q ( y ) -∇ 2 ˆ H ( y ) ⪰ 0 , then

Then an upper bound of ˆ H ( x ) can be found by inserting Eq. (45) into Eq. (44).

Then, insert Q = 2 ˆ Q into Eq. (46). Note that ˆ Q := 2( diag ( q ) + ˆ λ I ) , so ˆ Q ⪰ 0 and ˆ Q ⊤ = ˆ Q . Thereafter, the update rule x = y -ˆ Q -1 ∇ ˆ H ( y ) in Eq. (7) gives

Therefore, our QN-IRLS in Eq. (7) is guaranteed to descend.

C Computation Efficiency

Our RUNG model preserves advantageous efficiency even adopting the quasi-Newton IRLS algorithm.

C.1 Time Complexity Analysis

Each RUNG layer involves computing W , q , and the subsequent aggregations. We elaborate on them one by one. We denote the number of feature dimensions d , the number of nodes n , and the number of edges m , which are assumed to satisfy m ≫ 1 , n ≫ 1 and d ≫ 1 . The number of layers is denoted as k . The asymptotic computation complexity is denoted as O ( · ) .

̸

Computation of W ⊙ A and W ⊙ ˜ A . W := 1 i = j dρ γ ( y ij ) dy 2 ij is the edge weighting matrix dependent on the node feature matrix F . The computation of y ij = ∥ f i √ d i -f j √ d j ∥ 2 is O ( d ) and that of dρ γ ( y ij ) dy 2 ij is O (1) . W ij only needs computing when ( i, j ) ∈ E , because ∀ ( i, j ) / ∈ E , W ij will be masked out by A or ˜ A anyways. Each element of W involves computation time of O ( d ) and m elements are needed. In total, W costs O ( md ) , and W ⊙ A and W ⊙ ˜ A cost O ( md + m ) = O ( md ) .

Computation of ˆ Q -1 . ˆ Q -1 := 1 2 ( diag ( q ( k ) ) + λ I ) -1 is the inverse Hessian in our quasi-Newton IRLS. Because ˆ Q is designed to be a diagonal matrix, its inverse can be evaluated as element-wise reciprocal which is efficient. As for q := ∑ j W ( k ) mj A mj /d m , only existing edges ( i, j ) ∈ E need evaluation in the summation. Therefore, this computation costs O ( m ) . Thus, ˆ Q -1 costs O ( m ) in total.

Computation of aggregation. An RUNG layer follows

which combines the quantities calculated above. An extra graph aggregation realized by the matrix multiplication between W ⊙ ˜ A and F is required, costing O ( md ) . The subsequent addition to F (0) and the multiplication to the diagonal ˆ Q -1 both cost O ( nd ) .

Stacking layers. RUNGunrolls the QN-IRLS optimization procedure, which has multiple iterations. Therefore, the convergence increase that QN-IRLS introduces allows a RUNG with fewer layers and increases the overall complexity. It is worth noting that the QN-IRLS utilizes a diagonal approximated Hessian, and thus the computation per iteration is also kept efficient as discussed above.

Summing up all the costs, we have the total computational complexity of our RUNG, O (( m + n ) kd ) . Our RUNG thus scales well to larger graph datasets such as ogbn-arxiv.

Space Complexity Analysis The only notable extra storage cost is W whose sparse layout takes up O ( m ) . This is the same order of size as the adjacency matrix itself, thus not impacting the total asymptotic complexity.

C.2 Alternative Perspective

In fact, the above analysis can be simplified when we look at the local aggregation behavior of RUNG. For node i , it's updated via aggregation f i = 2 ˆ Q -1 ii (( ∑ j ∈N ( i ) W ij f j ) + λ f (0) i ) . The summation over neighbors' f j will give O ( m ) in the total time complexity in each feature dimension, and W ij involves O ( d ) computations for each neighbor. This sums up to O ( md ) as well. Essentially, the high efficiency of RUNG originates from that every edge weighting in our model involves only the 2 nodes on this edge.

D Comprehensive Bias Analysis

In this section, we provide multiple evidence from various perspectives to reveal the estimation bias of ℓ 1 -based estimation as follows.

(1) From the theoretical perspective, the extensive literature on high-dimensional statistics [40, 41] has proved that ℓ 1 regularization induces an estimation bias.

(2) From the algorithm perspective, in Section 2.3, we provide the explanation on the ℓ 1 -based estimation problem solver. Specifically, the soft-thresholding operator S λ ( θ ) := sign ( θ ) max( | θ | -λ, 0) induced by the ℓ 1 regularized problem causes a constant shrinkage for θ larger than λ , enforcing the estimator to be biased towards zero with the magnitude λ .

(3) From the numerical simulation in Section 2.3, we provide an example of mean estimation to verify this estimation bias. As shown in Figure 2, the ℓ 1 estimator (green) deviates further from the true mean as the ratio of outliers escalates. This can be clearly explained as the effect of the accumulation of estimation bias. In other words, each outlier results in a constant bias, and the bias accumulates with more outliers.

(4) From the performance perspective, ℓ 1 -based GNNs such as SoftMedian, TWIRLS, and RUNGℓ 1 (the ℓ 1 variant of our model) suffer from significant performance degradation when the attack budget increases.

(5) From our ablation study in Figure 6, we quantify the estimation bias of the aggregated feature f ⋆ i on the attacked graph from the feature f i on the clean graph: ∑ i ∈V ∥ f i -f ∗ i ∥ 2 2 . The results demonstrate that ℓ 1 -based GNN produces biased estimation under adversarial attacks and the bias indeed scales up with the attack budget. However, our proposed RUNG method exhibits a nearly zero estimation bias under the same attacking budgets.

All of this evidence can convincingly support our claim that ℓ 1 -based robust estimator suffers from the estimation bias, which validates the motivation of our new algorithm design.

E Additional experiment results

In this section, we present the experiment results that are not shown in the main paper due to space limits.

E.1 Adaptive Attacks

Table 3 and Table 4 are the results of adaptive local and global attacks on Citeseer, referred to in Section 4.2.

E.2 Transfer Attacks

Table 5 and Table 6 are the results of transfer global attacks on Cora ML and Citeseer. Figure 9 and Figure 10 are the experiment results of our RUNG attacked by transfer attacks generated on different surrogate models as mentioned in Section 4.3.

Figure 11 shows results of global evasion transfer attacks between different models on Cora ML. Our observations are summarized below:

The attacks generated by RUNG are stronger when applied to more robust models like SoftMedian, while are not strong against undefended or weakly defended models.

84.1

5%

F

83.683.783.784.183.983.283.783.6

82.182.382.382.6

79.6

79.5

10%

82.982.9 83.083.883.282.683.082.7

80.2 80.180.581.1

76.7

Accuracy / %

Budget

20%

81.4 81.3 81.3

83.1

81.7

81.281.280.8

76.576.676.377.6

72.5

75.0

30%7

79.779.279.8

82.2

80.1

78.6

80.1

78.9

73.573.272.574.2

68.5

70.5

40%

77.5

77.977.7

81.3

79.0

76.5

78.4

77.3

70.770.268.670.5

66.0

66.0

GCN

GAT

APPNP

RGCN

GRAND

ProGNN

TWIRLS

RUNG-l1

RUNG

GNNGuard

Jaccard-GCN

SoftMedian

Transferfrom

For ℓ 1 GNNs, the attacks are the strongest when transferred from ℓ 1 GNNs. This supports again our unified view on ℓ 1 GNNs. An exception is TWIRLS because it only has one attention layer and does not always converge to the actual ℓ 1 objective.

Accuracy / %

74.3

5%7

74.274.274.174.174.274.274.274.2

73.7

73.373.473.9

72.6

72.9

10%  74.2 74.274.074.274.2 74.274.374.273.773.273.373.8

71.8

Budget

20%

74.174.373.974.274.174.274.274.2

73.5

72.672.6

73.5

70.4

71.5

30% 74.174.273.974.174.274.174.174.2

73.4

472.672.2

73.4

69.5

70.1

40%74.074.11

73.774.074.174.174.174.2

73.2

72.3

71.9

73.2

68.6

68.6

GCN

GAT

APPNP

GNNGuard

RGCN

GRAND

Jaccard-GCN

ProGNN

SoftMedian

TWIRLS

TWIRLS-T

RUNG-l1

RUNG

Transferfrom

Budget:0.050

Budget:0.100

GCN -

76.0

78.0

77.7

80.2

79.9

82.3

81.5

81.8

83.7

81.9

70.6

72.7

72.7

76.4

78.1

79.7

79.7

79.0

82.9

79.0

APPNP

77.6

76.8

78.9

79.5

80.4

82.4

81.7

82.0

83.6

82.0

73.6

70.9

74.3

77.2

78.9

80.0

80.0

79.1

82.9

79.2

RGCN -

76.5

77.8

75.8

79.6

80.4

81.9

81.9

82.0

83.9

81.4

71.2

72.2

70.0

75.3

78.2

79.2

79.9

79.4

83.2

78.5

Transfer from

GRAND

79.4

78.7

79.2

76.9

81.1

82.8

82.0

82.0

83.2

82.3

75.4

73.1

74.6

71.7

79.5

80.6

80.3

79.0

82.6

79.3

GNNGuard

80.9

81.7

81.6

83.0

76.3

84.0

82.7

83.5

84.1

83.9

78.4

78.5

79.0

80.6

72.7

83.0

82.1

82.4

83.8

82.8

RUNG-f1

80.1

80.2

80.1

81.0

80.5

79.0

81.8

81.5

82.6

80.3

76.1

75.9

76.2

77.5

78.6

74.8

79.8

78.0

81.1

76.3

TWIRLS-f1

80.1

80.0

80.4

81.5

80.4

82.0

78.0

81.7

82.3

81.1

76.3

75.4

76.9

78.0

78.2

79.0

74.0

78.3

80.1

77.5

SoftMedian

81.6

81.8

82.3

82.2

81.3

82.5

81.9

79.0

82.1

82.1

78.6

78.4

79.3

79.3

79.7

79.9

80.2

75.8

80.2

79.7

RUNG -

81.7

81.8

81.9

82.1

81.1

82.4

81.6

80.9

79.6

81.5

79.2

78.8

79.4

79.3

79.7

79.4

79.6

77.7

76.7

78.7

Budget:0.200

Budget:0.400

GCN

62.1

64.4

64.4

70.0

74.8

75.1

76.8

73.0

81.3

73.3

49.8

53.0

52.5

60.1

69.9

67.1

72.8

60.4

77.9

63.6

APPNP-

67.0

61.9

68.0

71.3

76.0

75.7

77.4

72.7

81.4

73.8

55.3

50.7

57.3

59.8

70.7

66.5

72.4

61.7

77.5

63.4

RGCN-

62.1

63.1

60.8

68.2

74.8

74.0

77.3

73.2

81.7

72.0

48.7

51.9

46.9

59.0

70.6

66.5

73.9

62.9

79.1

62.5

Transfer from

GRAND-

69.5

65.5

68.0

63.3

77.5

77.7

78.1

75.1

81.2

75.1

59.0

55.0

57.6

52.6

73.0

68.0

73.3

62.5

76.5

64.6

GNNGuard

74.4

74.2

74.9

77.5

66.9

81.0

80.8

80.0

83.1

80.4

67.5

66.4

68.1

72.5

57.0

77.7

78.7

74.0

81.3

75.9

RUNG-f1

70.6

69.9

70.4

72.2

75.2

68.5

76.3

72.0

77.6

70.1

61.9

61.5

61.9

64.2

69.7

60.0

70.5

61.6

70.5

61.3

TWIRLS-f1

70.2

68.6

70.6

72.0

75.0

73.7

68.0

72.4

76.6

71.3

61.6

59.7

62.0

63.5

69.1

64.7

60.0

62.2

70.2

61.9

SoftMedian

74.6

74.0

75.3

75.4

77.3

76.2

77.8

72.1

76.5

75.8

60.7

60.5

61.3

63.9

69.8

60.9

71.4

58.7

70.5

62.4

RUNG

74.6

74.1

74.9

75.2

76.7

75.1

76.4

73.1

72.5

74.2

67.9

67.1

68.1

68.3

72.7

68.0

71.1

65.9

66.0

67.2

LS-l1

AND

ard

G

RUNG

GNN

GNN

RUNG-

TWIRLS

SoftMedian

Elastic

TWIRLS

Elastic

Transfer to

Transfer to

E.3 Poisoning Attacks

We provide the experiment results under poisoning attacks on Cora ML and Citeseer in Table 7 and Table 8, respectively.

E.4 Large Scale Ogbn-Arxiv

In the large scale datasets, we can not directly apply the vanilla PGD attack [36] on them due to excessive requirement of memory and computation on dense matrix. Alternatively, we leverage the PRBCD [42] to scale the PGD attack instead of manipulating on the dense adjacency matrix. We conduct experiments on large scale Ogbn-Arxiv and the results in Table 9 verified the superior robustness of our RUNG model. RUNG's robustness outperforms its ℓ 1 variant which delivers similar performance as SoftMedian and Elastic GNN due to their shared ℓ 1 graph smoothing.

E.5 Adversarial Training

Weconduct the adversarial training following [36] and present the results in Table 10. From the results, we can observe that with adversarial training, the robustness of RUNG can be further improved.

E.6 Graph Injection Attack

The injection attack was conducted following the settings in [37] to evaluate the robustness of different methods. We set up the budget on the number of injected nodes as 100 and the budget on degree as 200. The results in Table 11 show that our RUNG significantly outperforms the baseline models.

F Adaptive Attack Settings

For the adaptive PGD attack we adpoted in the experiments, we majorly followed the algorithm in [5] in the adaptive evasion attack. For the sake of completeness, we describe it below:

In summary, we consider the topology attack setting where the adjacency matrix A is perturbed by δ A whose element δ A ij ∈ { 0 , 1 } . The budget B is defined as B ≥ ∥ δ A ∥ 0 . The PGD attack involves first relaxing A from binary to continuous so that a gradient ascent attack can be conducted on the relaxed graph.

During the attack, the minimization problem below is solved:

where L is carefully designed attack loss function [4, 5], A , F and y target are respectively the graph, node feature matrix and ground truth labels in the dataset, θ is the parameters of the GNN under attack which are not altered in the evasion attack setting. ( I -2 A ) ⊙ δ A is the calculated perturbation that 'flips' the adjacency matrix between 0 and 1 when it is perturbed. The gradient of L attack δ A is computed and utilized to update the perturbation matrix δ A .

After the optimization problem is solved, δ A is projected back to the feasible domain of δ A ij ∈ { 1 } . The adjacency matrix serves as a probability matrix allowing a Bernoulli sampling of the binary adjacency matrix A ′ . The sampling is executed repeatedly so that an A ′ producing the strongest perturbation is finally generated.

G Additional Ablation Study of RUNG

G.1 Hyperparameters

The choice of the hyperparameters γ and λ is crucial to the performance of RUNG. We therefore experimented with their different combinations and conducted adaptive attacks on Cora as shown in Fig. 12.

Recall the formulation of RUNG in Eq.(8):

̸

In the formulation, λ controls the intensity of the regularization in the graph smoothing. In our experiments, we tune ˆ λ := 1 1+ λ which is normalized into (0 , 1) . In Figure 12, the optimal value of ˆ λ can be found almost always near 0 . 9 regardless of the attack budget. This indicates that our penalty function ρ γ is decoupled from γ which makes the tuning easier, contrary to the commonly used formulation of MCP [25].

On the other hand, γ has a more intricate impact on the performance of RUNG. Generally speaking, the smaller γ is, the more edges get pruned, which leads to higher robustness and a lower clean accuracy. We begin our discussion in three cases:

Small attack budget ( 0% , 5% , 10% ). The performance is largely dependent on clean accuracy. Besides, when γ → ∞ , RUNG becomes a state-of-the-art robust ℓ 1 model. Therefore, a small γ

likely introduces more harm to the clean performance than robustness increments over ℓ 1 models. The optimal γ thus at least recovers the performance of ℓ 1 models.

Large attack budget ( 20% , 30% , 40% ). In these cases, γ →∞ is no longer a good choice because ℓ 1 models are beginning to suffer from the accumulated bias effect. The optimal γ is thus smaller (near 0 . 5 ). However, for fairness, we chose the same γ under different budgets in our experiments, so the reported RUNG fixes γ = 3 . In reality, however, when we know the possible attack budgets in advance, we can tune γ for an even better performance.

Very large attack budget ( 50% , 60% ). We did not include these scenarios because almost all GNNs perform poorly in this region. However, we believe it can provide some insights into robust graph learning. Under these budgets, more than half of the edges are perturbed. In the context of robust statistics (e.g. mean estimation), the estimator will definitely break down. However, in our problem of graph estimation, the input node features offer extra information allowing us to exploit the graph information even beyond the breakdown point. In the 'peak' near (0 . 9 , 0 . 5) , RUNG achieves > 70% accuracy which is higher than MLP. This indicates that the edge weighting of RUNG is capable of securely harnessing the graph information even in the existence of strong adversarial attacks. The 'ridge' near a ˆ λ = 0 . 2 , on the other hand, emerges because of MLP. When the regularization dominates, λ →∞ , and ˆ λ → 0 . A small λ is then connected to a larger emphasis on the input node feature prior. Under large attack budgets, MLP delivers relatively good estimation, so a small ˆ λ is beneficial.

Budget: 0

Budget: 5

Budget: 10

Budget: 20

10

0.8558

0.8274

5

0.7989

0.7705

0

0.7421

Budget:30

Budget:40

Budget:50

Budget: 60

0.7137

10

0.6853

0.6568

5

0.6284

0.6000

0

0

0.5

1

0

0.5

1

0

0.5

1

0

0.5

1

V

G.2 GNN Layers

In RUNG, QN-IRLS is unrolled into GNN layers. We would naturally expect RUNG to have enough number of layers so that the estimator converges as desired. We conducted an ablation study on the performance (clean and adversarial) of RUNG with different layer numbers and the results are shown in Fig. Figure 13. We make the following observations:

As the layer number increases, RUNG exhibits better performance. This verifies the effectiveness of our proposed RUGE, as well as the stably converging QN-IRLS.

The performance of RUNG can achieve a reasonably good level even with a small layer number ( 3 -5 layers) with accelerated convergence powered by QN-IRLS. This can further reduce the computation complexity of RUNG.

0.80

一

0.75

0.70

0.65

budget: 5%

budget: 10%

0.60

budget:20%

budget:40%

5

10

15

20

Propagationsteps

H Robustness of GCN and APPNP

In addition to the formulation in section 3 the main text, we can simply apply our edge reweighting technique to the GCN architecture. Essentially the aggregation operation in GCN can be viewed as an APPNP layer with ˆ λ = 0 . The GCN version of a layer in our model can then be formulated as

where M is the learned weight matrix in GCN.

GCN-based defenses are less robust than APPNP-based defenses GCN consists of layers in which both feature transformation and message passing are included. This graph convolution operation will weaken defense methods that rely on edge weighting, such as GNNGuard, models using l 1 penalty as well as our method using MCP penalty.

Consider an edge that is added by the attacker 7 . A successful defense should detect the attack on this edge by the large difference of nodes connected by this edge, and then assign a zero weight or at least a smaller weight to this edge. However in GCN, even if this edge is detected in the first layer's message passing, the subsequent feature transformation makes the node difference less likely to be preserved until the second layer. This is where the attack can evade the defense and is thus a vulnerability allowing adaptive attacks through. According to our experiments, using different defense parameters in different layers of GCN, unfortunately, does not help much either.

On the other hand, in APPNP node features are also altered along the message-passing layers, but the node distance change is more regulated than in GCN since MLP is decoupled from the graph smoothing layers. In the latter submodule, node differences simply decrease, which allows our defense based on node differences.

Experiments It can be seen from Figure 14 that the correlation of node feature differences in different layers of GCN is about 4 times less than in APPNP, which means that an attack detected in the first layer is less likely to continue to be detected in the second layer than in APPNP. This property of GCN makes the many defense methods using GCN architecture as a backbone less robust, as shown in the experiment results in Table 12 and Table 13. Nevertheless, our GCN-based MCP model outperforms the SOTA models using l 1 methods.

20

30

30

cor:0.077

cor:0.252

cor:0.319

GCN

APPNP

25

25

15

6

after last layer of A

20

20

layer

layer

10

15

15

first

last

afterf

10

after I

10

5

5

5

0

2

T

4

6

8

10

0

0

10

20

30

0

0

5

10

15

20

beforefirstlayerof GCN

beforefirstlayerofAPPNP

before first layer of L21

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction (Section 1) closely follow the contribution (at the end of Section 1) in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in the Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the theory in Section 3.2 and the detailed proof in the Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the detailed experimental setting in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will organize the data and code after the submission.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https:// nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the detailed experimental setting in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because it would be too computationally expensive

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We include the theoretical computation analysis in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper closely follows NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the adversarial robustness of Graph Neural Networks (GNNs) against adaptive attacks. The authors propose a unified framework for robust estimation in GNNs to understand and improve their robustness. Specifically, \n\n1. The paper provides a unified view of $\\ell_1$-based robust graph signal smoothing, and tries to reveal the estimation bias leading to performance degradation under attacks.\n\n2. The authors propose a robust and unbiased graph signal estimator (RUGE) to mitigate the estimation bias.\n\n3. They develop an efficient Quasi-Newton Iterative Reweighted Least Squares (IRLS) algorithm to solve the non-smooth and non-convex estimation problem, as well as theoretical guarantees.\n\nSTRENGTHS:\n1. The topic of the paper, robust GNN, is of interest, and a way to alleviate the bias is proposed.\n\n2. A unified perspective on existing robust GNN works is built.\n\n3. A new optimization algorithm, QN-IRLS, is proposed, as well as the corresponding complexity analysis.\n\nWEAKNESSES:\n1. The manuscript is difficult to follow due to poor **writing**. Specifically,\n\n  1a. The phrase “without the false sense of security” in Sections 1 and 2 is unclear. What does it mean, and why is it significant in this context?\n\n  1b. The figures are hard to understand without additional details (either in captions or main text). For instance, in Figure 1, what does the x-axis represent, and how is 200% defined? How is an attack conducted? \nIn Figure 2, how are “clean sample” and “outliers” defined? Similar issues to Figures 5 and 7.\n\n  1c. Some of the equations are unclear. For example, in Equation (1), \\tilde{f} should be formatted similarly to Equation (2).\n\n2. The **contribution** is unclear.\n\n2a. Regarding the unified perspective: it seems inconsistent with the experimental results in Figures 1 and 2. As shown in Figure 2, the $\\ell_2$ estimator deviates further from the true mean, while in Figure 1, SoftMedian performs the worst, which is a special case of ElasticGNN with  p=1  ($\\ell_1$).\n2b. Regarding the optimization: the distinction and connection between QN-IRLS and other standard QN methods is missing.\n\n3.\tThe **experiments** can be further improved.\n\n3a. Only two datasets, Cora and CiteSeer, are included.\n\n3b. While it is understood that space constraints may limit the inclusion of all interesting details in the main text, lines 304-317 occupy significant space without providing primary conclusions or insights.\n\nQUESTIONS:\nIt would be great if the authors could respond to concerns mentioned in the *Weaknesses*.\n\nLIMITATIONS:\nThe authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

comments: SUMMARY:
This paper uncovers that while $\ell_1$-based models exhibit robustness against moderate adaptive attacks, their performance significantly degrades under strong adaptive attacks. The authors investigate the root cause of this phenomenon, identifying estimation bias as the culprit. To address this, they propose a robust and unbiased graph signal estimator. Extensive experiments and ablation studies confirm that their method significantly enhances GNN robustness without compromising clean accuracy.

STRENGTHS:
- This paper provides compelling motivating experiments, observations, and insights. Furthermore, the authors rigorously analyze the root cause of their findings, offering a persuasive and logical explanation.
- Based on these discoveries, the authors propose a simple yet effective method, well-supported both theoretically and empirically.
- Extensive experiments offer thorough analysis to demonstrate the effectiveness of the proposed method.
- The proposed method significantly outperforms other baselines under both local and global adaptive attacks.

WEAKNESSES:
- The datasets considered by the authors are limited to only three citation networks. While I acknowledge that implementing adaptive attacks on new datasets and baselines is significantly cumbersome and requires considerable effort, demonstrating that the proposed methods work well on datasets from other domains would greatly enhance the applicability and recognition of this work.

- There is no available source code for the reproduction,

QUESTIONS:
See the Weaknesses

LIMITATIONS:
The authors addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
The authors point out that some of the successful defenses for GNNs are related to l1-based graph smoothing. Motivated by the bias of l1-based estimators, the authors develop an unbiased variant (RUGE), rooted in high-dimensional statistics. To optimize under such a non-smooth objective, the authors devise a Quasi-Newton Iterative Reweighted Least Squares algorithm. In the empirical evaluation, the authors compare with a rich set of baseline GNNs and defenses on graphs of varying sizes. The authors report the performance on both non-adaptive and adaptive attacks. Moreover, the authors study the influence of hyperparameters and further complement their method with adversarial training.

STRENGTHS:
1. The authors provide a unified view of some of the important defenses in the GNN domain
1. The proposed method is well-motivated and empirically successful
1. The empirical evaluation is extensive and convincing

WEAKNESSES:
1. The paper only studies homophilic datasets
1. The adversarial training is performed in the biased transductive evasion setting of Xu et al [A]. It is better to follow an inductive evasion setting as in [B]. The authors should at least note that the studied setting has limitations.
1. Minor: The references in some cases do not mention the respective venue/journal (e.g., [36])

[A] Xu et al. "Topology Attack and Defense for Graph Neural Networks: An Optimization Perspective" NeurIPS 2019.
[B] Gosch et al. "Adversarial Training for Graph Neural Networks: Pitfalls, Solutions, and New Directions" NeurIPS 2023.

QUESTIONS:
1. How does RUGE perform on heterophilic graphs?

LIMITATIONS:
The limitations are sufficiently addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-11