Robustness of Graph Neural Networks at Scale

1 Introduction The evidence that Graph Neural Networks ( GNNs ) are not robust to adversarial perturbations is compelling [ 14 , 20 , 50 ] . However , the graphs in previous robustness studies are tiny . This is worrying , given that GNNs are already deployed in many real-world Internet-scale applications [ 5 , 44 ] . For example , PubMed [ 33 ] ( 19,717 nodes ) is often considered to be a large-scale graph and around 20 GB of memory is required for an attack based on its dense adjacency matrix . Such memory requirements are impractical and limit advancements of the field . In this work , we set the foundation for the holistic study of adversarial robustness of GNNs at scale . We study graphs with up to 111 million nodes for local attacks ( i.e . attacking a single node ) and 2.5 million nodes for global attacks ( i.e . attacking all nodes at once ) . As it turns out , GNNs at scale are also highly vulnerable to adversarial attacks . In Fig . 1 , we show the substantial improvement of memory efficiency of our attack over a popular prior work for attacking a GNN globally . Analogous to our attacks , our effective defense scales to graphs with 111 million nodes and beyond . Scope . We focus on adversarial robustness w.r.t . structure attacks on GNNs for node classification max Ã s.t . kÃ Ak0 < L ( f✓ ( Ã , X ) ) ( 1 ) with loss function L ( or its surrogate L0 ) , budget , and fixed model parameters ✓ . The GNN f✓ ( A , X ) is applied to a graph G = ( A , X ) with node attributes X 2 Rn⇥d , the adjacency matrix 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A 2 { 0 , 1 } n⇥n , and m edges . We focus on evasion ( test time ) attacks , but our methods can be used in poisoning ( train time ) attacks [ 49 ] . We distinguish between local attacks on a single node and global attacks that target a large fraction of nodes with a shared budget . We study white-box attacks since they have the most powerful threat model and can be used to understand the robustness w.r.t . “ worst-case noise ” of a model , as well as to assess the efficacy of defenses . For these reasons white-box attacks are important and practical from the perspective of a defender . Broader impact . Since we enable the study of robustness at scale , which previously was practically infeasible , an adversary can potentially abuse our attacks . The risk is minimized given that we assume perfect knowledge about the graph , model , and labels . Nonetheless , our findings suggest that one should be careful when deploying GNNs , and highlight that further research is needed . To mitigate this risk , we must be able to evaluate it . We also propose a scalable defense that shows strong performance empirically but we urge practitioners to consider potential trade-offs , e.g . improving robustness at the expense of accuracy for different groups of users . Contributions . We address three major challenges hindering the study of GNNs ’ adversarial robustness at scale and propose viable solutions including an extensive empirical evaluation : ( 1 ) Previous losses are not well-suited for global attacks on GNNs ; ( 2 ) Attacks on GNNs scale quadratically in the number of nodes or worse ; ( 3 ) Similarly , previous robust GNNs are typically not scalable . ( 1 ) Surrogate loss . We study the limitation of state-of-the-art surrogate losses for attacking the accuracy of a GNN over all nodes [ 8 , 9 , 17 , 27 , 41 , 43 , 49 ] in § 2 . Especially in combination with small/realistic budgets and on large graphs , previous surrogate losses lead to weak attacks . In particular , Cross Entropy ( CE ) or the widely used Carlini-Wagner loss [ 6 , 43 ] are weak surrogates for such global attacks . Our novel losses that overcome these limitations easily improve the strength of the attack by 100 % on common datasets . For larger graphs , this gap even becomes more significant . ( 2 ) Attacks . Attacks solving a discrete optimization problem easily become computationally infeasible because of the vast amount of potential adjacency matrices ( O ( 2n 2 ) ) . An approximate solution can be found with first-order optimization but we then still optimize over a quadratic number of parameters ( n2 ) . There is no trivial way to sparsify existing attacks as we need to represent each edge explicitly to obtain its gradient ( i.e . space complexity ⇥ ( n2 ) ) . Nevertheless , we overcome this limitation and propose two strategies to apply first-order optimization without the burden of a dense adjacency matrix . In § 3 , we describe how to add/remove edges between existing nodes based on Randomized Block Coordinate Descent ( R-BCD ) at an additional memory requirement of O ( ) . Due to the limited scalability of traditional GNNs , we also consider the case where we attack PPRGo [ 5 ] , a scalable GNN . Here , we even obtain an algorithm with constant complexity in the nodes n. ( 3 ) Defense . We propose Soft Median in § 4 – a computationally efficient , robust , differentiable aggregation function inspired by Geisler et al . [ 18 ] , by taking advantage of recent advancements in differentiable sorting [ 31 ] . Using our Soft Median we observe similar robustness to [ 18 ] , but with a significantly lower memory footprint , which enables us to defend GNNs at scale . 2 Surrogate Losses for Global Attacks . During training and in first-order attacks , we ideally wish to optimize a target metric which is often discontinuous ( e.g . accuracy and 0/1 loss L0/1 ) . However , for gradient-based optimization we commonly substitute the actual target loss by a surrogate L0 ⇡ L ( e.g . cross entropy for L0/1 ) . In the context of i.i.d . samples ( e.g . images ) , a single example is attacked in isolation with its own budget , which is similar to a local attack for GNNs . When a single node ’ s prediction is attacked , it is often sufficient to maximize the cross entropy for the attacked node/image ( untargeted attack ) : CE ( y , p ) = log ( pc⇤ ) . where y is the label and p is the vector of confidence scores . Many global attacks for GNNs [ 8 , 41 , 43 , 49 ] maximize the average CE to attack all nodes with a combined budget . However , a loss like CE can be ineffective , particularly when the number of nodes is large in comparison to the budget /n ! 0 . While experimenting on large graphs , we often observed that the CE loss increases even though the accuracy does not decline ( see § B ) . As we can see in Fig . 2 , this is due to CE ’ s bias towards nodes that have a low confidence score . With CE and a sufficiently small budget ⌧ n we primarily attack nodes that are already misclassified , which means that the classification margin = minc 6=c⇤ pc⇤ pc is already negative . Global attack . In contrast to attacking a single image/node , a global attack on a GNN has to ( 1 ) keep house with the budget and ( 2 ) find edges that degrade the overall accuracy maximally ( i.e . target “ fragile ” nodes ) . Without additional information , intuitively , one would first attack low-confidence nodes close to the decision boundary . Hence , the surrogate loss should have a minimal ( maximally negative ) gradient at ! 0+ ( i.e . approaching ! 0 from 0 ) . Moreover , if we solely want to lower the accuracy , then we can stop attacking a node once it is misclassified1 : Definition 1 A surrogate loss L0 for global attacks ( I ) should only incentivize perturbing nodes that are correctly classified : @ L0/ @ zc⇤ | < 0 = 0 and ( II ) should favour nodes close to the decision boundary : @ L0/ @ zc⇤ | 0 < @ L 0/ @ zc⇤ | 1 for any 0 < 0 < 1 . Since Eq . 1 is in general a discrete and non-convex optimization problem that is often NP-complete [ 1 , 3 , 38 , 45 ] , we propose to study the surrogate loss under the subsequent simplifying assumptions . Note that in an actual attack other influences ( e.g . node degree ) are still considered while solving the optimization problem . Assumption 1 : The set of attacked nodes is independent ( their receptive fields do not overlap ) . Particularly on large graphs with small budgets , /n ! 0 , deciding which node to attack becomes an increasingly local decision since the receptive field becomes insignificant in comparison to the rest of the graph . Assumption 2 : The budget required to change the prediction of node i depends ( only ) on the margin : i = g ( | i| ) for some increasing and non-negative function g ( · ) . That is , the larger the margin i , the harder it is to attack node i . As stated in Proposition 1 , with these assumptions , an optimizer with a surrogate loss compliant with Definition 1 is also optimizing the 0/1 loss L0/1 ( for proof see § B.4 ) . Proposition 1 Let L0 be the surrogate for the 0/1 loss L0/1 used to attack a node classification algorithm f✓ ( A , X ) with a global budget . Suppose we greedily attack nodes in order of @ L 0/ @ zc⇤ ( 0 ) @ L0/ @ zc⇤ ( 1 ) · · · @ L 0/ @ zc⇤ ( l ) until the budget is exhausted < Pl+1 i=0 i . Under Assumptions 1 & 2 , we then obtain the global optimum of maxÃ s.t . kÃ Ak0 < L0/1 ( f✓ ( Ã , X ) ) if L 0 has the properties ( I ) @ L0/ @ zc⇤ | < 0 = 0 and ( II ) @ L 0/ @ zc⇤ | 0 < @ L 0/ @ zc⇤ | 1 for any 0 < 0 < 1 . Even under the simplifying Assumptions 1 & 2 , the Cross Entropy ( CE ) is not guaranteed to obtain the global optimum . The ( CE ) violates property ( I ) and in the worst case only perturbs nodes that are already misclassified ( see Fig . 2 ) . The Carlini-Wagner ( CW ) [ 6 , 43 ] loss CW = min ( maxc 6=c⇤ zc zc⇤ , 0 ) violates property ( II ) . It is also not guaranteed to obtain the global optimum , i.e . CW loss lacks focus on nodes close to the decision boundary . In the worst case , an attack with CW spends all budget on confident nodes—without flipping a single one . We propose the Masked Cross Entropy MCE = 1/|V+| P i2V+ log ( p ( i ) c⇤ ) which fulfills both proper- ties by only considering correctly classified nodes V+ and , hence , reaches the global optimum under the stated assumptions . Empirically , for a greedy gradient-based attack the MCE comes with gains of more than 200 % in strength ( see Fig . 6 ) . Surprisingly , if we apply MCE to a Projected Gradient Descent ( PGD ) attack , we observe hardly any improvement over CE . We identify two potential reasons for that . The first is due to the learning dynamics of PGD . Suppose a misclassified node does 1We simply write p = f✓ ( A , X ) omitting that p belongs to specific node , i.e . pi of node i . We also overload this with the logits / pre-softmax activation as z = f✓ ( A , X ) . See § A for an overview of the notation . not receive any weight in the gradient update , now if the budget is exceeded after the update it is likely to be down-weighted . This can lead to nodes that oscillate around the decision boundary ( for more details see § B ) . A similar behavior occurs for to the Carlini-Wagner loss in e.g . Fig . B.1 ( e ) . In Definition 2 , we relax properties ( I ) / ( II ) and propose to overcome these limitations via enforcing confidently misclassified nodes , i.e . we want the attacked nodes to be at a “ safe ” distance from the decision boundary . We propose the tanh of the margin in logit space , i.e . tanh margin = tanh ( maxc 6=c⇤ zc zc⇤ ) . It obeys Definition 2 and its effectiveness is apparent from Fig . 2 . For the empirical evaluation see § 5 and for more results as well as details on all selected losses see § B . In the appendix we also study further losses to deepen the understanding about the required properties . Additionally , in § B.5 , we give an alternative Proposition 1 for a relaxed Assumption 2 s.t . E [ i| i ] = g ( | i| ) Definition 2 A surrogate loss L0 for global attacks that encourages confident misclassification ( A ) should saturate lim ! 1+ L0 < 1 and ( B ) should favor points close to the decision boundary : @ L0/ @ zc⇤ | 0 < @ L 0/ @ zc⇤ | 1 < 0 for any 0 < 0 < 1 < 1 or 1 < 1 < 0 < 0 .

Summary: The paper studies adversarial attacks against graph neural networks as well as adversarial robustness at scale.  The authors first propose two first-order optimization attacks, i.e., PR-BCD and GR-BCD. The authors then propose an empirical scalable defense based on soft median aggregation. Both attacks and defenses are evaluated on benchmark datasets and show their effectiveness.   Strengths +The studied problem is important +Propose a novel attack loss function +Evaluation is extensive  Weaknesses -The paper studies white-box attacks -Algorithm details are unclear -The proposed defense is empirical