Robustness of Graph Neural Networks at Scale

1 Introduction The evidence that Graph Neural Networks ( GNNs ) are not robust to adversarial perturbations is compelling [ 14 , 20 , 50 ] . However , the graphs in previous robustness studies are tiny . This is worrying , given that GNNs are already deployed in many real-world Internet-scale applications [ 5 , 44 ] . For example , PubMed [ 33 ] ( 19,717 nodes ) is often considered to be a large-scale graph and around 20 GB of memory is required for an attack based on its dense adjacency matrix . Such memory requirements are impractical and limit advancements of the field . In this work , we set the foundation for the holistic study of adversarial robustness of GNNs at scale . We study graphs with up to 111 million nodes for local attacks ( i.e . attacking a single node ) and 2.5 million nodes for global attacks ( i.e . attacking all nodes at once ) . As it turns out , GNNs at scale are also highly vulnerable to adversarial attacks . In Fig . 1 , we show the substantial improvement of memory efficiency of our attack over a popular prior work for attacking a GNN globally . Analogous to our attacks , our effective defense scales to graphs with 111 million nodes and beyond . Scope . We focus on adversarial robustness w.r.t . structure attacks on GNNs for node classification max Ã s.t . kÃ Ak0 < L ( f✓ ( Ã , X ) ) ( 1 ) with loss function L ( or its surrogate L0 ) , budget , and fixed model parameters ✓ . The GNN f✓ ( A , X ) is applied to a graph G = ( A , X ) with node attributes X 2 Rn⇥d , the adjacency matrix 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A 2 { 0 , 1 } n⇥n , and m edges . We focus on evasion ( test time ) attacks , but our methods can be used in poisoning ( train time ) attacks [ 49 ] . We distinguish between local attacks on a single node and global attacks that target a large fraction of nodes with a shared budget . We study white-box attacks since they have the most powerful threat model and can be used to understand the robustness w.r.t . “ worst-case noise ” of a model , as well as to assess the efficacy of defenses . For these reasons white-box attacks are important and practical from the perspective of a defender . Broader impact . Since we enable the study of robustness at scale , which previously was practically infeasible , an adversary can potentially abuse our attacks . The risk is minimized given that we assume perfect knowledge about the graph , model , and labels . Nonetheless , our findings suggest that one should be careful when deploying GNNs , and highlight that further research is needed . To mitigate this risk , we must be able to evaluate it . We also propose a scalable defense that shows strong performance empirically but we urge practitioners to consider potential trade-offs , e.g . improving robustness at the expense of accuracy for different groups of users . Contributions . We address three major challenges hindering the study of GNNs ’ adversarial robustness at scale and propose viable solutions including an extensive empirical evaluation : ( 1 ) Previous losses are not well-suited for global attacks on GNNs ; ( 2 ) Attacks on GNNs scale quadratically in the number of nodes or worse ; ( 3 ) Similarly , previous robust GNNs are typically not scalable . ( 1 ) Surrogate loss . We study the limitation of state-of-the-art surrogate losses for attacking the accuracy of a GNN over all nodes [ 8 , 9 , 17 , 27 , 41 , 43 , 49 ] in § 2 . Especially in combination with small/realistic budgets and on large graphs , previous surrogate losses lead to weak attacks . In particular , Cross Entropy ( CE ) or the widely used Carlini-Wagner loss [ 6 , 43 ] are weak surrogates for such global attacks . Our novel losses that overcome these limitations easily improve the strength of the attack by 100 % on common datasets . For larger graphs , this gap even becomes more significant . ( 2 ) Attacks . Attacks solving a discrete optimization problem easily become computationally infeasible because of the vast amount of potential adjacency matrices ( O ( 2n 2 ) ) . An approximate solution can be found with first-order optimization but we then still optimize over a quadratic number of parameters ( n2 ) . There is no trivial way to sparsify existing attacks as we need to represent each edge explicitly to obtain its gradient ( i.e . space complexity ⇥ ( n2 ) ) . Nevertheless , we overcome this limitation and propose two strategies to apply first-order optimization without the burden of a dense adjacency matrix . In § 3 , we describe how to add/remove edges between existing nodes based on Randomized Block Coordinate Descent ( R-BCD ) at an additional memory requirement of O ( ) . Due to the limited scalability of traditional GNNs , we also consider the case where we attack PPRGo [ 5 ] , a scalable GNN . Here , we even obtain an algorithm with constant complexity in the nodes n. ( 3 ) Defense . We propose Soft Median in § 4 – a computationally efficient , robust , differentiable aggregation function inspired by Geisler et al . [ 18 ] , by taking advantage of recent advancements in differentiable sorting [ 31 ] . Using our Soft Median we observe similar robustness to [ 18 ] , but with a significantly lower memory footprint , which enables us to defend GNNs at scale . 2 Surrogate Losses for Global Attacks . During training and in first-order attacks , we ideally wish to optimize a target metric which is often discontinuous ( e.g . accuracy and 0/1 loss L0/1 ) . However , for gradient-based optimization we commonly substitute the actual target loss by a surrogate L0 ⇡ L ( e.g . cross entropy for L0/1 ) . In the context of i.i.d . samples ( e.g . images ) , a single example is attacked in isolation with its own budget , which is similar to a local attack for GNNs . When a single node ’ s prediction is attacked , it is often sufficient to maximize the cross entropy for the attacked node/image ( untargeted attack ) : CE ( y , p ) = log ( pc⇤ ) . where y is the label and p is the vector of confidence scores . Many global attacks for GNNs [ 8 , 41 , 43 , 49 ] maximize the average CE to attack all nodes with a combined budget . However , a loss like CE can be ineffective , particularly when the number of nodes is large in comparison to the budget /n ! 0 . While experimenting on large graphs , we often observed that the CE loss increases even though the accuracy does not decline ( see § B ) . As we can see in Fig . 2 , this is due to CE ’ s bias towards nodes that have a low confidence score . With CE and a sufficiently small budget ⌧ n we primarily attack nodes that are already misclassified , which means that the classification margin = minc 6=c⇤ pc⇤ pc is already negative . Global attack . In contrast to attacking a single image/node , a global attack on a GNN has to ( 1 ) keep house with the budget and ( 2 ) find edges that degrade the overall accuracy maximally ( i.e . target “ fragile ” nodes ) . Without additional information , intuitively , one would first attack low-confidence nodes close to the decision boundary . Hence , the surrogate loss should have a minimal ( maximally negative ) gradient at ! 0+ ( i.e . approaching ! 0 from 0 ) . Moreover , if we solely want to lower the accuracy , then we can stop attacking a node once it is misclassified1 : Definition 1 A surrogate loss L0 for global attacks ( I ) should only incentivize perturbing nodes that are correctly classified : @ L0/ @ zc⇤ | < 0 = 0 and ( II ) should favour nodes close to the decision boundary : @ L0/ @ zc⇤ | 0 < @ L 0/ @ zc⇤ | 1 for any 0 < 0 < 1 . Since Eq . 1 is in general a discrete and non-convex optimization problem that is often NP-complete [ 1 , 3 , 38 , 45 ] , we propose to study the surrogate loss under the subsequent simplifying assumptions . Note that in an actual attack other influences ( e.g . node degree ) are still considered while solving the optimization problem . Assumption 1 : The set of attacked nodes is independent ( their receptive fields do not overlap ) . Particularly on large graphs with small budgets , /n ! 0 , deciding which node to attack becomes an increasingly local decision since the receptive field becomes insignificant in comparison to the rest of the graph . Assumption 2 : The budget required to change the prediction of node i depends ( only ) on the margin : i = g ( | i| ) for some increasing and non-negative function g ( · ) . That is , the larger the margin i , the harder it is to attack node i . As stated in Proposition 1 , with these assumptions , an optimizer with a surrogate loss compliant with Definition 1 is also optimizing the 0/1 loss L0/1 ( for proof see § B.4 ) . Proposition 1 Let L0 be the surrogate for the 0/1 loss L0/1 used to attack a node classification algorithm f✓ ( A , X ) with a global budget . Suppose we greedily attack nodes in order of @ L 0/ @ zc⇤ ( 0 ) @ L0/ @ zc⇤ ( 1 ) · · · @ L 0/ @ zc⇤ ( l ) until the budget is exhausted < Pl+1 i=0 i . Under Assumptions 1 & 2 , we then obtain the global optimum of maxÃ s.t . kÃ Ak0 < L0/1 ( f✓ ( Ã , X ) ) if L 0 has the properties ( I ) @ L0/ @ zc⇤ | < 0 = 0 and ( II ) @ L 0/ @ zc⇤ | 0 < @ L 0/ @ zc⇤ | 1 for any 0 < 0 < 1 . Even under the simplifying Assumptions 1 & 2 , the Cross Entropy ( CE ) is not guaranteed to obtain the global optimum . The ( CE ) violates property ( I ) and in the worst case only perturbs nodes that are already misclassified ( see Fig . 2 ) . The Carlini-Wagner ( CW ) [ 6 , 43 ] loss CW = min ( maxc 6=c⇤ zc zc⇤ , 0 ) violates property ( II ) . It is also not guaranteed to obtain the global optimum , i.e . CW loss lacks focus on nodes close to the decision boundary . In the worst case , an attack with CW spends all budget on confident nodes—without flipping a single one . We propose the Masked Cross Entropy MCE = 1/|V+| P i2V+ log ( p ( i ) c⇤ ) which fulfills both proper- ties by only considering correctly classified nodes V+ and , hence , reaches the global optimum under the stated assumptions . Empirically , for a greedy gradient-based attack the MCE comes with gains of more than 200 % in strength ( see Fig . 6 ) . Surprisingly , if we apply MCE to a Projected Gradient Descent ( PGD ) attack , we observe hardly any improvement over CE . We identify two potential reasons for that . The first is due to the learning dynamics of PGD . Suppose a misclassified node does 1We simply write p = f✓ ( A , X ) omitting that p belongs to specific node , i.e . pi of node i . We also overload this with the logits / pre-softmax activation as z = f✓ ( A , X ) . See § A for an overview of the notation . not receive any weight in the gradient update , now if the budget is exceeded after the update it is likely to be down-weighted . This can lead to nodes that oscillate around the decision boundary ( for more details see § B ) . A similar behavior occurs for to the Carlini-Wagner loss in e.g . Fig . B.1 ( e ) . In Definition 2 , we relax properties ( I ) / ( II ) and propose to overcome these limitations via enforcing confidently misclassified nodes , i.e . we want the attacked nodes to be at a “ safe ” distance from the decision boundary . We propose the tanh of the margin in logit space , i.e . tanh margin = tanh ( maxc 6=c⇤ zc zc⇤ ) . It obeys Definition 2 and its effectiveness is apparent from Fig . 2 . For the empirical evaluation see § 5 and for more results as well as details on all selected losses see § B . In the appendix we also study further losses to deepen the understanding about the required properties . Additionally , in § B.5 , we give an alternative Proposition 1 for a relaxed Assumption 2 s.t . E [ i| i ] = g ( | i| ) Definition 2 A surrogate loss L0 for global attacks that encourages confident misclassification ( A ) should saturate lim ! 1+ L0 < 1 and ( B ) should favor points close to the decision boundary : @ L0/ @ zc⇤ | 0 < @ L 0/ @ zc⇤ | 1 < 0 for any 0 < 0 < 1 < 1 or 1 < 1 < 0 < 0 .

Summary: This paper studies adversarial attack and defense of GNNs at scale. The authors investigate the relationship between surrogate losses and accuracy, and scalable attack and defense algorithms. Some theoretical intuitions and extensive empirical evaluation are provided to support the claims.