GARNET: A Spectral Approach to Robust and Scalable Graph Neural Networks

Graph neural networks ( GNNs ) have been increasingly deployed in various applications that involve learning on non-Euclidean data . However , recent studies show that GNNs are vulnerable to graph adversarial attacks . Although there are several defense methods to improve GNN adversarial robustness , they fail to perform well on low homophily graphs . In addition , few of those defense models can scale to large graphs due to their high computational complexity and memory usage . In this paper , we propose GARNET , a scalable spectral method to boost the adversarial robustness of GNN models for both homophilic and heterophilic graphs . GARNET first computes a reduced-rank yet sparse approximation of the adversarial graph by exploiting an efficient spectral graph embedding and sparsification scheme . Next , GARNET trains an adaptive graph filter on the reduced-rank graph for node representation refinement , which is subsequently leveraged to guide label propagation for further enhancing the quality of node embeddings . GARNET has been evaluated on both homophilic and heterophilic datasets , including a large graph with millions of nodes . Our extensive experiment results show that GARNET increases adversarial accuracy over state-of-the-art GNN ( defense ) models by up to 9.96 % and 18.06 % on homophilic and heterophilic graphs , respectively . 1 INTRODUCTION . Recent years have witnessed a surge of interest in graph neural networks ( GNNs ) , which incorporate both graph structure and node/edge attributes to produce low-dimensional embedding vectors that maximally preserve graph structural information ( Hamilton , 2020 ) . GNNs have achieved promising results in various real-world applications , such as recommendation systems ( Ying et al. , 2018 ) , self-driving car ( Casas et al. , 2020 ) , protein structure predictions ( Senior et al. , 2020 ) , and chip placements ( Mirhoseini et al. , 2021 ) . However , recent studies have shown that adversarial attacks on graph structure accomplished by inserting , deleting , or rewiring edges in an unnoticeable way , can easily fool GNN models and drastically degrade their accuracy in downstream tasks ( e.g. , node classification ) ( Zügner et al. , 2018 ; Zügner & Günnemann , 2019 ) . In literature , there are several attempts to defend GNNs against graph adversarial attacks . Entezari et al . ( 2020 ) first observed that graph adversarial attacks mainly affect high-rank properties of the graph ; consequently , a low-rank graph should be first constructed by performing truncated singular value decomposition ( SVD ) on the graph adjacency matrix , which can then be exploited for training a robust GNN model . Later , Jin et al . ( 2020 ) proposed Pro-GNN to jointly learn a new graph and its robust GNN model with the low-rank constraints imposed by the graph structure . However , such low-rank approximation methods involve dense adjacency matrices during the GNN training stage , which will lead to quadratic time and space complexity , prohibiting their applications in large-scale graph learning tasks . Another line of research aims at enhancing GNN robustness based on the graph homophily assumption , i.e. , adjacent nodes in a natural graph tend to have similar attributes , while the graph attacks essentially insert adversarial edges by connecting nodes with dissimilar attributes . As a result , researchers proposed to compute attribute similarity scores between adjacent nodes as edge weights , and drop edges with small weights to increase graph homophily ( Wu et al. , 2019 ; Zhang & Zitnik , 2020 ) . Although such approaches can successfully improve GNN robustness on high-homophily graphs , they may fail on low-homophily ( i.e. , heterophily ) graphs ( Zhu et al. , 2020 ) . In this paper , we propose GARNET , a spectral method for constructing GNN models that are robust to graph adversarial attacks for both homophilic and heterophilic graphs . In addition , GARNET scales comfortably to large graphs due to its nearly-linear algorithm complexity . More concretely , GARNET consists of three major kernels : ( 1 ) reduced-rank approximation , ( 2 ) adaptive filter learning , and ( 3 ) adaptive label propagation . The reduced-rank approximation kernel first performs truncated SVD on graph adjacency matrix to obtain a few dominant singular values and their corresponding singular vectors that are further leveraged to construct a sparse yet reduced-rank graph adjacency matrix ; the reduced-rank adjacency matrix can effectively mitigate the effects of adversarial attacks via connecting ( disconnecting ) nodes that are spectrally similar ( dissimilar ) . The adaptive filter learning kernel aims to learn a polynomial graph filter whose coefficients are trainable ; the learned graph filter can adapt to the homophilic/heterophilic properties of the underlying graph and thus will work effectively for both homophilic and heterophilic graphs . The adaptive label propagation stage will leverage the learned adaptive graph filter to guide the label propagation phase , which can further improve the adversarial accuracy by enhancing the quality of node representations . We evaluate the proposed GARNET model on three high-homophily datasets : Cora , Citeseer , and Pubmed as well as two low-homophily datasets : Chameleon and Squirrel , under strong graph adversarial attacks such as Nettack ( Zügner et al. , 2018 ) and Metattack ( Zügner & Günnemann , 2019 ) with various perturbation settings . Moreover , we further show the nearly-linear scalability of our approach on the ogbn-products dataset that consists of millions of nodes ( Hu et al. , 2020 ) . Our experimental results indicate that GARNET largely improves adversarial accuracy over baselines in most cases . The major advantages of GARNET are summarized as follows : • GARNET is robust to graph adversarial attacks . Exploiting the proposed reduced-rank approximation scheme allows GARNET to effectively filter out the noises potentially caused by adversarial attacks in the spectral domain . This immediately leads to substantial improvement ( up to 18.06 % ) of adversarial accuracy when comparing with the state-of-the-art baselines under the strong graph attacks on various datasets . • GARNET is agnostic to graph homophily . Unlike most existing defense models that do not work well on low-homophily adversarial graphs , GARNET leverages a trainable graph filter that can adapt to adversarial graphs with diverse levels of homophily . In addition , we theoretically demonstrate that the performance of the adaptive graph filter applied to an adversarial graph will be similar to the performance achieved on a clean graph . • GARNET is scalable to large graphs . GARNET has a nearly-linear runtime/space complexity and thus can scale comfortably to very large graph data sets with millions of nodes . We have conducted experiments on the ogbn-products dataset that contains 2 million nodes and 60 million edges , which is over 100× larger than the largest adversarial graph ever considered in the prior arts . 2 RELATED WORK . GNNs have received an increasing amount of attention in recent years due to its ability of learning on non-Euclidean ( graph ) data . In contrast to developing powerful GNN models on natural graph data , there is an active body of research focusing on adversarial attacks as well as defenses for GNNs . We summarize some of the recent efforts for graph adversarial attacks and defenses as follows . Graph adversarial attacks aim at degrading the accuracy of GNN models by perturbing the graph structure in an unnoticeable way . For instance , most existing graph adversarial attacks insert/delete edges while maintaining node degree distribution ( Sun et al. , 2018 ) . The most popular graph adversarial attacks fall into the following two categories : ( 1 ) targeted attack , ( 2 ) untargeted attack . The targeted attacks attempt to mislead a GNN model to produce a wrong prediction on a target sample ( e.g. , node ) , while the untargeted attacks strive to degrade the overall accuracy of a GNN model for the whole graph data set . Dai et al . ( 2018 ) first formulate the targeted attack as a combinatorial optimization problem and leverages reinforcement learning to insert/delete edges such that the target node is misclassified . Zügner et al . ( 2018 ) propose another targeted attack called Nettack , which produces an adversarial graph by maximizing the training loss of GNNs . Zügner & Günnemann ( 2019 ) further introduce Metattack , an untargeted attack that treats the graph as a hyperparameter and uses meta-gradients to perturb the graph structure . It is worth noting that graph adversarial attacks have two different settings : poison ( perturb a graph prior to GNN training ) and evasion ( perturb a graph after GNN training ) . As shown by Zhu et al . ( 2021 ) , the poison setting is typically more challenging to defend , as it changes graph structure that fools GNN training . Thus , in this work , we evaluate our model against both targeted and untargeted attacks under the poison setting . Graph adversarial defenses attempt to enhance GNN performance on the perturbed graphs generated by adversarial attacks . Entezari et al . ( 2020 ) first observe that Nettack , a strong targeted attack , only changes the high-rank information of the adjacency matrix after graph perturbation . Thus , they propose to construct a low-rank graph by performing truncated SVD to undermine the effects of adversarial attacks . Jin et al . ( 2020 ) propose Pro-GNN that adopts a similar idea yet jointly learns the low-rank graph and GNN model . However , such low-rank approximation based methods produce dense adjacency matrices that correspond to complete graphs , which would limit their applications for large graphs . Another line of research strives to purify the adversarial graph by assigning edge weights . Specifically , Wu et al . ( 2019 ) propose to modify the edge weights by computing the Jaccard similarity score per edge based on node attributes , which is followed by Zhang & Zitnik ( 2020 ) that propose GNNGuard to learn node attribute similarity score per edge through a trainable linear layer . Nonetheless , such approaches assume that nearby nodes should have similar attributes ( i.e. , graph homophily assumption ) , which is not valid for heterophilic graphs that have adjacent nodes with dissimilar attributes ( Zhu et al. , 2020 ) . In contrast to the prior arts , GARNET achieves highly robust yet scalable performance on both homophilic and heterophilic graphs under adversarial attacks by leveraging novel reduced-rank approximation and adaptive graph filtering schemes . 3 OUR APPROACH . Figure 1 gives an overview of our proposed approach , GARNET , which consists of three major phases . The first phase ( reduced-rank approximation ) constructs a reduced-rank yet sparse graph model by exploiting scalable truncated SVD and nearest-neighbor graph algorithms ( Baglama & Reichel , 2005 ; Malkov & Yashunin , 2018 ) . The second phase ( adaptive filter learning ) is the only phase that involves training , which outputs a learned graph filter that can adapt to the homophilic/heterophilic properties of the underlying graph . The last phase ( adaptive label propagation ) leverages the learned adaptive graph filter to guide the subsequent label propagation process . In the rest of this section , we will describe each of these three phases in more detail . 0 3.1 REDUCED-RANK APPROXIMATION ( PHASE 1 ) . The adjacency matrices of many real-world graphs ( e.g. , social network and biological network ) are naturally low rank and sparse , as the nodes typically tend to form communities and have a small number of neighbors ( Zhou et al. , 2013 ) . As a result , graph adversarial attacks can be viewed as compromising these properties by inserting edges that connect nodes from different communities ( Zügner et al. , 2018 ; Zügner & Günnemann , 2019 ) . Recently , Entezari et al . ( 2020 ) and Jin et al . ( 2020 ) have shown that the well-known graph adversarial attacks ( e.g. , Nettack and Metattack ) are essentially high-rank attacks , which mainly change the high-rank spectrum of the graph when perturbing the graph structure , while the low-rank spectrum remains almost the same . We empirically confirm that the graph rank indeed increases under adversarial attacks in Appendix A.7 . Consequently , a natural way for improving GNN robustness is to purify an adversarial graph by eliminating the high-rank components of its spectrum . To enhance GNN robustness , Entezari et al . ( 2020 ) and Jin et al . ( 2020 ) reconstruct a graph that only preserves the low-rank components of its spectrum to mitigate the effects of adversarial attacks , via performing truncated SVD on the adjacency matrix . Specifically , given a graph adjacency matrix A ∈ Rn×n , the rank-r approximated adjacency matrix can be obtained via truncated SVD : Â = UΣV T , where Σ ∈ Rr×r is a diagonal matrix consisting of r largest singular values of A. U ∈ Rn×r and V ∈ Rn×r contain the corresponding left and right singular vectors , respectively . However , the reconstructed low-rank adjacency matrix Â has two key issues : ( 1 ) Â is typically a dense matrix with O ( n2 ) nonzero elements , which may result in prohibitively expensive storage as well as GNN training , where n represents number of nodes ( Entezari et al. , 2020 ; Jin et al. , 2020 ) . Thus , existing low-rank approximation methods are not scalable to large graphs ; ( 2 ) Due to the high computational cost of SVD , Â is typically obtained by only leveraging top r singular values and singular vectors , where r is a relatively small number ( e.g. , r = 50 ) . Consequently , the rank of Â is only r = 50 , which loses too much important spectrum information and thus limits the performance of GNN training . Reduced-rank approximation via sparsification . To avoid the quadratic space complexity for obtaining Â , one simple solution is to sparsify Â on the fly . More concretely , instead of directly computing Â = UΣV T , we can compute Â row by row : Âi , : = Ui , :ΣV T . Once we obtain Âi , : , we can sparsify it by setting Ãi , j = 0 if Âi , j < δ and Ãi , j = Âi , j otherwise , where δ is a hyperparameter to control the sparsity . In this way , we only need to store a dense vector Âi , : at a time and the final adjacency matrix Ã is sparse . Although the space complexity is reduced to O ( m ) , where m denotes the number of non-zero elements in Ã , this sparsification method still has quadratic time complexity for computing Ã . To tackle this issue , in this work a nearly-linear complexity algorithm for constructing a reduced-rank yet sparse matrix Ã is proposed by exploiting the following connection between matrix sparsification and spectral graph embedding . Definition 1 . Given the top r smallest eigenvalues λ1 , λ2 , ... , λr and their corresponding eigenvectors v1 , v2 , ... , vr of normalized graph Laplacian matrix Lnorm = I −D− 1 2AD− 1 2 , where I and A are the identity matrix and graph adjacency matrix , respectively , and D is a diagonal matrix of node degrees , the weighted spectral embedding matrix is defined as : V def = [ √ |1− λ1|v1 , ... , √ |1− λr|vr ] , ( 1 ) whose i-th row Vi , : is the weighted spectral embedding of the corresponding i-th node in the graph . Theorem 1 . Given a normalized graph adjacency matrix Anorm = D− 1 2AD− 1 2 of an undirected graph , let Â be the rank-r approximation of Anorm via truncated SVD . If the top r dominant eigenvalues of Anorm are non-negative , then Âi , j corresponds to the dot product score between the weighted spectral embeddings of node i and j. Theorem 1 indicates that the reduced-rank approximation of the normalized adjacency matrix via truncated SVD captures the spectral similarity between nodes , which motivates us to leverage weighted spectral embeddings to capture graph spectral ( low-rank ) information , thereby identifying edges to be pruned ( sparsified ) from the graph . Unlike traditional spectral graph embedding methods that utilize the first r Laplacian eigenpairs to construct spectral embedding matrix , we exploit the top r largest singular values and their singular vectors of Anorm that may include both the smallest and largest Laplacian eigenpairs for capturing global and local structural information , respectively ( Shuman et al. , 2013 ) . More specifically , given a graph G = ( V , E ) and its normalized adjacency matrix Anorm , our approach first computes the r largest singular values ( e.g. , r = 50 ) and the corresponding singular vectors of Anorm leveraging efficient eigensolvers in O ( r|E| ) time to construct the weighted spectral embeddings ( Baglama & Reichel , 2005 ) ; next , the embedding results are then used to construct a nearest neighbor ( NN ) graph , where each node will be connected to its k most spectrally similar nodes . Afterwards , we compute the dot product similarity scores between adjacent nodes in the NN graph . If the similarity score is less than a threshold δ , we prune the corresponding edge from the NN graph to further sparsify the graph . In this work , we exploit an approximate k-nearest neighbor algorithm for constructing the NN graph , which has O ( |V| log |V| ) complexity and thus can scale to very large graphs ( Malkov & Yashunin , 2018 ) . We say a graph is a reduced-rank graph if its adjacency matrix is obtained via the proposed reduced-rank approximation method . As a result , we can obtain a reduced-rank and sparse graph in O ( r|E|+ |V| log |V| ) time . Apart from the advantage of scalability , our kNN-based reduced-rank graph also preserves much more important spectrum information than the truncated SVD-based low-rank graph . As shown by Entezari et al . ( 2020 ) , adversarial attacks only perturb the top few highest singular components in the graph spectrum , while the rest of spectral information corresponds to the clean graph structure in the spatial domain . Nonetheless , Figure 5 in Appendix A.8 shows the truncated SVD-based method aggressively reduces graph rank to 50 , which is two orders of magnitude smaller than the rank of input graph . In contrast , our reduced-rank method only removes the highest singular components , while retaining most of important spectral information . As a result , our reduced-rank approximation kernel leads to a significant accuracy improvement over the SVD-based low-rank approximation . More details are available in Appendices A.11 .

Summary: The paper proposes a reduced-rank and sparse approximation method to purify the graph structure for better robustness against adversarial graph attacks. The approximation is accelerated through spectral embedding and sparsification. Moreover, adaptive graph filter and label propagation are considered for further improvement on low homophily graphs.