GARNET: A Spectral Approach to Robust and Scalable Graph Neural Networks

Graph neural networks ( GNNs ) have been increasingly deployed in various applications that involve learning on non-Euclidean data . However , recent studies show that GNNs are vulnerable to graph adversarial attacks . Although there are several defense methods to improve GNN adversarial robustness , they fail to perform well on low homophily graphs . In addition , few of those defense models can scale to large graphs due to their high computational complexity and memory usage . In this paper , we propose GARNET , a scalable spectral method to boost the adversarial robustness of GNN models for both homophilic and heterophilic graphs . GARNET first computes a reduced-rank yet sparse approximation of the adversarial graph by exploiting an efficient spectral graph embedding and sparsification scheme . Next , GARNET trains an adaptive graph filter on the reduced-rank graph for node representation refinement , which is subsequently leveraged to guide label propagation for further enhancing the quality of node embeddings . GARNET has been evaluated on both homophilic and heterophilic datasets , including a large graph with millions of nodes . Our extensive experiment results show that GARNET increases adversarial accuracy over state-of-the-art GNN ( defense ) models by up to 9.96 % and 18.06 % on homophilic and heterophilic graphs , respectively . 1 INTRODUCTION . Recent years have witnessed a surge of interest in graph neural networks ( GNNs ) , which incorporate both graph structure and node/edge attributes to produce low-dimensional embedding vectors that maximally preserve graph structural information ( Hamilton , 2020 ) . GNNs have achieved promising results in various real-world applications , such as recommendation systems ( Ying et al. , 2018 ) , self-driving car ( Casas et al. , 2020 ) , protein structure predictions ( Senior et al. , 2020 ) , and chip placements ( Mirhoseini et al. , 2021 ) . However , recent studies have shown that adversarial attacks on graph structure accomplished by inserting , deleting , or rewiring edges in an unnoticeable way , can easily fool GNN models and drastically degrade their accuracy in downstream tasks ( e.g. , node classification ) ( Zügner et al. , 2018 ; Zügner & Günnemann , 2019 ) . In literature , there are several attempts to defend GNNs against graph adversarial attacks . Entezari et al . ( 2020 ) first observed that graph adversarial attacks mainly affect high-rank properties of the graph ; consequently , a low-rank graph should be first constructed by performing truncated singular value decomposition ( SVD ) on the graph adjacency matrix , which can then be exploited for training a robust GNN model . Later , Jin et al . ( 2020 ) proposed Pro-GNN to jointly learn a new graph and its robust GNN model with the low-rank constraints imposed by the graph structure . However , such low-rank approximation methods involve dense adjacency matrices during the GNN training stage , which will lead to quadratic time and space complexity , prohibiting their applications in large-scale graph learning tasks . Another line of research aims at enhancing GNN robustness based on the graph homophily assumption , i.e. , adjacent nodes in a natural graph tend to have similar attributes , while the graph attacks essentially insert adversarial edges by connecting nodes with dissimilar attributes . As a result , researchers proposed to compute attribute similarity scores between adjacent nodes as edge weights , and drop edges with small weights to increase graph homophily ( Wu et al. , 2019 ; Zhang & Zitnik , 2020 ) . Although such approaches can successfully improve GNN robustness on high-homophily graphs , they may fail on low-homophily ( i.e. , heterophily ) graphs ( Zhu et al. , 2020 ) . In this paper , we propose GARNET , a spectral method for constructing GNN models that are robust to graph adversarial attacks for both homophilic and heterophilic graphs . In addition , GARNET scales comfortably to large graphs due to its nearly-linear algorithm complexity . More concretely , GARNET consists of three major kernels : ( 1 ) reduced-rank approximation , ( 2 ) adaptive filter learning , and ( 3 ) adaptive label propagation . The reduced-rank approximation kernel first performs truncated SVD on graph adjacency matrix to obtain a few dominant singular values and their corresponding singular vectors that are further leveraged to construct a sparse yet reduced-rank graph adjacency matrix ; the reduced-rank adjacency matrix can effectively mitigate the effects of adversarial attacks via connecting ( disconnecting ) nodes that are spectrally similar ( dissimilar ) . The adaptive filter learning kernel aims to learn a polynomial graph filter whose coefficients are trainable ; the learned graph filter can adapt to the homophilic/heterophilic properties of the underlying graph and thus will work effectively for both homophilic and heterophilic graphs . The adaptive label propagation stage will leverage the learned adaptive graph filter to guide the label propagation phase , which can further improve the adversarial accuracy by enhancing the quality of node representations . We evaluate the proposed GARNET model on three high-homophily datasets : Cora , Citeseer , and Pubmed as well as two low-homophily datasets : Chameleon and Squirrel , under strong graph adversarial attacks such as Nettack ( Zügner et al. , 2018 ) and Metattack ( Zügner & Günnemann , 2019 ) with various perturbation settings . Moreover , we further show the nearly-linear scalability of our approach on the ogbn-products dataset that consists of millions of nodes ( Hu et al. , 2020 ) . Our experimental results indicate that GARNET largely improves adversarial accuracy over baselines in most cases . The major advantages of GARNET are summarized as follows : • GARNET is robust to graph adversarial attacks . Exploiting the proposed reduced-rank approximation scheme allows GARNET to effectively filter out the noises potentially caused by adversarial attacks in the spectral domain . This immediately leads to substantial improvement ( up to 18.06 % ) of adversarial accuracy when comparing with the state-of-the-art baselines under the strong graph attacks on various datasets . • GARNET is agnostic to graph homophily . Unlike most existing defense models that do not work well on low-homophily adversarial graphs , GARNET leverages a trainable graph filter that can adapt to adversarial graphs with diverse levels of homophily . In addition , we theoretically demonstrate that the performance of the adaptive graph filter applied to an adversarial graph will be similar to the performance achieved on a clean graph . • GARNET is scalable to large graphs . GARNET has a nearly-linear runtime/space complexity and thus can scale comfortably to very large graph data sets with millions of nodes . We have conducted experiments on the ogbn-products dataset that contains 2 million nodes and 60 million edges , which is over 100× larger than the largest adversarial graph ever considered in the prior arts . 2 RELATED WORK . GNNs have received an increasing amount of attention in recent years due to its ability of learning on non-Euclidean ( graph ) data . In contrast to developing powerful GNN models on natural graph data , there is an active body of research focusing on adversarial attacks as well as defenses for GNNs . We summarize some of the recent efforts for graph adversarial attacks and defenses as follows . Graph adversarial attacks aim at degrading the accuracy of GNN models by perturbing the graph structure in an unnoticeable way . For instance , most existing graph adversarial attacks insert/delete edges while maintaining node degree distribution ( Sun et al. , 2018 ) . The most popular graph adversarial attacks fall into the following two categories : ( 1 ) targeted attack , ( 2 ) untargeted attack . The targeted attacks attempt to mislead a GNN model to produce a wrong prediction on a target sample ( e.g. , node ) , while the untargeted attacks strive to degrade the overall accuracy of a GNN model for the whole graph data set . Dai et al . ( 2018 ) first formulate the targeted attack as a combinatorial optimization problem and leverages reinforcement learning to insert/delete edges such that the target node is misclassified . Zügner et al . ( 2018 ) propose another targeted attack called Nettack , which produces an adversarial graph by maximizing the training loss of GNNs . Zügner & Günnemann ( 2019 ) further introduce Metattack , an untargeted attack that treats the graph as a hyperparameter and uses meta-gradients to perturb the graph structure . It is worth noting that graph adversarial attacks have two different settings : poison ( perturb a graph prior to GNN training ) and evasion ( perturb a graph after GNN training ) . As shown by Zhu et al . ( 2021 ) , the poison setting is typically more challenging to defend , as it changes graph structure that fools GNN training . Thus , in this work , we evaluate our model against both targeted and untargeted attacks under the poison setting . Graph adversarial defenses attempt to enhance GNN performance on the perturbed graphs generated by adversarial attacks . Entezari et al . ( 2020 ) first observe that Nettack , a strong targeted attack , only changes the high-rank information of the adjacency matrix after graph perturbation . Thus , they propose to construct a low-rank graph by performing truncated SVD to undermine the effects of adversarial attacks . Jin et al . ( 2020 ) propose Pro-GNN that adopts a similar idea yet jointly learns the low-rank graph and GNN model . However , such low-rank approximation based methods produce dense adjacency matrices that correspond to complete graphs , which would limit their applications for large graphs . Another line of research strives to purify the adversarial graph by assigning edge weights . Specifically , Wu et al . ( 2019 ) propose to modify the edge weights by computing the Jaccard similarity score per edge based on node attributes , which is followed by Zhang & Zitnik ( 2020 ) that propose GNNGuard to learn node attribute similarity score per edge through a trainable linear layer . Nonetheless , such approaches assume that nearby nodes should have similar attributes ( i.e. , graph homophily assumption ) , which is not valid for heterophilic graphs that have adjacent nodes with dissimilar attributes ( Zhu et al. , 2020 ) . In contrast to the prior arts , GARNET achieves highly robust yet scalable performance on both homophilic and heterophilic graphs under adversarial attacks by leveraging novel reduced-rank approximation and adaptive graph filtering schemes . 3 OUR APPROACH . Figure 1 gives an overview of our proposed approach , GARNET , which consists of three major phases . The first phase ( reduced-rank approximation ) constructs a reduced-rank yet sparse graph model by exploiting scalable truncated SVD and nearest-neighbor graph algorithms ( Baglama & Reichel , 2005 ; Malkov & Yashunin , 2018 ) . The second phase ( adaptive filter learning ) is the only phase that involves training , which outputs a learned graph filter that can adapt to the homophilic/heterophilic properties of the underlying graph . The last phase ( adaptive label propagation ) leverages the learned adaptive graph filter to guide the subsequent label propagation process . In the rest of this section , we will describe each of these three phases in more detail . 0 3.1 REDUCED-RANK APPROXIMATION ( PHASE 1 ) . The adjacency matrices of many real-world graphs ( e.g. , social network and biological network ) are naturally low rank and sparse , as the nodes typically tend to form communities and have a small number of neighbors ( Zhou et al. , 2013 ) . As a result , graph adversarial attacks can be viewed as compromising these properties by inserting edges that connect nodes from different communities ( Zügner et al. , 2018 ; Zügner & Günnemann , 2019 ) . Recently , Entezari et al . ( 2020 ) and Jin et al . ( 2020 ) have shown that the well-known graph adversarial attacks ( e.g. , Nettack and Metattack ) are essentially high-rank attacks , which mainly change the high-rank spectrum of the graph when perturbing the graph structure , while the low-rank spectrum remains almost the same . We empirically confirm that the graph rank indeed increases under adversarial attacks in Appendix A.7 . Consequently , a natural way for improving GNN robustness is to purify an adversarial graph by eliminating the high-rank components of its spectrum . To enhance GNN robustness , Entezari et al . ( 2020 ) and Jin et al . ( 2020 ) reconstruct a graph that only preserves the low-rank components of its spectrum to mitigate the effects of adversarial attacks , via performing truncated SVD on the adjacency matrix . Specifically , given a graph adjacency matrix A ∈ Rn×n , the rank-r approximated adjacency matrix can be obtained via truncated SVD : Â = UΣV T , where Σ ∈ Rr×r is a diagonal matrix consisting of r largest singular values of A. U ∈ Rn×r and V ∈ Rn×r contain the corresponding left and right singular vectors , respectively . However , the reconstructed low-rank adjacency matrix Â has two key issues : ( 1 ) Â is typically a dense matrix with O ( n2 ) nonzero elements , which may result in prohibitively expensive storage as well as GNN training , where n represents number of nodes ( Entezari et al. , 2020 ; Jin et al. , 2020 ) . Thus , existing low-rank approximation methods are not scalable to large graphs ; ( 2 ) Due to the high computational cost of SVD , Â is typically obtained by only leveraging top r singular values and singular vectors , where r is a relatively small number ( e.g. , r = 50 ) . Consequently , the rank of Â is only r = 50 , which loses too much important spectrum information and thus limits the performance of GNN training . Reduced-rank approximation via sparsification . To avoid the quadratic space complexity for obtaining Â , one simple solution is to sparsify Â on the fly . More concretely , instead of directly computing Â = UΣV T , we can compute Â row by row : Âi , : = Ui , :ΣV T . Once we obtain Âi , : , we can sparsify it by setting Ãi , j = 0 if Âi , j < δ and Ãi , j = Âi , j otherwise , where δ is a hyperparameter to control the sparsity . In this way , we only need to store a dense vector Âi , : at a time and the final adjacency matrix Ã is sparse . Although the space complexity is reduced to O ( m ) , where m denotes the number of non-zero elements in Ã , this sparsification method still has quadratic time complexity for computing Ã . To tackle this issue , in this work a nearly-linear complexity algorithm for constructing a reduced-rank yet sparse matrix Ã is proposed by exploiting the following connection between matrix sparsification and spectral graph embedding . Definition 1 . Given the top r smallest eigenvalues λ1 , λ2 , ... , λr and their corresponding eigenvectors v1 , v2 , ... , vr of normalized graph Laplacian matrix Lnorm = I −D− 1 2AD− 1 2 , where I and A are the identity matrix and graph adjacency matrix , respectively , and D is a diagonal matrix of node degrees , the weighted spectral embedding matrix is defined as : V def = [ √ |1− λ1|v1 , ... , √ |1− λr|vr ] , ( 1 ) whose i-th row Vi , : is the weighted spectral embedding of the corresponding i-th node in the graph . Theorem 1 . Given a normalized graph adjacency matrix Anorm = D− 1 2AD− 1 2 of an undirected graph , let Â be the rank-r approximation of Anorm via truncated SVD . If the top r dominant eigenvalues of Anorm are non-negative , then Âi , j corresponds to the dot product score between the weighted spectral embeddings of node i and j. Theorem 1 indicates that the reduced-rank approximation of the normalized adjacency matrix via truncated SVD captures the spectral similarity between nodes , which motivates us to leverage weighted spectral embeddings to capture graph spectral ( low-rank ) information , thereby identifying edges to be pruned ( sparsified ) from the graph . Unlike traditional spectral graph embedding methods that utilize the first r Laplacian eigenpairs to construct spectral embedding matrix , we exploit the top r largest singular values and their singular vectors of Anorm that may include both the smallest and largest Laplacian eigenpairs for capturing global and local structural information , respectively ( Shuman et al. , 2013 ) . More specifically , given a graph G = ( V , E ) and its normalized adjacency matrix Anorm , our approach first computes the r largest singular values ( e.g. , r = 50 ) and the corresponding singular vectors of Anorm leveraging efficient eigensolvers in O ( r|E| ) time to construct the weighted spectral embeddings ( Baglama & Reichel , 2005 ) ; next , the embedding results are then used to construct a nearest neighbor ( NN ) graph , where each node will be connected to its k most spectrally similar nodes . Afterwards , we compute the dot product similarity scores between adjacent nodes in the NN graph . If the similarity score is less than a threshold δ , we prune the corresponding edge from the NN graph to further sparsify the graph . In this work , we exploit an approximate k-nearest neighbor algorithm for constructing the NN graph , which has O ( |V| log |V| ) complexity and thus can scale to very large graphs ( Malkov & Yashunin , 2018 ) . We say a graph is a reduced-rank graph if its adjacency matrix is obtained via the proposed reduced-rank approximation method . As a result , we can obtain a reduced-rank and sparse graph in O ( r|E|+ |V| log |V| ) time . Apart from the advantage of scalability , our kNN-based reduced-rank graph also preserves much more important spectrum information than the truncated SVD-based low-rank graph . As shown by Entezari et al . ( 2020 ) , adversarial attacks only perturb the top few highest singular components in the graph spectrum , while the rest of spectral information corresponds to the clean graph structure in the spatial domain . Nonetheless , Figure 5 in Appendix A.8 shows the truncated SVD-based method aggressively reduces graph rank to 50 , which is two orders of magnitude smaller than the rank of input graph . In contrast , our reduced-rank method only removes the highest singular components , while retaining most of important spectral information . As a result , our reduced-rank approximation kernel leads to a significant accuracy improvement over the SVD-based low-rank approximation . More details are available in Appendices A.11 .

Summary: This paper proposes a spectral approach towards robust and scalable graph learning, namely GARNET. There are three main components inspired by handling low-homophily graphs in GARNET: low rank and sparse approximation of the adversarial graph, a variant of spectral GNN with an adaptive filter, and label propagation with special interest to the learned filter. GARNET is claimed to be nearly-linear so that it can be scaled to very huge graphs. Experiments against both targeted and untargeted attacks under the poison setting over a collection of graph benchmark datasets demonstrate that GARNET could increase the adversarial accuracy more than state-of-the-art defending methods.