Not All Low-Pass Filters are Robust in Graph Convolutional Networks

1 Introduction . Graph Convolutional Networks ( GCNs ) elaborate the expressive power of deep learning from grid-like data to graph-structured data and have achieved remarkable success in a wide variety of domains [ 7 , 6 , 13 , 27 , 22 , 30 , 1 , 8 , 42 , 18 , 31 , 12 , 41 , 19 ] . Just like CNNs , modern GCNs could promisingly learn both the local and global structural patterns of graphs through designed convolutions . However , the vulnerability of GCNs against adversarial attacks has been revealed recently [ 70 , 11 , 9 ] . The lack of robustness arouses concerns on applying GCNs in a variety of fields pertaining to security and privacy . Adversarial attacks on graphs aim to degrade the ability of representation learning of GCNs , and fool them to make wrong decisions by perturbing either node features or graph structures . Given the complexity of the underlying structural information and the ease of operation in practice [ 50 ] , the majority of literature focuses on the adversarial attacks on structures by inserting/removing/rewiring adversarial edges , i.e. , structural attacks . In other words , ∗Heng Chang is supported by 2020 Tencent Rhino-Bird Elite Training Program . Work done during Heng ’ s internship at Tencent AI Lab . †Corresponding authors . 3Code available at https : //github.com/SwiftieH/LFR . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . structural attacks on graphs during the training stage of GCNs with node features unchanged becomes a commonly considered setting by attackers [ 24 ] . Therefore , it is of emerging importance to enhance the robustness of GCNs against structural attacks , which is also the aim of this work . For a better understanding of the structural attacks , it naturally leads to the question : Do the adversarial edges post equal influences on the graph spectrum ? Here , graph spectrum plays a significant role in Graph Signal Processing ( GSP ) [ 45 , 39 ] and graph learning . Empirical observations from recent papers suggest the answer could be “ no ” . We can observe that the perturbations resulting from structural attacks express an implicit trend on the graph spectrum [ 14 ] . As the toy example shown in Figure 1 , low-frequency components ( far left ) tend to be more robust in comparison with high-frequency ones ( far right ) . Similarly , [ 29 ] observes that the low-pass filters seem to be more stable than high-pass filters when we randomly add/remove edges . These observations point to a promising direction for developing principled defense approaches based on the theoretical analysis of graph spectrum . Contributions . To mitigate the vulnerability of GCNs against adversarial attacks , in this paper , we aim to bridge the gap between the spectral analysis and the robustness of GCNs under structural perturbations . Specifically , for the symmetric normalized Laplacian , which is commonly chosen as the graph filter in GCNs , we prove that the low-frequency components could be more robust against both one-edge and multiedge perturbations when their eigenvalues fall into a robust interval . Interestingly , this result reveals that not all low-frequency components are robust to adversarial attacks . Considering the benefit from robust interval in low frequencies , we further propose GCN-LFR ( Low-Frequency based Regularization ) , a general robust co-training paradigm that can transfer the robustness from the eligible low-frequency components . GCN-LFR is a general defender that can work with any GCN-based model against various training-time attacks . In particular , GCN-LFR regularizes the training process of any given GCN with the robust information from an auxiliary regularization netMLFR , which injects a set of learnable parameters to imitate the robust interval for low-frequency components . Moreover , instead of jointly training two branches , we design an alternative training scheme to accelerate the training process . We observe that the alternative training scheme also benefits the final performance in practice . In experiments , to demonstrate the flexibility of GCN-LFR , we integrate GCN-LFR with five popular GCN-based models and compare GCN-LFR with four state-of-the-art defenders across five benchmark datasets under a variety of settings , including one-edge targeted , multi-edge targeted , and nontargeted attacks . Extensive results demonstrate that GCN-LFR consistently outperforms the defending baselines under all settings . Remarkably , we also show that GCN-LFR successfully enhances the robustness of GCNs without deduction of performance on benign graphs . It further reveals the broad applicability and relevance of GCN-LFR in the area of deep graph learning . 2 Related Work . Adversarial attacks on graphs . Adversarial attacks on graph neural networks have drawn unprecedented attention from researchers recently [ 47 , 24 , 49 ] . Based on the stage where the attack happens , adversarial attacks on graph-structured data can be divided into 1 ) poisoning attacks ( e.g. , Nettack [ 70 ] ) that perturb the graph in the training stage and 2 ) evasion attacks ( e.g. , RL-S2V [ 11 ] ) that perturb the graph at testing . A series of studies integrates various of techniques to construct persuasive adversarial samples , such as gradient based methods [ 55 , 60 ] , meta-learning [ 71 ] , matrix perturbation theory [ 3 ] , graph generation [ 32 ] , graph signal processing [ 9 ] and reinforcement learning [ 11 , 48 ] . Defenses on graphs . Other than the adversarial attacks on graphs , researchers are also concerned about the robustness of graph models . Generally , we can category the existing methods into two families : spatial-based and spectral-based defense . 1 ) Spatial-based defense . GCN-Jaccard [ 55 ] proposes to examine fake edges in the step of preprocessing by utilizing similarity metrics . RGCN [ 67 ] , Pro-GNN [ 25 ] , and VPN [ 23 ] focus on proposing new variants of GCNs that can effectively defend against attacks on VanillaGCN . GNNGUARD [ 63 ] detects the adverse effects existing in the relationship between the graph structure and node features by neighbor importance estimation . Low-Pass [ 53 ] considers low-pass message passing from neighborhoods , which is similar to the attention mechanism in GNNGUARD to defense structural attacks . 2 ) Spectral-based defense . GCN-SVD [ 14 ] empirically verifies that only the high-rank singular components of the graph are affected by Nettack , and preprocesses the poisoned graph via SVD with pure structural information for defense . In this paper , we first explore the theoretical connection between robustness and graph spectrum , and propose GCN-LFR to bring the robustness information from eligible low-frequency components in the spectral domain . The proposed framework GCN-LFR is a general defender and can protect various GCN-based models from adversarial attacks . 3 Preliminaries . We consider an undirected attributed graph G = ( V , E , X ) , where |V| = N is the set of N nodes , E ⊆ V × V is the set of E edges , and X = { x1 , ... , xN } is the associated feature matrix , in which xu ∈ RM is the M -dimensional node feature for node u ∈ V . We utilize Nu to denote the neighbors of node u . We denote the adjacency matrix as A ∈ RN×N , where Auv ∈ { 0 , 1 } indicates whether an edge euv ∈ E exists . The node degree matrix is defined as D = diag ( d1 , · · · , dn ) , where du = ∑ vAuv represents the degree of node u . An adversarial attacker deliberately perturbs edges in G and results in the poisoned version of G , which we denote as G′ = ( V ′ , E ′ , X ) , and A′ is the perturbed version of A . Let L = D−1/2 ( D −A ) D−1/2 be the symmetric normalized graph Laplacian . L has eigenvalues ranging from 0 to 2 . Â = D− 12AD− 12 = In −L is the symmetric normalized adjacency matrix , whose eigenvalues range from −1 to 1 . Specifically , the low-frequency components refer to the ones of L with eigenvalues in [ 0 , 1 ) ( of Â with eigenvalues in ( 0 , 1 ] ) , and the high-frequency ones refer to that of L with eigenvalues in ( 1 , 2 ] ( of Â with eigenvalues in [ −1 , 0 ) ) . 3.1 Graph Convolutional Networks . VanillaGCN can be viewed as the spectral convolution based on the Fourier transform on graphs with first-order Chebyshev polynomial filter g ( L ) . The layer-wise update rule is given as : feature transformation : H ′ ( l ) = H ( l ) Θ , graph convolution : H ( l+1 ) = σ ( g ( L ) H ′ ( l ) ) , ( 1 ) where H ( l+1 ) ∈ RN×q ( l+1 ) is the output from hidden layer l with input as H ( l ) ∈ RN×q ( l ) . q ( l ) refers to the dimension of output at layer l. For the input layer , we have H ( 0 ) = X . σ ( · ) refers to the activation function , such as ReLU . Θl ∈ Rq ( l ) ×q ( l+1 ) refers to the weight matrix . In Eq . ( 1 ) , different filter g ( L ) leads to different GCNs . For example , VanillaGCN along with its variants [ 61 , 54 , 16 , 10 ] usually use the symmetric normalized adjacency matrix Â = In −L as g ( L ) . Therefore , we mainly focus on the analysis of filter Â , and try to find out the connection between the robustness of GCNs and the perturbation on Â . To ease the notation , we denote U and Λ = diag ( λ1 , · · · , λN ) as the eigen-pairs of Â in the following sections unless otherwise indicated . 3.2 Poisoning Structural Attacks . Poisoning structural attacks towards graph-structured data focus on corrupting the topology during training stage [ 24 ] . Specifically , the attacker is able to delete or insert a small number of edges on the benign graph to generate a perturbed graph G′ with adjacency matrix A′ . Through this manipulation , the attacker intends to fool the prediction of GCNs and damage the performance of the downstream learning tasks . Formally , given a fixed budget of perturbation ∆ on edges , i.e. , ||A′ −A||0 ≤ 2∆ , the adversarial attack on a GCN-based modelM can be formulated as a bi-level optimization problem [ 3 , 9 ] : argmin A′∈PG∆ Lattack ( M ( A′ , X ; Θ∗ ) , y ) s.t . Θ∗ = argmin Θ LGCN ( M ( A′ , X ; Θ ) , y ) , where PG∆ denotes the possible perturbations within budget ∆ . y is the ground-truth label . Lattack and LGCN denote the loss of adversarial attack and modelM , respectively.M ( A′ , X ; Θ∗ ) refers to the prediction of modelM on the poisoned graph G′ , where the weights Θ∗ keep trained to optimal . We denote T as the set of target nodes aimed by the attacker . We consider the following two types of attacks : 1 ) Targeted attacks . The attacker aims to destroy prediction for a specific node u by manipulating the adjacent edges of u [ 70 , 55 , 9 ] , and T = { u } in this situation . 2 ) Non-targeted attacks . The attacker aims to degrade the overall classification performance but does not care about which node is being targeted [ 71 , 60 ] . Under this setting , T = Vtest , where Vtest is the full test set .

Summary: This work tries to study the robustness of GCNs under structural perturbations from the perspective of graph spectrum, which for me is interesting. In theory, they proved that the low-frequency components can be more robust against structural perturbations if their eigenvalues fall into a robust interval. For algorithm design, they developed a regularized learning framework that combines a regular GCN-based objective and a low-frequency regularization term, named GCN-LFR. Finally, the authors tested the capability of GCN-LFR in resisting adversarial attacks on several benchmark datasets.