Defending Graph Neural Networks via Tensor-Based Robust Graph Aggregation

1 INTRODUCTION . In real-world applications , many complex structures from various domains can be represented by graph data , such as social networks , chemistry ( molecular ) , biology ( protein-protein interaction networks ) , knowledge databases , and recommender systems . Graph Neural Networks ( GNNs ) have achieved outstanding success for graph data in a wide range of application areas ( Kipf & Welling , 2017 ; Battaglia et al. , 2018 ; Wu et al. , 2021 ; Zhou et al. , 2020 ) . GNNs are equipped with a messagepassing scheme ( Gilmer et al. , 2017 ) , where the node representation is updated by transforming the aggregated representations propagated from its neighbor nodes via edges , and are optimized for performance on downstream tasks . In this fashion , GNNs can generate representations capturing both graph structural information and node feature information . In this paper , we focus on the semi-supervised task of node classification ( Kipf & Welling , 2017 ) by GNNs . GNNs have achieved promising performance in recent years , but have also shown to be vulnerable to adversarial attacks ( Bojchevski & Günnemann , 2019 ; Dai et al. , 2018 ; Jin et al. , 2020a ; Liu et al. , 2019 ; Ma et al. , 2019 ; Wu et al. , 2019b ; Zügner et al. , 2018 ; Zügner & Günnemann , 2019b ) . Most of adversarial attack methods for GNNs focus on poisoning attacks , which deliberately rebuild the graph topology by changing an unnoticeable number of edges before the training process of GNNs . These unnoticeable perturbations can significantly degrade learned graph representations and confuse GNNs to catastrophically misclassify nodes . To defend the GNNs from adversarial edges , existing methods attempt to learn robust graphs , which are extracted from the perturbed graphs and perform solidly with GNNs . One way is to learn robust graph by reducing the attention weights of potentially malicious edges ( Zhu et al. , 2019 ; Tang et al. , 2019 ; 2020 ; Zhang & Zitnik , 2020 ) . Preprocessing the perturbed graphs to obtain robust graphs is another way to defend GNNs from adversarial edges ( Wu et al. , 2019b ; Entezari et al. , 2020 ) . Most of the existing works only pay attention to a specific property of graph data ( e.g. , low-rank property ( Entezari et al. , 2020 ; Al-Sayouri et al. , 2020 ) , feature similarities ( Wu et al. , 2019b ) ) . Pro-GNN ( Jin et al. , 2020b ) regularizes the learned graph to fit a set of the corresponding graph properties and learns the weights for each node instead of the whole graph . These methods show improvements in defending GNNs from poisoning attacks but ignore the connection and balance between different graph properties or different nodes . The learned robust graph has every single edge with a weight close to the original graph and following the specific graph properties , however , when looking through the point of view from the whole graph , it is either ignoring the original structure of the graph or not able to remove most of the perturbed edges . To learn better robust graph from adversarial graphs , we propose a Tensor-based framework for Graph Neural Networks ( TGNN ) , which can aggregate common information from the predefined robust graphs and the adversarial graphs and thus enhance the robustness of GNNs with improved classification performance . TGNN approximates the graph tensor , which consists of the attacked graph and several predefined robust graphs focusing on specific graph properties ( e.g. , low-rank , feature similarity properties ) , via low-rank approximation by applying tensor decomposition . By preserving the low-rank tensor representation , TGNN generates the robust graph and maintains the common core tensor space between the learned robust graph and predefined robust graphs . The learned robust graph is no longer calculated individually by each entry but aggregated with the common core tensors , balancing not only the different graph properties but also varied structural graph information . The learned robust graph is fed into GNNs and jointly leaned with GNN parameters . Extensive empirical results show that our proposed TGNN framework consistently improves the robustness of the learned graph and significantly boosts the performance of node classification task for GNNs against adversarial graphs . 2 RELATED WORKS . 2.1 GRAPH NEURAL NETWORKS AND ADVERSARIAL ATTACKS ON GRAPHS . GNNs have achieved great success in solving machine learning problems on graph data in recent years . Existing GNNs can be divided into two categories : spectral-based methods where graph spectral theory ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Wu et al. , 2019a ; Chen et al. , 2018 ) was applied to learn node representations ; and spatial-based GNNs based on local-dependence assumption of graph-structured data that graph convolution is defined in the spatial domain as aggregating and transforming local information ( Gilmer et al. , 2017 ; Hamilton et al. , 2017 ; Velickovic et al. , 2018 ; Atwood & Towsley , 2016 ) . Extensive studies have demonstrated that deep learning models are vulnerable to adversarial attacks ( Goodfellow et al. , 2015 ; Kurakin et al. , 2017a ; b ; Rakhsha et al. , 2020 ) . Attackers add unnoticeable perturbations to the input for altering the output of the deep learning model . GNNs are also fragile to adversarial attacks ( Bojchevski & Günnemann , 2019 ; Dai et al. , 2018 ; Jin et al. , 2020a ; Liu et al. , 2019 ; Ma et al. , 2019 ; Wu et al. , 2019b ; Zügner et al. , 2018 ; Zügner & Günnemann , 2019b ) . It is more challenging to investigate adversarial attack on graphs due to the discrete graph and dependent nodes compared with independent samples or grid data . Nettack ( Zügner et al. , 2018 ) preserves degree distribution and constrains feature co-occurrence to generate unnoticeable perturbations to poison GNNs in training time . Reinforcement learning was employed in RL-S2V ( Dai et al. , 2018 ) to generate adversarial attacks in testing-time . Both of the above two methods can only degrade the performance of GNN on target nodes . To attack the graph globally , metattack ( Zügner & Günnemann , 2019b ) is proposed to generate non-targeted poisoning attacks based on meta-learning . 2.2 DEFENSE ON GRAPHS . The studies for improving the robustness of GNNs have received increasing attention recently ( Tang et al. , 2019 ; Wu et al. , 2019b ; Zhu et al. , 2019 ; Zügner & Günnemann , 2019a ; Bojchevski et al. , 2020 ; Geisler et al. , 2020 ) . In RGCN ( Zhu et al. , 2019 ) , Gaussian distribution is adopted in hidden layers to absorb the effects of adversarial attacks into the variances . A meta-optimization way is proposed in PA-GNN ( Tang et al. , 2019 ) to learn attention scores for robust GNNs with the help of supervision knowledge from clean graphs . Transfer learning ( Tang et al. , 2020 ) is leveraged to improve the robustness of GNNs against poisoning attack with additional graph data from the similar domain during training . It is observed in Wu et al . ( 2019b ) that attackers tend to connect to nodes with different features occurrence , GNN-Jaccard is proposed to remove the links between dissimilar nodes to identify the manipulated edges . GNN-SVD ( Entezari et al. , 2020 ) is proposed to preprocess the graph with its low-rank approximations to remove noisy information through an SVD decomposition . Pro-GNN ( Jin et al. , 2020b ) proposes three different regularizers which correspondingly regularize the graph to be sparse , low-rank and smooth the features of connected nodes . In Zhang & Zitnik ( 2020 ) , GNNGuard is proposed to quantify the relationship between the graph and node features , and assigns less weight to suspicious edges , which mitigates negative effects of the adversarial perturbations . Different from the aforementioned defense methods , we aim to aggregates predefined robust adversarial graphs to form a powerful regularization , which can be extended to different GNNs straightforwardly to enhance the robustness of GNNs . 3 METHODOLOGY . Before presenting the proposed method , we first introduce the problem statement for node classification with GNN , including some basic notations and concepts . Let G = ( V , E , X ) be a graph , where V represents the set of N nodes { v1 , v2 , · · · , vN } , E represents the set of M edges { ( i1 , j1 ) , ( i2 , j2 ) , · · · , ( iM , jM ) } , and X ∈ RN×d denotes the node feature matrix composed of the feature vector xi of the node vi . The edges E describe the graph by the pair-wise relations between nodes , which can also be represented by an adjacency matrix A ∈ RN×N , one of the most popular representation for graph , where Aij = 1 ⇐⇒ ( i , j ) ∈ E ( otherwise Aij = 0 ) . In this work , we alternately use the concept of adjacency matrix and graph . Thus , a graph can also be denoted as G = ( A , X ) . In this work , we focus on the common semi-supervised node classification setting for graph representation . Given a single ( attributed ) graph G = ( A , X ) , the part of nodes VP = { v1 , v2 , · · · , vP } ⊆ V are associated with the corresponding labels YP = { y1 , y2 , · · · , yP } , where yi is the class label ( e.g. , the topic of a paper in a citation graph ) of vi , the goal of node classification in GNN is to learn a neural network representing a function fθ : VP → YP which maps the nodes to the corresponding labels so that fθ can be generalized to predict the labels of unlabeled nodes ( i.e. , nodes not in VP ) , where θ is the parameters of fθ . With LCE ( · , · ) denoting the cross entropy loss , the objective function of node classification task in GNN can be formulated as : min θ Lcls ( θ , A , X , YP ) = 1 P P∑ i=1 LCE ( yi , fθ ( X , A ) i ) , ( 1 ) where fθ ( X , A ) i is the prediction of node vi . Note that our proposed method can be extended to different GNNs straightforwardly , in this work , we focus on Graph Convolution Network ( GCN ) proposed in Kipf & Welling ( 2017 ) among numbers of different GNNs . Specifically , we adopt a two-layer GCN with layers ( fθ1 , fθ2 ) and the corresponding parameters ( θ1 , θ2 ) = ( W1 , W2 ) can be denoted as : fθ ( X , A ) = softmax ( fθ2 ( fθ1 ( X , A ) , A ) ) = softmax ( Âσ ( ÂXW1 ) W2 ) , ( 2 ) where σ is the activation function and Â = D̂− 1 2 ( A + I ) D̂− 1 2 represents diagonal normalized A + I . Here , self-loop I is added to A and D̂ is the diagonal degree matrix of A + I such that D̂ii = 1 + ∑ jAij .

Summary: The paper proposes a tensor-based defense strategy against adversarial perturbations to graph neural networks. Specifically, the authors propose to aggregate several "robust" adjacency matrices obtained from previously proposed methods into a tensor, which is then decomposed. Based on this decomposition, we can then obtain a new robust graph, with the idea of combining the strengths of the different robust approaches.