A General Unified Graph Neural Network Framework Against Adversarial Attacks

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have drawn great attention as graphs can represent complex relationships among nodes . Graphs are ubiquitous in different domains , which are usually applied in recommender systems ( Ying et al. , 2018a ) , chemistry ( Duvenaud et al. , 2015 ) , social media ( Qiu et al. , 2018 ) and so on . Utilizing the strong representation capacity of graphs , we can enhance performance of down-stream tasks such as node classification ( Kipf & Welling , 2017 ; Velickovic et al. , 2018 ; Klicpera et al. , 2019 ) , link prediction ( Grover & Leskovec , 2016 ; Bojchevski et al. , 2018 ) and graph classification ( Defferrard et al. , 2016 ; Ying et al. , 2018b ) . A GNN model often consists of several graph convolution layers . A common practice of convolution layers is utilizing a feed-forward network to transform features and then aggregating transformed features . A series of convolution layers have been proposed and achieved great success such as GCN ( Kipf & Welling , 2017 ) , GAT ( Velickovic et al. , 2018 ) and PPNP ( Klicpera et al. , 2019 ) . However , GNN models composed of these convolution layers are vulnerable to adversarial attacks . Attacks can be conducted on either node features or the graph structure , while most existing adversarial attacks on graph data focus on modifying the graph structure ( Xu et al. , 2020 ) . They always try to add , delete , or rewire edges to change the graph structure . Although these perturbations are unnoticeable , they can easily degrade the performance of GNN models , which may cause bad consequences . For example , spammers may create virtual followers to increase the chance of false messages being recommended and spread . The lack of GNNs ’ robustness raises increasing concerns for applying it in some risk-sensitive domains . Therefore , it is necessary to develop graph defense techniques . Many existing defense methods focus on cleaning perturbed graphs by detecting properties of clean graphs and effects of specific attacks on graphs ( Entezari et al. , 2020 ; Jin et al. , 2020b ) . Prior knowledge according to these researches can help GNN models defend against adversarial attacks to a certain extent . The study ( Jin et al. , 2020b ) has proved that adversarial attacks could lead perturbed graphs to violate some properties of real graphs . For example , the rank of attacked graph increases and adversarial attacks often connect nodes with large feature differences . However , existing studies only focus on cleaning the perturbed graph structure , and almost none of them simultaneously consider denoising features . As the graph and features are closely tied and contain perturbations and noises , in this paper , we propose a General Unified Graph Neural Network ( GUGNN ) framework to jointly clean the graph and denoise features . Based on the GUGNN framework , we further introduce two kinds of operations . One operation is reconstructing the graph with its properties that real-world graphs are sparse ( Zhou et al. , 2013 ) , the features of two adjacent nodes tend to be similar ( McPherson et al. , 2001 ) and perturbed graphs have many slight noises with small eigenvalues . Nodes with more neighbors are hard to attack ( Zügner et al. , 2018 ) . Though we can not change the number of nodes ’ first-order neighbors , we can adjust the size of nodes ’ neighborhood to add some high-order neighbors to nodes . According to this principle , from the denoising perspective , we design the convolution operation for features to find the optimal solution . Utilizing the two kinds of operations above , we develop a robust model ( R-GUGNN ) , which can be applied for defending against different adversarial attacks . The contributions of this paper are summarized as follows : • We propose GUGNN framework to jointly clean the graph and denoise features for perturbations and noises existing in the graph and features . • We introduce two kinds of operations to clean attacked graphs and denoise features respectively based on the GUGNN framework . • For defending against adversarial attacks , we develop a concrete model R-GUGNN to realize the goal of the GUGNN framework utilizing the two kinds of operations . • Experiments show that R-GUGNN has a strong capacity for defending against different adversarial attacks and stably outperforms the state-of-the-art defense models . 2 RELATED WORK . This section has two parts , including graph neural networks , and adversarial attacks and defenses for GNNs . 2.1 GRAPH NEURAL NETWORKS . In this subsection , we review some famous graph neural network models , including GCN , GAT , PPNP , and a unified GNN framework UGNN . For more knowledge about GNNs , you can refer to some reviews ( Wu et al. , 2020 ; Zhang et al. , 2020 ) . The convolution of GCN ( Kipf & Welling , 2017 ) is defined in the graph spectral domain . Avoiding computing the full eigenvectors of the graph Laplacian matrix , based on Chebyshev polynomials , GCN only uses the first-order polynomial to simplify the graph convolution , which has been an accepted good graph convolution layer for its performance . The convolution of GAT ( Velickovic et al. , 2018 ) is defined in the spatial domain . The difference between GAT and GCN is that GAT learns different attention scores for neighbors when aggregating features . PPNP ( Klicpera et al. , 2019 ) derives a propagation scheme based on personalized PageRank . It propagates information from a large and adjustable neighborhood instead of the first-order neighborhood directly . The neighborhood can be adjusted via a hyper-parameter . UGNN ( Ma et al. , 2020 ) is a unified GNN framework available for different feature propagation processes from a denoising perspective . It proposes a denoising optimization problem with the Laplacian regularization term and tries to solve it in different ways utilizing the first derivative or the optimal solution . Original Laplacian matrix can be also replaced with different normalized forms . Different solutions to the optimization problem are corresponding to various convolution layers with different feature aggregation processes such as GCN , GAT , PPNP and so on . 2.2 ADVERSARIAL ATTACKS AND DEFENSES FOR GNNS . We recommend a repository DeepRobust ( Li et al. , 2020 ) for readers . It contains many adversarial attacks and defenses on the graph , which is quite useful for researchers . For more knowledge about adversarial attacks and defenses for GNNs , you can refer to the review ( Jin et al. , 2020a ) . Some adversarial attack methods have been proposed to show the vulnerability of GNNs with some unnoticeable perturbations added to the graph structure or node attributes . In the field of node classification , the aim of adversarial attacks is fooling GNNs into classifying nodes incorrectly . Poisoning attacks change the graph structure before we train GNN models , which is one of the most common settings of adversarial attacks on graph data . Poisoning attacks have various types , including global attack , targeted attack and random attack . Based on the whole graph , the goal of global attack is to degrade the overall performance of GNNs . One of the state-of-the-art global attacks is metattack ( Zügner & Günnemann , 2019 ) , which generates the poisoning attacks based on metalearning . Targeted attack generates attacks on some specific nodes and aims to fool GNNs on these target nodes . The nettack ( Zügner et al. , 2018 ) is one of the state-of-the-art targeted attacks , which aims to change the graph structure and features of target nodes or nearby nodes with perturbations remaining unnoticeable . Random attack adds random noises to the clean graph whose concrete practice is adding , removing or flipping edges randomly . Methods about preventing GNNs from adversarial attacks are also developed to improve the robustness of GNNs recently . To mitigate the effects of adversarial attacks on the graph , RGCN ( Zhu et al. , 2019 ) uses Gaussian distributions as hidden representations of nodes instead of plain vectors in other GNNs . Considering that nettack is a high-rank attack , GCN-SVD ( Entezari et al. , 2020 ) is proposed to reconstruct the perturbed graph with only the top-k largest singular components . Using such a low-rank approximation , GCN-SVD can reduce the effects of nettack . In fact , the practice of ensuring low-rank is removing noises with small singular values of the graph . Pro-GNN ( Jin et al. , 2020b ) jointly optimizes a structural graph and a robust GNN model from the perturbed graph with some properties of clean graphs . Pro-GNN has big improvement over other defense models with these properties . However , like other defense models , Pro-GNN doesn ’ t take denoising features into account jointly . 3 THE PROPOSED FRAMEWORK . In this section , we first present GUGNN framework , and then we introduce a novel graph reconstruction operation . At last , we show the convolution operation for features and our concrete design of R-GUGNN model , which is used to realize the goal of GUGNN framework . 3.1 NOTIONS . We denote some notations here . Denote X ∈ RN×d as the feature matrix , where N , d represent the number of samples and the dimension of features respectively . Denote G = { V , E } as the graph , where V represents node sets and E represents edge sets . We also use the adjacency matrix A ∈ RN×N to represent G. Algorithm 1 : R-GUGNN Input : Adjacency matrix A , Feature matrix X , Labels y , Hyper-parameters m , c , β , λ , Learning rate η . Output : GNN parameters θ 1 Ã = A + I ; Initialize S = Ã ; 2 for i= 1 to m do 3 Calculating Z in formula ( 5 ) ; 4 S = S− c2Z ; 5 S = prox∗β ( S ) ; 6 S = proxs ( S ) ; 7 Initialize F = X ; Randomly initialize θ ; 8 while Stopping condition is not met do 9 Forward propagation using two convolution layers : 10 Using feature transformation formula : F = FW , 11 and feature aggregation formula : F = ( I + λL̂ ) −1F ; 12 Getting output y′ ; 13 Calculating gradient g according to y and y′ ; 14 Backward propagation : θ = θ − ηg ; 15 return θ ; 3.2 THE GENERAL UNIFIED GNN FRAMEWORK . To discard perturbations and noises in the graph and features , considering the tight connection between them , we propose our general united graph neural network ( GUGNN ) framework to solve such problem , which is shown as follows : argmin S , F L = ‖S− A‖2F + γ‖F− X‖2F + c · tr ( FTLF ) + β · f ( S ) ( 1 ) where S and F are the learned adjacency and feature matrix . L is the Laplacian matrix of S. L = D − S , where D is a diagonal matrix and Dii = ∑N j=1 Sij . L can be also replaced with different normalized forms . tr ( FTLF ) is Laplacian regularization term for both denoising features and cleaning the graph . f ( S ) is a flexible regularization term to enforce some prior over S. γ , c and β are hyper-parameters to balance different components . From a united perspective , we view that both features and the graph contain noises and our goal is jointly optimizing F and S. tr ( FTLF ) can be rewritten as 12 ∑N i , j=1 Sij ( fi − fj ) 2 , where fi is the i-th row of F. This term represents that features of two adjacent nodes should be similar , which is the guidance for both learning F and S. Although X and A have some noises , they can represent the real features and the graph to a large extent . So , the learned F and S should be similar to X and A respectively , which are the meanings of ‖F − X‖2F and ‖S − A‖2F . In addition , we add some prior to the graph in f ( S ) to make it more accurate .

Summary: Considering the problem of adversarial manipulations of graphs, the authors propose a framework for "cleaning" the graph structure and its features to obtain more robust predictions. The core contribution, compared to existing works, is the additional considering of the feature cleaning. Technically, the authors first perform the graph cleaning (phrased as some kind of sparse, trace minimization problem) followed by some feature cleaning (phrased as some kind of feature diffusion/propagation). Experiments on standard (small) benchmark datasets shows some improvements.