A Unified View on Graph Neural Networks as Graph Signal Denoising

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have shown great capacity in learning representations for graphstructured data and thus have facilitated many down-stream tasks such as node classification ( Kipf & Welling , 2016 ; Veličković et al. , 2017 ; Ying et al. , 2018a ; Klicpera et al. , 2018 ) and graph classification ( Defferrard et al. , 2016 ; Ying et al. , 2018b ) . As traditional deep learning models , a GNN model is usually composed of several stacking GNN layers . Given a graph G with N nodes , a GNN layer typically contains a feature transformation and a feature aggregation operation as : Feature Transformation : X′in = ftrans ( Xin ) ; Feature Aggregation : Xout = fagg ( X ′ in ; G ) ( 1 ) where Xin ∈ RN×din and Xout ∈ RN×dout denote the input and output features of the GNN layer with din and dout as the corresponding dimensions , respectively . Note that the non-linear activation is not included in Eq . ( 1 ) to ease the discussion . The feature transformation operation ftrans ( · ) transforms the input of Xin to X′in ∈ RN×dout as its output ; and the feature aggregation operation fagg ( · ; G ) updates the node features by aggregating the transformed node features via the graph G. In general , different GNN models share similar feature transformations ( often , a single feed-forward layer ) , while adopting different designs for aggregation operation . We raise a natural question – is there an intrinsic connection among these feature aggregation operations and their assumptions ? The significance of a positive answer to this question is two-fold . Firstly , it offers a new perspective to create a uniform understanding on representative aggregation operations . Secondly , it enables us to develop a general GNN framework that not only provides a unified view on multiple existing representative GNN models , but also has the potential to inspire new ones . In this paper , we aim to build the connection among feature aggregation operations of representative GNN models including GCN ( Kipf & Welling , 2016 ) , GAT ( Veličković et al. , 2017 ) , PPNP and APPNP ( Klicpera et al. , 2018 ) . In particular , we mathematically establish that the aggregation operations in these models can be unified as the process of exactly , and sometimes approximately , addressing a graph signal denoising problem with Laplacian regularization ( Shuman et al. , 2013 ) . This connection suggests that these aggregation operations share a unified goal : to ensure feature smoothness of connected nodes . With this understanding , we propose a general GNN framework , UGNN , which not only provides a straightforward , unified view for many existing aggregation operations , but also suggests various promising directions to build new aggregation operations suitable for distinct applications . To demonstrate its potential , we build an instance of UGNN called ADA-UGNN , which is suited for handling varying smoothness properties across nodes , and conduct experiments to show its effectiveness . 2 REPRESENTATIVE GRAPH NEURAL NETWORKS . In this section , we introduce notations for graphs and briefly summarize several representative GNN models . A graph can be denoted as G = { V , E } , where V and E are its corresponding node and edge sets . The connections in G can be represented as an adjacency matrix A ∈ RN×N , with N the number of nodes in the graph . The Laplacian matrix of the graph G is denoted as L. It is defined as L = D −A , where D is a diagonal degree matrix corresponding to A . There are also normalized versions of the Laplacian matrix such as L = I −D− 12AD− 12 or L = I −D−1A . In this work , we sometimes adopt different Laplacians to establish connections between different GNNs and the graph denoising problem , clarifying in the text . In this section , we generally use Xin ∈ RN×din and Xout ∈ RN×dout to denote input and output features of GNN layers . Next , we describe a few representative GNN models . 2.1 GRAPH CONVOLUTIONAL NETWORKS ( GCN ) . Following Eq . ( 1 ) , a single layer in GCN ( Kipf & Welling , 2016 ) can be written as follows : Feature Transformation : X′in = XinW ; Feature Aggregation : Xout = ÃX ′ in , ( 2 ) where W ∈ Rdin×dout is a feature transformation matrix , and Ã is a normalized adjacency matrix which includes a self-loop , defined as follows : Ã = D̂− 1 2 ÂD̂− 1 2 , with Â = A+ I and D = diag ( ∑ j Â1 , j , . . . , ∑ j ÂN , j ) . ( 3 ) In practice , multiple GCN layers can be stacked , where each layer takes the output of its previous layer as input . Non-linear activation functions are included between consecutive layers . 2.2 GRAPH ATTENTION NETWORKS ( GAT ) . Graph Attention Networks ( GAT ) adopts the same feature transformation operation as GCN in Eq . ( 2 ) . The feature aggregation operation ( written node-wise ) for a node i is as : Xout [ i , : ] = ∑ j∈Ñ ( i ) αijX ′ in [ j , : ] , with αij = exp ( eij ) ∑ k∈Ñ ( i ) exp ( eik ) . ( 4 ) where Ñ ( i ) = N ( i ) ∪ { i } denotes the neighbors ( self-inclusive ) of node i , and Xout [ i , : ] is the i-th row of the matrix Xout , i.e . the output node features of node i . In this aggregation operation , αij is a learnable attention score to differentiate the importance of distinct nodes in the neighborhood . Specifically , αij is a normalized form of eij , which is modeled as : eij = LeakyReLU ( [ X′in [ i , : ] ‖X′in [ j , : ] ] a ) ( 5 ) where [ ·‖· ] denotes the concatenation operation and a ∈ R2d is a learnable vector . Similar to GCN , a GAT model usually consists of multiple stacked GAT layers . 2.3 PERSONALIZED PROPAGATION OF NEURAL PREDICTIONS ( PPNP ) . Personalized Propagation of Neural Predictions ( PPNP ) ( Klicpera et al. , 2018 ) introduces an aggregation operation based on Personalized PageRank ( PPR ) . Specifically , the PPR matrix is defined as α ( I − ( 1 − α ) Ã ) −1 , where α ∈ ( 0 , 1 ) is a hyper-parameter . The ij-th element of the PPR matrix specifies the influence of node i on node j . The feature transformation operation is modeled as Multi-layer Perception ( MLP ) . The PPNP model can be written in the form of Eq . ( 1 ) as follows : Feature Transformation : X′in = MLP ( Xin ) ; Feature Aggregation : Xout = α ( I− ( 1− α ) Ã ) −1X′in . ( 6 ) Unlike GCN and GAT , PPNP only consists of a single feature aggregation layer , but with a poten- tially deep feature transformation . Since the matrix inverse in Eq . ( 6 ) is costly , Klicpera et al . ( 2018 ) also introduces a practical , approximated version of PPNP , called APPNP , where the aggregation operation is performed in an iterative way as : X ( k ) out = ( 1− α ) ÃX ( k−1 ) out + αX ′ in k = 1 , . . .K , ( 7 ) where X ( 0 ) out = X ′ in and X ( K ) out is the output of the feature aggregation operation . As proved in Klicpera et al . ( 2018 ) , X ( K ) out converges to the solution obtained by PPNP , i.e. , Xout in Eq . ( 6 ) . 3 GNNS AS GRAPH SIGNAL DENOISING . In this section , we aim to establish the connections between the introduced GNN models and a graph signal denoising problem with Laplacian regularization . We first introduce the problem . Problem 1 ( Graph Signal Denoising with Laplacian Regularization ) . Suppose that we are given a noisy signal X ∈ RN×d on a graph G. The goal of the problem is to recover a clean signal F ∈ RN×d , assumed to be smooth over G , by solving the following optimization problem : argmin F L = ‖F−X‖2F + c · tr ( F > LF ) , ( 8 ) Note that the first term guides F to be close to X , while the second term tr ( F > LF ) is the Laplacian regularization that guides the smoothness of F over the graph . c > 0 is a balancing constant . Assuming we adopt the unnormalized version of Laplacian matrix with L = D−A ( the adjacency matrix A is assumed to be binary ) , the second term in Eq . ( 8 ) can be written in an edge-centric way or a node-centric way as : edge-centric : c ∑ ( i , j ) ∈E ‖F [ i , : ] − F [ j , : ] ‖22 ; node-centric : 1 2 c ∑ i∈V ∑ j∈Ñ ( i ) ‖F [ i , : ] − F [ j , : ] ‖22 . ( 9 ) Clearly , from the edge-centric view , the regularization term measures the global smoothness of F , which is small when connected nodes share similar features . On the other hand , we can view the term∑ j∈Ñ ( i ) ‖F [ i , : ] − F [ j , : ] ‖ 2 2 as a local smoothness measure for node i as it measures the difference between node i and all its neighbors . The regularization term can then be regarded as a summation of local smoothness over all nodes . Note that the adjacency matrix A is assumed to be binary when deriving Eq . ( 9 ) . Similar formulations can also be derived to other types of Laplacian matrices . In the following subsections , we demonstrate the connections between aggregation operations in various GNN models and the graph signal denoising problem . 3.1 CONNECTION TO PPNP AND APPNP . In this subsection , we establish the connection between the graph signal denoising problem ( 8 ) and the aggregation propagations in PPNP and APPNP in Theorem 1 and Theorem 2 , respectively . Theorem 1 . When we adopt the normalized Laplacian matrix L = I−Ã , with Ã defined in Eq . ( 3 ) , the feature aggregation operation in PPNP ( Eq . ( 6 ) ) can be regarded as exactly solving the graph signal denoising problem ( 8 ) with X′in as the input noisy signal and c = 1 α − 1 . Proof . Note that the objective in Eq . ( 8 ) is convex . Hence , its closed-form solution F∗ to exactly solve the graph signal denosing problem can be obtained by setting its derivative to 0 as : ∂L ∂F = 2 ( F−X ) + 2cLF = 0⇒ F∗ = ( I+ cL ) −1X ( 10 ) Given L = I− Ã , F∗ can be reformulated as : F∗ = ( I+ cL ) −1 X = ( I+ c ( I− Ã ) ) −1 X = 1 1 + c ( I− c 1 + c Ã ) −1 X ( 11 ) The feature aggregation operation in Eq . ( 6 ) is equivalent to the closed-form solution in Eq . ( 11 ) when we set α = 1/ ( 1 + c ) and X = X′in . This completes the proof . Theorem 2 . When we adopt the normalized Laplacian matrix L = I − Ã , the feature aggregation operation in APPNP ( Eq . ( 7 ) ) approximately solves the graph signal denoising problem ( 8 ) by iterative gradient descent with X′in as the input noisy signal , c = 1 α − 1 and stepsize b = 1 2+2c . Proof . To solve the graph signal denoising problem ( 8 ) , we take iterative gradient method with the stepsize b . Specifically , the k-th step gradient descent on problem ( 8 ) is as follows : F ( k ) ← F ( k−1 ) − b · ∂L ∂F ( F = F ( k−1 ) ) = ( 1− 2b− 2bc ) F ( k−1 ) + 2bX+ 2bcÃF ( k−1 ) ( 12 ) where F ( 0 ) = X . When we set the stepsize b as 12+2c , we have the following iterative steps : F ( k ) ← 1 1 + c X+ c 1 + c ÃF ( k−1 ) , k = 1 , . . .K , ( 13 ) which is equivalent to the iterative aggregation operation of the APPNP model in Eq . ( 7 ) with X = X′in and α = 1 1+c . This completes the proof . These two connections provide a new explanation on the hyper-parameter α in PPNP and APPNP from the graph signal denoising perspective . Specifically , a smaller α indicates a larger c , which means the obtained Xout is enforced to be smoother over the graph .

Summary: of the paper: In this paper, the authors make the following new argument: The aggregation processes of current popular GNN models such as GCN, GAT, PPNP, and APPNP can be treated as a graph denoising problem where the objective is to minimize a recovery error (a norm of noisy feature matrix, i.e. ||F-X||) plus a graph-based regularization (smoothness). This new view provides a way to build a GNN model, namely (Ada-)UGNN. Experimental results show the effectiveness of Ada-UGNN on the task of node classification and the task of preventing adversarial attacks on graphs.