A Unified View on Graph Neural Networks as Graph Signal Denoising

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have shown great capacity in learning representations for graphstructured data and thus have facilitated many down-stream tasks such as node classification ( Kipf & Welling , 2016 ; Veličković et al. , 2017 ; Ying et al. , 2018a ; Klicpera et al. , 2018 ) and graph classification ( Defferrard et al. , 2016 ; Ying et al. , 2018b ) . As traditional deep learning models , a GNN model is usually composed of several stacking GNN layers . Given a graph G with N nodes , a GNN layer typically contains a feature transformation and a feature aggregation operation as : Feature Transformation : X′in = ftrans ( Xin ) ; Feature Aggregation : Xout = fagg ( X ′ in ; G ) ( 1 ) where Xin ∈ RN×din and Xout ∈ RN×dout denote the input and output features of the GNN layer with din and dout as the corresponding dimensions , respectively . Note that the non-linear activation is not included in Eq . ( 1 ) to ease the discussion . The feature transformation operation ftrans ( · ) transforms the input of Xin to X′in ∈ RN×dout as its output ; and the feature aggregation operation fagg ( · ; G ) updates the node features by aggregating the transformed node features via the graph G. In general , different GNN models share similar feature transformations ( often , a single feed-forward layer ) , while adopting different designs for aggregation operation . We raise a natural question – is there an intrinsic connection among these feature aggregation operations and their assumptions ? The significance of a positive answer to this question is two-fold . Firstly , it offers a new perspective to create a uniform understanding on representative aggregation operations . Secondly , it enables us to develop a general GNN framework that not only provides a unified view on multiple existing representative GNN models , but also has the potential to inspire new ones . In this paper , we aim to build the connection among feature aggregation operations of representative GNN models including GCN ( Kipf & Welling , 2016 ) , GAT ( Veličković et al. , 2017 ) , PPNP and APPNP ( Klicpera et al. , 2018 ) . In particular , we mathematically establish that the aggregation operations in these models can be unified as the process of exactly , and sometimes approximately , addressing a graph signal denoising problem with Laplacian regularization ( Shuman et al. , 2013 ) . This connection suggests that these aggregation operations share a unified goal : to ensure feature smoothness of connected nodes . With this understanding , we propose a general GNN framework , UGNN , which not only provides a straightforward , unified view for many existing aggregation operations , but also suggests various promising directions to build new aggregation operations suitable for distinct applications . To demonstrate its potential , we build an instance of UGNN called ADA-UGNN , which is suited for handling varying smoothness properties across nodes , and conduct experiments to show its effectiveness . 2 REPRESENTATIVE GRAPH NEURAL NETWORKS . In this section , we introduce notations for graphs and briefly summarize several representative GNN models . A graph can be denoted as G = { V , E } , where V and E are its corresponding node and edge sets . The connections in G can be represented as an adjacency matrix A ∈ RN×N , with N the number of nodes in the graph . The Laplacian matrix of the graph G is denoted as L. It is defined as L = D −A , where D is a diagonal degree matrix corresponding to A . There are also normalized versions of the Laplacian matrix such as L = I −D− 12AD− 12 or L = I −D−1A . In this work , we sometimes adopt different Laplacians to establish connections between different GNNs and the graph denoising problem , clarifying in the text . In this section , we generally use Xin ∈ RN×din and Xout ∈ RN×dout to denote input and output features of GNN layers . Next , we describe a few representative GNN models . 2.1 GRAPH CONVOLUTIONAL NETWORKS ( GCN ) . Following Eq . ( 1 ) , a single layer in GCN ( Kipf & Welling , 2016 ) can be written as follows : Feature Transformation : X′in = XinW ; Feature Aggregation : Xout = ÃX ′ in , ( 2 ) where W ∈ Rdin×dout is a feature transformation matrix , and Ã is a normalized adjacency matrix which includes a self-loop , defined as follows : Ã = D̂− 1 2 ÂD̂− 1 2 , with Â = A+ I and D = diag ( ∑ j Â1 , j , . . . , ∑ j ÂN , j ) . ( 3 ) In practice , multiple GCN layers can be stacked , where each layer takes the output of its previous layer as input . Non-linear activation functions are included between consecutive layers . 2.2 GRAPH ATTENTION NETWORKS ( GAT ) . Graph Attention Networks ( GAT ) adopts the same feature transformation operation as GCN in Eq . ( 2 ) . The feature aggregation operation ( written node-wise ) for a node i is as : Xout [ i , : ] = ∑ j∈Ñ ( i ) αijX ′ in [ j , : ] , with αij = exp ( eij ) ∑ k∈Ñ ( i ) exp ( eik ) . ( 4 ) where Ñ ( i ) = N ( i ) ∪ { i } denotes the neighbors ( self-inclusive ) of node i , and Xout [ i , : ] is the i-th row of the matrix Xout , i.e . the output node features of node i . In this aggregation operation , αij is a learnable attention score to differentiate the importance of distinct nodes in the neighborhood . Specifically , αij is a normalized form of eij , which is modeled as : eij = LeakyReLU ( [ X′in [ i , : ] ‖X′in [ j , : ] ] a ) ( 5 ) where [ ·‖· ] denotes the concatenation operation and a ∈ R2d is a learnable vector . Similar to GCN , a GAT model usually consists of multiple stacked GAT layers . 2.3 PERSONALIZED PROPAGATION OF NEURAL PREDICTIONS ( PPNP ) . Personalized Propagation of Neural Predictions ( PPNP ) ( Klicpera et al. , 2018 ) introduces an aggregation operation based on Personalized PageRank ( PPR ) . Specifically , the PPR matrix is defined as α ( I − ( 1 − α ) Ã ) −1 , where α ∈ ( 0 , 1 ) is a hyper-parameter . The ij-th element of the PPR matrix specifies the influence of node i on node j . The feature transformation operation is modeled as Multi-layer Perception ( MLP ) . The PPNP model can be written in the form of Eq . ( 1 ) as follows : Feature Transformation : X′in = MLP ( Xin ) ; Feature Aggregation : Xout = α ( I− ( 1− α ) Ã ) −1X′in . ( 6 ) Unlike GCN and GAT , PPNP only consists of a single feature aggregation layer , but with a poten- tially deep feature transformation . Since the matrix inverse in Eq . ( 6 ) is costly , Klicpera et al . ( 2018 ) also introduces a practical , approximated version of PPNP , called APPNP , where the aggregation operation is performed in an iterative way as : X ( k ) out = ( 1− α ) ÃX ( k−1 ) out + αX ′ in k = 1 , . . .K , ( 7 ) where X ( 0 ) out = X ′ in and X ( K ) out is the output of the feature aggregation operation . As proved in Klicpera et al . ( 2018 ) , X ( K ) out converges to the solution obtained by PPNP , i.e. , Xout in Eq . ( 6 ) . 3 GNNS AS GRAPH SIGNAL DENOISING . In this section , we aim to establish the connections between the introduced GNN models and a graph signal denoising problem with Laplacian regularization . We first introduce the problem . Problem 1 ( Graph Signal Denoising with Laplacian Regularization ) . Suppose that we are given a noisy signal X ∈ RN×d on a graph G. The goal of the problem is to recover a clean signal F ∈ RN×d , assumed to be smooth over G , by solving the following optimization problem : argmin F L = ‖F−X‖2F + c · tr ( F > LF ) , ( 8 ) Note that the first term guides F to be close to X , while the second term tr ( F > LF ) is the Laplacian regularization that guides the smoothness of F over the graph . c > 0 is a balancing constant . Assuming we adopt the unnormalized version of Laplacian matrix with L = D−A ( the adjacency matrix A is assumed to be binary ) , the second term in Eq . ( 8 ) can be written in an edge-centric way or a node-centric way as : edge-centric : c ∑ ( i , j ) ∈E ‖F [ i , : ] − F [ j , : ] ‖22 ; node-centric : 1 2 c ∑ i∈V ∑ j∈Ñ ( i ) ‖F [ i , : ] − F [ j , : ] ‖22 . ( 9 ) Clearly , from the edge-centric view , the regularization term measures the global smoothness of F , which is small when connected nodes share similar features . On the other hand , we can view the term∑ j∈Ñ ( i ) ‖F [ i , : ] − F [ j , : ] ‖ 2 2 as a local smoothness measure for node i as it measures the difference between node i and all its neighbors . The regularization term can then be regarded as a summation of local smoothness over all nodes . Note that the adjacency matrix A is assumed to be binary when deriving Eq . ( 9 ) . Similar formulations can also be derived to other types of Laplacian matrices . In the following subsections , we demonstrate the connections between aggregation operations in various GNN models and the graph signal denoising problem . 3.1 CONNECTION TO PPNP AND APPNP . In this subsection , we establish the connection between the graph signal denoising problem ( 8 ) and the aggregation propagations in PPNP and APPNP in Theorem 1 and Theorem 2 , respectively . Theorem 1 . When we adopt the normalized Laplacian matrix L = I−Ã , with Ã defined in Eq . ( 3 ) , the feature aggregation operation in PPNP ( Eq . ( 6 ) ) can be regarded as exactly solving the graph signal denoising problem ( 8 ) with X′in as the input noisy signal and c = 1 α − 1 . Proof . Note that the objective in Eq . ( 8 ) is convex . Hence , its closed-form solution F∗ to exactly solve the graph signal denosing problem can be obtained by setting its derivative to 0 as : ∂L ∂F = 2 ( F−X ) + 2cLF = 0⇒ F∗ = ( I+ cL ) −1X ( 10 ) Given L = I− Ã , F∗ can be reformulated as : F∗ = ( I+ cL ) −1 X = ( I+ c ( I− Ã ) ) −1 X = 1 1 + c ( I− c 1 + c Ã ) −1 X ( 11 ) The feature aggregation operation in Eq . ( 6 ) is equivalent to the closed-form solution in Eq . ( 11 ) when we set α = 1/ ( 1 + c ) and X = X′in . This completes the proof . Theorem 2 . When we adopt the normalized Laplacian matrix L = I − Ã , the feature aggregation operation in APPNP ( Eq . ( 7 ) ) approximately solves the graph signal denoising problem ( 8 ) by iterative gradient descent with X′in as the input noisy signal , c = 1 α − 1 and stepsize b = 1 2+2c . Proof . To solve the graph signal denoising problem ( 8 ) , we take iterative gradient method with the stepsize b . Specifically , the k-th step gradient descent on problem ( 8 ) is as follows : F ( k ) ← F ( k−1 ) − b · ∂L ∂F ( F = F ( k−1 ) ) = ( 1− 2b− 2bc ) F ( k−1 ) + 2bX+ 2bcÃF ( k−1 ) ( 12 ) where F ( 0 ) = X . When we set the stepsize b as 12+2c , we have the following iterative steps : F ( k ) ← 1 1 + c X+ c 1 + c ÃF ( k−1 ) , k = 1 , . . .K , ( 13 ) which is equivalent to the iterative aggregation operation of the APPNP model in Eq . ( 7 ) with X = X′in and α = 1 1+c . This completes the proof . These two connections provide a new explanation on the hyper-parameter α in PPNP and APPNP from the graph signal denoising perspective . Specifically , a smaller α indicates a larger c , which means the obtained Xout is enforced to be smoother over the graph .

Summary: 1). The novelty and contribution are very limited. In literature, many papers have discussed the connection between different GNNs, typically, including aggregators and Updaters, such as discussed in “Deep Learning on Graphs: A Survey”. The submission only provides a kind of connection between GCN, GAT, PPPN and APPNP in the perspective of denoising. Compared with that, the survey paper actually connects many different GNNs.