Power up! Robust Graph Convolutional Network based on Graph Powering

1 INTRODUCTION . Graph convolutional networks ( GCNs ) are powerful extensions of convolutional neural networks ( CNN ) to graph-structured data . Recently , GCNs and variants have been applied to a wide range of domains , achieving state-of-the-art performances in social networks ( Kipf & Welling , 2017 ) , traffic prediction ( Rahimi et al. , 2018 ) , recommendation systems ( Ying et al. , 2018 ) , applied chemistry and biology ( Kearnes et al. , 2016 ; Fout et al. , 2017 ) , and natural language processing ( Atwood & Towsley , 2016 ; Hamilton et al. , 2017 ; Bastings et al. , 2017 ; Marcheggiani & Titov , 2017 ) , just to name a few ( Zhou et al. , 2018 ; Wu et al. , 2019 ) . GCNs belong to a family of spectral methods that deal with spectral representations of graphs ( Zhou et al. , 2018 ; Wu et al. , 2019 ) . A fundamental ingredient of GCNs is the graph convolution operation defined by the graph Laplacian in the Fourier domain : gθ ? x : = ĝθ ( L ) x , ( 1 ) where x ∈ Rn is the graph signal on the set of vertices V and ĝθ is a spectral function applied to the graph Laplacian L : = D −A ( whereD andA are the degree matrix and the adjacency matrix , respectively ) . Because this operation is computational intensive for large graphs and non-spatially localized ( Bruna et al. , 2014 ) , early attempts relied on a parameterization with smooth coefficients ( Henaff et al. , 2015 ) or a truncated expansion in terms of of Chebyshev polynomials ( Hammond et al. , 2011 ) . By further restricting the Chebyshev polynomial order by 2 , the approach in ( Kipf & Welling , 2017 ) referred henceforth as the vanilla GCN pushed the state-of-the-art performance of semi-supervised learning . The network has the following layer-wise update rule : H ( l+1 ) : = ψ ( AH ( l ) W ( l ) ) , ( 2 ) where H ( l ) is the l-th layer hidden state ( with H ( 1 ) : = X as nodal features ) , W ( l ) is the l-th layer weight matrix , ψ is the usual point-wise activation function , and A is the convolution operator chosen to be the degree weighted Laplacian with some slight modifications ( Kipf & Welling , 2017 ) . Subsequent GCN variants have different architectures , but they all share the use of the Laplacian matrix as the convolution operator ( Zhou et al. , 2018 ; Wu et al. , 2019 ) . 1.1 WHY NOT GRAPH LAPLACIAN ? . Undoubtedly , the Laplacian operator ( and its variants , e.g. , normalized/powered Laplacian ) plays a central role in spectral theory , and is a natural choice for a variety of spectral algorithms such as principal component analysis , clustering and linear embeddings ( Chung & Graham , 1997 ; Belkin & Niyogi , 2002 ) . So what can be problematic ? From a spatial perspective , GCNs with d layers can not acquire nodal information beyond its ddistance neighbors ; hence , it severely limits its scope of data fusion . Recent works ( Lee et al. , 2018 ; Abu-El-Haija et al. , 2018 ; 2019 ; Wu et al. , 2019 ) alleviated this issue by directly powering the graph Laplacian . From a spectral perspective , one could demand better spectral properties , given that GCN is fundamentally a particular ( yet effective ) approximation of the spectral convolution ( 1 ) . A key desirable property for generic spectral methods is known as “ spectral separation , ” namely the spectrum should comprise a few dominant eigenvalues whose associated eigenvectors reveal the sought structure in the graph . A well-known prototype is the Ramanujan property , for which the second leading eigenvalue of a r-regular graph is no larger than 2 √ r − 1 , which is also enjoyed asymptotically by random r-regular graphs ( Friedman , 2004 ) and Erdős-Rényi graphs that are not too sparse ( Feige & Ofek , 2005 ) . In a more realistic scenario , consider the stochastic block model ( SBM ) , which attempts to capture the essence of many networks , including social graphs , citation graphs , and even brain networks ( Holland et al. , 1983 ) . Definition 1 ( Simplified stochastic block model ) . The graph G with n nodes is drawn under SBM ( n , k , aintra , ainter ) if the nodes are evenly and randomly partitioned into k communities , and nodes i and j are connected with probability aintra/n ∈ [ 0 , 1 ] if they belong to the same community , and ainter/n ∈ [ 0 , 1 ] if they are from different communities . It turns out that for community detection , the top k leading eigenvectors of the adjacency matrix A play an important role . In particular , for the case of 2 communities , spectral bisection algorithms simply take the second eigenvector to reveal the community structure . This can be also seen from the expected adjacency matrix E [ A ] under SBM ( n , 2 , aintra , ainter ) , which is a rank-2 matrix with the top eigenvalue 12 ( aintra + ainter ) and eigenvector 1 , and the second eigenvalue 1 2 ( aintra − ainter ) and eigenvector σ such that σi = 1 if i is in community 1 and σi = −1 otherwise . More generally , the second eigenvalue is of particular theoretical interests because it controls at the first order how fast heat diffuses through graph , as depicted by the discrete Cheeger inequality ( Lee et al. , 2014 ) . While one would expect taking the second eigenvector of the adjacency matrix suffices , it often fails in practice ( even when it is theoretically possible to recover the clusters given the signal-to-noise ratio ) . This is especially true for sparse networks , whose average nodal degrees is a constant that does not grow with the network size . This is because the spectrum of the Laplacian or adjacency matrix is blurred by “ outlier ” eigenvalues in the sparse regime , which is often caused by high degree nodes ( Kaufmann et al. , 2016 ) . Unsurprisingly , powering the Laplacian would be of no avail , because it does not change the eigenvectors or the ordering of eigenvalues . In fact , those outliers can become more salient after powering , thereby weakening the useful spectral signal even further . Besides , pruning the largest degree nodes in the adjacency matrix or normalizing the Laplacian can not solve the issue . To date , the best results for pruning does not apply down to the theoretical recovery threshold ( Coja-Oghlan , 2010 ; Mossel et al. , 2012 ; Le et al. , 2015 ) ; either outliers would persist or one could prune too much that the graph is destroyed . As for normalized Laplacian , it may overcorrect the large degree nodes , such that the leading eigenvectors would catch the “ tails ” of the graph , i.e. , components weakly connected to the main graph . See Figure A.3 in the appendix for an experimental illustration . In summary , graph Laplacian may not be the ideal choice due to its limited spatial scope of information fusion , and its undesirable artefacts in the spectral domain .. 1.2 IF NOT LAPLACIAN , THEN WHAT ? . In searching for alternatives , potential choices are many , so it is necessary to clarify the goals . In view of the aforementioned pitfalls of the graph Laplacian , one would naturally ask the question : Can we find an operator that has wider spatial scope , more robust spectral properties , and is meanwhile interpretable and can increase the expressive power of GCNs ? From a perspective of graph data analytics , this question gauges how information is propagated and fused on a graph , and how we should interpret “ adjacency ” in a much broader sense . An image can be viewed as a regular grid , yet the operation of a CNN filter goes beyond the nearest pixel to a local neighborhood to extract useful features . How to draw an analogy to graphs ? From a perspective of robust learning , this question sheds light on the basic observation that realworld graphs are often noisy and even adversarial . The nice spectral properties of a graph topology can be lost with the presence or absence of edges . What are some principled ways to robustify the convolution operator and graph embeddings ? In this paper , we propose a graph learning paradigm that aims at achieving this goal , as illustrated in Figure 1 . The key idea is to generate a sequence of graphs from the given graph that capture a wide range of spectral and spatial behaviors . We propose a new operator based on this derived sequence . Definition 2 ( Variable power operator ) . Consider an unweighted and undirected graph G. Let A [ k ] denote the k-distance adjacency matrix , i.e. , [ A [ k ] ] ij = 1 if and only if the shortest distance ( in the original graph ) between nodes i and j is k. The variable power operator of order r is defined as : A ( r ) θ = r∑ k=0 θkA [ k ] , ( 3 ) where θ : = ( θ0 , . . . , θr ) is a set of parameters . Clearly , A ( r ) θ is a natural extension of the classical adjacency matrix ( i.e. , r = 1 and θ0 = θ1 = 1 ) . With power order r > 1 , one can increase the spatial scope of information fusion on the graph when applying the convolution operation . The parameters θk also has a natural explanation—the magnitude and the sign of θk can be viewed as “ global influence strength ” and “ global influence propensity ” at distance k , respectively , which also determines the participation factor of each graph in the sequence in the aggregated operator . Furthermore , we provide some theoretical justification of the proposed operator by establishing the following asymptotic property of spectral separation under the important SBM setting , which is , nevertheless , not enjoyed by the classical Laplacian operator or its normalized or powered versions . ( All proofs are given in the appendix . ) Theorem 3 ( Asymptotic spectral separation of variable power operator ) . Consider a graph G drawn from SBM ( n , 2 , aintra , ainter ) . Assume that the signal-to-noise ratio ξ22/ξ1 > 1 , where ξ1 = 1 2 ( aintra + ainter ) and ξ2 = 12 ( aintra − ainter ) ( c.f. , ( Decelle et al. , 2011 ) ) . Suppose r is on the order of c log ( n ) for a constant c , such that c log ( ξ1 ) < 1/4 . Given nonvanishing θk for k > r/2 , the variable power operator A ( r ) θ has the following spectral properties : ( i ) the leading eigenvalue is on the order of Θ ( ‖θ‖1ξr1 ) , the second leading eigenvalue is on the order of Θ ( ‖θ‖1ξr2 ) , and the rest are bounded by ‖θ‖1n ξr/21 O ( log ( n ) ) for any fixed > 0 ; and ( ii ) the two leading eigenvectors are sufficient to recover the two communities asymptotically ( i.e. , as n goes to infinity ) . Intuitively , the above theoretical result suggests that the variable power operator is able to “ magnify ” benign signals from the latent graph structure while “ suppressing ” noises due to random artefacts . This is expected to improve spectral methods in general , especially when the benign signals tend to be overwhelmed by noises . For the rest of the paper , we will apply this insight to propose a robust graph learning paradigm in Section 2 , as well as a new GCN architecture in Section 3 . We also provide empirical evidence of the gain from this theory in Section 4 and conclude in Section 5 .

Summary: This paper proposes a graph convolutional operator based on graph powering and applies it to GCN architecture to improve the performance and robustness. This work is mainly motivated by the paper (Graph powering and spectral robustness, Abbe et al., 2018). The authors introduce the graph powering to graph convolution neural network domain to replace the original Laplacian operator. They further propose a graph sparsification/pruning strategy on the powered adjacency matrices in order to reduce the complexity and increase the robustness against adversarial attacks. They also provide theoretical analysis to prove that the proposed powering operator and subsequent methods have some spectral properties and theoretical feasibility. However, some conclusions are limited to the ideal situations or seem subjective. Extensive experiments are conducted to show better or comparable performance in both benign and adversarial situations.