SoGCN: Second-Order Graph Convolutional Networks

1 INTRODUCTION . Deep localized convolutional filters have achieved great success in the field of deep learning . In image recognition , the effectiveness of 3 × 3 kernels as the basic building block in Convolutional Neural Networks ( CNNs ) is shown both experimentally and theoretically ( Zhou , 2020 ) . We are inspired to search for the maximally localized Graph Convolution ( GC ) kernel with full expressiveness power for Graph Convolutional Networks ( GCNs ) . Most existing GCN methods utilize localized GCs based on one-hop aggregation scheme as the basic building block . Extensive works have shown performance limitations of such design due to over-smoothing ( Li et al. , 2018 ; Oono & Suzuki , 2019 ; Cai & Wang , 2020 ) . In vanilla GCNs ( Kipf & Welling , 2017 ) the root cause of its deficiency is the lumping of the graph node self-connection with pairwise neighboring connections . Recent works of Xu et al . ( 2019 ) ; Dehmamy et al . ( 2019 ) ; Ming Chen et al . ( 2020 ) disentangle the effect of self-connection by adding an identity mapping ( so-called first-order GC ) . However , its lack of expressive power in filter representation remains ( Abu-El-Haija et al. , 2019 ) . The work of ( Ming Chen et al. , 2020 ) conjectured that the ability to express a polynomial filter with arbitrary coefficients is essential for preventing over-smoothing . A longer propagation distance in the graph facilitates GCNs to retain its expressive power , as pointed out by ( Liao et al. , 2019 ; Luan et al. , 2019 ; Abu-El-Haija et al. , 2019 ) . The minimum propagation distance needed to construct our building block of GCN remains the open question . We show that the minimum propagation distance is two : a two-hop graph kernel with the second-order polynomials in adjacency matrices is sufficient . We call our graph kernel Second-Order GC ( SoGC ) . We introduce a Layer Spanning Space ( LSS ) framework to quantify the expressive power of multilayer GCs for modeling a polynomial filter with arbitrary coefficients . By relating low-pass filtering on the graph spectrum ( Hoang & Maehara , 2019 ) with over-smoothing , one can see the lack of filter representation power ( Ming Chen et al. , 2020 ) can lead to the performance limitation of GCN . Using the LSS framework , we show that SoGCs can approximate any linear GCNs in channel-wise filtering . Furthermore , higher-order GCs do not contribute more expressiveness , and vanilla GCN or first-order GCs can not represent all polynomial filters in general . In this sense , SoGC is the maximally localized graph kernel with the full representation power . To validate our theory , we build Second-Order Graph Convolutional Networks ( SoGCN ) using SoGC kernels . Our SoGCN using simple graph topological features consistently achieves state- of-the-art performance on the GNN benchmark datasets ( Dwivedi et al. , 2020 ) , including citation networks , super-pixel classification , and molecule regression . To our best knowledge , this work is the first study that identifies the importance of the two-hop neighborhood in the context of GCNs ’ ability to express a polynomial filter with arbitrary coefficients . Our model is a special but non-trivial case of Defferrard et al . ( 2016 ) . Kipf & Welling ( 2017 ) conducted an ablation study with GC kernels of different orders but missed the effectiveness of the second-order relationships . The work of Abu-El-Haija et al . ( 2019 ) talked about muti-hop graph kernels ; however , they did not identify the critical importance of the second-order form . In contrast , we clarify the prominence of SoGCs in theories and experiments . Our research on graph convolution using pure topologically relationship is orthogonal to those uses geometric relations ( Monti et al. , 2017 ; Fey et al. , 2018 ; Pei et al. , 2020 ) , or those with expressive edge features ( Li et al. , 2016 ; Gilmer et al. , 2017 ; Corso et al. , 2020 ) , and hyper-edges ( Morris et al. , 2019 ; Maron et al. , 2018 ; 2019 ) . It is also independent with graph sampling procedures ( Rong et al. , 2019 ; Hamilton et al. , 2017 ; Li et al. , 2019 ) . 2 PRELIMINARIES . We begin by reformulating spectral GCNs and introducing our notation . We are interested in a finite graph set G = { G1 , · · · , G|G| } . Assume each graph G ∈ G is simple and undirected , associated with a finite vertex set V ( G ) , an edge set E ( G ) = { ( u , v ) : ∀u ↔ v } , and a symmetric normalized adjacency matrix A ( G ) ( Chung & Graham , 1997 ; Shi & Malik , 2000 ) . Without loss of generality and for simplicity , |V ( G ) | = N for every G ∈ G. Single-channel features x ∈ RN supported in graph G ∈ G is a vectorization of function V ( G ) → R. Graph Convolutions ( GCs ) is known as Linear Shift-Invariant ( LSI ) operators to adjacency matrices ( Sandryhaila & Moura , 2013 ) . By this definition , GCs can extract features regardless of where local structures fall . Given parameter space Ω ⊆ R , we write a single-channel GC ( Sandryhaila & Moura , 2013 ; Defferrard et al. , 2016 ) as a mapping fθ : G × RN → RN such that 1 : fθ ( G , x ) = K∑ k=0 θkA ( G ) kx , ( 1 ) where θ = [ θ0 · · · θK ] T ∈ ΩK+1 parameterizes the GC . K reflects the localization of fθ : a linear combination of features aggregated by A ( G ) k. Moreover , we reformulate two popular models , vanilla GC ( Figure 1a ) and first-order GC ( Figure 1b ) , as below : f0 ( G , x ) = θ ( A ( G ) + I ) x , f1 ( G , x ) = ( θ1A ( G ) + θ0I ) x . ( 2 ) The general spectral GCNs stack L layers of GCs ( Equation 1 ) with nonlinear activations . Let f ( l ) be GC layers with parameters θ ( l ) ∈ ΩK+1 , l ∈ [ L ] , the single-channel GCNs can be written as : F ( G , x ) = g ◦ f ( L ) ◦ σ ◦ f ( L−1 ) ◦ · · · ◦ σ ◦ f ( 1 ) ( G , x ) , ( 3 ) 1We can replace the Laplacian matrix L in Defferrard et al . ( 2016 ) with the normalized adjacency matrix A since L = I −A . where σ is an element-wise activation function , the superscripts denote the corresponding layer number , g is a task-specified readout function ( e.g. , softmax ) , the inputs are graph G ∈ G and signals x ∈ RN . The compositionality principle of deep learning suggests L being large , while K being small and localized ( LeCun et al. , 2015 ) . 3 OVERVIEW : SECOND-ORDER GRAPH CONVOLUTION . We are interested in the overall graph convolution network ’ s representation power of expressing a polynomial filter ( Equation 1 ) with arbitrary coefficients . A multi-layer GCN approximate a Korder polynomial filter ∑K k=0 θkA ( G ) k , θk ∈ Ω , k = 0 , · · ·K , by stacking basic building blocks of graph convolution ( GC ) layers ( Wu et al. , 2019 ; Ming Chen et al. , 2020 ) . We formally define the second-order GC ( SoGC ) using the second-order polynomials of adjacency matrices : f2 ( G , x ) = ( θ2A ( G ) 2 + θ1A ( G ) + θ0I ) x , ( 4 ) where θi ∈ R , i = 0 , 1 , 2 are trainable paremeters in the context of machine learning . Its vertexdomain interpretation is illustrated in Figure 1c . At first glance , it seems that we could stack two one-hop graph convolution ( GC ) kernels to approximate a SoGC . However , as shown in Section 4.3 , that is not the case . The critical insight is that graph filter approximation can be viewed as a polynomial factorization problem . It is known that any univariate polynomial can be factorized into sub-polynomials of degree two . Based on this fact , we show by stacking enough SoGCs ( and varying their parameters ) can achieve decomposition of any polynomial filters . In contrast , first-order GCs are not universal approximators ; two stacked one-hop GCs can not model every two-hop GC . Polynomial filter completeness of SoGC leads to better performance of GCNs . As shown in Figure 2 , networks built with SoGC can overcome over-smoothing and extract features on high-frequency bands . In the next section , we demonstrate our formal arguments on polynomial approximation . 4 REPRESENTATION POWER ANALYSIS . 4.1 LAYER SPANNING SPACE FRAMEWORK . To illustrate the representation power of GC layers , we establish a Layer Spanning Space ( LSS ) framework to study the graph filter space spanned by stacking multiple graph kernels . First , we present our mathematical devices in Definition 1 , 2 with Lemma 1 as below . Definition 1 . Suppose the parameter space Ω = R. The Linear Shift-Invariant ( LSI ) graph filter space of degree K > 0 with respect to a finite graph set G is defined as A = { fθ : G × RN → RN , ∀θ ∈ RK+1 } , where fθ follows the definition in Equation 1 . Definition 2 . Let spectrum set S ( G ) = { λ : λ ∈ S ( A ( G ) ) , ∀G ∈ G } , where S ( A ) denotes the eigenvalues of A . Define spectrum capacity Γ = |S ( G ) | . In particular , Γ = ( N − 1 ) |G| if every graph adjacency matrix has no common eigenvalues other than 1 . Lemma 1 . A of degree K > 0 has dimension min { K + 1 , Γ } as a vector space . Proof of Lemma 1 follows from Theorem 3 of Sandryhaila & Moura ( 2013 ) . The complete version can be found in Appendix C. Here , we induce a finite-dimension filter space by Lemma 1 . For simplicity , we will model the linear composition of filters to analyze its representation power . The nonlinear activation effects are beyond the scope of this work . Following Definition 1 , let A be the full filter space of degree Γ− 1 and B be the low-level filter space as a set of polynomials in adjacency matrices ( Equation 1 ) . Denote the GC at l-th layer by f ( l ) ∈ B , then we yield the LSS of stacking L layers as below : BL = { f : f ( G , x ) = f ( L ) ◦ · · · ◦ f ( 1 ) ( G , x ) = L∏ l=1 p ( l ) ( A ( G ) ) x } , ( 5 ) where p ( l ) ( x ) varies in a certain class of polynomials . We can assess the expressive capability of GC layers by comparing the LSS with A. Kernels in B have full representation power if A ⊆ BL . We are interested in BK , which denotes all localized filters of degree at most K. The LSS of BK is modeled as : BLK = { f : f ( G , x ) = L∏ l=1 K∑ k=0 θ ( l ) k A ( G ) kx , θ ( l ) k ∈ R } , ( 6 ) where the number of layers is bounded by L ≤ d ( Γ− 1 ) /Ke , according to Lemma 1 .

Summary: This is yet another paper on graph convolutional networks (GCNs). The investigated SoGCN is a second-order GCN, thus a special case of high-order GCNs (namely with multi-hop graph kernels), which have has been proposed earlier by many researchers, such as by Defferrard et al. (2016), by Kipf & Welling (2017) and by Abu-El-Haija et al. (2019).