Connecting Graph Convolution and Graph PCA

1 INTRODUCTION . Graph neural networks ( GNNs ) are neural networks designed for the graph domain . Since the breakthrough of GCN ( Kipf & Welling , 2017 ) , which notably improved performance on the semisupervised node classification problem , many GNN variants have been proposed ; including GAT ( Veličković et al. , 2018 ) , GraphSAGE ( Hamilton et al. , 2017 ) , DGI ( Veličković et al. , 2019 ) , GIN ( Xu et al. , 2019 ) , PPNP and APPNP ( Klicpera et al. , 2019 ) , to name a few . Despite the empirical successes of GNNs in both node-level and graph-level tasks , they remain not well understood due to limited systematic and theoretical analysis of GNNs . For example , researchers have found that GNNs , unlike their non-graph counterparts , suffer from performance degradation with increasing depth , their expressive power decaying exponentially in number of layers ( Oono & Suzuki , 2020 ) . Such behavior is only partially explained by the oversmoothing phenomenon ( Li et al. , 2018 ; Zhao & Akoglu , 2020 ) . Another surprising observation shows that a Simplified Graph Convolution model , named SGC ( Wu et al. , 2019 ) , can achieve similar performance to various more complex GNNs on a variety of node classification tasks . Moreover , a simple baseline that does not utilize the graph structure altogether performs similar to state-of-the-art GNNs on graph classification tasks ( Errica et al. , 2020 ) . These observations call attention to studies for a better understanding of GNNs ( NT & Maehara , 2019 ; Morris et al. , 2019 ; Xu et al. , 2019 ; Oono & Suzuki , 2020 ; Loukas , 2020 ; Srinivasan & Ribeiro , 2020 ) . ( See Sec . 2 for more on understanding GNNs . ) Toward a systematic analysis and better understanding of GNNs , we establish a connection between the graph convolution operator of GCN ( and PPNP ) and Graph-regularized PCA ( GPCA ) ( Zhang & Zhao , 2012 ) , and show the similarity between GCN and stacking GPCA . This connection provides a deeper understanding of GCN ’ s power and limitation . Empirically , we also find that GPCA performance matches that of many GNN baselines on benchmark semi-supervised node classification tasks . We argue that the simple GPCA should be a strong baseline in future . What is more , the unsupervised stacking GPCA can be viewed as “ unsupervised GCN ” and provides a straightforward , yet systematic way to initialize GCN training . We summarize our contributions as follows : • Connection between Graph Convolution and GPCA : We establish the connection between the graph convolution operator of GCN ( also PPNP ) and the closed-form solution of graph-regularized PCA ( GPCA ) formulation . We demonstrate that a simple graph-regularized PCA paired with 1- or 2-layer MLP can achieve similar or even better results than state-of-the-art GNN baselines over several benchmark datasets . We further extend GPCA to ( semi- ) supervised setting which can generate embeddings using information of labels , which yields better performance on 3 out of 5 datasets . The outstanding performance of simple GPCA supports that the prowess of GCN on node classification task comes from graph based regularization . This motivates the study and design of other graph regularization techniques in the future . • GPCANET : New Stacking GPCA model : Capitalizing on the connection between GPCA and graph convolution , we design a new GNN model called GPCANET shaped by ( 1 ) stacking multiple GPCA layers and nonlinear transformations , and ( 2 ) fine-tuning end-to-end via supervised training . GPCANET is a generalized GCN model with adjustable hyperparameters that control the strength of graph regularization of each layer . We show that with stronger regularization , we can train GPCANET with fewer ( 1–3 ) layers and achieve comparable performance to much deeper GCNs . • First initialization strategy for GNNs : Capitalizing on the connection between GCN and GPCANET , we design a new strategy to initialize GCN training based on stacking GPCA , outperforming the popular Xaiver initialization ( Glorot & Bengio , 2010 ) . We show that the GPCANET-initialization is extremely effective for training deeper GCNs , that significantly improves the convergence speed , performance , and robustness . Notably , GPCANET-initialization is general-purpose and also applies to other GNNs . To our knowledge , it is the first initialization method specifically designed for GNNs . We open-source code at http : //bit.ly/GPCANet . All datasets are public-domain . 2 RELATED WORK . Understanding GNNs . Our work concerns learning on a single graph , hence we limit discussion of related work to node-level GNNs . GCN ’ s graph convolution is originally motivated from the approximation of graph filters in graph signal processing ( Kipf & Welling , 2017 ) . NT & Maehara ( 2019 ) show that graph convolution only performs low-pass filtering on original feature vectors , and also state a connection between graph filtering and Laplacian regularized least squares . Motivated by the oversmoothing phenomenon of graph convolution , Oono & Suzuki ( 2020 ) theoretically prove that GCN can only preserve information of node degrees and connected components when the number of layers goes to infinity , under some conditions of GCN weights . Recently several papers revisited the connection of graph convolution to graph-regularized optimization problem ( Li et al. , 2019 ; Ma et al. , 2020 ; Pan et al. , 2021 ; Zhao & Akoglu , 2020 ; Zhu et al. , 2021 ) , which is originally discussed in graph signal processing ( Shuman et al. , 2013 ) . More specifically , both Ma et al . ( 2020 ) and Zhu et al . ( 2021 ) relate graph-regularization optimization to several GNNs such as GCN ( Kipf & Welling , 2017 ) , APPNP ( Klicpera et al. , 2019 ) , and GAT ( Veličković et al. , 2018 ) . However , all previous work study these connections while ignoring the learnable parameters , which are essential for high-performance deep learning . Our work differs from these by establishing a stronger and closer connection to graph-regularized PCA that also takes learnable parameters into account . Graph-regularized PCA . PCA and its variants are standard linear dimensionality reduction approaches . Several work extend PCA to graph-structured data , such as Graph-Laplacian PCA ( Jiang et al. , 2013 ) and Manifold-regularized Matrix Factorization ( Zhang & Zhao , 2012 ) . For other variants , see Shahid et al . ( 2016 ) . Stacking Models and Deep Learning . The connection between CNN and stacking PCA has been explored in PCANet ( Chan et al. , 2015 ) , which demonstrated that the ( unsupervised ) simple stacking PCA works as well as supervised CNN over a large variety of vision tasks . The original PCANet is shallow and does not have nonlinear transformations , while PCANet+ ( Low et al. , 2017 ) overcomes these limitations and pushes the architecture much deeper . The idea of layerwise stacking for feature extraction is not new and was empirically observed to exhibit better representation ability in terms of classification . For a comprehensive review , we refer to Bengio et al . ( 2013 ) . Initialization . Traditionally , neural networks ( NNs ) were initialized with random weights generated from Gaussian distribution with zero mean and a small standard deviation ( Krizhevsky et al. , 2012 ) . As training deeper NNs became extremely difficult due to vanishing gradient and activation functions , Glorot & Bengio ( 2010 ) provided a specific weight initialization formula , named Xavier initialization , based on variance analysis without considering activation function . Xavier initialization is widely used for any type of NN even today , and it is the main initialization strategy used for GNNs . Later , He et al . ( 2015 ) adapted Xavier initialization to ReLU activation by considering a multiplier . Taking another direction , Saxe et al . ( 2013 ) analyzed the dynamics of training deep NNs and proposed random orthonormal initialization . Mishkin & Matas ( 2015 ) further improved orthonormal initialization for batch normalization ( Ioffe & Szegedy , 2015 ) . Different from these data-independent approaches , others ( Krähenbühl et al. , 2016 ; Seuret et al. , 2017 ; Wagner et al. , 2013 ) have employed datadependent techniques , like PCA , to initialize deep NNs . Although initialization has been widely studied for general NNs , no specific initialization has been proposed for GNNs . In this work , we propose a data-driven initialization technique ( based on GPCA ) , specific to GNNs for the first time . 3 GRAPH CONVOLUTION AND GPCA . 3.1 GRAPH CONVOLUTION . Consider a node-attributed input graph G = ( V , E , X ) with |V | = n nodes and |E| = m edges , where X ∈ Rn×d denotes the node feature matrix with d features . Broadly , graph convolution operation convolves the features ( or representations ) over the graph structure . GCN . Similar to other neural networks stacked with repeated layers , GCN contains multiple graph convolution layers each of which is followed by a nonlinear activation . Let H ( l ) be the l-th hidden layer representation , then , each GCN layer performs H ( l+1 ) = σ ( ÃsymH ( l ) W ( l ) ) ( 1 ) where Ãsym = D̃− 1 2 ( A+ I ) D̃− 1 2 denotes the n×n symmetrically normalized adjacency matrix with self-loops , D̃ is the diagonal degree matrix where D̃ii = 1 + ∑n j=1 Aij , W ( l ) depicts the l-th layer parameters ( to be learned ) , and σ is the nonlinear activation function . Formally , graph convolution is parameterized with W and maps an input X to a new representation Z as Z = ÃsymXW . ( 2 ) PPNP . For PPNP ( Klicpera et al. , 2019 ) , the features are first transformed by an MLP before convolving over the graph . Formally , the operation is revised as Z = µ ( I − ( 1− µ ) Ãsym ) −1 MLPW ( X ) = ( I + αL̃ ) −1 MLPW ( X ) ( 3 ) where we replace µ with α = ( 1− µ ) /µ , L̃ : = I − Ãsym denotes the normalized graph Laplacian , and W depicts the learnable MLP parameters . As matrix inverse is expensive , an approximate version called APPNP that employs the power method ( Golub & Van Loan , 1989 ) is often used in practice . 3.2 GRAPH-REGULARIZED PCA ( GPCA ) . As stated by Bengio et al . ( 2013 ) , “ Although depth is an important part of the story , many other priors are interesting and can be conveniently captured when the problem is cast as one of learning a representation. ” GPCA is one such representation learning technique with a graph-based prior . Standard PCA learns k-dimensional projections Z ∈ Rn×k of feature matrix X ∈ Rn×d , aiming to minimize the reconstruction error ∥X − ZWT ∥2F , ( 4 ) subject to W ∈ Rd×k being an orthonormal basis . GPCA extends this formalism to graph-structured data by additionally assuming either smoothing bases ( Jiang et al. , 2013 ) or smoothing projections ( Zhang & Zhao , 2012 ) over the graph . In this work we consider the latter case where low-dimensional projections are smooth over the input graph G , where L̃ = I− Ãsym denotes its normalized Laplacian matrix . The objective formulation of GPCA is then given as min Z , W ∥X − ZWT ∥2F + αTr ( ZT L̃Z ) s.t . WTW = I ( 5 ) where α is a hyperparameter that balances reconstruction error and the variation of the projections over the graph . Note that the first part of Eq . equation 5 , along with the constraint , corresponds to the objective of the original PCA , while the second part is a graph regularization term that aims to “ smooth ” the learned representations Z over the graph structure . As such , GPCA becomes the standard PCA when α = 0 . Similar to PCA , the problem ( 5 ) is non-convex but has a closed-form solution ( Zhang & Zhao , 2012 ) . Surprisingly , as we show , it has a close connection with the graph convolution formulation in Eq . equation 2 . In the following , we give the GPCA solution and then detail its connection to graph convolution . Theorem 3.1 . GPCA with formulation shown in ( 5 ) has the optimal solution ( Z∗ , W ∗ ) following Z∗ = ( I + αL̃ ) −1XW ∗ , and W ∗ = ( w1 , w2 , ... , wk ) ( 6 ) where w1 , ... , wk are the eigenvectors of XT ( I+αL̃ ) −1X corresponding to the largest k eigenvalues . Proof . The proof can be found in Appendix . A.1 .

Summary: This paper relates GCN to PCA from the perspective of optimization. The authors propose Graph PCA that is a general form of GCN. They further introduce a regularization term that enforces nodes with same labels close to each other.