Generalizing Graph Convolutional Networks via Heat Kernel

Graph convolutional networks ( GCNs ) have emerged as a powerful framework for mining and learning with graphs . A recent study shows that GCNs can be simplified as a linear model by removing nonlinearities and weight matrices across all consecutive layers , resulting the simple graph convolution ( SGC ) model . In this paper , we aim to understand GCNs and generalize SGC as a linear model via heat kernel ( HKGCN ) , which acts as a low-pass filter on graphs and enables the aggregation of information from extremely large receptive fields . We theoretically show that HKGCN is in nature a continuous propagation model and GCNs without nonlinearities ( i.e. , SGC ) are the discrete versions of it . Its low-pass filter and continuity properties facilitate the fast and smooth convergence of feature propagation . Experiments on million-scale networks show that the linear HKGCN model not only achieves consistently better results than SGC but also can match or even beat advanced GCN models , while maintaining SGC ’ s superiority in efficiency . 1 INTRODUCTION . Graph neural networks ( GNNs ) have emerged as a powerful framework for modeling structured and relational data ( Gori et al. , 2005 ; Scarselli et al. , 2008 ; Gilmer et al. , 2017 ; Kipf & Welling , 2017 ) . A wide range of graph mining tasks and applications have benefited from its recent emergence , such as node classification ( Kipf & Welling , 2017 ; Veličković et al. , 2018 ) , link inference ( Zhang & Chen , 2018 ; Ying et al. , 2018 ) , and graph classification ( Xu et al. , 2019b ) . The core procedure of GNNs is the ( discrete ) feature propagation operation , which propagates information between nodes layer by layer based on rules derived from the graph structures . Take the graph convolutional network ( GCN ) ( Kipf & Welling , 2017 ) for example , its propagation is performed through the normalized Laplacian of the input graph . Such a procedure usually involves 1 ) the non-linear feature transformation , commonly operated by the activation function such as ReLU , and 2 ) the discrete propagation layer by layer . Over the course of its development , various efforts have been devoted to advancing the propagation based architecture , such as incorporating self-attention in GAT ( Veličković et al. , 2018 ) , mixing high-order neighborhoods in MixHop ( Abu-El-Haija et al. , 2019 ) , and leveraging graphical models in GMNN ( Qu et al. , 2019 ) . Recently , Wu et al . ( Wu et al. , 2019 ) observe that the non-linear part of GCNs ’ feature propagation is actually associated with excess complexity and redundant operations . To that end , they simplify GCNs into a linear model SGC by removing all non-linearities between consecutive GCN layers . Surprisingly , SGC offers comparable or even better performance to advanced GCN models , based on which they argue that instead of the non-linear feature transformation , the repeated graph propagation may contribute the most to the expressive power of GCNs . Though interesting results generated , SGC still inherits the discrete nature of GCNs ’ propagation , which can lead to strong oscillations during the procedure . Take , for example , a simple graph of two nodes v1 and v2 with one-dimension input features x1 = 1 & x2 = 2 and one weighted edge between them , the feature updates of x1 and x2 during the GCN propagation is shown in Figure 1 ( a ) , from which we can clearly observe the oscillations of x1 and x2 step by step . This indicates that though the features from multi-hops away may seem to be taken into consideration during the GCN propagation , it is still far away to learn patterns from them . In this work , we aim to generalize GCNs into a continuous and linear propagation model , which is referred to as HKGCN . We derive inspiration from Newton ’ s law of cooling by assuming graph feature propagation follow a similar process . Straightforwardly , this leads us to leverage heat kernel for feature propagation in HKGCN . Theoretically , we show that the propagation matrix of GCNs is equivalent to the finite difference version of the heat kernel . In other words , using heat kernel as the propagation matrix will lead to smooth feature convergence . In the same example above , we show the heat kernel based propagation in HKGCN can prevent oscillations , as illustrated in Figure 1 ( b ) . Finally , from the graph spectral perspective , heat kernel acts as a low-pass filter and the cutoff frequency of heat kernel can be adjusted by changing the propagation time . Empirically , we demonstrate the performance of HKGCN for both transductive and inductive semi-supervised node classification tasks . The experiments are conducted on both traditional GNN datasets , such as Cora , CiteSeer , Pubmed , and Reddit , and latest graph benchmark data indexed by Open Graph Benchmark ( Hu et al. , 2020 ) . The results suggest that the simple and linear HKGCN model can consistently outperform SGC on all six datasets and match or even beat the performance of advanced graph neural networks on both tasks , while at the same time maintaining the order-of-magnitude efficiency superiority inherited from SGC . 2 RELATED WORK . Graph Neural Networks . Graph neural networks ( GNNs ) have emerged as a new paradigm for graph mining and learning , as significant progresses have been made in recent years . Notably , the spectral graph convolutional network ( Bruna et al. , 2013 ) is among the first to directly use back propagation to learn the kernel filter , but this has the shortcoming of high time complexity . Another work shows how to use Chebyshev polynomial approximation to fast compute the filter kernel ( Hammond et al. , 2011 ) . Attempts to further this direction leverage Chebyshev Expansion to achieve the same linear computational complexity as classical CNNs ( Defferrard et al. , 2016 ) . Later , the graph convolutional network ( GCN ) ( Kipf & Welling , 2017 ) simplifies the filter kernel to the second-order of Chebyshev Expansion , inspiring various advancements in GNNs . GAT brings the attention mechanisms into graph neural networks ( Veličković et al. , 2018 ) . GMNN combines the benefits of statistical relational learning and GNNs into a unified framework ( Qu et al. , 2019 ) . To enable fast and scalable GNN training , FastGCN interprets graph convolutions as integral transforms of features and thus uses Monte Carlo method to simulate the feature propagation step ( Chen et al. , 2018 ) . GraphSage treats the feature propagation as the aggregation from ( sampled ) neighborhoods ( Hamilton et al. , 2017 ) . LADIES ( Zou et al. , 2019 ) further introduces the layer-dependent importance sampling technique for efficient training . Recently , there are also research efforts devoting on the theoretical or deep understanding of GCNs ( Xu et al. , 2019b ; Battaglia et al. , 2018 ) . For example , the feature propagation in GNNs can be also explained as neural message passing ( Gilmer et al. , 2017 ) . In addition , studies also find that the performance of GNNs decreases with more and more layers , known as the over-smoothing issue ( Li et al. , 2018 ; Zhao & Akoglu , 2020 ) . To reduce GCNs ’ complexity , SGC turns the GCN model into a linear model by removing the non-linear activation operations between consecutive GCN layers ( Wu et al. , 2019 ) , producing promising results in terms of both efficacy and efficiency . Heat Kernel . The properties of heat kernel for graphs are reviewed in detail by Chuang in ( Chung & Graham , 1997 ) . Recently , heat kernel has been frequently used as the feature propagation modulator . In ( Kondor & Lafferty , 2002 ) , the authors show that heat kernel can be regarded as the discretization of the familiar Gaussian kernel of Euclidean space . Additionally , heat kernel is often used as the window function for windowed graph Fourier transform ( Shuman et al. , 2016 ) . In ( Zhang et al. , 2019 ) , the second-order heat kernel is used as the band-pass filter kernel to amplify local and global structural information for network representation learning . Concurrent work . Several recent works have developed similar idea . ( Poli et al. , 2020 ; Zhuang et al. , 2020 ) use the Neural ODE framework and parametrize the derivative function using a 2 or 3 layer GNN directly . ( Xhonneux et al. , 2020 ) improved ODE by developing a continuous messagepassing layer . All ODE models make feature converge to stable point by adding residual connection . In contrast , our model outputs an intermediate state of feature , which is a balance between local and global features . Some recent works ( Xu et al. , 2019a ; Klicpera et al. , 2019 ) propose to leverage heat kernel to enhance low-frequency filters and enforce a smooth feature propagation . However , they do not realize the relationship between the feature propagation of GCNs and heat kernel . 3 GENERALIZING ( SIMPLE ) GRAPH CONVOLUTION VIA HEAT KERNEL . 3.1 PROBLEM AND BACKGROUND . We focus on the problem of semi-supervised node classification on graphs , which is the same as GCN ( Kipf & Welling , 2017 ) . Without loss of generality , the input to this problem is an undirected network G = ( V , E ) , where V denotes the node set of n nodes { v1 , ... , vn } and E represents the edge set . The symmetric adjacency matrix of G is defined as A and its diagonal degree matrix as D with Dii = ∑ j Aij . For each node vi ∈ V , it is associated with a feature vector xi ∈ X ∈ Rn×d and a one-hot label vector yi ∈ Y ∈ { 0 , 1 } n×C , where C is the number of classes . The problem setting of semi-supervised graph learning is given the labels YL of a subset of nodes VL , to infer the labels YU of the remaining nodes VU , where VU = V \VL . Graph Convolutional Networks . Given the input graph G = ( V , E ) with A , D , X , and YL , GCN can be understood as feature propagation over the graph structure . Specifically , it follows the following propagation rule : H ( l+1 ) = σ ( D̃− 1 2 ÃD̃− 1 2H ( l ) W ( l ) ) , ( 1 ) where Ã = A + IN is the adjacency matrix with additional self-connections with IN as the identity matrix , W ( l ) is a trainable weight matrix in the lth layer , σ ( · ) is a nonlinear function such as ReLU , and H ( l ) denotes the hidden node representation in the lth layer with the first layer H ( 0 ) = X . The essence of GCN is that each GCN layer is equivalent to the first-order Chebyshev expansion of spectral convolution ( Kipf & Welling , 2017 ) . It also assumes that the first-order coefficient a1 is equal to the 0-th order coefficient a0 multiplied by −1 , i.e. , a1 = −a0 . We will later prove that this is just a discrete solution of heat equation . Simple Graph Convolution . Since its inception , GCNs have drawn tremendous attention from researchers ( Chen et al. , 2018 ; Veličković et al. , 2018 ; Qu et al. , 2019 ) . A recent study shows that GCNs can be simplified as the Simple Graph Convolution ( SGC ) model by simply removing the nonlinearities between GCN layers ( Wu et al. , 2019 ) . Specifically , the SGC model is a linear model and can be formalized by the following propagation rule : Y = softmax ( ( D̃− 1 2 ÃD̃− 1 2 ) K XW ) ( 2 ) Surprisingly , the linear SGC model yields comparable prediction accuracy to the sophisticated GCN models in various downstream tasks , with significant advantages in efficiency and scalability due to its simplicity . Heat Equation and Heat Kernel . The heat equation , as a special case of the diffusion equation , is used to describe how heat distributes and flows over time ( Widder & Vernon , 1976 ) . Image a scenario of graph , in which each node has a temperature and heat energy could only transfer along the edge between connected nodes , and the heat propagation on this graph follows Newton ’ s law of cooling . So the heat propagation between node vi and node vj should be proportional to 1 ) the edge weight and 2 ) the temperature difference between vi and vj . Let x ( t ) i denote the temperature of vi at time t , the heat diffusion on graph G can be described by the following heat equation : dx ( t ) i dt = −k ∑ j Aij ( x ( t ) i − x ( t ) j ) = −k [ Diix ( t ) i − ∑ j Aijx ( t ) j ] . ( 3 ) The equation under the matrix form is dX ( t ) dt = −kLX ( t ) , where L = D−A is the graph Laplacian matrix . By reparameterizing t and k into a single term t′ = kt , the equation can be rewritten as : dX ( t ′ ) dt′ = −LX ( t ′ ) ( 4 ) A heat kernel is the fundamental solution of the heat equation ( Chung & Graham , 1997 ) . The heat kernel Ht is defined to be the n× n matrix : Ht = e −Lt ( 5 ) Given the initial status X ( 0 ) = X , the solution to the heat equation in Eq . 4 can be written as X ( t ) = HtX ( 6 ) Naturally , the heat kernel can be used as the feature propagation matrix in GCNs .

Summary: This submission introduced a new graph convolutional operator based on heat diffusion, named heat kernel GCN (HKGCN). First, continuous-time heat diffusion on graphs is reviewed, where the solution is given by the heat equation (6). Then, the authors showed that classical GCN can be approximated in the same formulation through discretization.