Fractional Graph Convolutional Networks (FGCN) for Semi-Supervised Learning

Due to high utility in many applications , from social networks to blockchain to power grids , deep learning on non-Euclidean objects such as graphs and manifolds continues to gain an ever increasing interest . Most currently available techniques are based on the idea of performing a convolution operation in the spectral domain with a suitably chosen nonlinear trainable filter and then approximating the filter with finite order polynomials . However , such polynomial approximation approaches tend to be both non-robust to changes in the graph structure and to capture primarily the global graph topology . In this paper we propose a new Fractional Generalized Graph Convolutional Networks ( FGCN ) method for semisupervised learning , which casts the Lévy Flights into random walks on graphs and , as a result , allows to more accurately account for the intrinsic graph topology and to substantially improve classification performance , especially for heterogeneous graphs . 1 INTRODUCTION . Adaptation of deep learning ( DL ) to graphs and other non-Euclidean objects has recently witnessed an ever increasing interest , leading to the new subfield of geometric deep learning . In particular , geometric deep learning is an emerging direction in machine learning which aims at generalizing the concepts of deep learning for data in non-Euclidean spaces , e.g. , graphs and manifolds , by bridging the gap between graph theory and deep neural networks ( see discussion by Bronstein et al. , 2017 ; Monti et al. , 2017 ; Monti , 2019 , and references therein ) . Many such DL approaches for non-Euclidian objects are based on the idea of performing a convolution operation in the spectral domain with a suitably chosen nonlinear trainable filter . As a result , node features are mapped into some Euclidian space . Next , graph filters are approximated with various finite order polynomials , e.g. , Chebyshev polynomials ( i.e. , the ChebNet model family of Defferrard et al. , 2016 ; Kipf & Welling , 2017 ) , Cayley transform ( i.e. , CayleyNet of Levie et al. , 2018 ) or the generalization of polynomial filters in a form of Auto-Regressive Moving Average ( ARMA ) models ( Bianchi et al. , 2019 ) . However , deep learning approaches based on approximation with finite order polynomials tend to be non-robust to even minor changes in the graph structure and to largely disregard the local graph topology that often plays the critical role for learning on heterogeneous graphs . In contrast , as noted by Bianchi et al . ( 2019 ) , one of the primary benefits of the ARMA filters over polynomial ones is that ARMA filters are not computed in the Fourier space induced by a graph Laplacian , and as a result , ARMA filters are local in the node space and enable to more flexibly and accurately capture the underlying graph topology . In this paper we further advance this localized approach to DL on graphs and propose a fractional generalized graph-based convolutional filter for semi-supervised learning , which casts the Lévy Flights into random walks on graphs . As a result , our new Fractional Generalized Graph Convolutional Networks ( FGCN ) method allows to more accurately account for the intrinsic local graph topology and to substantially improve classification performance , especially for heterogeneous graphs . The key contributions of our paper can be summarized as follows : • we propose a new fractional generalized graph-based convolutional filter for semisupervised learning , which casts the Lévy Flights into random walks on graphs and , as a result , provides a more efficient exploration of the graph structure . • we develop a new Fractional Generalized Graph Convolutional Networks ( FGCN ) method that substantially improves accuracy of node classification for graphs of smaller sizes and with a lower number of node features . • the proposed architecture of FGCN uses two state-of-the-art operations – gated maxaverage pooling and residual block . The architecture has been shown to significantly improve the training convergence and model output stability . 2 RELATED WORK . Many earlier semi-supervised learning approaches on graphs , e.g. , Gaussian mixture models , cotraining , harmonic function , and label propagation , employed only the label information ( i.e. , labeled instances ) to train models based on the smoothness assumption over the labels ( Zhu & Goldberg , 2009 ) and largely disregarded the graph structure . To improve performance , several learning methods on graphs propose to incorporate intrinsic “ graph-based ” information by designing a classifying function via generalizing the normalized cut and adding smooth function with respect to the intrinsic structure ( Zhou et al. , 2004 ; Zhou & Burges , 2007 ) . A recent approach of Avrachenkov et al . ( 2012 ) proposes a generalized optimization framework through considering the above two methods as particular cases . However , one of the major limitations to these graph-based semi-supervised learning methods is disregarding important information contained in graph edges . To address this disadvantage , Defferrard et al . ( 2016 ) propose a formulation of convolutional neural networks ( CNN ) based on spectral graph theory – ChebNet , which employs approximation via finite order polynomials and , in particular , is based in the Chebyshev expansion for fast filtering instead of the expensive eigen-decomposition . Graph Convolutional Networks ( GCN ) of Kipf & Welling ( 2017 ) simplifies ChebNet and further addresses the gradient vanishing problem and reduces the number of optimization . Other related approaches to graph learning with deep neural networks include , for instance , mixture model networks ( MoNet ) ( Monti et al. , 2017 ) , graph attention networks ( GAT ) ( Veličković et al. , 2017 ) , graph convolutional recurrent networks ( Seo et al. , 2018 ) , dual graph convolutional networks ( Zhuang & Ma , 2018 ) , FastGCN ( Chen et al. , 2018 ) , and simplified version of GCN ( Wu et al. , 2019 ) . To extend the success of GCN on undirected graphs to directed graphs , MotifNet of Monti et al . ( 2018 ) replaces the normalized Laplacian with the motif Laplacian in a multivariate polynomial filter , where the motifs information can help capture the network structure . Finally , the most recent approach of Bianchi et al . ( 2019 ) provides more flexible responses than GCN by using parallel and periodic concatenations of the convolutional kernel via the ARMA filter . As a result , the ARMA approach which is applicable to both directed and undirected networks allows to more accurately incorporate the underlying local graph structure into the graph learning process . 3 METHODOLOGY . Given a graph structure G = { V , E , W } , with V the set of nodes ( N = |V| ) and E ⊆ V × V the set of edges between any two nodes . Let square matrix W represents the adjacency matrix of a graph whose entries { ωij } 1≤i , j≤N are the edge weight function such that each edge eij ∈ E has a weight ωij . We call W the symmetric adjacency matrix ( ωij 6= 0 if there is an edge between vertex vi and vertex vj ) if we assume that graph G is undirected . In reality , however , undirected graph is a simplified representation of complex networks , we also consider directed graph ( in which case W > 6= W ) and we can easily transform the non-symmetric W to the symmetric one through W ′ = ( W > +W ) /2 . We takeN×Q feature matrixX as the input to a semi-supervised learning algorithm whereQ is the number of different node features . Suppose we would like to classify N data points into K classes ( communities ) , we define N ×K label matrix Y as : Yik = { 1 , if vertex i is labelled as a class k , 0 , otherwise . ( 1 ) Here we refer to each column Y·k of matrix Y as a labeling function . Also define an N ×K matrix F and call its columns F·k classification functions . 3.1 GRAPH SIGNAL PROCESSING . In graph signal processing , a real-symmetric matrix has N real eigenvalues and its N real eigenvectors form an orthonormal basis . Given the symmetric adjacency matrix W of a graph , let D be the degree matrix where dii = ∑N j=1 wij and denote L = U > ΛU the Standard Laplacian matrix , where Λ = diag ( λ0 , . . . , λN−1 ) and U = [ u0 , . . . , uN−1 ] is the matrix of eigenvectors . In the following , we will revisit three popular semi-supervised learning methods - graph-based semisupervised learning , fractional graph-based semi-supervised learning , and graph convolutional networks and gain new insights for improving their modeling capabilities . Graph-based semi-supervised learning . Graph-based semi-supervised learning ( G-SSL ) has received much attention as an alternative approach to the population paradigm of supervised learning in recent years . G-SSL develops a generalized optimization framework , which has three particular cases ( i ) the Standard Laplacian ( SL ) ; ( ii ) Normalized Laplacian ( NL ) ; ( iii ) PageRank ( PR ) . The optimization formulation with the following expression : min F { 2FT·kD σ−1LDσ−1F·k + µ ( F·k − Y·k ) TD2σ−1 ( F·k − Y·k ) } , ( 2 ) where µ is a regularization parameter . The minimization of the first term in Eq . 2 corresponds to the idea that if two nodes are close in graph with respect to some metric , they should belong to the same class ; and by minimizing the second term we would like to bring the classification function F·k as close as possible to the labelling function Y·k . The Eq . 2 allows us to obtain the Standard Laplacian based formulation ( σ = 1 ) , the Normalized Laplacian formulation ( σ = 12 ) , and PageRank formulation ( σ = 0 ) . The objective of the generalized optimization framework for G-SSL is a convex function and the corresponding classification function : F·k = ( 1− α ) ( I − αD−σWDσ−1 ) −1 Y·k , ( 3 ) where α = 22+µ and for k = 1 , . . . , K. Tuning the parameter σ on the power of degree matrix D , we can obtain three mentioned above particular semi-supervised learning methods : • SL method ( σ = 1 ) : F·k = ( 1− α ) ( I − αD−1W ) −1 Y·k • NL method ( σ = 12 ) : F·k = ( 1− α ) ( I − αD−12 WD−12 ) −1 Y·k • PR method ( σ = 0 ) : F·k = ( 1− α ) ( I − αWD−1 ) −1 Y·k From above formulations , the classification function F , i.e. , the result of random walk process which provides the connection to the probabilistic interpretation of G-SSL . The parameter α controls the strength of the ground truth label matrix Y in the generalized optimization framework . Fractional graph-based semi-supervised learning . To improve the classification performance ( in particular , fuzzy graphs and unbalanced labeled data ) of G-SSL , fractional graph-based semisupervised learning ( De Nigris et al. , 2017 ) embeds Lévy Flights into random walks on graphs by constructing from powers of the Laplacian matrix , i.e. , the Lγ operator . This operation can be used to generate different transition probabilities ( i.e. , corresponding to stochastic adjacency matrix ) based on different γ values . Intuitively , embedding Lévy Flights into random walks allows for better capturing mixing properties ( i.e. , dependence ) in the data . Based on a fractional Laplacian matrix , 0 < γ < 1 , the anomalous ( fractional ) diffusion processes on networks can be constructed from the spectra data and eigenvectors of the Laplacian matrix . The fractional powers of L allows Lévy random walks with long-range navigation on a network . For example , the long-range transitions on a network can directly move node u and node v with the transition probability m ( γ ) u→v through a random walker , where m ( γ ) u→v is an element in the fractional transition matrix M ( γ ) ( see the graph within dotted circle in Figure 1 ) . The transition probability m ( γ ) u→v between any two nodes whose geodesic distance is not infinite can be summarized as follow : m ( γ ) u→v = δuv − ( Lγ ) uv k ( γ ) u , ( 4 ) where δuv represents the Kronecker delta , k ( γ ) u denotes the fractional degree of the node u and k ( γ ) u ≡ ( Lγ ) uu . Eq . 4 provides transition probabilities for the Lévy Flights . Unlike the standard random walk , the Lévy Flights can jump immediately over several hops in a graph . This feature makes Lévy Flights a very good exploratory process . There is a price to pay for this : the typically sparse transition probability matrix becomes non-sparse . We can mitigate non-sparsity by taking a reasonable number of principal singular eigenvectors or limiting the number of terms in the Taylor expansion . Through replacing the L operator with Lγ = U > ΛγU , the new optimization formulation leaves us with the following expression : min F { 2FT·kD σ−1 γ L γDσ−1γ F·k + µ ( F·k − Y·k ) TD2σ−1γ ( F·k − Y·k ) } , ( 5 ) where ( Dγ ) ii = ( Lγ ) ii . Let 0 < γ < 1 , the closed form solution for Eq . 5 can be obtain as fellow : F·k = ( 1− α ) ( I − αD−σγ WγDσ−1γ ) −1 Y·k , ( 6 ) for k = 1 , . . . , K. Therefore , we can conclude three particular fractional semi-supervised learning mehtods like G-SSL : • Fractional SL method ( σ = 1 ) : F·k = ( 1− α ) ( I − αD−1γ Wγ ) −1 Y·k • Fractional NL method ( σ = 12 ) : F·k = ( 1− α ) ( I − αD −1 2 γ WγD −1 2 γ ) −1 Y·k • Fractional PR method ( σ = 0 ) : F·k = ( 1− α ) ( I − αWγD−1γ ) −1 Y·k

Summary: This paper proposes a fractional graph convolutional networks for semi-supervised learning. The proposed method used a classification function of a fractional graph semi-supervised learning (GSSL) [De Nigris et al., 2017] as a graph filter. In addition, the authors adopt a parallel system and weighted combination of max and average pool. Experimental results show that the proposed method (FGCN) shows the best accuracy compared to other recent graph-based neural networks for all datasets except one.