GraphCGAN: Convolutional Graph Neural Network with Generative Adversarial Networks

1 INTRODUCTION . Graph-based semi-supervised learning ( SSL ) aims to classify nodes in graph , where only small amounts of nodes are labeled due to the expensive and time-consuming label collection process . To solve such task , various graph neural networks ( GNNs ) have been proposed using the idea of convolutional neural networks ( CNN ) to implicitly propagate the information of labeled nodes to unlabeled nodes through the linkage between nodes ( Kipf & Welling , 2016 ; Veličković et al. , 2017 ; Hamilton et al. , 2017 ) . These convolution-based graph neural networks have achieved superior performance on multiple benchmark datasets in graph-based SSL tasks ( Wu et al. , 2019 ) . Recently , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have been shown a power in improving the performance of image-based SSL problems ( Odena , 2016 ; Salimans et al. , 2016 ; Li et al. , 2019b ) . In semi-GAN ( Salimans et al. , 2016 ) , authors converted the M -class classification task into solving ( M + 1 ) -class problem where the synthetic ( M + 1 ) th class is generated by the GAN ’ s generator . Later on , Dai et al . provided a theoretical insight that the generated data are able to boost the performance of classifier under certain assumptions . Our work is motivated by the the semi-GAN . GraphSGAN ( Ding et al. , 2018 ) first investigated the adversarial learning over graph , where the graph is embedding into an embedding space and synthetic data are generated in the corresponding space . The multi-layer perceptron ( MLP ) is trained as the classifier on the embedding vectors . However , to our knowledge , there is still no existed method to combine the adversarial learning to convolution-based GNNs on graph-based SSL task . In this work , we explore the potential of incorporating the convolution-based GNN and GAN . The challenges of constructing a general framework have three folds : first , the attributed graph data are non-Euclidean whose distribution contains information of graph topology structure as well as the attributes of nodes . Hence , it is not trivial to construct generator to model the distribution . Second , even the generator can model the graph ’ s distribution , the generator should be trained properly to boost the performance of the classifier . A poor-quality generator would introduce noise to the existed graph and affect the classifier . Third , many variants of GCN have been proposed continuously . The framework should be built with flexibility to adapt to different convolution-based GNNs . We construct a novel approach called GraphCGAN to deal with above challenges . First , to model the distribution of graph , the generator is built sequentially from two sub-generators : one models the attribute information ( node ’ s attribute ) and another one models the graph topology structure ( adjacency relation of node ) . Details can be found in Section 3.1 . Second , in GraphCGAN , the generator is trained based on the feature matching technique ( Salimans et al. , 2016 ) which minimizes the distance between generated nodes and real nodes in the constructed feature space . This technique showed a good performance in SSL tasks in practice . The details for construction of loss functions can be found in Section 3.3 . For GCN , the attributes of nodes are aggregated convolutionally by multiple layers . The representation of the last layer is usually considered as the prediction for the labels . For variants of GCN , the main differences exist in the strategy of layer aggregation ( Hamilton et al. , 2017 ) . In our framework , we choose the second to the last layer of convolution-based GNN as the feature matching functions . Therefore , our framework is easily extended to variants of GCN . More discussions can be found in Section 3.2 . 2 PRELIMINARY . We first introduce the notation about graph . Let G = ( V , E ) denote a graph , where V is the set of nodes with |V | = n and E ⊂ V × V is a set of edges with |E| = m. The adjacency matrix A ∈ R|V |×|V | is defined as Aij = 1 if node vi and vj has edge , otherwise Aij = 0 . Suppose each node vi has a d-dimensional feature xi ∈ Rd and a single value label yi ∈ { 1 , 2 , .. , M } . In the semi-supervised learning setting , there is a disjoint partition for the nodes , V = V L ∪ V U , such that , for vi ∈ V L , the corresponding label is known and for vj ∈ V U the corresponding label is unknown . The distributions of node in labeled set V L and unlabeled set V U are denoted as pV L and pV U , respectively . The semi-supervised learning is to learn the label for unlabeled set { yj |vj ∈ V U } given adjacency matrix A , feature matrix X = [ xi ] vi∈V and labels for labeled sets { yi|vi ∈ V L } . 2.1 CONVOLUTION BASED GRAPH NEURAL NETWORK CLASSIFIER . Based on the Laplacian smoothing , the convolution-based GNN models propagate the information of nodes features across the nodes ’ neighbors in each layer . Specifically , in GCN , the layer-wise propagation rule can be defined as follows : H ( l+1 ) = σ ( D−1AH ( l ) W ( l ) + b ( l ) ) , l = 0 , 1 , 2 .. , L− 1 ( 1 ) where W ( l ) and b ( l ) are layer-specific trainable weight matrix and bias , respectively . σ ( · ) is an activation function . D is the diagonal degree matrix with Dii = ∑ j Aij . Hence , D −1A represents normalization of adjacency matrix A . The initial layer H ( 0 ) is the feature matrix X . The final layer H ( L ) followed by a softmax layer can be viewed as the prediction of one-hot representation for the true label y . Recently , many variants of the GCN layer-wise propagation rule had been proposed , including graph attention network , cluster GCN ( Veličković et al. , 2017 ; Chiang et al. , 2019 ) , which achieved stateof-the-art performances in many benchmark datasets . 2.2 GENERATIVE ADVERSARIAL NETWORK BASED SEMI-SUPERVISED LEARNING . In semi-GAN , the classifier C and generator G play a non-cooperative game , where classifier aims to classify the unlabeled data as well as distinguish the generated data from real data ; generator attempts to match feature of real data and that of generated data . Therefore , the objective function for classifier can be divided into two parts ( Salimans et al. , 2016 ) . The first part is the supervised loss function Lsup = Ev , y∼pV L logPC ( y|v , y ≤M ) which is the log probability of the node label given the real nodes . The second part is the unsupervised loss function Lun−sup = Ev∼pV U logPC ( y ≤M |v ) + Ev∼pV G logPC ( y =M + 1|v ) which is the sum of log probability of the first M classes for real nodes and the log probability of the ( M + 1 ) th class for generated nodes V G. The classifier C can be trained by maximize the objective function LC = Lsup + Lun−sup . ( 2 ) For objective function of generator , Salimans et al . ( 2016 ) found minimizing feature matching loss in Equation 3 achieved superior performance in practice LG = ||Ev∼pV U ( f ( v ) ) − Ez∼pz ( z ) ( f ( G ( z ) ) ) || 2 2 , ( 3 ) where the feature matching function f ( · ) maps the input into a feature space and z ∼ pz ( z ) is drawn from a given distribution like uniform distribution . Furthermore , Dai et al . ( 2017 ) provided a theoretical justification that complementary generator G was able to boost the performance of classifier C in SSL task . 3 FRAMEWORK OF GRAPHCGAN . To combine the aforementioned Laplacian smoothing on graph and semi-GAN on SSL together , we develop GraphCGAN model , using generated nodes to boost the performance of convolution-based GNN models . 3.1 CONSTRUCTION OF GENERATOR FOR GRAPHCGAN . The generator G generates fake node v0 by generating feature vector x0 ∈ Rd and adjacency relation a0 ∈ Rn jointly , where a0 , i = 1 if the fake node is connected to real node vi , otherwise a0 , i = 0 . Therefore , the distribution for generated node pG ( v0 ) can be expressed by the joint distribution of the corresponding feature and adjacency relation pG ( x0 , a0 ) . From the conditional distribution formula , the joint distribution can be written as pG ( x0 , a0 ) = pG1 ( x0 ) pG2 ( a0|x0 ) . We use sub-generators G1 and G2 to generate fake feature x0 and a0|x0 , respectively . In practice , a0|x0 can be modeled by G2 ( z ; x0 ) = G2 ( z ; G1 ( z ) ) where the adjacency relation a0 is constructed by sub-generator G2 given the input of x0 . The distribution of generated node can be denoted by pG ( v0 ) = pG ( x0 , a0 ) = pG ( x0 ) p ( a0|x0 ) = p ( G1 ( z ) ) p ( G2 ( z ; G1 ( z ) ) ) = : p ( G ( z ) ) . ( 4 ) If B nodes ( v0,1 , v0,2 , .. , v0 , B ) are generated , the generated feature matrix is denoted as X0 = ( xT0,1 , x T 0,2 , .. , x T 0 , B ) T and generated adjacency matrix has form A0 = ( aT0,1 , a T 0,2 , .. , a T 0 , B ) T . Hence , the combined adjacency matrix can be denoted as Ã = [ A AT0 A0 IB ] ∈ R ( n+B ) × ( n+B ) , ( 5 ) The combined feature vector is X̃ = [ X X0 ] ∈ R ( n+B ) ×d . ( 6 ) The diagonal degree matrix D̃ ∈ R ( n+B ) × ( n+B ) can be denoted as [ D∗ 0 0 DB ] where D∗ ∈ Rn×n with D∗ , ii = ∑ j Aij + ∑ bA0 , bi and DB ∈ RB×B with DB , bb = ∑ j A0 , bj + 1 . 3.2 ANALYSIS OF CLASSIFIER FOR GRAPHCGAN . In GraphCGAN , we adopt the convolution-based GNN , such as GCN , GraphSage ( Hamilton et al. , 2017 ) or GAT ( Veličković et al. , 2017 ) , as the the classifier . The classifier is applied to the enlarged graph G̃ = [ X̃ , Ã ] to obtain the prediction ỹ of nodes V ∪ V G. Specially , considering the layer-wise propagation of GCN ( Equation 1 ) as the classifier in GraphCGAN , the propogation rule can be denoted as H̃ ( l+1 ) = σ ( D̃−1ÃH̃ ( l ) W ( l ) + b̃ ( l ) ) = σ ( [ D−1∗ 0 0 D−1B ] [ A AT0 A0 IB ] [ H ( l ) ∗ H ( l ) 0 ] W ( l ) + [ b ( l ) b ( l ) B ] ) = σ ( [ D−1∗ AH ( l ) ∗ +D −1 ∗ A T 0 H ( l ) 0 D−1B A0H ( l ) ∗ +D −1 B H ( l ) 0 ] W ( l ) + [ b ( l ) b ( l ) B ] ) = σ ( [ D−1∗ AH ( l ) ∗ W ( l ) + b ( l ) ∗ ( D−1B A0H ( l ) ∗ +D −1 B W ( l ) ) W ( l ) + b ( l ) B ] ) = : [ H ( l+1 ) ∗ H ( l+1 ) 0 ] . ( 7 ) where the first layer is chosen as the enlarged feature matrix H̃ ( 0 ) = X̃ . Weight matrix W ( l ) has the same in Equation 1 . Bias vector b̃ ( l ) has dimension ( n+B ) which is denoted as [ b ( l ) T , b ( l ) TB ] T . We denote b ( l ) ∗ = D−1∗ A T 0 H ( l ) ∗ W ( l ) + b ( l ) to make the format clear . From Equation 7 , the layer propagation of real nodes ( first n rows ) follows the same format as the GCN layer propagation in Equation 1 . As a special case , for the zero generator A0 = 0 or X0 = 0 , the performance of classifier on V ∪ V G would be the same as that of original classifier on V . For the last layer H̃ ( L ) ∈ R ( n+B ) ×M , we adopt the strategy in Salimans et al . ( 2016 ) to obtain the ( M + 1 ) class label ỹ by ỹ = softmax ( H̃ ( L ) ||0 ( n+B ) ×1 ) , ( 8 ) where || denotes concatenation and 0 ( n+B ) ×1 ∈ R ( n+B ) ×1 is a zero matrix . The loss function for classifier in GraphCGAN follows the same format in Equation 2 .

Summary: The paper presents a method to combine graph convolutional neural networks (GCNs) with generative adversarial networks (GANs). The authors focus on the problem of semi-supervised learning on graphs and propose an end-to-end framework in which the generative model is followed by direct convolutions on the graph nodes. Experiments are conducted on standard benchmark datasets and the proposed method, GraphCGAN is compared against several state-of-the-art approaches.