GraphMix: Regularized Training of Graph Neural Networks for Semi-Supervised Learning

We present GraphMix , a regularized training scheme for Graph Neural Network based semi-supervised object classification , leveraging the recent advances in the regularization of classical deep neural networks . Specifically , we propose a unified approach in which we train a fully-connected network jointly with the graph neural network via parameter sharing , interpolation-based regularization and self-predicted-targets . Our proposed method is architecture agnostic in the sense that it can be applied to any variant of graph neural networks which applies a parametric transformation to the features of the graph nodes . Despite its simplicity , with GraphMix we can consistently improve results and achieve or closely match state-of-the-art performance using even simpler architectures such as Graph Convolutional Networks , across three established graph benchmarks : the Cora , Citeseer and Pubmed citation network datasets , as well as three newly proposed datasets : Cora-Full , Co-author-CS and Co-author-Physics . 1 INTRODUCTION . Due to the presence of graph structured data across a wide variety of domains , such as biological networks , social networks and telecommunication networks , there have been several attempts to design neural networks that can process arbitrarily structured graphs . Early work includes ( Gori et al . ; Scarselli et al. , 2009 ) which propose a neural network that can directly process most type of graphs e.g. , acyclic , cyclic , directed , and undirected graphs . More recent approaches include ( Bruna et al. , 2013 ; Henaff et al. , 2015 ; Defferrard et al. , 2016 ; Kipf & Welling , 2016 ; Gilmer et al. , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ; 2019 ; Qu et al. , 2019 ; Gao & Ji , 2019 ; Ma et al. , 2019 ) , among others . Many of these approaches are designed for addressing the important problem of Semi-supervised learning over graph structured data ( Zhou et al. , 2018 ) . Much of this research effort has been dedicated to developing novel architectures . Unlike many existing works which try to come up with the new architectures , we re-synthesize ideas from the recent advances in the regularization of the classical neural network , and propose an architecture-agnostic framework for regularized training of graph neural network based semisupervised object classification . Recently , Data-Augmentation based regularization has been shown to be very effective in other types of neural networks but how to apply these techniques in graph neural networks is still under-explored . Our proposed method GraphMix 1is a unified framework that utilizes interpolation based data augmentation ( Zhang et al. , 2018 ; Verma et al. , 2019a ) and self-targetprediction based data-augmentation techniques ( Laine & Aila , 2016 ; Tarvainen & Valpola , 2017 ; Verma et al. , 2019b ; Berthelot et al. , 2019 ) . We show that with our proposed unified framework , we can achieve state-of-the-art performance even when using simpler graph neural network architectures such as Graph Convolutional Networks ( Kipf & Welling , 2017 ) and without incurring any significant additional computation cost . 1code available at https : //github.com/anon777000/GraphMix 2 PROBLEM DEFINITION AND PRELIMINARIES . 2.1 PROBLEM SETUP . We are interested in the problem of semi-supervised object classification using graph structured data . We can formally define such graph structured data as G = ( V , A ) , where V represents the set of nodes { v1 , . . . , vn } , and A is the adjacency matrix representing the edges between the nodes of V . Each node vi in the graph has a corresponding d-dimensional feature vector xi ∈ Rd . The feature vectors of all the nodes X = [ x1 , . . . , xn ] > are stacked together to form the entire feature matrix X ∈ Rn×d . Each node belongs to one out of C classes and can be labeled with a C-dimensional one-hot vector yi ∈ { 0 , 1 } C . Given the labels of YL for few of the labeled nodes VL ⊂ V , the task is to predict the labels YU of the remaining nodes VU = V \ VL . 2.2 GRAPH NEURAL NETWORKS . Graph Neural Networks ( GNN ) learn the lth layer representations of a sample i by leveraging the representations of the samplesNB ( i ) in the neighbourhood of i . This is done by using an aggregation function that takes as an input the representations of all the samples and the graph structure and outputs the aggregated representation . The aggregation function can be defined using the Graph Convolution layer ( Kipf & Welling , 2017 ) , Graph Attention Layer ( Veličković et al. , 2018 ) , or any general message passing layer ( Gilmer et al. , 2017 ) . Formally , let Hl ∈ Rn×k be a matrix containing the k-dimensional representation of n nodes in the lth layer , then : Hl+1 = a ( HlW , A ) ( 1 ) where W ∈ Rk×k′ is a linear transformation matrix , k′ is the dimension of ( l + 1 ) th layer and a is the aggregation function that utilizes the graph adjacency matrix A . 2.3 INTERPOLATION BASED REGULARIZATION TECHNIQUES . Recently , interpolation-based techniques have been proposed for regularizing neural networks . We briefly describe some of these techniques here . Mixup ( Zhang et al. , 2018 ) trains a neural network on the convex combination of input and targets , whereas Manifold Mixup ( Verma et al. , 2019a ) trains a neural network on the convex combination of the hidden states of a randomly chosen hidden layer and the targets . While Mixup regularizes a neural network by enforcing that the model output should change linearly in between the examples in the input space , Manifold Mixup regularizes the neural network by learning better ( more discriminative ) hidden states . Formally , suppose g : x −→ h is a function that maps input sample to hidden states , f : h −→ ŷ is a function that maps hidden states to predicted output , λ is a random variable drawn from Beta ( α , α ) distribution , Mixλ ( a , b ) = λ ∗ a+ ( 1− λ ) ∗b is an interpolation function , D is the data distribution , ( x , y ) and ( x′ , y′ ) is a pair of examples sampled from distribution D and ` be a loss function such as cross-entropy loss , then the Manifold Mixup Loss is defined as : L = E ( x , y ) ∼D E ( x′ , y′ ) ∼D E λ∼Beta ( α , α ) ` ( f ( Mixλ ( g ( x ) , g ( x′ ) ) ) , Mixλ ( y , y′ ) ) . ( 2 ) We use above Manifold Mixup loss for training an auxiliary Fully-connected-network as described in Section 3 and Line 6 and 12 of Algorithm 1 . 3 GRAPHMIX . 3.1 MOTIVATION . Data Augmentation is arguably the simplest and most efficient technique for regularizing a neural network . In some domains , such as computer vision , speech and text , there exist efficient data augmentation techniques , for example , random cropping , translation or Cutout ( Devries & Taylor , 2017 ) for computer vision , Ko et al . ( 2015 ) and Park et al . ( 2019 ) for speech and Xie et al . ( 2017 ) for text domain . However , data augmentation for the graph-structured data remains under-explored . There exists some recent work along these lines but the prohibitive additional computation cost ( see Section 5.3 ) introduced by these methods make them impractical for real-world large graph datasets . Based on these limitations , our main objective is to propose an efficient data augmentation technique for graph datasets . Recent work based on interpolation-based data augmentation ( Zhang et al. , 2018 ; Verma et al. , 2019a ) has seen sizable improvements in regularization performance across a number of tasks . However , these techniques are not directly applicable to graphs for an important reason : Although we can create additional nodes by interpolating the features and corresponding labels , it remains unclear how these new nodes must be connected to the original nodes via synthetic edges such that the structure of the whole graph is preserved . To alleviate this issue , we propose to train an auxiliary Fully-connected-net ( FCN ) using Manifold Mixup as discussed in Section 3.2 . Note that the FCN only uses the node features ( not the graph structure ) , thus the Manifold mixup loss in Eq . 2 can be directly used for training the FCN . Furthermore , drawing inspiration from the success of selfsupervised semi-supervised learning algorithms ( self-predicted-targets based algorithms which can be also interpreted as a form of data-augmentation techniques ) ( Verma et al. , 2019b ; Berthelot et al. , 2019 ) , we explore self-supervision in the training of GNNs . We note that self-supervision has already been explored for unsupervised representation learning from graph structured data ( Veličković et al. , 2019 ) , but not for semi-supervised object classification over graph structured data . Based on these challenges and motivations we present our proposed approach GraphMix for training Graph Neural Networks in the following Section . 3.2 METHOD . GraphMix augments the vanilla GNN with a Fully Connected Network ( FCN ) . The FCN loss is computed using the Manifold Mixup as discussed in Section 2.3 and the GNN loss is computed in the standard way . The Manifold Mixup training of FCN facilitates learning more discriminative node representations . How to transfer these discriminative node representations to the GNN ? We apply parameter sharing between FCN and GNN to facilitate this . Using these more discriminative representations of the nodes , as well as the graph structure , GNN loss is computed in the usual way . Algorithm 1 GraphMix : A procedure for improved training of Graph Neural Networks ( GNN ) 1 : Input : A GCN : g ( X , A , θ ) , a FCN : f ( X , θ , λ ) which shares parameters with the GCN . Beta distribution parameter α for Manifold Mixup . Number of random perturbations K , Sharpening temperature T . Consistency parameter γ . Number of epochs N . γ ( t ) : rampup function for increasing the importance of consistency regularization . ( XL , YL ) represents labeled samples and XU represents unlabeled samples . 2 : for t = 1 to N do 3 : i = random ( 0,1 ) // generate randomly 0 or 1 4 : if i=0 then 5 : λ ∼ Beta ( α , α ) // Sample a mixing coefficient from Beta distribution 6 : Lsup = L ( f ( XL , θ , λ ) , YL ) // supervised loss from FCN using the Manifold Mixup 7 : for k = 1 to K do 8 : X̂U , k = RandomPerturbations ( XU ) // Apply kth round of random perturbation to XU 9 : end for 10 : ȲU = 1K ∑ k g ( Y | X̂U , k ; θ , A ) // Compute average predictions across K perturbations of XU using the GCN 11 : YU = Sharpen ( ȲU , T ) // Apply temperature sharpening to the average prediction 12 : Lusup = L ( f ( XU , θ , λ ) , YU ) // unsupervised loss from FCN using the Manifold Mixup 13 : L = Lsup + γ ( t ) ∗ Lusup // Total loss is the weighted sum of supervised and unsupervised FCN loss 14 : else 15 : L = L ( g ( XL , θ , A ) , YL ) // Loss using the vanilla GCN 16 : end if 17 : end for 18 : return L Both the FCN loss and GNN loss are optimized in an alternating fashion during training . Furthermore , the predicted targets from the GNN are used to augment the training set of the FCN . In this way , both FCN and GNN facilitate each other ’ s learning process . At inference time , the predictions are made using only GNN . The diagrammatic representation of GraphMix is presented in Figure 1 and the full algorithm is presented in Algorithm 1 . The GraphMix framework can be applied to any underlying GNN as long as the underlying GNN applies parametric transformations to the node features . In our experiments , we show the improvements over GCN ( Kipf & Welling , 2016 ) , GAT ( Veličković et al. , 2018 ) and Graph U-Net using GraphMix . Furthermore , GraphMix ( GCN ) framework does not add any significant computation cost over the underlying GNN , because the underlying GNN is trained in the standard way and the FCN training requires trivial additional computation cost for computing the predicted-targets ( Section 3.2.1 and 3.2.1 ) and the interpolation function ( Mixλ ( a , b ) in Section 2.3 ) . There are no additional memory requirements for GraphMix ( GCN ) , since FCN and GNN share the parameters . Some implementation considerations . For Manifold Mixup training of FCN , we apply mixup only in the hidden layer . Note that in Verma et al . ( 2019a ) , the authors recommended applying mixing in a randomly chosen layer ( which also includes the input layer ) at each training update . However , we observed under-fitting when applying mixup randomly at the input layer or hidden layer . Applying mixup only in the input layer also resulted in underfitting and did not improve test accuracy . The performance of self-supervision based algorithms such as GraphMix is greatly affected by the accuracy of the predicted targets . To improve the accuracy of the predicted targets , we applied the average of the model prediction on K random perturbations of an input sample as discussed in Section 3.2.1 and sharpening as described in Section 3.2.2 . Further , we draw similarities and difference of GraphMix w.r.t . Co-training framework in the Section 3.2.3 .

Summary: In this paper, the authors developed GNN by integrating an interpolation based regularization. In particular, the proposed GraphMix model has two components, one GNN and one MLP. For the MLP, the manifold mixup loss is used for training. The two components share parameters so that the GCN training can be regularized by the manifold mixup loss. Moreover, the predicted samples of GCN are used to augment the data of MLP for training in a self-supervised manner. In the experiments, GraphMix was compared with several recent GNN models on several benchmark datasets and demonstrates effectiveness for node classification and link classification.