Metropolis-Hastings Data Augmentation for Graph Neural Networks

1 Introduction . Graph Neural Networks ( GNNs ) [ 1 ] have been widely used for representation learning on graphstructured data due to their superior performance in various applications such as node classification [ 2– 4 ] , link prediction [ 5–7 ] and graph classification [ 8 , 9 ] . They have been proven effective by achieving impressive performance for diverse datasets such as social networks [ 10 ] , citation networks [ 4 ] , physics [ 11 ] , molecules [ 12 ] , and knowledge graphs [ 10 ] . However , GNNs often suffer from weakgeneralization due to their small and sparsely labeled graph datasets . One prevalent remedy to address the problem is data augmentation . Data augmentation increases the diversity of data and improves the generalization power of machine learning models trained on randomly augmented samples . It is widely used to enhance the generalization ability of models in many domains . For instance , in image recognition , advanced methods like [ 13–15 ] as well as simple transformations such as random cropping , cutout , Gaussian noise , or blurring have been used to achieve competitive performance . However , unlike image recognition , designing effective and label-preserving data augmentation for individual samples on graphs is challenging due to their non-Euclidean nature and the dependencies between data samples . In image recognition , it is straightforward to identify operations that preserve labels . For instance , human can verify that rotation , translation , and small color jittering do not change the labels in image classification . In contrast , graphs are less interpretable and it is non-trivial ∗First two authors have equal contribution †is the corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for even human to check whether the augmented samples belong to the original class or not . In addition , due to the dependencies between nodes and edges in a graph , it is hard to control the degree of augmentation for individual samples . For instance , a simple operation on a graph , e.g. , dropping a node , may result in a completely different degree of augmentation depending on the graph structure . If a hub node is removed , the single perturbation affects a substantial amount of other nodes , which are data samples in node classification . To address these challenges , learning-based data augmentation methods for graphs have been proposed . AdaEdge [ 16 ] optimizes the graph topology based on the model prediction . [ 17 ] proposes GAug-M and GAug-O that generate augmented graphs via a differentiable edge predictor . GraphMix [ 18 ] presents interpolation-based regularization by jointly train a fully connected network and graph neural networks . However , they require additional models for augmentation and more importantly do not explicitly guarantee that augmentation has a proper strength and diversity . In this paper , we proposed a novel framework called Metropolis-Hastings Data Augmentation ( MHAug ) that draws augmented graphs from an ‘ explicit ’ target distribution with the desired strength and diversity for semi-supervised learning . Since the direct sampling from the complex distribution is challenging , we adopt the Metropolis-Hastings algorithm to obtain the augmented samples . Recently , the importance of leveraging unlabeled data as well as adopting advanced augmentation has emerged [ 19 , 20 ] . Inspired by that , we also adopt the consistency training by utilizing the regularizers for unlabeled data . Our extensive experiments demonstrate that MH-Aug can generate a sequence of samples according to the desired distribution and be combined with the consistency training and it significantly improves the performance of graph neural networks . Our contributions are summarized as follows : • We proposed a novel framework Metropolis-Hastings Data Augmentation that draws augmented samples from an ‘ explicit ’ target distribution . To the best of our knowledge , this is the first work that studies data augmentation for graph-structured data from a perspective of a Markov chain Monte Carlo sampling . • We theoretically and experimentally prove that our MH-Aug generates the augmented samples according to the desired distribution with respect to the strength and diversity . • We propose a target distribution that flexibly controls the strength and diversity of augmentation . This includes an efficient way to measure the strength of augmentation reflecting the structural changes of ego-graphs ( or samples in node classification ) . • Lastly , we propose a simple and effective semi-supervised learning strategy leveraging sequentially generated samples from our method . 2 Related Works . Semi-Supervised Learning on Graphs . GNNs have been widely adopted in representation learning on graphs [ 2–4 ] . However , existing works only utilize a small subset of nodes . To fully utilize a large amount of unlabeled data , recent studies for semi-supervised learning have emerged inspired by semisupervised frameworks in other domains [ 19 , 20 ] . GraphMix [ 18 ] is a regularization method based on semi-supervised learning by linear interpolation between two data on graphs , and SSL [ 21 ] proposes self-supervised learning strategies to exploit available information from graph structure . BVAT [ 22 ] promotes the smoothness of GNNs by generating virtual adversarial perturbations . Likewise , we follow semi-supervised strategy to leverage unlabeled data while considering sequentially generated samples from our augmentation . Data Augmentation on Graphs . Data augmentation is an effective technique to improve generalization by increasing the diversity of data . It is becoming the de facto necessity for model training to employ simple data augmentation ( e.g. , image rotation , flipping , translation , and so on ) . Despite the effectiveness of data augmentation , few approaches have been explored in graph domain due to its non-Euclidean nature and dependencies between data samples . Simple approaches exist such as DropEdge [ 23 ] to randomly remove a certain number of edges and AdaEdge [ 16 ] to adaptively control the inter-class/intra-class edges . Similarly , a method to propagate the perturbed node features by randomly dropping on a node-based was proposed in [ 24 ] . GAug [ 17 ] proposes the neural edge predictors as an augmentation module . Unlike existing methods employing simple perturbation [ 23 ] or extra augmentor model [ 17 , 25 ] , we propose the sampling-based augmentation , where a sequence of augmented samples are drawn from the explicitly designed target distribution for augmentation . 3 Method . We present a novel data augmentation framework for graph-structured data via Metropolis-Hastings algorithm . MH-Aug is a sampling-based augmentation , where a sequence of augmented samples are drawn from the explicit target distribution that enables flexible control of strength and diversity of augmentation . The overall sampling process of MH-Aug is described in Figure 1 . In this section , we first summarize the basics for our framework and delineate the components of MH-Aug. Then , we outline the training procedure with proposed consistency regularizers for semi-supervised learning . Lastly , we theoretically prove the distribution of augmented samples by MH-Aug converges to the desired target distribution . 3.1 Preliminaries . Ego-graph , Gi . A graph is denoted as G = ( V , E ) , where V and E are the sets of nodes and edges . A k-hop ego-graph Gi [ 26 ] is a subgraph of G centered at node vi ∈ V , consisting of neighbors within k hops from node vi and all edges between the neighbors including vi . In other words , the k-hop ego-graph of a node vi is defined as Gi = ( Vi , Ei ) , Vi = { u|S ( u , vi ) ≤ k , u ∈ V } , Ei = { ( u , v ) | ( u , v ) ∈ E and u , v ∈ Vi } , where S ( u , v ) is the length of the shortest path between nodes u and v. In this paper , we do not explicitly specify k for ego-graphs since 2-hop ego-graphs are used in all experiments . Change ratio of graph , ∆G′ . The change ratio of graph G to G′ = ( V ′ , E ′ ) is measured by the number of added/deleted edges ( or nodes ) divided by the number of original edges ( or nodes ) , i.e. , ∆G′E = ( |E ′−E|+|E−E ′| ) /|E| and ∆G′V = ( |V ′−V|+|V−V ′| ) /|V| . Since , in this work , we consider only subgraphs of the original input graph as augmented samples , which is similar to DropEdge [ 23 ] and DropNode [ 24 ] , the change ratio can be equivalently written as ∆G′E = 1 − |E ′|/|E| and ∆G′V = 1− |V ′|/|V| . Thereby ∆G′E and ∆G′V are always ranged in [ 0 , 1 ] . Metropolis-Hastings ( MH ) algorithm . MH algorithm is a Markov chain Monte Carlo method to draw random samples from a target distribution when direct sampling is difficult [ 27 ] . The algorithm comprises three components : the target distribution P , the proposal distribution Q , and the acceptance ratio A . The MH algorithm iteratively draws samples from the target distribution P being only dependent on the current sample . The MH algorithm uses a proposal distribution Q to draw a candidate sample and evaluates the acceptance ratio A to decide whether to accept or reject the candidate sample . The accepted samples by the MH algorithm follow the target distribution P . 3.2 Metropolis-Hastings Data Augmentation . Our objective is to sample the augmented graph G′ from the target distribution P given the original graph G and can be written as G′ ∼ P ( G′ ; G ) . ( 1 ) Since direct sampling from the target distribution P is challenging , we propose a novel data augmentation method based on Metropolis-Hastings algorithm . Target Distribution . We design the target distribution P to control the strength and diversity of augmentation for effective learning . The strength and diversity can be discussed from two perspectives : a full graph and ego-graphs . In our framework , the strength of augmentation is measured by the change ratio of ego-graphs , i.e. , ∆G′i , since most existing GNNs with k-layers learn node representations based on their k-hop ego-graphs . On the other hand , the diversity of augmentation is controlled by ∆G′ and ∆G′i from both full graph and ego-graph perspectives . The diversity of augmentation is adaptively adjusted for each ego-graph by the standard deviation σ ( i ) that is a simple linear function of the entropy i of the prediction at node vi . Given the expected strength µE ∈ R and ego-graph level diversity σE ( i ) ∈ R , the target distribution P w.r.t . edges is given as follows : PE ( G′ ) ∝ |V|∏ i exp ( − ( ∆G′i , ( E ) − µE ) 2 2 { σE ( i ) } 2 ) λ1 · 1 ( |E| |E|·∆G′E ) λ2 , ( 2 ) where ∆G′i , ( E ) is the change ratio of G ′ i w.r.t . E and λs are hyperparamters for controlling the influence of the two components . To have various full graph change ratios ∆G′E , the normalization by the number of possible augmented graphs corresponding to the same change ratio , ( |E| |E|·∆G′E ) , is necessary . As the size of graph increases , without the normalization , it becomes extremely difficult to generate augmented samples with a low ( or high ) full graph change ratio . For more details , see Section 4.2 . Similarly , The target distribution P with respect to nodes can be written as follows : PV ( G′ ) ∝ |V|∏ i exp ( − ( ∆G′i , ( V ) − µV ) 2 2 { σV ( i ) } 2 ) λ3 · 1 ( |V| |V|·∆G′V ) λ4 , ( 3 ) where ∆G′i , ( V ) is the change ratio of G ′ i w.r.t . the nodes , V . With combining the two distributions , the overall target distribution is defined as : P ( G′ ) = PE ( G′ ) · PV ( G′ ) . ( 4 ) In our experiment , unlike the change ratio of the full graph ∆G′E ( and ∆G′V ) , we define the ego-graph change ratio ∆G′i , ( E ) ( and ∆G ′ i , ( V ) ) with the change of the number of received messages from k-hop ego-graphs . Figure 2 illustrates the calculation of ∆G′i , ( E ) regarding two different cases : ( b ) dropping distant ( 2-hop ) edges and ( c ) dropping near ( 1-hop ) edges . In this definition , even if the number of dropped edges is the same , dropping edges connecting nodes closer to the center node vi leads to a larger ∆G′i , ( E ) than the case of distant nodes ( 0.6 > 0.13 ) , which can be regarded as a much stronger augmentation . It indicates that the amount of received messages depends on not only the number of removed edges but also which edges are dropped . This structural property can only be properly handled from ego-graph perspective . In practice , this definition allows time and memory efficient implementation using matrix multiplications as ∆G′i , ( E ) = 1− ( Ã′k1 ) i ( Ãk1 ) i , and ∆G′i , ( V ) = 1− ( Ãkm ) i ( Ãk1 ) i , ( 5 ) where Ã and Ã′ are adjacency matrices of the original graph G and the current graph G′ , where both graphs include a self-connection for every node , 1 ∈ R|V| is a vector of ones , and m ∈ R|V| is a mask vector for DropNode . Proposal Distribution . For efficient sampling and a theoretical guarantee of convergence to the target distribution , a proposal distribution is crucial . A proposal distribution Q ( G′|G ( t ) ) suggests a candidate augmented sample G′ , given the current sample G ( t ) . To draw diversely augmented graphs with various edge/node change ratios ∆G′E and ∆G′V , a candidate augmented sample is generated by three steps : 1 ) change ratio sampling , 2 ) graph modification and 3 ) merging . We first independently sample change ratios ∆G′ ( · ) for edges and nodes from Gaussian distributions truncated to the range [ 0 , 1 ] given mean ∆G ( t ) ( · ) and standard deviation σ∆ , ( · ) . Then , we modify the original graph G to generate augmented samples G′E and G′V , which can be viewed as a uniform sampling from all possible augmented graphs with ∆G′E and ∆G′V respectively . Finally , the two graphs G′E and G′V are merged to construct the candidate augmented sample G′ . Formally , the proposal distribution is given by : Q ( G′|G ( t ) ) ∝ φ ( ξ ′ E ) Φ ( βE ) − Φ ( αE ) · φ ( ξ ′ V ) Φ ( βV ) − Φ ( αV ) · 1 ( |E| |E|·∆G′E ) · 1 ( |V| |V|·∆G′V ) , ( 6 ) where α ( · ) = a−∆G ( t ) ( · ) σ∆ , β ( · ) = b−∆G ( t ) ( · ) σ∆ , ξ′ ( · ) = ∆G′ ( · ) −∆G ( t ) ( · ) σ∆ , φ ( x ) = 1√ 2π exp ( − 12x 2 ) as the probability density function of the standard normal distribution and Φ ( x ) = 12 ( 1 + erf ( x√ 2 ) ) as its cumulative distribution function . a and b represent the extremes of Gaussian distribution . Since the change ratio should be ranged in [ 0 , 1 ] , a is 0 and b is 1 . In ( 6 ) , the first and second terms denote the likelihood of the change ratios ∆G′E and ∆G′V given ∆G ( t ) . The third and fourth terms are for the probability of a sample with ∆G′E and ∆G′V . Acceptance Ratio . Starting with the original graph G , MH-Aug draws the candidate graph G′ from the proposal distribution Q . Then , with an acceptance ratio A , MH-Aug decides whether to accept or reject the candidate G′ . A is given as : A = min { 1 , P ( G′ ) Q ( G ( t ) |G′ ) P ( G ( t ) ) Q ( G′|G ( t ) ) } . ( 7 ) The computation of A with target distribution in ( 4 ) and proposal distribution in ( 6 ) is described in the supplement . MH-Aug generates a sequence of augmented graphs { G ( t ) } 0≤t≤T , where T is the number of accepted samples .

Summary: This paper introduces a MH-style data augmentation to boost the generalization ability of the existing GNNs by generating sequence of augmented graphs. The generation is proven to be converging to the desired target distribution. The experimental results shows a promising result for applying augmented graph during the GNN training.