Metropolis-Hastings Data Augmentation for Graph Neural Networks

1 Introduction . Graph Neural Networks ( GNNs ) [ 1 ] have been widely used for representation learning on graphstructured data due to their superior performance in various applications such as node classification [ 2– 4 ] , link prediction [ 5–7 ] and graph classification [ 8 , 9 ] . They have been proven effective by achieving impressive performance for diverse datasets such as social networks [ 10 ] , citation networks [ 4 ] , physics [ 11 ] , molecules [ 12 ] , and knowledge graphs [ 10 ] . However , GNNs often suffer from weakgeneralization due to their small and sparsely labeled graph datasets . One prevalent remedy to address the problem is data augmentation . Data augmentation increases the diversity of data and improves the generalization power of machine learning models trained on randomly augmented samples . It is widely used to enhance the generalization ability of models in many domains . For instance , in image recognition , advanced methods like [ 13–15 ] as well as simple transformations such as random cropping , cutout , Gaussian noise , or blurring have been used to achieve competitive performance . However , unlike image recognition , designing effective and label-preserving data augmentation for individual samples on graphs is challenging due to their non-Euclidean nature and the dependencies between data samples . In image recognition , it is straightforward to identify operations that preserve labels . For instance , human can verify that rotation , translation , and small color jittering do not change the labels in image classification . In contrast , graphs are less interpretable and it is non-trivial ∗First two authors have equal contribution †is the corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for even human to check whether the augmented samples belong to the original class or not . In addition , due to the dependencies between nodes and edges in a graph , it is hard to control the degree of augmentation for individual samples . For instance , a simple operation on a graph , e.g. , dropping a node , may result in a completely different degree of augmentation depending on the graph structure . If a hub node is removed , the single perturbation affects a substantial amount of other nodes , which are data samples in node classification . To address these challenges , learning-based data augmentation methods for graphs have been proposed . AdaEdge [ 16 ] optimizes the graph topology based on the model prediction . [ 17 ] proposes GAug-M and GAug-O that generate augmented graphs via a differentiable edge predictor . GraphMix [ 18 ] presents interpolation-based regularization by jointly train a fully connected network and graph neural networks . However , they require additional models for augmentation and more importantly do not explicitly guarantee that augmentation has a proper strength and diversity . In this paper , we proposed a novel framework called Metropolis-Hastings Data Augmentation ( MHAug ) that draws augmented graphs from an ‘ explicit ’ target distribution with the desired strength and diversity for semi-supervised learning . Since the direct sampling from the complex distribution is challenging , we adopt the Metropolis-Hastings algorithm to obtain the augmented samples . Recently , the importance of leveraging unlabeled data as well as adopting advanced augmentation has emerged [ 19 , 20 ] . Inspired by that , we also adopt the consistency training by utilizing the regularizers for unlabeled data . Our extensive experiments demonstrate that MH-Aug can generate a sequence of samples according to the desired distribution and be combined with the consistency training and it significantly improves the performance of graph neural networks . Our contributions are summarized as follows : • We proposed a novel framework Metropolis-Hastings Data Augmentation that draws augmented samples from an ‘ explicit ’ target distribution . To the best of our knowledge , this is the first work that studies data augmentation for graph-structured data from a perspective of a Markov chain Monte Carlo sampling . • We theoretically and experimentally prove that our MH-Aug generates the augmented samples according to the desired distribution with respect to the strength and diversity . • We propose a target distribution that flexibly controls the strength and diversity of augmentation . This includes an efficient way to measure the strength of augmentation reflecting the structural changes of ego-graphs ( or samples in node classification ) . • Lastly , we propose a simple and effective semi-supervised learning strategy leveraging sequentially generated samples from our method . 2 Related Works . Semi-Supervised Learning on Graphs . GNNs have been widely adopted in representation learning on graphs [ 2–4 ] . However , existing works only utilize a small subset of nodes . To fully utilize a large amount of unlabeled data , recent studies for semi-supervised learning have emerged inspired by semisupervised frameworks in other domains [ 19 , 20 ] . GraphMix [ 18 ] is a regularization method based on semi-supervised learning by linear interpolation between two data on graphs , and SSL [ 21 ] proposes self-supervised learning strategies to exploit available information from graph structure . BVAT [ 22 ] promotes the smoothness of GNNs by generating virtual adversarial perturbations . Likewise , we follow semi-supervised strategy to leverage unlabeled data while considering sequentially generated samples from our augmentation . Data Augmentation on Graphs . Data augmentation is an effective technique to improve generalization by increasing the diversity of data . It is becoming the de facto necessity for model training to employ simple data augmentation ( e.g. , image rotation , flipping , translation , and so on ) . Despite the effectiveness of data augmentation , few approaches have been explored in graph domain due to its non-Euclidean nature and dependencies between data samples . Simple approaches exist such as DropEdge [ 23 ] to randomly remove a certain number of edges and AdaEdge [ 16 ] to adaptively control the inter-class/intra-class edges . Similarly , a method to propagate the perturbed node features by randomly dropping on a node-based was proposed in [ 24 ] . GAug [ 17 ] proposes the neural edge predictors as an augmentation module . Unlike existing methods employing simple perturbation [ 23 ] or extra augmentor model [ 17 , 25 ] , we propose the sampling-based augmentation , where a sequence of augmented samples are drawn from the explicitly designed target distribution for augmentation . 3 Method . We present a novel data augmentation framework for graph-structured data via Metropolis-Hastings algorithm . MH-Aug is a sampling-based augmentation , where a sequence of augmented samples are drawn from the explicit target distribution that enables flexible control of strength and diversity of augmentation . The overall sampling process of MH-Aug is described in Figure 1 . In this section , we first summarize the basics for our framework and delineate the components of MH-Aug. Then , we outline the training procedure with proposed consistency regularizers for semi-supervised learning . Lastly , we theoretically prove the distribution of augmented samples by MH-Aug converges to the desired target distribution . 3.1 Preliminaries . Ego-graph , Gi . A graph is denoted as G = ( V , E ) , where V and E are the sets of nodes and edges . A k-hop ego-graph Gi [ 26 ] is a subgraph of G centered at node vi ∈ V , consisting of neighbors within k hops from node vi and all edges between the neighbors including vi . In other words , the k-hop ego-graph of a node vi is defined as Gi = ( Vi , Ei ) , Vi = { u|S ( u , vi ) ≤ k , u ∈ V } , Ei = { ( u , v ) | ( u , v ) ∈ E and u , v ∈ Vi } , where S ( u , v ) is the length of the shortest path between nodes u and v. In this paper , we do not explicitly specify k for ego-graphs since 2-hop ego-graphs are used in all experiments . Change ratio of graph , ∆G′ . The change ratio of graph G to G′ = ( V ′ , E ′ ) is measured by the number of added/deleted edges ( or nodes ) divided by the number of original edges ( or nodes ) , i.e. , ∆G′E = ( |E ′−E|+|E−E ′| ) /|E| and ∆G′V = ( |V ′−V|+|V−V ′| ) /|V| . Since , in this work , we consider only subgraphs of the original input graph as augmented samples , which is similar to DropEdge [ 23 ] and DropNode [ 24 ] , the change ratio can be equivalently written as ∆G′E = 1 − |E ′|/|E| and ∆G′V = 1− |V ′|/|V| . Thereby ∆G′E and ∆G′V are always ranged in [ 0 , 1 ] . Metropolis-Hastings ( MH ) algorithm . MH algorithm is a Markov chain Monte Carlo method to draw random samples from a target distribution when direct sampling is difficult [ 27 ] . The algorithm comprises three components : the target distribution P , the proposal distribution Q , and the acceptance ratio A . The MH algorithm iteratively draws samples from the target distribution P being only dependent on the current sample . The MH algorithm uses a proposal distribution Q to draw a candidate sample and evaluates the acceptance ratio A to decide whether to accept or reject the candidate sample . The accepted samples by the MH algorithm follow the target distribution P . 3.2 Metropolis-Hastings Data Augmentation . Our objective is to sample the augmented graph G′ from the target distribution P given the original graph G and can be written as G′ ∼ P ( G′ ; G ) . ( 1 ) Since direct sampling from the target distribution P is challenging , we propose a novel data augmentation method based on Metropolis-Hastings algorithm . Target Distribution . We design the target distribution P to control the strength and diversity of augmentation for effective learning . The strength and diversity can be discussed from two perspectives : a full graph and ego-graphs . In our framework , the strength of augmentation is measured by the change ratio of ego-graphs , i.e. , ∆G′i , since most existing GNNs with k-layers learn node representations based on their k-hop ego-graphs . On the other hand , the diversity of augmentation is controlled by ∆G′ and ∆G′i from both full graph and ego-graph perspectives . The diversity of augmentation is adaptively adjusted for each ego-graph by the standard deviation σ ( i ) that is a simple linear function of the entropy i of the prediction at node vi . Given the expected strength µE ∈ R and ego-graph level diversity σE ( i ) ∈ R , the target distribution P w.r.t . edges is given as follows : PE ( G′ ) ∝ |V|∏ i exp ( − ( ∆G′i , ( E ) − µE ) 2 2 { σE ( i ) } 2 ) λ1 · 1 ( |E| |E|·∆G′E ) λ2 , ( 2 ) where ∆G′i , ( E ) is the change ratio of G ′ i w.r.t . E and λs are hyperparamters for controlling the influence of the two components . To have various full graph change ratios ∆G′E , the normalization by the number of possible augmented graphs corresponding to the same change ratio , ( |E| |E|·∆G′E ) , is necessary . As the size of graph increases , without the normalization , it becomes extremely difficult to generate augmented samples with a low ( or high ) full graph change ratio . For more details , see Section 4.2 . Similarly , The target distribution P with respect to nodes can be written as follows : PV ( G′ ) ∝ |V|∏ i exp ( − ( ∆G′i , ( V ) − µV ) 2 2 { σV ( i ) } 2 ) λ3 · 1 ( |V| |V|·∆G′V ) λ4 , ( 3 ) where ∆G′i , ( V ) is the change ratio of G ′ i w.r.t . the nodes , V . With combining the two distributions , the overall target distribution is defined as : P ( G′ ) = PE ( G′ ) · PV ( G′ ) . ( 4 ) In our experiment , unlike the change ratio of the full graph ∆G′E ( and ∆G′V ) , we define the ego-graph change ratio ∆G′i , ( E ) ( and ∆G ′ i , ( V ) ) with the change of the number of received messages from k-hop ego-graphs . Figure 2 illustrates the calculation of ∆G′i , ( E ) regarding two different cases : ( b ) dropping distant ( 2-hop ) edges and ( c ) dropping near ( 1-hop ) edges . In this definition , even if the number of dropped edges is the same , dropping edges connecting nodes closer to the center node vi leads to a larger ∆G′i , ( E ) than the case of distant nodes ( 0.6 > 0.13 ) , which can be regarded as a much stronger augmentation . It indicates that the amount of received messages depends on not only the number of removed edges but also which edges are dropped . This structural property can only be properly handled from ego-graph perspective . In practice , this definition allows time and memory efficient implementation using matrix multiplications as ∆G′i , ( E ) = 1− ( Ã′k1 ) i ( Ãk1 ) i , and ∆G′i , ( V ) = 1− ( Ãkm ) i ( Ãk1 ) i , ( 5 ) where Ã and Ã′ are adjacency matrices of the original graph G and the current graph G′ , where both graphs include a self-connection for every node , 1 ∈ R|V| is a vector of ones , and m ∈ R|V| is a mask vector for DropNode . Proposal Distribution . For efficient sampling and a theoretical guarantee of convergence to the target distribution , a proposal distribution is crucial . A proposal distribution Q ( G′|G ( t ) ) suggests a candidate augmented sample G′ , given the current sample G ( t ) . To draw diversely augmented graphs with various edge/node change ratios ∆G′E and ∆G′V , a candidate augmented sample is generated by three steps : 1 ) change ratio sampling , 2 ) graph modification and 3 ) merging . We first independently sample change ratios ∆G′ ( · ) for edges and nodes from Gaussian distributions truncated to the range [ 0 , 1 ] given mean ∆G ( t ) ( · ) and standard deviation σ∆ , ( · ) . Then , we modify the original graph G to generate augmented samples G′E and G′V , which can be viewed as a uniform sampling from all possible augmented graphs with ∆G′E and ∆G′V respectively . Finally , the two graphs G′E and G′V are merged to construct the candidate augmented sample G′ . Formally , the proposal distribution is given by : Q ( G′|G ( t ) ) ∝ φ ( ξ ′ E ) Φ ( βE ) − Φ ( αE ) · φ ( ξ ′ V ) Φ ( βV ) − Φ ( αV ) · 1 ( |E| |E|·∆G′E ) · 1 ( |V| |V|·∆G′V ) , ( 6 ) where α ( · ) = a−∆G ( t ) ( · ) σ∆ , β ( · ) = b−∆G ( t ) ( · ) σ∆ , ξ′ ( · ) = ∆G′ ( · ) −∆G ( t ) ( · ) σ∆ , φ ( x ) = 1√ 2π exp ( − 12x 2 ) as the probability density function of the standard normal distribution and Φ ( x ) = 12 ( 1 + erf ( x√ 2 ) ) as its cumulative distribution function . a and b represent the extremes of Gaussian distribution . Since the change ratio should be ranged in [ 0 , 1 ] , a is 0 and b is 1 . In ( 6 ) , the first and second terms denote the likelihood of the change ratios ∆G′E and ∆G′V given ∆G ( t ) . The third and fourth terms are for the probability of a sample with ∆G′E and ∆G′V . Acceptance Ratio . Starting with the original graph G , MH-Aug draws the candidate graph G′ from the proposal distribution Q . Then , with an acceptance ratio A , MH-Aug decides whether to accept or reject the candidate G′ . A is given as : A = min { 1 , P ( G′ ) Q ( G ( t ) |G′ ) P ( G ( t ) ) Q ( G′|G ( t ) ) } . ( 7 ) The computation of A with target distribution in ( 4 ) and proposal distribution in ( 6 ) is described in the supplement . MH-Aug generates a sequence of augmented graphs { G ( t ) } 0≤t≤T , where T is the number of accepted samples .

Summary: The paper proposes a new data augmentation method for Graph Neural Networks based on Markov Chain Monte Carlo sampling and in particular the Metropolis-Hastings algorithm in the context of semi-supervised learning. The main intuition of the paper is to define a target distribution for the whole graph which is representative of the changes in the number of messages that are exchanged in the k-hop neighborhood of each node and then apply the Metropolis-Hastings algorithm for obtaining samples that are representative of this distribution. The experimental section compares the proposed method (MH-Aug) to other recent data augmentation methods showing favorable results w.r.t. the baselines.