Intrusion-Free Graph Mixup

1 INTRODUCTION . Graph Neural Networks ( GNNs ) ( Kipf & Welling , 2017 ; Veličković et al. , 2018 ; Xu et al. , 2019 ) have recently shown its profound successes in many challenging applications , including predicting molecule properties for drug and material discovery ( Gilmer et al. , 2017 ; Wu et al. , 2018 ) , forecasting protein functions for biological networks ( Alvarez & Yan , 2012 ; Jiang et al. , 2017 ) , and estimating circuit functionality in modern circuit design ( Zhang et al. , 2019 ) . Nevertheless , like other successfully deployed deep networks such as those for image classification ( Krizhevsky et al. , 2012 ) , speech recognition ( Graves et al. , 2013 ) and machine translation ( Sutskever et al. , 2014 ; Bahdanau et al. , 2014 ) , GNNs are also suffering from the data-hungry issue due to their high modeling freedom . Consequently , researchers have been actively seeking effective regularization techniques for GNNs , aiming to power their learning but to avoid over-smoothing ( Li et al. , 2018 ; Wu et al. , 2019 ) and over-fitting ( Goodfellow et al. , 2016 ; Zhang et al. , 2021 ) for better model generalization . To this end , data augmentation schemes for regularizing GNNs mostly involve edge and node manipulation ( e.g. , deletion and addition ) on a single graph ( Rong et al. , 2020 ; Zhou et al. , 2020 ; You et al. , 2020 ) . In this paper , we look into a very successful pairwise data augmentation technique for image recognition ( Zhang et al. , 2018a ; Verma et al. , 2018 ; Guo et al. , 2019a ; Kim et al. , 2020 ) and natural text classification ( Guo et al. , 2019b ; Guo , 2020 ; Jindal et al. , 2020 ) , called Mixup . Mixup was originally introduced by Zhang et al . ( 2018a ) as an interpolation-based regularizer for image classification . It regularizes the learning of deep classification models by training with synthetic samples , which are created by linearly interpolating a pair of randomly selected training samples as well as their modeling targets . Neverthe- less , unlike images or natural sentences , which embrace a grid or linear sequence format , graph data have arbitrary structure and topology , which play a critical role on the semantics of a graph , and consequently even simply removing or adding one edge can dramatically change the semantic meaning of a graph . As a result , mixing a graph pair is challenging and may naturally cause the manifold intrusion issue in Mixup as identified by Guo et al . ( 2019a ) . Manifold intrusion results from conflicts between the synthetic labels of the mixed-up samples or between the synthetic labels and the labels of the original training data . For example , consider the graph on the left of Figure 1 , and its application of the popular graph perturbation action of node and edge deletion ( gray lines in the figure ) . In this case , the resulting two graphs in the middle will have the same structure as the two on the right from the original training set , but with different labels ( indicated by the color bars under the graphs ) . Note : an illustration of manifold intrusion when mixing two graphs is presented in A.5 . As discussed in ( Guo et al. , 2019a ) , when such intrusions occur , regularization using these synthetic graphs will contradict with the original training data . This essentially induces a form of under-fitting , resulting in the degradation of the model performance . To address the aforementioned challenge , we propose the first input mixing schema for Mixup on graph learning , coined ifMixup ( intrusion-free Mixup ) . As illustrated in Figure 2 , ifMixup first samples a random graph pair ( top subfigure ) from the training data . For a given mixing ratio λ sampled from a Beta distribution , ifMixup then creates a synthetic graph ( bottom subfigure ) , for each sample pair . ifMixup treats a graph pair as two sets of ordered nodes in the same size ( middle subfigure , where the unfilled node depicts an added dummy node ) . As a result , it can linearly interpolate the node features and the edges of the input pair . The newly generated graphs , which have a much larger number of graphs with changing local neighborhood properties than the original training dataset , are then used for training to regularize the GNNs learning . Theoretically , we prove that our mixing strategy can recover the source graphs from the mixed graph , and such invertibility property guarantees that the mixed graphs are intrusion free . That is , our method eliminates the possibility for the graphs resulting from mixing to coincide with any other graph in the training set or with a mixed graph from any other graph pairs . Experimentally , we show , using eight benchmarking tasks from different domains , that our strategy effectively regularizes the graph classification to improve its predictive performance , outperforming popular graph augmentation approaches and existing pair-wise graph mixing methods 1 . Our key contributions are as follows : 1 ) To the best of our knowledge , we are the first to propose an input Mixup for graph classification . 2 ) We prove that our Mixup schema is free of manifold intrusion , which can significantly degrade a Mixup-like model ’ s predictive accuracy . 3 ) We obtain the SOTA performance among popular graph perturbation baselines . 2 RELATED WORK . GNNs have been shown to be very effective for graph classification in a variety of domains ( Kipf & Welling , 2017 ; Ying et al. , 2018 ; Veličković et al. , 2018 ; Klicpera et al. , 2019 ; Xu et al. , 2019 ; Bianchi et al. , 2020 ) . One of the key challenges of these successes is to leverage strong regularization techniques such as data augmentation to regularize those GNNs models with high modeling freedom . Nonetheless , data augmentation is still a less touched area in graph data due to the arbitrary structure and topology . Most of the graph augmentation strategies are for node classification tasks , and heavily focus on perturbing nodes and edges in one given graph ( Hamilton et al. , 2017 ; Zhang et al. , 2018b ; Rong et al. , 2020 ; Chen et al. , 2020 ; Zhou et al. , 2020 ; Qiu et al. , 2020 ; You et al. , 2020 ; Wang et al . ; Fu et al. , 2020 ; Wang et al. , 2020 ; Song et al. , 2021 ; Zhao et al. , 2021 ) . For example , DropEdge ( Rong et al. , 2020 ) randomly removes a set of edges of a given graph . GAUG ( Zhao et al. , 2021 ) learns to perturb graph edges for node classification . DropNode , representing node 1Our PyTorch code will be released upon the acceptance of the paper . sampling based methods ( Hamilton et al. , 2017 ; Chen et al. , 2018 ; Huang et al. , 2018 ) , samples a set of nodes from a given graph . Unlike these approaches , our proposed strategy leverages a pair of graphs , instead of one graph , to augment the learning of graph level classification . Despite its great success in augmenting data for image recognition and text processing ( Zhang et al. , 2018a ; Verma et al. , 2018 ; Guo et al. , 2019a ; Guo , 2020 ; Jindal et al. , 2020 ; Kim et al. , 2020 ) , Mixup has been less explored for graph learning . To the best of our knowledge , there are two methods that apply Mixup to GNNs . GraphMix ( Verma et al. , 2019 ) leverages the idea of mixing on the embedding layer , with an additional fully-connected network to share parameters with the GNNs , for graph node classification in semi-supervised learning . MixupGraph ( Wang et al. , 2021 ) also leverages a simple way to avoid dealing with the arbitrary structure in the input space for mixing a graph pair , through mixing the graph representation resulting from passing the graph through the GNNs . Our paper here introduces the first input mixing method for Mixup to augment training data for graph classification . Furthermore , our method guarantees that the mixed graphs are manifold intrusion free for Mixup . 3 INTRUSION-FREE GRAPH MIXING . 3.1 GRAPH CLASSIFICATION AND MIXUP INTERPOLATION . Graph Classification In the graph classification problem we consider , the input G is a graph labelled with node features , or a “ node-featured graph ” . Specifically , assume that the graph has n nodes , each identified with an integer ID in [ n ] : = { 1 , 2 , . . . , n } . The set of edges E of the graph is a collection of unordered pairs ( i , j ) ’ s of node-IDs , as the current paper considers only undirected graphs ( although there is no difficulty to extend the setting to directed graphs ) . Associated with node i , there is a feature vector v ( i ) of dimension d. We will use v to denote the collection of all feature vectors and one may simply regards v as a matrix of dimension n× d. Thus , each input node featured graph G is essentially specified via the pair ( v , E ) . The output y is a class label in finite set Y : = { 1 , 2 , . . . , C } , which will be expressed as a ( one-hot ) probability vector over the label set Y . The classification problem is to find a mapping that predicts the label y for a node-featured graph G. The training data G of this learning task is a collection of such ( G , y ) pairs . Modern GNNs use the graph structure and node features to learn a distributed vector to represent a graph . The learning follows the “ message passing ” mechanism for neighborhood aggregation . It iteratively updates the embedding of a node hv by aggregating representations/embeddings of its neighbors . The entire graph representation hG is then obtained through a READOUT function , which aggregates embeddings from all nodes of the graph . Formally , representation hki of node i at the k-th layer of a GNN is defined as : hki = AGGREGATE ( h k−1 i , h k−1 j |j ∈ N ( i ) , W k ) , ( 1 ) where W k denotes the trainable weights at layer k , N ( i ) denotes the set of all nodes adjacent to i , and AGGREGATE is an aggregation function implemented by the specific GNN model ( popular ones include Max , Mean , Summation pooling operations ) , and h0i is typically initialized as the input node feature v ( i ) . The graph representation hG aggregates node representations hv using the READOUT graph pooling function : hG = READOUT ( hki |i ∈ [ n ] ) . ( 2 ) This graph representation is then mapped to label y using a standard classification network ( for example , a softmax layer ) . Mixup Interpolation Mixup was introduced by ( Zhang et al. , 2018a ) as an interpolation-based regularizer for image classification . It regularizes the learning of deep classification models by training with synthetic samples , which are created by linearly interpolating a pair of randomly selected training samples as well as their modeling targets . In detail , let ( xA , yA ) and ( xB , yB ) be two training instances , in which xA and xB refer to the input images and yA and yB refer to their corresponding labels . For a randomly chosen such training pair , Mixup generates a synthetic sample as follows . x̃ = λxA + ( 1− λ ) xB , ( 3 ) ỹ = λyA + ( 1− λ ) yB , ( 4 ) where λ is a scalar mixing ratio , sampled from a Beta ( α , β ) distribution with hyper-parameters α and β . Such synthetic instances ( x̃ , ỹ ) ’ s are then used for training . Motivated by the effectiveness of Mixup in regularizing image classification models , we are naturally motivated to design a similar “ Mixup ” scheme for graph data , in particular , the node-featured graphs , as are the interest of this paper . When this is possible , we may use the synthetic instances ( G̃ , ỹ ) ’ s to learn the modele parameter θ by minimizing the loss L : min θ E ( GA , yA ) ∼G , ( GB , yB ) ∼G , λ∼Beta ( α , β ) dL ( G̃ , ỹ ) e. ( 5 ) To mix ( GA , yA ) and ( GB , yB ) , it is straight-forward to apply Equation ( 4 ) to obtain the mixed label ỹ . The key question of investigation is how to mix GA and GB to obtain G̃ . It is worth noting that , unlike images or natural sentences which embrace a rigid structure of spatial coordinates or time axis , the underlying “ coordinate system ” of graph data may have different and arbitrary topology across different instances . Consequently even simply deleting or adding one edge can invalid the semantic meaning of a graph . One simple way to avoid dealing with the arbitrary structure in the input space for mixing a graph pair is to mix their fixed-size graph representation that results from the READOUT function as depicted in Equation 2 , namely mixing the two graphs by G̃ = λhGA + ( 1− λ ) hGB , as proposed by Wang et al . ( 2021 ) . We here propose to directly mix the graph inputs with arbitrary sizes for Mixup . Our mixing strategy can recover the source graphs from the mixed graph , and such invertibility guarantees that the mixed graphs are free of manifold intrusion ( Guo et al. , 2019a ) , which can cause severe underfitting for Mixup learning .

Summary: This manuscript develops a mixup strategy for graphs. Specifically, as claimed by the authors, the proposed method is manifold intrusion-free. This is achieved by mixing two input graph pairs (some dummy nodes are needed, see Figure2) based on the standard mixup formulation.