Intrusion-Free Graph Mixup

1 INTRODUCTION . Graph Neural Networks ( GNNs ) ( Kipf & Welling , 2017 ; Veličković et al. , 2018 ; Xu et al. , 2019 ) have recently shown its profound successes in many challenging applications , including predicting molecule properties for drug and material discovery ( Gilmer et al. , 2017 ; Wu et al. , 2018 ) , forecasting protein functions for biological networks ( Alvarez & Yan , 2012 ; Jiang et al. , 2017 ) , and estimating circuit functionality in modern circuit design ( Zhang et al. , 2019 ) . Nevertheless , like other successfully deployed deep networks such as those for image classification ( Krizhevsky et al. , 2012 ) , speech recognition ( Graves et al. , 2013 ) and machine translation ( Sutskever et al. , 2014 ; Bahdanau et al. , 2014 ) , GNNs are also suffering from the data-hungry issue due to their high modeling freedom . Consequently , researchers have been actively seeking effective regularization techniques for GNNs , aiming to power their learning but to avoid over-smoothing ( Li et al. , 2018 ; Wu et al. , 2019 ) and over-fitting ( Goodfellow et al. , 2016 ; Zhang et al. , 2021 ) for better model generalization . To this end , data augmentation schemes for regularizing GNNs mostly involve edge and node manipulation ( e.g. , deletion and addition ) on a single graph ( Rong et al. , 2020 ; Zhou et al. , 2020 ; You et al. , 2020 ) . In this paper , we look into a very successful pairwise data augmentation technique for image recognition ( Zhang et al. , 2018a ; Verma et al. , 2018 ; Guo et al. , 2019a ; Kim et al. , 2020 ) and natural text classification ( Guo et al. , 2019b ; Guo , 2020 ; Jindal et al. , 2020 ) , called Mixup . Mixup was originally introduced by Zhang et al . ( 2018a ) as an interpolation-based regularizer for image classification . It regularizes the learning of deep classification models by training with synthetic samples , which are created by linearly interpolating a pair of randomly selected training samples as well as their modeling targets . Neverthe- less , unlike images or natural sentences , which embrace a grid or linear sequence format , graph data have arbitrary structure and topology , which play a critical role on the semantics of a graph , and consequently even simply removing or adding one edge can dramatically change the semantic meaning of a graph . As a result , mixing a graph pair is challenging and may naturally cause the manifold intrusion issue in Mixup as identified by Guo et al . ( 2019a ) . Manifold intrusion results from conflicts between the synthetic labels of the mixed-up samples or between the synthetic labels and the labels of the original training data . For example , consider the graph on the left of Figure 1 , and its application of the popular graph perturbation action of node and edge deletion ( gray lines in the figure ) . In this case , the resulting two graphs in the middle will have the same structure as the two on the right from the original training set , but with different labels ( indicated by the color bars under the graphs ) . Note : an illustration of manifold intrusion when mixing two graphs is presented in A.5 . As discussed in ( Guo et al. , 2019a ) , when such intrusions occur , regularization using these synthetic graphs will contradict with the original training data . This essentially induces a form of under-fitting , resulting in the degradation of the model performance . To address the aforementioned challenge , we propose the first input mixing schema for Mixup on graph learning , coined ifMixup ( intrusion-free Mixup ) . As illustrated in Figure 2 , ifMixup first samples a random graph pair ( top subfigure ) from the training data . For a given mixing ratio λ sampled from a Beta distribution , ifMixup then creates a synthetic graph ( bottom subfigure ) , for each sample pair . ifMixup treats a graph pair as two sets of ordered nodes in the same size ( middle subfigure , where the unfilled node depicts an added dummy node ) . As a result , it can linearly interpolate the node features and the edges of the input pair . The newly generated graphs , which have a much larger number of graphs with changing local neighborhood properties than the original training dataset , are then used for training to regularize the GNNs learning . Theoretically , we prove that our mixing strategy can recover the source graphs from the mixed graph , and such invertibility property guarantees that the mixed graphs are intrusion free . That is , our method eliminates the possibility for the graphs resulting from mixing to coincide with any other graph in the training set or with a mixed graph from any other graph pairs . Experimentally , we show , using eight benchmarking tasks from different domains , that our strategy effectively regularizes the graph classification to improve its predictive performance , outperforming popular graph augmentation approaches and existing pair-wise graph mixing methods 1 . Our key contributions are as follows : 1 ) To the best of our knowledge , we are the first to propose an input Mixup for graph classification . 2 ) We prove that our Mixup schema is free of manifold intrusion , which can significantly degrade a Mixup-like model ’ s predictive accuracy . 3 ) We obtain the SOTA performance among popular graph perturbation baselines . 2 RELATED WORK . GNNs have been shown to be very effective for graph classification in a variety of domains ( Kipf & Welling , 2017 ; Ying et al. , 2018 ; Veličković et al. , 2018 ; Klicpera et al. , 2019 ; Xu et al. , 2019 ; Bianchi et al. , 2020 ) . One of the key challenges of these successes is to leverage strong regularization techniques such as data augmentation to regularize those GNNs models with high modeling freedom . Nonetheless , data augmentation is still a less touched area in graph data due to the arbitrary structure and topology . Most of the graph augmentation strategies are for node classification tasks , and heavily focus on perturbing nodes and edges in one given graph ( Hamilton et al. , 2017 ; Zhang et al. , 2018b ; Rong et al. , 2020 ; Chen et al. , 2020 ; Zhou et al. , 2020 ; Qiu et al. , 2020 ; You et al. , 2020 ; Wang et al . ; Fu et al. , 2020 ; Wang et al. , 2020 ; Song et al. , 2021 ; Zhao et al. , 2021 ) . For example , DropEdge ( Rong et al. , 2020 ) randomly removes a set of edges of a given graph . GAUG ( Zhao et al. , 2021 ) learns to perturb graph edges for node classification . DropNode , representing node 1Our PyTorch code will be released upon the acceptance of the paper . sampling based methods ( Hamilton et al. , 2017 ; Chen et al. , 2018 ; Huang et al. , 2018 ) , samples a set of nodes from a given graph . Unlike these approaches , our proposed strategy leverages a pair of graphs , instead of one graph , to augment the learning of graph level classification . Despite its great success in augmenting data for image recognition and text processing ( Zhang et al. , 2018a ; Verma et al. , 2018 ; Guo et al. , 2019a ; Guo , 2020 ; Jindal et al. , 2020 ; Kim et al. , 2020 ) , Mixup has been less explored for graph learning . To the best of our knowledge , there are two methods that apply Mixup to GNNs . GraphMix ( Verma et al. , 2019 ) leverages the idea of mixing on the embedding layer , with an additional fully-connected network to share parameters with the GNNs , for graph node classification in semi-supervised learning . MixupGraph ( Wang et al. , 2021 ) also leverages a simple way to avoid dealing with the arbitrary structure in the input space for mixing a graph pair , through mixing the graph representation resulting from passing the graph through the GNNs . Our paper here introduces the first input mixing method for Mixup to augment training data for graph classification . Furthermore , our method guarantees that the mixed graphs are manifold intrusion free for Mixup . 3 INTRUSION-FREE GRAPH MIXING . 3.1 GRAPH CLASSIFICATION AND MIXUP INTERPOLATION . Graph Classification In the graph classification problem we consider , the input G is a graph labelled with node features , or a “ node-featured graph ” . Specifically , assume that the graph has n nodes , each identified with an integer ID in [ n ] : = { 1 , 2 , . . . , n } . The set of edges E of the graph is a collection of unordered pairs ( i , j ) ’ s of node-IDs , as the current paper considers only undirected graphs ( although there is no difficulty to extend the setting to directed graphs ) . Associated with node i , there is a feature vector v ( i ) of dimension d. We will use v to denote the collection of all feature vectors and one may simply regards v as a matrix of dimension n× d. Thus , each input node featured graph G is essentially specified via the pair ( v , E ) . The output y is a class label in finite set Y : = { 1 , 2 , . . . , C } , which will be expressed as a ( one-hot ) probability vector over the label set Y . The classification problem is to find a mapping that predicts the label y for a node-featured graph G. The training data G of this learning task is a collection of such ( G , y ) pairs . Modern GNNs use the graph structure and node features to learn a distributed vector to represent a graph . The learning follows the “ message passing ” mechanism for neighborhood aggregation . It iteratively updates the embedding of a node hv by aggregating representations/embeddings of its neighbors . The entire graph representation hG is then obtained through a READOUT function , which aggregates embeddings from all nodes of the graph . Formally , representation hki of node i at the k-th layer of a GNN is defined as : hki = AGGREGATE ( h k−1 i , h k−1 j |j ∈ N ( i ) , W k ) , ( 1 ) where W k denotes the trainable weights at layer k , N ( i ) denotes the set of all nodes adjacent to i , and AGGREGATE is an aggregation function implemented by the specific GNN model ( popular ones include Max , Mean , Summation pooling operations ) , and h0i is typically initialized as the input node feature v ( i ) . The graph representation hG aggregates node representations hv using the READOUT graph pooling function : hG = READOUT ( hki |i ∈ [ n ] ) . ( 2 ) This graph representation is then mapped to label y using a standard classification network ( for example , a softmax layer ) . Mixup Interpolation Mixup was introduced by ( Zhang et al. , 2018a ) as an interpolation-based regularizer for image classification . It regularizes the learning of deep classification models by training with synthetic samples , which are created by linearly interpolating a pair of randomly selected training samples as well as their modeling targets . In detail , let ( xA , yA ) and ( xB , yB ) be two training instances , in which xA and xB refer to the input images and yA and yB refer to their corresponding labels . For a randomly chosen such training pair , Mixup generates a synthetic sample as follows . x̃ = λxA + ( 1− λ ) xB , ( 3 ) ỹ = λyA + ( 1− λ ) yB , ( 4 ) where λ is a scalar mixing ratio , sampled from a Beta ( α , β ) distribution with hyper-parameters α and β . Such synthetic instances ( x̃ , ỹ ) ’ s are then used for training . Motivated by the effectiveness of Mixup in regularizing image classification models , we are naturally motivated to design a similar “ Mixup ” scheme for graph data , in particular , the node-featured graphs , as are the interest of this paper . When this is possible , we may use the synthetic instances ( G̃ , ỹ ) ’ s to learn the modele parameter θ by minimizing the loss L : min θ E ( GA , yA ) ∼G , ( GB , yB ) ∼G , λ∼Beta ( α , β ) dL ( G̃ , ỹ ) e. ( 5 ) To mix ( GA , yA ) and ( GB , yB ) , it is straight-forward to apply Equation ( 4 ) to obtain the mixed label ỹ . The key question of investigation is how to mix GA and GB to obtain G̃ . It is worth noting that , unlike images or natural sentences which embrace a rigid structure of spatial coordinates or time axis , the underlying “ coordinate system ” of graph data may have different and arbitrary topology across different instances . Consequently even simply deleting or adding one edge can invalid the semantic meaning of a graph . One simple way to avoid dealing with the arbitrary structure in the input space for mixing a graph pair is to mix their fixed-size graph representation that results from the READOUT function as depicted in Equation 2 , namely mixing the two graphs by G̃ = λhGA + ( 1− λ ) hGB , as proposed by Wang et al . ( 2021 ) . We here propose to directly mix the graph inputs with arbitrary sizes for Mixup . Our mixing strategy can recover the source graphs from the mixed graph , and such invertibility guarantees that the mixed graphs are free of manifold intrusion ( Guo et al. , 2019a ) , which can cause severe underfitting for Mixup learning .

Summary: This paper shows a very simple technique for performing input-space mixup on graphs, in which the vertices are put into an arbitrary order and dummy vertices are added to make the two graphs being mixed have the same number of vertices.  Mixing is then performed on the vertices and edges.  The improvements are excellent and seem to be SOTA on all benchmarks considered, often by nice margins.    Detailed Notes from reading paper:    -Mixing on graphs.  Theoretical guarantee that mixed graphs avoid intrusions.     -Example of intrusion on mixed graphs is interesting, simple, and logical.     -Proposed algorithm is "Intrusion-Free Mixup" ifMixup.     -Edges and nodes take interpolated values.  Labels are interpolated based on edges.     -Can recover source graphs from mixed graphs.     -Task is to convert each graph into a single class (as opposed to each node having a class label).     -Each node is assigned a distinct index, dummy nodes are created to make each "graph" same size.  Dummy nodes have no edges.     -First construct "soft" edges by mixing edges between two graphs.  Network needs to be able to take soft edges.     -Proof that this graph mixing operation is invertible.     -ifMixup also shows benefit from more depth, as compared to baseline (which is probably badly overfitting).