FLAG: Adversarial Data Augmentation for Graph Neural Networks

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have emerged as powerful architectures for learning and analyzing graph representations . The Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2016 ) and its variants have been applied to a wide range of tasks , including visual recognition ( Zhao et al. , 2019 ; Shen et al. , 2018 ) , meta-learning ( Garcia & Bruna , 2017 ) , social analysis ( Qiu et al. , 2018 ; Li & Goldwasser , 2019 ) , and recommender systems ( Ying et al. , 2018 ) . However , the training of GNNs on large-scale datasets usually suffers from overfitting , and realistic graph datasets often involve a high volume of out-of-distribution test nodes ( Hu et al. , 2020 ) , posing significant challenges for prediction problems . One promising solution to combat overfitting in deep neural networks is data augmentation ( Krizhevsky et al. , 2012 ) , which is commonplace in computer vision tasks . Data augmentations apply label-preserving transformations to images , such as translations and reflections . As a result , data augmentation effectively enlarges the training set while incurring negligible computational overhead . However , it remains an open problem how to effectively generalize the notion of data augmentation to GNNs . Transformations on images rely heavily on image structures , and it is challenging to design low-cost transformations that preserve semantic meaning for non-visual tasks like natural language processing ( Wei & Zou , 2019 ) and graph learning . Generally speaking , graph data for machine learning comes with graph structure ( or edge features ) and node features . In the limited cases where data augmentation can be done on graphs , it generally focuses exclusively on the graph structure by adding/removing edges ( Rong et al. , 2019 ) . To date , there is no study on how to manipulate graphs in node feature space for enhanced performance . In the meantime , adversarial data augmentation , which happens in the input feature space , is known to boost neural network robustness and promote resistance to adversarially chosen inputs ( Goodfellow et al. , 2014 ; Madry et al. , 2017 ) . Despite the wide belief that adversarial training harms standard generalization and leads to worse accuracy ( Tsipras et al. , 2018 ; Balaji et al. , 2019 ) , recently a growing amount of attention has been paid to using adversarial perturbations to augment datasets and ultimately alleviate overfitting . For example , Volpi et al . ( 2018 ) showed adversarial data augmentation is a data-dependent regularization that could help generalize to out-of-distribution samples , and its effectiveness has been verified in domains including computer vision ( Xie et al. , 2020 ) , language understanding ( Zhu et al. , 2019 ; Jiang et al. , 2019 ) , and visual question answering ( Gan et al. , 2020 ) . Despite the rich literature about adversarial training of GNNs for security purposes ( Zügner et al. , 2018 ; Dai et al. , 2018 ; Bojchevski & Günnemann , 2019 ; Zhang & Zitnik , 2020 ) , it remains unclear how to effectively and efficiently improve GNN ’ s clean accuracy using adversarial augmentation . Present work . We propose FLAG , Free Large-scale Adversarial Augmentation on Graphs , to tackle the overfitting problem . While existing literature focuses on modifying graph structures to augment datasets , FLAG works purely in the node feature space by adding gradient-based adversarial perturbations to the input node features with graph structures unchanged . FLAG leverages “ free ” methods ( Shafahi et al. , 2019 ) to conduct efficient adversarial training so that it is highly scalable on large-scale datasets . We verify the effectiveness of FLAG on the Open Graph Benchmark ( OGB ) ( Hu et al. , 2020 ) , which is a collection of large-scale , realistic , and diverse graph datasets for both node and graph property prediction tasks . We conduct extensive experiments across OGB datasets by applying FLAG to prestigious GNN models , which are GCN , GraphSAGE , GAT , and GIN ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Veličković et al. , 2017 ; Xu et al. , 2019 ) and show that FLAG brings consistent and significant improvements . For example , FLAG lifts the test accuracy of GAT on ogbn-products by an absolute value of 2.31 % . DeeperGCN ( Li et al. , 2020 ) is another strong baseline that achieves top performance on several OGB benchmarks . FLAG enables DeeperGCN to generalize further and reach new state-of-the-art performance on ogbg-molpcba and ogbg-ppa . FLAG is simple ( adding just a dozen lines of code ) , general ( can be directly applied to any GNN model ) , versatile ( works in both transductive and inductive settings ) , and efficient ( able to bring salient improvement at tractable or even no extra cost ) . Our main contributions are summarized as follows : • We propose adversarial perturbations as a data augmentation in the input node feature space to efficiently boost GNN performance . The resulting FLAG framework is a scalable and flexible augmentation scheme for GNN , which is easy to implement and applicable to any GNN architecture for both node and graph classification tasks . • We advance the state-of-the-art on a number of large-scale OGB datasets , often by large margins . • We provide a detailed analysis and deep insights on the effects adversarial augmentation has on GNNs . 2 PRELIMINARIES . Graph Neural Networks ( GNNs ) . We denote a graph as G ( V , E ) with initial node features xv for v ∈ V and edge features euv for ( u , v ) ∈ E . GNNs are built on graph structures to learn representation vectors hv for every node v ∈ V and a vector hG for the entire graph G. The k-th iteration of message passing , or the k-th layer of GNN forward computation is : h ( k ) v = COMBINE ( k ) ( h ( k−1 ) v , AGGREGATE ( k ) ( { ( h ( k−1 ) v , h ( k−1 ) u , euv ) : u ∈ N ( v ) } ) ) , ( 1 ) where h ( k ) v is the embedding of node v at the k-th layer , euv is the feature vector of the edge between node u and v , N ( v ) is node v ’ s neighbor set , and h ( 0 ) v = xv . COMBINE ( · ) and AGGREGATE ( · ) are functions parameterized by neural networks . To simplify , we view the holistic message passing pipeline as an end-to-end function fθ ( · ) built on graph G : H ( K ) = fθ ( X ; G ) , ( 2 ) whereX is the input node feature matrix . After K rounds of message passing we get the final-layer node matrix H ( K ) . To obtain the representation of the entire graph hG , the permutation-invariant READOUT ( · ) function pools node features from the final iteration K as : hG = READOUT ( { h ( K ) v | v ∈ V } ) , ( 3 ) Additionally from the spectral convolution point of view , the k-th layer of GCN is : I +D− 1 2AD− 1 2 → D̃− 12 ÃD̃− 12 , S = D̃− 12 ÃD̃− 12 , ( 4 ) H ( k+1 ) = σ ( SH ( k ) Θ ( k ) ) , ( 5 ) where H ( k ) is the node feature matrix of the k-th layer with H0 = X , Θk is the trainable weight matrix of layer k , and σ is the activation function . D and A denote the diagonal degree matrix and adjacency matrix , respectively . Here , we view S as a normalized adjacency matrix with self-loops added . Adversarial training . Standard adversarial training seeks to solve the min-max problem as : min θ E ( x , y ) ∼D [ max ‖δ‖p≤ L ( fθ ( x+ δ ) , y ) ] , ( 6 ) where D is the data distribution , y is the label , ‖ · ‖p is some ` p-norm distance metric , is the perturbation budget , and L is the objective function . Madry et al . ( 2017 ) showed that this saddlepoint optimization problem could be reliably tackled by Stochastic Gradient Descent ( SGD ) for the outer minimization and Projected Gradient Descent ( PGD ) for the inner maximization . In practice , the typical approximation of the inner maximization under an l∞-norm constraint is as follows , δt+1 = Π‖δ‖∞≤ ( δt + α · sign ( ∇δL ( fθ ( x+ δt ) , y ) ) ) , ( 7 ) where perturbation δ is updated iteratively , and Π‖δ‖∞≤ performs projection onto the -ball in the l∞-norm . For maximum robustness , this iterative updating procedure usually loops M times , which makes PGD computationally expensive . While there are M forward and backward steps within the process , θ gets updated just once using the final δM . 3 PROPOSED METHOD : FLAG . Adversarial training is a form of data augmentation . By hunting for and stamping out small perturbations that cause the classifier to fail , one may hope that adversarial training should be beneficial to standard accuracy ( Goodfellow et al. , 2014 ; Tsipras et al. , 2018 ; Miyato et al. , 2018 ) . With an increasing amount of attention paid to leverage adversarial training for better clean performance in varied domains ( Xie et al. , 2020 ; Zhu et al. , 2019 ; Gan et al. , 2020 ) , we conduct the first study on how to effectively generalize GNNs using adversarial data augmentation . Here we introduce FLAG , Free Large-scale Adversarial Augmentation on Graphs , to best exploit the power of adversarial augmentation . Note that our method differs from other augmentations for graphs in that it happens in the input node feature space . Augmentation for “ free ” . We leverage the “ free ” adversarial training method ( Shafahi et al. , 2019 ) to craft adversarial data augmentations . PGD is a strong but inefficient way to solve the inner maximization of ( 6 ) . While computing the gradient for the perturbation δ , free training simultaneously computes the model parameter θ ’ s gradient . This “ free ” parameter gradient is then used to compute the ascent step . The authors proposed to train on the same minibatch M times in a row to simulate the inner maximization in ( 6 ) , while compensating by performing M times fewer epochs of training . The resulting algorithm yields accuracy and robustness competitive with standard adversarial training , but with the same runtime as clean training . Gradient accumulation . When doing “ free ” adversarial training , the inner/adversarial loop is usually run M times , each time computing both the gradient for δt and θt−1 . Rather than updating the model parameters in each loop , Zhang et al . ( 2019 ) proposed to accumulate the gradients for θt−1 during the inner loop and applied them all at once during the outer/parameter update . The same idea was used by Zhu et al . ( 2019 ) , who proposed FreeLB to tackle this optimization issue on language understanding tasks . FreeLB ran multiple PGD steps to craft adversaries , and meanwhile accumulated the gradients ∇θL of model parameters . The gradient accumulation behavior can be approximated as optimizing the objective below : min θ E ( x , y ) ∼D [ 1 M M−1∑ t=0 max δt∈It L ( fθ ( x+ δt ) , y ) ] , ( 8 ) where It = Bx+δ0 ( αt ) ∩ Bx ( ) . The gradient accumulation algorithm largely empowers FLAG to further improve GNN with efficient gradient usage for optimization . Algorithm 1 FLAG : Free Large-scale Adversarial Augmentation on Graphs Require : Graph G = ( V , E ) ; input feature matrix X ; learning rate τ ; ascent steps M ; ascent step size α ; training epochs N ; forward function on graph fθ ( · ) denoted in ( 2 ) ; L ( · ) as objective function . We omit the READOUT ( · ) function in ( 3 ) for the inductive scenario here . 1 : Initialize θ 2 : for epoch = 1 . . . N do 3 : δ0 ← U ( −α , α ) . initialize from uniform distribution 4 : g0 ← 0 5 : for t = 1 . . .M do 6 : gt ← gt−1 + 1M · ∇θL ( fθ ( X + δt−1 ; G ) , y ) . θ gradient accumulation 7 : gδ ← ∇δL ( fθ ( X + δt−1 ; G ) , y ) 8 : δt ← δt−1 + α · gδ/ ‖gδ‖F . perturbation δ gradient ascent 9 : end for 10 : θ ← θ − τ · gM . model parameter θ gradient descent 11 : end for Unbounded attack . Usually on images , the inner maximization is a constrained optimization problem . The largest perturbation one can add is bounded by the hyperparameter , typically 8/255 under the l∞-norm . This encourages the visual imperceptibility of the perturbations , thus making defenses realistic and practical . However , graph node features or language word embeddings do not have such straightforward semantic meanings , which makes the selection of highly heuristic . In light of the positive effect of large perturbations on generalization ( Volpi et al. , 2018 ) , and also to simplify hyperparameter search , FLAG drops the projection step when performing the inner maximization . Note that , although the perturbation is not bounded by an explicit , it is still implicitly bounded in the furthest distance that δ can reach , i.e . the step size α times the number of ascending steps M . Biased perturbation for node classification . Conventional conv nets treat each test sample independently during inference , whereas this is not the case in transductive graph learning scenarios . When classifying one target node , messages from the whole k-hop neighborhood are aggregated and combined into its embedding . It is natural to believe that a further neighbor should have lower impact , i.e . higher smoothness , on the final decision of the target node , which can also be intuitively reflected by the message passing view of GNNs in ( 1 ) . To promote more invariance for furtheraway neighbors when doing node classification , we perturb unlabeled nodes with larger step sizes αu than αl for target nodes . We show the effectiveness of this biased perturbation in the ablation study section . The overall augmentation pipeline is presented in Algorithm 1 . Note that when doing transductive node classification , we use diverse step sizes αl and αu to craft adversarial augmentation for target and unlabeled nodes , respectively . In the following sections , we verify FLAG ’ s effectiveness through extensive experiments . In addition , we provide detailed discussions for a deep understanding of the effects of adversarial augmentation .

Summary: This paper proposes FLAG (Free Large-scale Adversarial Augmentation on Graphs), an adversarial data augmentation technique that can be applied to different GNN models in order to improve their generalization. The proposed technique consists on adding adversarial perturbations to the nodes’ features solving the standard min-max problem for adversarial training. In this setup, a noise vector is added to the input features which tries to maximize the loss by performing gradient ascent, while the classifier is trained to minimize the loss despite the added adversarial noise.