Graph Adversarial Self-Supervised Learning

1 Introduction . Learning effective representations of graph-structured data plays an essential role in a variety of real-world applications , including social , biological , molecules , and financial networks [ 1 ] . Recently , graph neural networks ( GNNs ) have emerged as powerful architectures for learning and analyzing graph representations [ 2 , 3 , 4 , 5 , 6 ] . GNNs typically learn graph representations in a supervised or semi-supervised setting . In practice , obtaining a large number of labels is often difficult or even impossible , especially in specific areas that are very costly , such as in biochemistry . The labeled graphs may be limited , while unlabeled graphs are easy to collect . Self-supervised learning utilizing unlabeled data has made significant progress in computer vision [ 7 , 8 , 9 , 10 , 11 , 12 , 13 ] and shows great potential in exploring unlabeled data to enhance graph deep learning [ 14 , 15 , 16 , 17 , 18 , 19 , 20 ] . Despite their success , existing self-supervised learning methods rely heavily on handcrafted view , where the view here refers to human-defined data transformations to preserve the invariance of their intrinsic properties . In recent years , researchers have designed views of graphs from various levels , including nodes dropping , edge perturbation , attribute masking , subgraph [ 15 ] , and graph diffusion ∗Corresponding authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 14 ] . However , the handcrafted views require expert knowledge and trial and error but also do not yield consistent performance gains across multiple tasks [ 15 ] . Therefore , how to automatically search for augmentations for graph data remains an open problem . GNNs are vulnerable to adversarial attacks , as are deep neural networks . Adversarial attacks usually exploit the gradient information to generate imperceptibly small perturbations that alter the model ’ s output . Adding these adversarial samples to the training set , i.e. , adversarial training , can improve the neural network to generalize to out-of-distribution samples [ 21 , 22 , 23 ] . Adversarial training usually leads to a trade-off between robustness and generalization . There has been much research on adversarial training for security purposes [ 24 ] , in particular , it is still unclear how to combine adversarial training in self-supervised learning of GNNs to improve the classification accuracy . In this paper , we are motivated to address the drawbacks mentioned above and propose a selfsupervised learning framework to train a graph neural network without any class labels . We refer to this novel adversarial self-supervised learning approach as Graph Adversarial Self-Supervised Learning ( GASSL ) . GASSL directly maximizes the similarity of a graph and its perturbed adversarial graph , relying on neither negative pairs nor handcrafted augmented views . In the training phase , we use the gradient accumulation strategy [ 25 , 21 ] to accelerate the model training . We verify the effectiveness of GASSL on 10 datasets for the graph classification task including the TU datasets [ 26 ] and the Open Graph Benchmark ( OGB ) [ 27 ] . We conduct extensive experiments across graph datasets by applying classical GNN models ( GCN [ 4 ] and GIN [ 5 ] ) as encoders . Our approach automatically generates challenging views to yield performance gains on multiple tasks compared to handcrafted views . The results show that our method outperforms state-of-the-art graph self-supervised learning and is close to the performance of the supervised GNNs . Our contribution could be summarized as : ( 1 ) We propose a self-supervised learning method GASSL for graph representation learning without human supervision . ( 2 ) We use adversarial training to automatically generate challenging views for self-supervised learning in place of handcrafted views , which yield performance gains on multiple datasets . ( 3 ) We show that GASSL consistently outperforms state-of-the-art self-supervised models with a significant margin in graph classification tasks . When compared to supervised baselines , GASSL performs on par with or superior to the strong baselines . 2 Related work . Graph neural network ( GNN ) GNN is built on graph structures to learn representation vector Hv for each node v ∈ V , which are formalized as the following function : H ( k+1 ) v = COMBINE ( k ) ( H ( k ) v , AGGREGATE ( k ) ( { H ( k ) u , ∀u ∈ N ( v ) } ) ) , ( 1 ) where H ( k ) v is the embedding of node v at the k-th layer , N ( v ) denotes a neighbor set of node v , and H ( 0 ) v = Xv . COMBINE and AGGREGATE are functions parameterized by neural networks . After K rounds of message passing , we obtain the final-layer node representations . To obtain the representation of the entire graph hG ∈ Rd , we need the permutation-invariant READOUT function as follows : hG = READOUT ( { Hv | v ∈ V } ) . Various GNNs have been proposed [ 4 , 28 , 5 ] with various pooling [ 29 , 30 , 31 ] , achieving state-of-the-art performance in graph tasks . Adversarial robustness For adversarial training , more attention is paid to improving model robustness and less on improving generalization performance . For image classification tasks , combining a contrast learning framework with perturbation of the input samples is effective . CLAE [ 32 ] employs the Fast Gradient Sign Method ( FSGM ) , and RoCL [ 33 ] adopts the Projected Gradient Descent ( PGD ) method to improve the generalization performance of the model . Concurrent to our work , Tamkin et al . [ 34 ] proposed a model-agnostic network ( ViewMaker ) that perturbs the input by adding an ` p constraint to produce useful views and has successful applications on image , speech , and time-series data . Kong et al . [ 24 ] proposed to perturb the features of the input nodes of GNN for better generalization performance and utilized a gradient accumulation strategy to accelerate adversarial training in a supervised learning setting . Out of positive view , Hu et al . [ 35 ] proposed to directly learn a set of negative adversaries playing against the self-trained representation . Graph self-supervised learning Self-supervised learning has recently made new advances in graph representation learning , in which contrast learning [ 36 , 17 , 15 , 14 , 37 , 38 , 39 , 40 , 41 ] has achieved the state-of-the-art performance . Infograph [ 17 ] maximizes the mutual information between the graph-level representation and the representations of substructures of different scales ( e.g. , nodes , edges , triangles ) . By doing so , the graph-level representations encode aspects of the data shared across different scales of substructures . GraphCL [ 15 ] designed four types of graph augmentations to incorporate various priors , including node dropping , edge perturbation , attribute masking , and subgraph . MVGRL [ 14 ] utilized graph diffusion for graph augmentation and found no performance gain for more than two views or multi-scales of encoding . GCC [ 16 ] performs a random walk with a restart for each node to sample subgraph as augmentation . GRACE [ 40 ] adopt two augmentations , including removing edges and node feature masking . For a thorough review , we refer the reader to the recent survey [ 42 ] . However , one limitation shared by all these successful approaches is the handcrafted view , which is the primary goal of our GASSL – how to learn a view automatically without resorting to handcrafting or expert domain knowledge . 3 Methodology . We now show how to learn the representation of a graph without handcrafted views with domain expert knowledge . Before that , we will briefly introduce adversarial training in supervised learning . A graph can be represented as G = ( V , E ) , where V is the set of |V| = n nodes , the adjacency matrix A ∈ { 0 , 1 } n×n , along with the c dimensional node attribute X ∈ Rn×c . Our goal is to learn from multiple graphs in a dataset and predict the property of a single graph G. The learned d dimensional distributed representation hG ∈ Rd is applied for downstream tasks ( e.g . graph classification task ) . Adversarial robustness We start with the definition of adversarial attacks under supervised settings . Given a dataset D = ( X , Y ) , let x ∈ X and y ∈ Y denote a training sample and the corresponding label , respectively . Given a supervised learning model fθ : X → Y with parameters θ . Traditional adversarial attacks maximize the loss within a certain radius from the sample as follows : xi+1 = ΠB ( x , ) ( xi + αsign ( ∇xiLCE ( θ , xi , y ) ) ) ( 2 ) where B ( x , ) is the ` ∞ norm-ball around x with radius , and Π is the projection function for norm-ball , α is the step size , i is the attack iterations , sign ( · ) returns the sign of the vector , LCE is the cross-entropy loss for supervised training . The straightforward way to defend against adversarial attacks is to minimize the loss of adversarial samples . Authors [ 43 ] proposed to seek to find optimal parameters θ∗ to minimize the maximum risk for any δ within a norm ball as follows : min θ E ( x , y ) ∼D [ max δ∈B ( x , ) LCE ( θ , x+ δ , y ) ] ( 3 ) where δ is the perturbation of the adversarial example . The conventional adversarial attacks [ 43 , 44 ] require to have a class label y ∈ Y , which is not applicable to unlabeled data . 3.1 Self-supervised learning on graphs . Self-supervised learning typically design pretext tasks to bring different views of the same instance ( positive view ) closer and push views of different samples ( negative view ) farther apart . The simple and performant BYOL [ 9 ] does not need to maintain negative views explicitly and depends only on positive views . Inspired by BYOL , we propose GASSL framework ( Figure 3.1 ) to learn graph representation . GASSL comprises the two networks : the teacher and student networks . The two networks shared the same architecture with different parameters . In detail , the teacher network is defined by a set of weights θ , while the student network using a different set of weights ξ , which are an exponential moving average of parameters θ . Given a target decay rate β ∈ [ 0 , 1 ] , after each training step , we perform the following update , ξ ← βξ + ( 1 − β ) θ . Note that the predictor qθ is only applied to the teacher network to avoid collapse , leading to an asymmetric architecture . Encoders . In order to learn the graph representation z , we used GNN ( defined in ( 2 ) ) following with a two-layer multi-layer perceptron ( MLP ) as an encoder f ( · ) . Our framework allows various choices of the network architecture without any constraints . We opt for simplicity and adopt the commonly used GCN [ 4 ] and GIN [ 5 ] . Similarity loss Our method relies on the positive view of the input graph . The traditional way to produce positive view need to be designed manually , such as node dropping , edge perturbation , subgraph . We will introduce to learn a challenging positive view automatically in the next subsection . Here suppose we have obtained the positive view , denoted as G′ . We feed G′ and G to the teacher network and the student network , and obtained the output representations qθ ( z′θ ) and zξ , respectively . We then ` 2-normalized both qθ ( z′θ ) and zξ to q̄θ ( z ′ θ ) = qθ ( z ′ θ ) /||qθ ( z′θ ) ||2 and z̄ξ = zξ/||zξ||2 . We define the mean squared error as follows , Lθ , ξ = ‖q̄θ ( z′θ ) − z̄ξ‖22 = 2− 2 · 〈qθ ( z′θ ) , zξ〉 ‖qθ ( z′θ ) ‖2 · ‖zξ‖2 ( 4 ) We symmetrize the loss Lθ , ξ by separately feeding G and G′ to student network and teacher network to compute L̂θ , ξ . At each training step , we optimize the loss as follows , LGASSLθ , ξ = Lθ , ξ + L̂θ , ξ . ( 5 )

Summary: This paper presents a self-supervised learning method GASSL for graph representation learning. The authors have designed 2 key points:  a) Utilizing BYOL framework to learn graph representation; b) Using adversarial training to automatically generate challenging views for self-supervised learning. The idea seems to be simple but effective. Sufficient experimental analyses are conducted.