Interrogating Paradigms in Self-supervised Graph Representation Learning

1 INTRODUCTION . Analyzing graph-structured data is essential for many real-world applications and graph neural networks ( GNNs ) have emerged as a popular solution for challenging prediction tasks . However , these tasks often have limited labeled data due to prohibitive procuration costs and require models to possess strong generalization abilities to be practically useful . For example , in molecular property prediction , obtaining training labels requires expensive wet lab experiments and , at test time , models must predict properties for novel candidate molecules ( Hwang et al. , 2020 ; Duvenaud et al. , 2015 ; Zitnik et al. , 2018 ) . Unsupervised graph representation learning is a natural paradigm in such cases , where graph contrastive learning ( GCL ) is a promising approach over using pre-training tasks ( Hu et al. , 2020 ) . While recent findings ( Arora et al. , 2019 ; HaoChen et al. , 2021 ; Tian et al. , 2020 ; von Kügelgen et al. , 2021 ; Zimmermann et al. , 2021 ; Wang & Isola , 2020 ; Purushwalkam & Gupta , 2020 ) have investigated what makes for successful visual contrastive learning , a similar understanding remains lacking for GCL . Furthermore , in computer vision , reconstruction-based approaches using an autoencoder framework ( Kingma & Welling , 2014 ) , updated with modern encoder architectures and data augmentation , are emerging as an alternative paradigm that avoids negative sampling or large batch sizes ( Falcon et al. , 2021 ) . Reconstruction-based approaches ( Kipf & Welling , 2016 ) have not yet been similarly revisited with improved GNN architecture designs or augmentations for selfsupervised graph representation learning . Moreover , due to complications arising from the discrete , non-Euclidean nature of graph datasets , analysis from visual contrastive learning ( VCL ) can not be straightforwardly extended to a graph setting . Therefore , it remains unclear under what circumstances a reconstruction or CL-based approach is expected to perform well on a given task . In this paper , we investigate , theoretically and empirically , the conditions that enable a given unsupervised learning approach to perform well . Specifically , we first show theoretically that the success of GCL with generic graph augmentations ( GGA ) introduced by GraphCL ( You et al. , 2020a ) is dependent on the graph edit distance between classes . Next , we address the elephant in the room : that untrained GNNs have enough inductive bias to nullify the benefits of unsupervised pre-training on benchmark graph classification datasets . Therefore , we introduce a carefully designed synthetic dataset and conduct an extensive evaluation to better understand the behavior of both reconstruction and CL approaches . Here , we demonstrate that the effectiveness of different unsupervised approaches over strong untrained baselines can be understood through a style vs. content decomposition : the proportion of relevant information for a task ( content ) and irrelevant information ( style ) that each example in a dataset contains . Our contributions are as follows : ( i ) We provide theoretical analysis of when contrastive learning is expected to work well , showing that this depends on the graph edit distance of samples within and across classes ; ( ii ) We systematically evaluate reconstruction as an alternative unsupervised paradigm to contrastive learning for graphs , including introducing augmentation-augmented graph autoencoders ( AAGAE ) . Further , we empirically show to what extent untrained GNN models are a competitive baseline in terms of accuracy , invariance , and sample complexity ; and ( iii ) mWe identify a style vs. content trade-off in graphs and introduce an extensive benchmark setup that can carefully control this trade-off . Using our benchmark , we show how not only the ratio of style vs. content but also how generic versus content-aware augmentation impacts different learning paradigms . 2 BACKGROUND : UNSUPERVISED REPRESENTATION LEARNING FOR GNNS . While the general approach of pre-training can be either supervised ( Hu et al. , 2020 ) ) or unsupervised , we focus on the latter . In this section , we formalize unsupervised representation learning ( URL ) for graphs and discuss two widely adopted approaches , namely graph contrastive learning and reconstruction-based learning . Appendix A.6 contains additional related work on self-supervised learning and data augmentation for graphs . Graph Contrastive Learning . Contrastive learning ( CL ) frameworks learn representations by maximizing similarity between positive or augmented examples , and at the same time minimizing similarity between negative or uncorrelated examples . Formally , let X = { x1 , . . . , xn } denote a dataset consisting of un-augmented ( clean ) samples , where each xi = ( Gi , Fi ) corresponds to a tuple containing the adjacency matrix Gi ∈ [ 0 , 1 ] n×n and node feature matrix Fi . Let A represent a set of augmentations over X ; namely the generic graph augmentations ( GGA ) introduced by ( You et al. , 2020a ) : ( i ) node dropping , ( ii ) edge perturbation , ( iii ) attribute masking and ( iv ) sub-graph sampling . Furthermore , let X be the set of all augmented samples given X , and { xi = A ( xi ) , xj = A ( xi ) } be considered a positive pair . GraphCL parallels SimCLR ( Chen et al. , 2020a ) and uses the normalized temperature scaled cross entropy ( NT-XENT ) loss to learn representations by contrasting the representations of positive pairs and negative samples . Specifically , let f be a graph feature learner , such that f ( xi ) = zi ∈ Rd , e.g. , a GNN with a global READOUT layer , i.e. , READOUT : Rn×d → Rd . Given batch size B , similarity function , sim : ( Rd , Rd ) → [ 0 , 1 ] , temperature parameter , τ , and positive pair , { xi , xj } , the NT-XENT loss is defined as : ℓi , j = − log exp ( sim ( zi , zj ) /τ ) ∑2B k=1 1 [ k ̸=i ] exp ( sim ( zi , zk ) /τ ) . ( 1 ) Here , the numerator encourages the representations for xi , xj to have high similarity , while the denominator encourages representations of negative pairs ( k ̸= i ) to have low similarity . By maximizing similarity between positive samples , we expect the representations to become invariant to the properties modified by augmentations . Correspondingly , models learn to perform instance discrimination where each sample defines its own class and the augmented samples also belong to this class . Other recent GCL frameworks follow this general formulation but differ in the choice of the objective function and the augmentation strategy . For example , InfoGraph ( Sun et al. , 2020 ) maximizes the mutual information between sampled subgraphs ( local ) and pooled graph ( global ) representations . MVGRL ( Hassani & Ahmadi , 2020 ) contrasts representations of graphs augmented through diffusion processes at node and graph scales . While this work focuses on GraphCL , other formulations including InfoGraph , DGI ( Hjelm et al. , 2019 ) , and GMI ( Peng et al. , 2020 ) , can be derived as instances of this general framework . Reconstruction-Based Approaches . In computer vision , auto-encoding frameworks enriched with sophisticated model architectures and strong data augmentations ( Falcon et al. , 2021 ) are currently being revisited as an alternative unsupervised learning paradigm that is less dependent on large batch-sizes ( Chen et al. , 2020a ) and negative sampling strategies ( Kalantidis et al. , 2020 ) when compared to CL . Given that graph datasets are often significantly smaller than vision datasets and negative sampling strategies may be difficult to design , such approaches are particularly relevant to self-supervised graph representation learning and bear revisiting in light of stronger GNN architectures and augmentations . Formally , let g : ( Gi , Fi ) → Rn×d be an encoder that outputs node representations , h : Rn×d → [ 0 , 1 ] n×n be a decoder that predicts the edges of Gi given node representations , and READOUT : Rn×d → Rd provide graph representations given node representations . Then , a vanilla graph autoencoder minimizes : ||h ( g ( xi ) − Gi||22 , where h is often defined as σ ( g ( xi ) g ( xi ) T ) and the aggregated graph representation zi = READOUT ( g ( Gi , Fi ) ) is used to perform downstream tasks . Given that graphs are generally sparse , standard implementations sample an equivalent number of positive and negative edges to ensure stable training . Variational graph auto-encoders ( Kipf & Welling , 2016 ) reparameterize node representations and add a KL divergence term to the reconstruction loss similar to ( Kingma & Welling , 2014 ) . As we will argue in this paper , reconstruction-based approaches enhanced with suitable augmentations can be an effective alternative for GCL , under specific conditions . We choose to focus on reconstruction tasks as it is more general than task-specific pretraining tasks and more amenable to theoretical analysis ( Khemakhem et al. , 2020 ) . In subsequent sections , we seek to understand when and why a given method performs well . We begin by taking a closer look at the performance of graph URL using benchmark datasets . 3 A CLOSER LOOK INTO GRAPH URL USING BENCHMARK DATASETS . Designing an unsupervised representation learning pipeline for graph-structured data requires selecting from a number of components , including encoder architecture , data augmentation , strategies for leveraging inductive bias , and training paradigms . It is important to understand the impact of different components on downstream performance , so practitioners can appropriately select them for their needs and determine if URL or pre-training will improve representation quality . To this end , we begin by performing an empirical study on benchmark datasets that considers different : ( i ) levels of inductive biases by incorporating data augmentation and varying the amount of training ; ( ii ) representation learning paradigms ( CL , reconstruction-based ) ( iii ) GNN architectures ( GIN , GCN , GAT etc. ) . We note that we are the first to implement and benchmark the augmentation augmented graph auto-encoders ( AAGAE ) as a stable reconstruction-based approach for graph URL . While our empirical analysis offers several insights into different components of graph URL pipelines , we are unable to evaluate context-aware augmentations ( CAA ) as they are difficult to realize on standard graph benchmarks . CAA have been critical to the advancements in visual CL and the remainder of this paper investigates if CAA have similar promise for graph URL . In Section 4 , we extend analytical tools from visual CL to characterize the behaviour of generic graph augmentations ( GGA ) . In Section 5 , we introduce a novel , synthetic benchmark that gives us control over the style vs. content ratio in the synthesized samples . This enables systematic evaluation of the potential gains to be obtained from CAA . We begin by evaluating the quality of representations obtained not only through GCL and reconstruction-based approaches but also from surprisingly effective , untrained GNN encoders . 3.1 REPRESENTATION LEARNING STRATEGIES . We consider three flavors of unsupervised representation learning : ( i ) Graph contrastive learning : For a representative GCL approach , we select GraphCL with GGA as it is a popular and effective method for graph classification . Following You et al . ( 2020a ) , we use random node-dropping or subgraph sampling at 20 % of the graph size as the augmentation strategy ; ( ii ) Reconstruction-based : In addition to standard graph autoencoders ( GAEs ) , we extend AAVAE ( Falcon et al. , 2021 ) to the graph domain and introduce augmentation-augmented graph autoencoders ( AAGAE ) . While GAE minimizes the reconstruction loss with respect to the original sample , AAGAE minimizes the reconstruction loss between the original and the reconstruction for an augmented sample to learn representations that are consistent with respect to augmentations : ||Gi −h ( g ( A ( Gi , Xi ) ) ) ||22 . We use the same augmentations and encoder architecture as GraphCL for fair comparison and include a straightthrough estimator in the decoder for better training ( Jang et al. , 2017 ) ; and finally ( iii ) Untrained representations : We include untrained GNNs as an important baseline and find that representations from randomly initialized models often perform comparably to unsupervised approaches . There is some folk wisdom that untrained GNNs can be surprisingly competitive with trained models . Early GNN works noted that even an untrained model may have strong inductive bias suitable for node-level tasks ( Kipf & Welling , 2017 ) . For graph-level transfer learning , Xu et al . ( 2021 ) consider an untrained GIN model as a baseline , finding it obtains competitive although generally inferior results . A recent blog post performs a limited exploratory analysis of graph classification using untrained embeddings obtained from a simple GCN model ( Safronov , 2021 ) . Our analysis here ( extended to consider different architectures in Appendix A.3-A.4 ) is far more comprehensive , showing precisely in what ways untrained models may be competitive .

Summary: This paper focuses on the augmentation of graph contrastive learning. Authors make further explorations on graph contrastive learning based on the previous CV contrastive work (HaoChen et al. (2021) ) and hope to design a good benchmark by analyzing the relationship between GCL/auto-encoding methods and augmentations. In the analysis part, authors mainly use the perspective of population augmentation graph work to find connections with existing graph augmentations (mainly You et al., 2020a) and analyze the relationship between GED and task performance/downstream labels. In the benchmark part, authors proposes a benchmark based on the idea of "STYLE VS. CONTENT" which is inherited from CV contrastive work. The analysis of the experimental results is okay.