InfoGCL: Information-Aware Graph Contrastive Learning

1 Introduction . Inspired by their success in the vision and language domains , contrastive learning methods have been wildly adopted by recent progress in graph learning to improve the performance of a variety of tasks [ 42 , 13 , 22 ] . In a nutshell , these methods typically learn representations by creating two augmented views of a graph and maximizing the feature consistency between the two views . Inheriting the advantages of self-supervised learning , contrastive learning relieves graph representation learning from its reliance on label information in graph domain , where label information can be very costly or even impossible to collect while unlabeled/partially labeled data is common , such as chemical graph data [ 28 ] . Graph contrastive learning methods have achieved similar ( and even better ) performance as compared to the equivalent methods trained with labels on benchmark graph datasets [ 42 , 13 , 6 ] . Despite being effective and prevalent , existing graph contrastive learning models differ mostly in augmented view design , encoding architecture , and contrastive objective ( refer to Table 1 in Appendix for more comparisons ) . For a learning task , it usually requires a substantial degree of domain expertise to carefully design and customize these modules for the specific dataset . For example , while both DGI [ 33 ] and InfoGraph [ 28 ] seek to obtain graph representations by maximizing the mutual information between patch-level and graph-level representations , they adopt different graph encoders , GCN [ 15 ] and GIN [ 38 ] respectively . mvgrl [ 13 ] applies graph diffusion convolution to construct the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . augmented view , while GCC [ 22 ] and GRACE [ 45 ] adopt subgraph sampling and graph perturbation , respectively . The main question this paper attempts to answer is : how to perform contrastive learning for your learning tasks on specific graph datasets ? However , answering this question is challenging . First , contrastive learning consists of multiple components , such as view augmentation and information encoding . For each of them , there are various choices . Numerous variations make it difficult to design models that are both robust and efficient . Existing graph contrastive learning approaches are carefully designed for different learning tasks on different datasets , however , none of them studies the guiding principles for choosing the best components . Second , graph data has unique properties that distinguish it from other types of data , such as rich structural information and highly diverse distribution [ 33 , 28 , 11 , 10 ] . Thus , it is desirable to design the contrastive learning model that fits your graph data properties , even without any domain knowledge of the data . We propose to address these challenges via Information Bottleneck ( IB ) [ 31 ] , which provides a crucial principle for representation learning . Specifically , IB encourages the representations to be maximally informative about the target in the downstream task , which helps keep task-relevant information . Concurrently , IB discourages the representation learning from acquiring the task-irrelevant information from the input data , which is related to the idea of minimal sufficient statistics [ 27 ] . However , different from the typical representation learning , there are two information flows involved in the two augmented views in contrastive learning . Therefore , we extend the previous IB work [ 37 , 43 ] and propose InfoGCL , an information-aware contrastive learning framework for graph data . To study how information is transformed and transferred , we decouple a typical graph contrastive learning model into three sequential modules ( as shown in Figure 1 ) : view augmentation , view encoding , and representation contrasting . We further formalize how to find the optimal of the three modules into three optimization problems . To build the optimal graph contrastive learning model for the particular dataset and task , we argue that it is necessary and sufficient to minimize the mutual information between contrastive representations while maximizing task-relevant information at the levels of both individual module and entire framework . Our work is also motivated by the InfoMin theory [ 30 ] , which suggests that a good set of views for contrastive learning in the vision domain should share the minimal information necessary to perform well at the downstream task . Beyond view selection , our work extends InfoMin to suggest principles of selecting view encodings and contrastive modes for graph learning considering the unique properties of graph data . We suggest practically feasible principles to find the optimal modules in graph contrastive learning and show that all recent graph contrastive learning methods can be unified by these principles : i ) the augmented views should contain as much task-relevant information as possible , while they should share as little information as possible ; ii ) the view encoder should be task-relevant and simple as much as possible ; iii ) the contrastive mode should keep task-relevant information as much as possible after contrasting . Besides , we also investigate the role of negative samples in graph contrastive learning and argue that negative samples are not necessarily required , especially when graph data is not extremely sparse . Our proposed method , InfoGCL , is validated on a rich set of benchmark datasets for both node-level and graph-level tasks , where we analyze its ability to capture the unique structural properties of the graph data . The results demonstrate that our algorithm achieves highly competitive performance with up to 5.2 % relative improvement in accuracy on graph classification task and competitive results on node classification task over the state-of-the-art unsupervised methods . 2 Related Work . Graph Contrastive Learning . Some recent research efforts in graph domain have been attracted by the success of contrastive learning in vision and language domains [ 3 , 8 , 4 ] . A number of graph contrastive learning approaches have been proposed [ 28 , 22 , 42 , 13 ] . Despite all of them creating two views and targeting at maximizing the feature disagreement between the two views , these methods are carefully designed and differ in various aspects . Deep graph Infomax ( DGI ) [ 33 ] applied the InfoMax principle [ 18 ] to graph data by contrasting the representations of node-level and graph-level for node classification tasks . Different from DGI , InfoGraph [ 28 ] aims at node classification tasks and it contrasts the representations of graph-level and substructure-level of different granularity . In addition , DGI and InfoGraph use different graph encoders to extract latent representations . mvgrl [ 13 ] studies both node and graph classification tasks . It transforms the adjacency matrix to a diffusion matrix and treat the two matrices as two congruent views . However , in GCC [ 22 ] and GRACE [ 45 ] , subgraph sampling and graph perturbation are used to create the augmented views . GraphCL [ 42 ] explores the view augmentations approaches for graph contrastive learning . Specifically , it studies the approaches of node dropping , edge perturbation , attribute masking , and subgraph sampling . A recent work [ 44 ] also studies graph data augmentations . However , it focuses on graph neural networks for node classification and does not study the contrastive learning framework . Our method differs from them . We aim to answer the question how to perform contrastive learning for your graph data and tasks . Instead of carefully designing the architectures , we decouple typical graph contrastive learning into three stages and provide our InfoGCL principles to analyze the optimality theoretically and practically . Information Bottleneck . Our method is related to the Information Bottleneck ( IB ) theory [ 31 ] , which aims to find the best trade-off between accuracy and complexity when summarizing a random variable . IB has been recently used to study the deep learning approaches [ 26 , 37 , 43 ] . Specifically , IB expresses the trade-off between the mutual information measures I ( D , Z ) and I ( Z , y ) as max IBβ = −I ( D ; Z ) + βI ( Z ; y ) , ( 1 ) where D , Z , y are the input , the latent representation and the task label , respectively . θ is a hyperparameter . In other words , IB aims to learn representation Z that is maximally expressive about y , while being minimally expressive about D. More recently , there are some efforts applying the IB theory to graph representation learning . [ 37 ] aims to generate both expressive and robust graph representations , while [ 43 ] studies the subgraph recognition problem . In contrast , our work focuses on graph contrastive learning , where the two augmented views make the optimization objective ( information trade-off ) different . Our work is also related to the idea of minimal sufficient statistics [ 27 ] , which has been recently studied in the vision domain [ 30 ] , claiming that a good set of image views should share the minimal information necessary to perform well at the downstream task . Different from [ 30 ] , we focus on the graph domain and propose three stages considering the unique properties of graph data . 3 Preliminaries and Notations . Graph Representation Learning . A graph is denoted by G = ( A , X ) . A ∈ Rn×n is the adjacency matrix . X ∈ Rn×d is the node attribute matrix , where d is the attribute dimension . In this work , we focus on both node-level and graph-level tasks . For node-level task , given graph G and the labels of a subset of nodes , denoted by Yv , the goal is to learn the latent representation zv for each node v such that zv preserves both network structures and node attributes , which can be further used to predict Yv . For graph-level task , given a set of graphs G = { G1 , G2 , · · · } and the labels of some graphs , denoted by Yg , the goal is to learn the latent representation zg for each graph such that zg can be used to predict Yg . Typically , the graph data is fed into graph neural networks ( GNNs ) to generate the representations , such as zg = GNNs ( G ) . Graph Contrastive Learning . Given an input graph , graph contrastive learning aims to learn the representations of graph or nodes ( for graph-level or node-level tasks respectively ) through maximizing the feature consistency between two augmented views of the input graph via contrastive loss in the latent space . We decouple a typical graph contrastive learning model into three sequential modules . ( i ) View augmentation . Graph G undergoes data augmentation q ( · ) to obtain two views vi , vj , i.e. , vi ∼ qi ( G ) and vj ∼ qj ( G ) . A view is represented as graph data , such as vi = ( Avi , Xvi ) , where Avi ∈ Rn×n and Xvi ∈ Rn×d . In practice , view augmentation approaches include node dropping , edge perturbation , subgraph sampling , etc . ( ii ) View encoding . Graph-level or node-level latent representation is extracted from views vi , vj by using the view encoder networks f ( · ) ( a GNN backbone plus a projection MLP ) , i.e. , zi ∼ fi ( vi ) and zj ∼ fj ( vj ) . The two encoders might or might not share parameters depending on whether they are from the same domain . ( iii ) Representation contrasting . Given the latent representations , a contrastive loss is optimized to score the positive pairs zi , zj higher compared to other negative pairs . Typically , the negative pairs are constructed from the augmented views of other graphs in the same minibatch . The InfoNCE loss [ 21 ] has been adopted as one of popular contrastive losses , which is defined as : LNCE = −E [ log exp ( h ( zi , n , zj , n ) ) ∑N n′=1 exp ( h ( zi , n , zj , n′ ) ) ] , ( 2 ) where h ( · ) is a contrasting operation to score the agreement between two representations . Theoretically , minimizing the InfoNCE loss equivalently maximizes a lower bound on the mutual information between the views of positive pairs . In other words , I ( zi , zj ) > log ( N ) -LNCE , where I ( · ) measures the mutual information .

Summary: In this paper, the authors study the graph contrastive learning problem and propose an information-aware approach called InfoGCL. The motivation for this work is that all recent graph contrastive learning approaches are customized, and it is an open question how to build the graph contrastive learning method for particular tasks and datasets. The authors' method decouples the typical graph contrastive learning into three modules and formalizes how to find the optimal of the three modules into three optimization problems. Experimental results on both node-level and graph-level tasks demonstrate the advantages of the proposed method.