Rethinking Temperature in Graph Contrastive Learning

1 INTRODUCTION . Self-supervised learning provides a good learning paradigm without high-cost label information for computer vision ( Chen et al . ( 2020 ) ; Chen & He ( 2021 ) ; Grill et al . ( 2020 ) ) , natural language processing ( Wu et al . ( 2019 ) ; Gao et al . ( 2021 ) ) , and speech recognition ( Ravanelli et al . ( 2020 ) ; Kharitonov et al . ( 2021 ) ) . Contrastive-based methods have a prominent place among numerous self-supervised learning methods ( Jaiswal et al . ( 2021 ) ; Le-Khac et al . ( 2020 ) ) . Recently , some researchers have explored graph contrastive frameworks ( consisting of data augmentation , network encoding , and contrastive learning ) for self-supervised learning on graphs ( Velickovic et al . ( 2019 ) ; Peng et al . ( 2020 ) ; Zhu et al . ( 2020 ; 2021c ) ; Thakoor et al . ( 2021 ) ) . On benchmark datasets , the stateof-the-art graph contrastive learning ( GCL ) algorithms have been verified to be competitive with or even superior to the supervised learning algorithms in downstream tasks , such as node classification and graph classification . Investigating the quality of contrastive representations is important for contrastive learning , but it is absent in the domain of graphs . Wang & Isola ( 2020 ) propose two novel loss functions , alignment loss Lalign and uniformity loss Luniform , for measuring the image representation quality . By optimizing the integration of Lalign , Luniform , and contrastive loss , the performance on the downstream task ( such as image classification on IMAGENET ( Tian et al . ( 2019 ) ) ) becomes better . However , Lalign and Luniform designed for independent and identically distributed ( IID ) data are not suitable for Non-IID data such as nodes in a graph . The InfoNCE ( here “ NCE ” denotes “ Noise-Contrastive Estimation ” ) contrastive loss ( Oord et al . ( 2018 ) ) , as a universal loss function of contrastive learning , generally samples negative examples uniformly from the whole training data . To optimize the InfoNCE loss , the strategy of hard negative sampling ( Zhuang et al . ( 2019 ) ; Robinson et al . ( 2021 ) ) collects highly similar but true negative examples in advance to construct contrastive pairs and is widely used in image processing and metric learning ( Duan et al . ( 2018 ) ; Robinson et al . ( 2021 ) ; Wang & Liu ( 2021 ) ) . This strategy makes the model be able to correct its mistakes quickly and improve the semantic closeness between represen- tations ( Wang & Liu ( 2021 ) ) . Despite hard negative sampling ’ s practical importance in the image domain , Yang et al . ( 2020 ) have proved that it is sub-optimal in the graph domain . In this paper , we investigate how to generate high-quality node representations under the InfoNCE contrastive objective in graphs . We highlight the importance of embeddings ’ global uniformity and local separation for GCL . The temperature coefficient τ , as a key component of InfoNCE loss , decides how the current learning state focuses on global uniformity and local separation . We illustrate this conclusion in Figure 1 and prove it by a gradient analysis in Section 3 . The dynamic setting of τ can generate different learning states for the same task and help the algorithm smoothly transit from one state to another . Therefore , inspired by the optimizer called Momentum ( Qian ( 1999 ) ) , we develop a Graph contrastive Learning algorithm with dynAmic Temperature Estimation ( GLATE ) . Compared with the fixed setting of τ , GLATE develops to its full potential on contrastive learning by further maximizing the self-supervised Information Bottleneck objective . In a nutshell , the main contributions of this work are as follows : • To evaluate the node representations in GCL , we propose two new metrics : global uniformity ( Eq . ( 9 ) ) and local separation ( Eq . ( 10 ) ) . We prove the importance of temperature τ to them by theoretical analysis ( Section 3.1 ) . • We develop a simple but very effective GCL algorithm GLATE ( Section 3.2 ) . In the training phase , GLATE dynamically adjusts τ ’ s value with its momentum to learn high-quality representations . With the help of the information bottleneck principle , we analyze the connections between GLATE and entropy information ( Section 3.3 ) . • The experimental results on node and graph classification tasks indicate that GLATE is superior to the state-of-the-art GCL algorithms ( Section 4 ) . For example , for the task of transductive node classification , GLATE outperforms baselines by 2.8 percent on average . 2 BACKGROUND AND RELATED WORK . Preliminaries of Graph Contrastive Learning . Let us review an overall graph contrastive learning process with contrastive pairs . We consider a graph G = ( A , X ) where A ∈ RN×N denotes the adjacency matrix , X ∈ RN×M denotes the attribute distribution matrix , N is the number of nodes , and M is the dimension of attributes . The raw graph G is distorted via random data augmentation strategies ( such as edge removing and attribute masking ) to generate two new graphs G̃1 = ( A′ , X ′ ) as well as G̃2 = ( A′′ , X ′′ ) . Then , the message-passing-based graph neural network ( e.g. , GCN proposed in ( Kipf & Welling ( 2016 ) ) ) is used as a shared encoder of G̃1 and G̃2 . It aims to learn node embeddings from G̃1 and G̃2 . The message passing form of a two-layer GCN is : Z′ = softmax ( Â′ReLU ( Â′X ′W 0 ) W 1 ) , ( 1 ) whereZ ′ , Â′ , andW 0 ( W 1 ) are node embedding matrix , renormalizedA′ , and first-layer ( secondlayer ) neural network parameters , respectively . Next , in each training epoch , the pre-defined contrastive objective encourages GCN to minimize the distance between Z ′ and Z ′′ ( another node embedding matrix learned from G̃2 ) , and meanwhile to maximize the distance between different nodes ( Zhu et al . ( 2020 ) ; Zbontar et al . ( 2021b ) ; Bardes et al . ( 2021 ) ) . After n rounds of iteration , we use trained GCN to infer each node ’ s embedding on G , which is used as the initial value of new node embedding in different downstream tasks , such as node classification . Graph Contrastive Objectives . To make positive examples closer and negative examples farther , different graph contrastive loss functions are defined , such as : 1 ) Jason-Shannon Divergence ( JSD ) loss ( Velickovic et al . ( 2019 ) ; Hassani & Khasahmadi ( 2020 ) ) , 2 ) InfoNCE loss ( Zhu et al . ( 2020 ) ; You et al . ( 2020 ) ; Zhu et al . ( 2021c ) ; Pan & Kang ( 2021 ) ; Xu et al . ( 2021 ) ) , and 3 ) Triplet Margin ( TM ) loss ( Zhang et al . ( 2019 ) ) . According to a recent empirical study ( Zhu et al . ( 2021b ) ) , the models under the InfoNCE loss generally achieve better performance compared to those under the other graph contrastive objectives with the participation of negative examples . The important role of InfoNCE loss in graph contrastive learning motivates our work to have deep insights into a series of GCL algorithms ( e.g . GRACE ( Zhu et al . ( 2020 ) ) ) derived by it . Understanding Contrastive Representation Learning . Another main area of concern is understanding contrastive representation learning and exploring what high-quality representations are . In the image domain , alignment , uniformity , and semantic closeness ( Wang & Isola ( 2020 ) ; Wang & Liu ( 2021 ) ) are thought to be three important metrics for measuring the quality of learned representations . Hard negative sampling ( Robinson et al . ( 2021 ) ; Wang & Liu ( 2021 ) ) has been proved to be a successful strategy to meet the needs of the above three metrics . However , recent studies ( Xia et al . ( 2021 ) ; Yang et al . ( 2020 ) ) have confirmed the existence of sampling bias when using hard negative sampling for a graph , which limits its application to the graph domain . 3 METHOD . 3.1 GRADIENT ANALYSIS . We start with a general contrastive loss function , i.e. , InfoNCE loss ( Oord et al . ( 2018 ) ; Zhu et al . ( 2020 ) ) . In the graph domain , it can be formulated as : L1,2i = −log exp ( 1 τ · S ( i′ , i′′ ) ) exp ( 1 τ · S ( i′ , i′′ ) ) + ∑ k 6=i exp ( 1 τ · S ( i′ , k′′ ) ) ︸ ︷︷ ︸ inter−view term + ∑ k 6=i exp ( 1 τ · S ( i′ , k′ ) ) ︸ ︷︷ ︸ intra−view term . ( 2 ) The vector similarity function S ( i′ , k′′ ) = f ( ||h ( Z ′i , : ) ||2 , ||h ( Z ′′ k , : ) ||2 ) where f ( · ) and h ( · ) are respectively cosine similarity and non-linear projection transformation ( two layers of multilayer perception ) . As h ( · ) is not shared between different channels , the whole network structure is technically asymmetric , which leads to the asymmetric result of learned node representations . To eliminate the bias between different channels , G̃1 and G̃2 are swapped to use , and thus the total loss function is L = 12N ∑N i=1 ( L 1,2 i + L 2,1 i ) . To simplify the symbols in Eq . ( 2 ) , we replace exp ( 1 τ · S ( i ′ , k′′ ) ) by Ei′ , k′′ , and then derive the following gradient results : ∇S ( i′ , i′′ ) L1,2i = − ∑ k 6=i Ei′ , k′′ + ∑ k 6=i Ei′ , k′ τ · ( Ei′ , i′′ + ∑ k 6=i Ei′ , k′′ + ∑ k 6=i Ei′ , k′ ) , ( 3 ) ∇S ( i′ , j′ ) L1,2i = − Ei′ , j′ τ · ( Ei′ , i′′ + ∑ k 6=i Ei′ , k′′ + ∑ k 6=i Ei′ , k′ ) , ( 4 ) ∇S ( i′ , j′′ ) L1,2i = − Ei′ , j′′ τ · ( Ei′ , i′′ + ∑ k 6=i Ei′ , k′′ + ∑ k 6=i Ei′ , k′ ) . ( 5 ) These gradient results have the same denominator , which can be removed when we calculate the gradient ratio between any two of them . Lemma 1 . Following Eq . ( 2 ) , the ratio of intra-view negative gradient to positive gradient is R ( i′ , j′ ; i′ , i′′ ) = ∇S ( i′ , j′ ) L1,2i ∇S ( i′ , i′′ ) L1,2i = Ei′ , j′∑ k 6=i Ei′ , k′′ + ∑ k 6=i Ei′ , k′ . ( 6 ) The result in Eq . ( 6 ) obeys the Boltzmann distribution , as shown in Figure 2 ( a , b ) ( note that we omit the specific form of R ( i′ , j′′ ; i′ , i′′ ) because it has the same property as R ( i′ , j′ ; i′ , i′′ ) ) . It means that if τ is low ( τ ∈ ( 0 , 0.5 ) ) , the optimizer will punish more on the hard negative examples than the easy ones ; otherwise , it will nearly make no difference of them . So high τ is good at distinguishing similar and dissimilar embeddings and low τ is good at partitioning highly similar embeddings to ensure local separation . It is also worth noting the rate between the negative gradients of different views , which is defined as : R ( i′ , j′ ; i′ , j′′ ) = ∇S ( i′ , j′ ) L1,2i ∇S ( i′ , j′′ ) L1,2i = Ei′ , j′ Ei′ , j′′ = exp ( 1 τ · ( S ( i′ , j′ ) − S ( i′ , j′′ ) ) ) . ( 7 ) In Eq . ( 7 ) , τ still plays an important role ( note that we omit the specific form of R ( i′ , j′′ ; i′ , j′ ) because it has the same property as R ( i′ , j′ ; i′ , j′′ ) ) . We have the following proposition : Proposition 1 . ( Different views ’ imbalanced update. ) . The temperature τ is crucial to balance the rate between the negative gradients of different views . If τ → ∞ , then ∇S ( i′ , j′ ) L1,2i → ∇S ( i′ , j′′ ) L1,2i , i.e. , the negative examples of inter-view and intra-view keep the gradient update at the same speed . Otherwise , there exists an imbalance between ∇S ( i′ , j′ ) L1,2i and ∇S ( i′ , j′′ ) L 1,2 i . The imbalance will deepen as τ decreases . We refer the reader to Appendix A.1 for detailed proof . We illustrate the phenomenon of the imbalanced gradient update in Figure 2 ( c ) and ( d ) . Based on the above analysis , we conclude that 1 ) high τ is helpful to the embedding ’ s global uniformity and low τ is helpful to the embedding ’ s local separation ; 2 ) high τ can alleviate the negative effect of imbalanced update . Therefore , dynamic estimation of τ with a relatively high initial value is recommended .

Summary: This paper investigates a crucial problem, i.e., how to generate good node representations under the InfoNCE contrastive loss in graphs. By studying the feature of loss function’s gradients, it finds that the dynamic temperature change is beneficial to learn uniform node representations. The proposed method GLATE is simple but effective. The analysis of the connection between GLATE and the information bottleneck principle helps us understand why GLATE is so effective. The presentation of this paper is clear and well-organized. For example, Figure 1 makes us easy to understand the problem generated by the static temperature and the significance of this work. The experiments are laid out in detail. The results on the tasks of transductive and inductive learning show GLATE’s advantage over the SOTA graph contrastive learning algorithms.