Year: 2021

Abstract:
Due to not relying on the rare human-labeled information, self-supervised learning, especially contrastive learning, attracted much attention from researchers. It has begun to show its strong advantages on both IID data (independent and identically distributed data, such as images and texts) and Non-IID data (such as nodes in graphs). Recently, researchers have begun to explore the quality of contrastive representations and proposed some metrics for measuring it, such as alignment, uniformity, and semantic closeness. However, current studies only consider IID data and ignore the evaluation of representations quality in graph contrastive learning. In this paper, we investigate and discuss how to generate high-quality representations for a general loss (InfoNCE) in graph contrastive learning. We argue that the properties of global uniformity and local separation are both necessary to the representation quality. By theoretical analysis, we find that the two properties can be naturally regulated by temperature τ in InfoNCE loss. Based on this point, we develop a simple but effective algorithm GLATE to dynamically adjust the temperature value in the training phase. On node and graph classification tasks, GLATE is validated to be competitive with the state-of-the-art graph contrastive learning algorithms.

heading: 1 INTRODUCTION
text: Self-supervised learning provides a good learning paradigm without high-cost label information for computer vision (Chen et al. (2020); Chen & He (2021); Grill et al. (2020)), natural language processing (Wu et al. (2019); Gao et al. (2021)), and speech recognition (Ravanelli et al. (2020); Kharitonov et al. (2021)). Contrastive-based methods have a prominent place among numerous self-supervised learning methods (Jaiswal et al. (2021); Le-Khac et al. (2020)). Recently, some researchers have explored graph contrastive frameworks (consisting of data augmentation, network encoding, and contrastive learning) for self-supervised learning on graphs (Velickovic et al. (2019); Peng et al. (2020); Zhu et al. (2020; 2021c); Thakoor et al. (2021)). On benchmark datasets, the stateof-the-art graph contrastive learning (GCL) algorithms have been verified to be competitive with or even superior to the supervised learning algorithms in downstream tasks, such as node classification and graph classification.
Investigating the quality of contrastive representations is important for contrastive learning, but it is absent in the domain of graphs. Wang & Isola (2020) propose two novel loss functions, alignment loss Lalign and uniformity loss Luniform, for measuring the image representation quality. By optimizing the integration of Lalign, Luniform, and contrastive loss, the performance on the downstream task (such as image classification on IMAGENET (Tian et al. (2019))) becomes better. However, Lalign and Luniform designed for independent and identically distributed (IID) data are not suitable for Non-IID data such as nodes in a graph.
The InfoNCE (here “NCE” denotes “Noise-Contrastive Estimation”) contrastive loss (Oord et al. (2018)), as a universal loss function of contrastive learning, generally samples negative examples uniformly from the whole training data. To optimize the InfoNCE loss, the strategy of hard negative sampling (Zhuang et al. (2019); Robinson et al. (2021)) collects highly similar but true negative examples in advance to construct contrastive pairs and is widely used in image processing and metric learning (Duan et al. (2018); Robinson et al. (2021); Wang & Liu (2021)). This strategy makes the model be able to correct its mistakes quickly and improve the semantic closeness between represen-
tations (Wang & Liu (2021)). Despite hard negative sampling’s practical importance in the image domain, Yang et al. (2020) have proved that it is sub-optimal in the graph domain.
In this paper, we investigate how to generate high-quality node representations under the InfoNCE contrastive objective in graphs. We highlight the importance of embeddings’ global uniformity and local separation for GCL. The temperature coefficient τ , as a key component of InfoNCE loss, decides how the current learning state focuses on global uniformity and local separation. We illustrate this conclusion in Figure 1 and prove it by a gradient analysis in Section 3. The dynamic setting of τ can generate different learning states for the same task and help the algorithm smoothly transit from one state to another. Therefore, inspired by the optimizer called Momentum (Qian (1999)), we develop a Graph contrastive Learning algorithm with dynAmic Temperature Estimation (GLATE). Compared with the fixed setting of τ , GLATE develops to its full potential on contrastive learning by further maximizing the self-supervised Information Bottleneck objective.
In a nutshell, the main contributions of this work are as follows:
• To evaluate the node representations in GCL, we propose two new metrics: global uniformity (Eq. (9)) and local separation (Eq. (10)). We prove the importance of temperature τ to them by theoretical analysis (Section 3.1).
• We develop a simple but very effective GCL algorithm GLATE (Section 3.2). In the training phase, GLATE dynamically adjusts τ ’s value with its momentum to learn high-quality representations. With the help of the information bottleneck principle, we analyze the connections between GLATE and entropy information (Section 3.3).
• The experimental results on node and graph classification tasks indicate that GLATE is superior to the state-of-the-art GCL algorithms (Section 4). For example, for the task of transductive node classification, GLATE outperforms baselines by 2.8 percent on average.

heading: 2 BACKGROUND AND RELATED WORK
text: Preliminaries of Graph Contrastive Learning. Let us review an overall graph contrastive learning process with contrastive pairs. We consider a graph G = (A,X) where A ∈ RN×N denotes the adjacency matrix, X ∈ RN×M denotes the attribute distribution matrix, N is the number of nodes, and M is the dimension of attributes. The raw graph G is distorted via random data augmentation strategies (such as edge removing and attribute masking) to generate two new graphs G̃1 = (A′,X ′) as well as G̃2 = (A′′,X ′′). Then, the message-passing-based graph neural network (e.g., GCN proposed in (Kipf & Welling (2016))) is used as a shared encoder of G̃1 and G̃2. It aims to learn node
embeddings from G̃1 and G̃2. The message passing form of a two-layer GCN is: Z′ = softmax(Â′ReLU(Â′X ′W 0)W 1), (1)
whereZ ′, Â′, andW 0 (W 1) are node embedding matrix, renormalizedA′, and first-layer (secondlayer) neural network parameters, respectively. Next, in each training epoch, the pre-defined contrastive objective encourages GCN to minimize the distance between Z ′ and Z ′′ (another node embedding matrix learned from G̃2), and meanwhile to maximize the distance between different nodes (Zhu et al. (2020); Zbontar et al. (2021b); Bardes et al. (2021)). After n rounds of iteration, we use trained GCN to infer each node’s embedding on G, which is used as the initial value of new node embedding in different downstream tasks, such as node classification.
Graph Contrastive Objectives. To make positive examples closer and negative examples farther, different graph contrastive loss functions are defined, such as: 1) Jason-Shannon Divergence (JSD) loss (Velickovic et al. (2019); Hassani & Khasahmadi (2020)), 2) InfoNCE loss (Zhu et al. (2020); You et al. (2020); Zhu et al. (2021c); Pan & Kang (2021); Xu et al. (2021)), and 3) Triplet Margin (TM) loss (Zhang et al. (2019)). According to a recent empirical study (Zhu et al. (2021b)), the models under the InfoNCE loss generally achieve better performance compared to those under the other graph contrastive objectives with the participation of negative examples. The important role of InfoNCE loss in graph contrastive learning motivates our work to have deep insights into a series of GCL algorithms (e.g. GRACE (Zhu et al. (2020))) derived by it.
Understanding Contrastive Representation Learning. Another main area of concern is understanding contrastive representation learning and exploring what high-quality representations are. In the image domain, alignment, uniformity, and semantic closeness (Wang & Isola (2020); Wang & Liu (2021)) are thought to be three important metrics for measuring the quality of learned representations. Hard negative sampling (Robinson et al. (2021); Wang & Liu (2021)) has been proved to be a successful strategy to meet the needs of the above three metrics. However, recent studies (Xia et al. (2021); Yang et al. (2020)) have confirmed the existence of sampling bias when using hard negative sampling for a graph, which limits its application to the graph domain.

heading: 3 METHOD

heading: 3.1 GRADIENT ANALYSIS
text: We start with a general contrastive loss function, i.e., InfoNCE loss (Oord et al. (2018); Zhu et al. (2020)). In the graph domain, it can be formulated as:
L1,2i = −log exp( 1 τ · S(i′, i′′))
exp( 1 τ · S(i′, i′′)) + ∑ k 6=i exp( 1 τ · S(i′, k′′))
︸ ︷︷ ︸ inter−view term
+ ∑ k 6=i exp( 1 τ · S(i′, k′))
︸ ︷︷ ︸ intra−view term
. (2)
The vector similarity function S(i′, k′′) = f(||h(Z ′i,:)||2, ||h(Z ′′ k,:)||2) where f(·) and h(·) are respectively cosine similarity and non-linear projection transformation (two layers of multilayer perception). As h(·) is not shared between different channels, the whole network structure is technically asymmetric, which leads to the asymmetric result of learned node representations. To eliminate the bias between different channels, G̃1 and G̃2 are swapped to use, and thus the total loss function is L = 12N ∑N i=1(L 1,2 i + L 2,1 i ). To simplify the symbols in Eq. (2), we replace exp( 1 τ · S(i
′, k′′)) by Ei′,k′′ , and then derive the following gradient results:
∇S(i′,i′′)L1,2i = − ∑ k 6=i Ei′,k′′ + ∑ k 6=i Ei′,k′
τ · (Ei′,i′′ + ∑ k 6=i Ei′,k′′ + ∑ k 6=i Ei′,k′) , (3)
∇S(i′,j′)L1,2i = − Ei′,j′ τ · (Ei′,i′′ + ∑ k 6=i Ei′,k′′ + ∑ k 6=i Ei′,k′) , (4)
∇S(i′,j′′)L1,2i = − Ei′,j′′ τ · (Ei′,i′′ + ∑ k 6=i Ei′,k′′ + ∑ k 6=i Ei′,k′) . (5)
These gradient results have the same denominator, which can be removed when we calculate the gradient ratio between any two of them.
Lemma 1. Following Eq. (2), the ratio of intra-view negative gradient to positive gradient is
R(i′, j′; i′, i′′) = ∇S(i′,j′)L1,2i ∇S(i′,i′′)L1,2i
= Ei′,j′∑ k 6=i Ei′,k′′ + ∑ k 6=i Ei′,k′ . (6)
The result in Eq. (6) obeys the Boltzmann distribution, as shown in Figure 2 (a,b) (note that we omit the specific form of R(i′, j′′; i′, i′′) because it has the same property as R(i′, j′; i′, i′′)). It means that if τ is low (τ ∈ (0, 0.5)), the optimizer will punish more on the hard negative examples than the easy ones; otherwise, it will nearly make no difference of them. So high τ is good at distinguishing similar and dissimilar embeddings and low τ is good at partitioning highly similar embeddings to ensure local separation.
It is also worth noting the rate between the negative gradients of different views, which is defined as:
R(i′, j′; i′, j′′) = ∇S(i′,j′)L1,2i ∇S(i′,j′′)L1,2i = Ei′,j′ Ei′,j′′ = exp( 1 τ · (S(i′, j′)− S(i′, j′′))). (7)
In Eq. (7), τ still plays an important role (note that we omit the specific form of R(i′, j′′; i′, j′) because it has the same property as R(i′, j′; i′, j′′)). We have the following proposition:
Proposition 1. (Different views’ imbalanced update.). The temperature τ is crucial to balance the rate between the negative gradients of different views. If τ → ∞, then ∇S(i′,j′)L1,2i → ∇S(i′,j′′)L1,2i , i.e., the negative examples of inter-view and intra-view keep the gradient update at the same speed. Otherwise, there exists an imbalance between ∇S(i′,j′)L1,2i and ∇S(i′,j′′)L 1,2 i . The imbalance will deepen as τ decreases.
We refer the reader to Appendix A.1 for detailed proof. We illustrate the phenomenon of the imbalanced gradient update in Figure 2 (c) and (d). Based on the above analysis, we conclude that 1) high τ is helpful to the embedding’s global uniformity and low τ is helpful to the embedding’s local separation; 2) high τ can alleviate the negative effect of imbalanced update. Therefore, dynamic estimation of τ with a relatively high initial value is recommended.

heading: 3.2 DYNAMIC TEMPERATURE ESTIMATION WITH MOMENTUM
text: Some studies (Robinson et al. (2021); Wang & Liu (2021)) have confirmed that the properties of uniformity and semantic closeness are both necessary for image contrastive learning. In the graph domain, the closeness of nodes embodies the proximity of the topology information and the similarity of the attribute information. Both of them are incorporated into Z ′, Z ′′ by the encoder of graph neural networks with the message passing mechanism.
The strategy of hard negative sampling is successful to discriminate the images of similar semantics. It only updates the hard or informative examples by truncating the uninformative ones. Following Eq. (2), the loss under hard negative sampling can be formulated as: L1,2i,hard = −log(Ei′,i′′/(Ei′,i′′ + ∑ Si′,k′′>δ Ei′,k′′ + ∑ Si′,k′>δ
Ei′,k′)). Although this strategy is rational in the image domain (Robinson et al. (2021); Wang & Liu (2021); Zhu et al. (2021a)), it is not suitable for the graph (we also compare the proposed algorithm with its variant using hard negative sampling, refer to Table 1). Current negative sampling methods in the graph have a common shortcoming of
sampling bias, i.e., most of the negative examples being highly similar to the anchor example are actually positive examples (Xia et al. (2021)). Yang et al. (2020) have proved that a nice negative sampling distribution should be sub-linearly correlated to the positive sampling distribution in the graph domain: pneg(i|j) ∝ ppos(i|j)α, α ∈ (0, 1). As the real ppos is unknown and its approximation is hard to define, we retain all negative examples and employ dynamic temperature estimation to control global uniformity and local separation.
Based on the gradient analysis in Section 3.1, we propose a new algorithm called GLATE whose main idea is dynamic temperature estimation. More precisely, at the early stage, the temperature τ is set high to ensure that all embeddings are split uniformly by imposing nearly equal punishments on negative examples. As the training goes on, the dissimilar examples are separated gradually and the distance between similar examples becomes close. At the later stage, τ is set relatively lower to distinguish the similar and hard negative examples. Therefore, the change of temperature should be influenced by the uniformity of the node representations.
Inspired by Momentum (Qian (1999)), a classical stochastic gradient descent optimization method, we calculate the temperature’s current value τt by integrating its last state’s momentum ∆τt−1 and the current embedding’s global uniformity degree Dtglobal together:
τt = τt−1−ε(p∆τt−1 + 1
Dtglobal )︸ ︷︷ ︸
∆τt
. (8)
Also, the global uniformity degree at epoch t Dtglobal is defined as:
Dtglobal = −log 2 N(N − 1) ∑ i ∑ j<i exp−||Z t i−Z t j || 2 2 , (9)
where ε is the learning rate and p is the momentum parameter. In addition, we define the local separation degree at epoch t and use it as an important metric to measure the model’s uniformity from a local perspective (Figure 4):
Dtlocal = −log 2 N(N − 1) ∑ i ∑ j<i S(Zti,Z t j) · exp−||Z t i−Z t j || 2 2 , (10)
where S(Zti,Z t j) is the cosine similarity between node embeddings Zi and Zj at epoch t.
Compared with the steepest descent ∆τt = −ε/Dtglobal, the update rule in Eq. (8) makes τ decrease at an adjustable pace. At the early stage, τ decreases slowly due to little information accumulation, while it decreases faster later to distinguish harder negative examples.

heading: 3.3 ANALYSIS FROM THE INFORMATION BOTTLENECK PRINCIPLE
text: Here we analyze GLATE’s intrinsic mechanism and reveal the connection between GLATE and the information bottleneck principle (Tishby & Zaslavsky (2015); Amjad & Geiger (2019)). Assuming the node representation matrix obeys the D-dimensional Gaussian distribution: Z ′ ∼ N (µZ ′ ,ΣZ ′) (µZ ′ ∈ R 1×D, ΣZ ′ ∈ R D×D), the self-supervised learning’s information bottleneck (IB) principle (Zbontar et al. (2021a)) is to maximize the following Lagrange’s function (we omit Z ′′ in this section because it has the same properties as Z ′): IBssl = I(Z′,X)− βI(Z′,X ′) (β > 0), (11) where I(a,b) is the mutual information between a and b. The motivation behind IBssl is to make the learned embedding preserve as much feature information of the raw input dataX as possible and evade the noise in the augmented dataX ′. In addition, a basic property between mutual information and entropy information is as follow:
Property 1. The relation between mutual information and conditional entropy is I(X,Y ) = H(X)−H(X|Y ) = H(Y )−H(Y |X).
According to Property 1, Eq. (11) can be rewritten as (note that H(Z ′|X ′) is determined): IBssl = [H(Z′)−H(Z′|X)]− β[H(Z′)−H(Z′|X ′)] = (1− β)H(Z′)−H(Z′|X). (12) If 1−β 6 0, then the optimal solution in Eq. 12 will be trivial (it is also known as complete collapse or dimensional collapse (Hua et al. (2021))). To avoid this issue, we define λ = 1− β > 0 and have IBssl = λH(Z ′) − H(Z ′|X). So the connection between information bottleneck principle and
entropy information is max IBssl = max λH(Z′)−H(Z′|X) (λ ∈ (0, 1)). (13)
Zhu et al. (2020) have proved that the loss in Eq. (2) is the upper bound of InfoNCE loss: L1,2i ≥ LInfoNCE, so minimizing L 1,2 i is equivalent to minimizing LInfoNCE’s upper bound. Because InfoNCE’s principle is to pull positive pairs to be closer (i.e., to minimize H(Z ′|X)) and push negative pairs to be farther (i.e., to maximize H(Z ′)), it can be viewed as a specific form of IBssl. Therefore, we can conclude that minimizing L1,2i is equivalent to maximizing the lower bound of IBssl. Next, we discuss the effect of temperature τ for H(Z ′). We have the following proposition: Proposition 2. (Temperature τ is crucial to H(Z ′).) When each column of Z ′ is normalized, the node embedding’s entropy H(Z ′) increases with the decrease of τ .
We refer the reader to Appendix A.2 for detailed proof. In Proposition 2, the condition of the normalized column inZ ′ is actually hard to meet. Also, if always keeping a low τ and only focusing on the local separation (as shown in Figure 1 (c)), it is easy to generate the problem of embedding misplacement. So the local separation (low τ ) should be arranged after the global uniformity (high τ ).

heading: 4 EXPERIMENTS
text: In this section, we verify GLATE’s performance on two most important graph-related tasks: node classification and graph classification. The node classification task includes two types of learning: transductive learning and inductive learning1. With the help of the loss curve, global uniformity metric and local separation metric, we design some ablation studies to understand the contributions of different components in GLATE to the overall system. For the detailed dataset descriptions and experimental setups, please refer to Appendix A.4 and A.5. More experimental designs and results can be seen in Appendix A.6.

heading: 4.1 NODE CLASSIFICATION TASK

heading: 4.1.1 RESULTS OF TRANSDUCTIVE LEARNING
text: We compare GLATE with several baseline models including the supervised GCN (Kipf & Welling (2016)) model and self-supervised GCL models, as well as several variants of GLATE.
We compare six different GCL models which implement a variety of techniques. DGI (Velickovic et al. (2019)) and GMI (Peng et al. (2020)) use graph contrastive objectives based on the mutual information maximization. MVGRL (Hassani & Khasahmadi (2020)) learns node’s and graph’s representations from two structural views: first-order neighbors and a graph diffusion. Following SimCLR (Chen et al. (2020)), GRACE (Zhu et al. (2020)) and GCA (Zhu et al. (2021c)) employ InfoNCE loss to learn node representations for graph data. Inspired by BYOL (Grill et al. (2020)), BGRL (Thakoor et al. (2021)) imitatively uses an asymmetric encoder structure to get rid of GCL’s dependence on negative pairs. The variants of GLATE include four GLATE models with fixed temperature values and a GLATE model with the hard negative sampling. The hard negative sampling strategy only collects hard negative examples (the similarity threshold δ = 0.8) to construct contrastive pairs.
Table 1 shows the accuracy scores on the task of transductive node classification. We can find that GLATE outperforms all the baseline models on the three datasets with an average improvement of 2.8 percent. Unsurprisingly, an extremely high (or low) temperature value cannot make the model converge. By setting dynamic temperature values, GLATE performs better than the models with a fixed temperature value, especially on Citeseer. Other experiments about the temperature values can be seen in Section 4.3.

heading: 4.1.2 RESULTS OF INDUCTIVE LEARNING
text: 1Our code is available at: https://github.com/anonymousICLR22/GLATE.
Methods PPI Raw features 42.20
Supervised GraphSAGE-GCN 96.90±0.20 GraphSAGE-GCN 46.50 GraphSAGE-mean 48.60
GraphSAGE-LSTM 48.20 GraphSAGE-pool 50.20
DGI+GraphSAGE−GCN 63.80±0.20 GMI+GraphSAGE−GCN 65.00±0.02 GRACE+GraphSAGE−GCN 66.20±0.10 BGRL+GraphSAGE−GCN 67.52±0.05 GLATE+GraphSAGE−GCN 68.41±0.01
Table 2: Comparison with the state-of-the-art GCL methods on inductive node classification.
For the task of inductive learning, we use GraphSAGE-GCN (Hamilton et al. (2017)) as the encoder of GLATE and other GCL baseline models including DGI, GMI, GRACE, and BGRL. We also compare our model with unsupervised GraphSAGE models with different aggregation functions, which are denoted as GraphSAGE-*. The method of “Raw features” only uses node features as inputs, so its accuracy score can be regarded as the lower-bound of all methods. “Supervised GraphSAGEGCN” uses supervised learning with the encoder of GraphSAGE-GCN and its accuracy score can be seen as the upper-bound of all methods. From Table 2, we find that
GLATE+GraphSAGE−GCN outperforms all the self-supervised baselines and is 0.89 percent ahead of the nearest competitor BGRL. Moreover, all the GCL models perform better than unsupervised GraphSAGE models, which shows the effectiveness of GCL on inductive node classification.

heading: 4.1.3 RESULTS ON LARGER GRAPH DATASET
text: To verify GLATE’s scalability, we select a larger graph dataset ogbn-arxiv2 to test its performance. The ogbn-arxiv dataset is a directed graph, representing the citation network between all Computer Science (CS) arXiv papers indexed by Microsoft Academic Graph (MAG). There are 169,343 nodes and 1,166,243 edges in ogbn-arxiv. The results are shown in Table (3). Apart from GLATE,
all of the reported experimental results are from Thakoor et al. (2021). For GRACE and GLATE, we subsample 2048 nodes from the full graph as each anchor node’s negative examples. From Table 3, we can see that GLATE outperforms the three GCL methods and is competitive with the supervised GCN model on validation and test sets.
2https://ogb.stanford.edu/docs/nodeprop/

heading: 4.2 GRAPH CLASSIFICATION TASK
text: Next, we evaluate GLATE’s performance on the graph classification task using four graph benchmark datasets. Here we choose the following methods as baseline models: node2vec (Grover & Leskovec (2016)), sub2vec (Adhikari et al. (2018)), graph2vec (Narayanan et al. (2017)), InfoGraph (Sun et al. (2020)), and GraphCL (You et al. (2020)). All the baselines and GLATE are used to generate graph embeddings by unsupervised learning. These embeddings are as the inputs of the downstream SVM classifier. In GLATE, we use graph isomorphism network (Xu et al. (2018)) as its encoder networks, which is the same as GraphCL’s. The results are shown in Table 4. Apart from GLATE, all of the reported results are from (You et al. (2020)). From Table 4, we can see that GLATE outperforms all the baselines on most datasets except on IMDB-B with small graph size (the number of average nodes is less than 20). Besides, compared with another GCL method GraphCL, the average accuracy result of GLATE is higher than it 1.2 percent.

heading: 4.3 ABLATION STUDIES

heading: 4.3.1 PERFORMANCE ON THE TRAINING LOSS
text: In this section, we use different temperature mechanisms to visualize the effects of temperature and momentum on the training loss. Specifically, we design different training strategies to analyze the components of temperature and momentum, including 1) removing dynamic temperature estimation by fixing τ as 0.4 or 0.8; 2) replacing the momentum with an alternative dynamic solution, i.e, multiplying τ by a fixed decay factor at each epoch; 3) removing the constraint of the temperature’s lower-bound (denoted as “GLATE w/o τlower” in Figure 3); 4) our proposed model GLATE (denoted as “GLATE w/ τlower” in Figure 3). We visualize the change of training loss under the above strategies for transductive learning in Figure 3.
To evaluate the static and dynamic settings, we mainly focus on the performances of “τ = 0.4”, “τ = 0.8”, and “GLATE w/ τlower” in Figure 3. Although GLATE converges slower than the settings of τ = 0.4 and τ = 0.8, it eventually obtains a lower loss value. Compared with the static setting, the dynamic temperature estimation can explore a larger learning space, which improves the model’s learning ability.
To evaluate the effect of the momentum mechanism, we mainly focus on “dynamic decay”, “GLATE w/o τlower”, and “GLATE w/ τlower” in Figure 3. Both GLATE w/o τlower and GLATE w/ τlower
have faster convergence speeds than the setting of dynamic decay, which indicates the effectiveness of momentum. In addition, when without the constraint of τlower, the model’s convergence speed becomes very fast during training, which eventually leads to the overflow of loss (“NaN”). So we can conclude that both momentum and τlower are necessary to GLATE.

heading: 4.3.2 PERFORMANCE ON GLOBAL UNIFORMITY AND LOCAL SEPARATION
text: As we discussed in previous sections, global uniformity and local separation are two important metrics for measuring the quality of graph contrastive representations. We use −Dtglobal and −Dtlocal (global uniformity degree Dtglobal and local separation degree D t local are defined in Eq. (9) and Eq. (10), respectively) as the coordinates to evaluate the quality of learned representations in GLATE. We plot the changing trend of results under these two metrics in Figure 4. The datum of each epoch is a single point. Because the coordinates mean the negative uniformity degree and negative separation degree, the ideal result lies in the left lower corner. From Figure 4, we find all of the three experimental settings (GLATE, GLATE with a fixed temperature τinitial = 0.8, as well as GLATE with a fixed temperature τlower = 0.2) can make the results of uniformity and separation better in the training phase. Moreover, the performance of GLATE is significantly better than those of the other two methods. Note that the changing trend on Pubmed is different from those on Cora and Citeseer because we implement mini-batch training for the dataset of Pubmed and full-batch training for the other two datasets, respectively.

heading: 5 CONCLUSION AND DISCUSSION
text: In this paper, we discuss the limitations of the classical InfoNCE objective for graph contrastive learning: when using a fixed temperature, the InfoNCE objective can only capture a certain state of embedding distribution and ignore the others. By theoretical analysis, we find that the temperature τ is crucial to control the representation distribution’s global uniformity and local separation and thus introduce GLATE, a simple but effective algorithm for graph contrastive learning. GLATE uses dynamic temperature estimation with momentum to adjust the gradient rates so that it can cater to different distribution states. On both tasks of node classification and graph classification, we show GLATE’s advantages over the state-of-the-art GCL algorithms. By some ablation studies, we illustrate the positive effects brought by the dynamic temperature setting and visualize the results under the two new metrics in the training phase.
Broader impacts. Studying the temperature’s property is an interesting topic in contrastive learning, even in deep learning. Considering the dominant role of InfoNCE objectives in graph contrastive learning, we believe that deep insights into the temperature in InfoNCE objectives have an important impact on the development of graph contrastive learning. In our work, the gradient analysis and dynamic temperature estimation are not only suitable to GRACE but also any graph contrastive learning algorithm based on InfoNCE objectives. More experimental details about GLATE’s generality can be seen in Appendix A.6.1.

heading: A APPENDIX

heading: A.1 PROOF OF PROPOSITION 1
text: Proof. Using Eq. (4) and (5), we derive: ∇S(i′,j′)L1,2i ∇S(i′,j′′)L1,2i = Ei′,j′ Ei′,j′′ = exp( 1 τ · (S(i′, j′)− S(i′, j′′)). (14) Here 1/τ is a scaling coefficient of S(i′, j′) − S(i′, j′′). If τ → ∞, the whole exponential term tends to 1, so we conclude ∇S(i′,j′)L1,2i → ∇S(i′,j′′)L 1,2 i . If τ decreases, the difference between ∇S(i′,j′)L1,2i and ∇S(i′,j′′)L 1,2 i will be larger.

heading: A.2 PROOF OF PROPOSITION 2
text: Property 2. IfZ obeys a D-dimensional Gaussian distribution: Z ∼ N (µZ ,ΣZ ) (µZ ∈ R 1×D,,
ΣZ ∈ R D×D), then H(Z) =
D 2 (ln2π + 1) + 1 2 ln(detΣZ ).
Proof. If τ decreases, then the learning algorithm will punish more on hard examples than easy examples. So it makes the similarity between hard negative examples lower. Besides, the easy negative examples are always dissimilar. So we have Z ′iZ ′T j → 0 (i 6= j) and Z ′ iZ ′T i → 1. Due to n d (dense node representations), the (normalized) columns ofZ ′ tend to be linearly independent and Z ′TZ ′ → ID (ID is an identity matrix, ID ∈ RD×D). Due to (ΣZ ′)i,j = Cov[Z ′ i,Z ′ j ], we can rewrite the determinant of ΣZ ′ as:
det(ΣZ ′) = det(Z ′TZ ′) = D∏ i=1 λi = exp( D∑ i=1 logλi) (15)
where λ1, λ1, · · ·λD are ΣZ ′ ’s eigenvalues. According to Jensen inequality, as log(·) is a concave function, we have 1D ∑D i=1 logλi 6 log ∑D i=1 λi D = 0 (note that ∑D i=1 λi = trace(ΣZ ′) = D). So det(ΣZ ′) 6 exp(0) = 1. Note that if Z ′TZ ′ → ID, then det(ΣZ ′) → 1 (i.e., its upper bound) and H(Z ′) tends to be its global maximum (according to Property 2).

heading: A.3 THEORETICAL DIFFERENCE WITH THE CURRENT WORK
text: We investigate another work (https://arxiv.org/abs/2106.05819) using the analysis of the graph information bottleneck principle. Here we give some explanation about the difference between our theoretical basis and theirs. The main difference is the negative terms in the information bottleneck when ignoring the impact of the coefficients. We analyze the information bottleneck principles of https://arxiv.org/abs/2106.05819 (denoted as IB1ssl) and our paper (denoted as IB 2 ssl) in detail and give the proof below. For the task of learning node representations Z ′ and Z ′′, Equation (5) in https://arxiv.org/abs/2106.05819 can be represented as: IB1ssl = I(Z ′, Z ′′) = H(Z ′)−H(Z ′|Z ′′). As H(Z ′|Z ′′) = H(Z ′, Z ′′)−H(Z ′′), then
IB1ssl = H(Z ′) +H(Z ′′)−H(Z ′, Z ′′). (16)
While, in our paper (Equation (11)): IB2ssl = λH(Z ′)−H(Z ′|X) (λ ∈ (0, 1)). Because Z ′′ has the same property as Z ′, we have:
IB2ssl = λ 2 ∗ (H(Z ′) +H(Z ′′))− 1 2 ∗ (H(Z ′|X) +H(Z ′′|X)). (17)
Regardless of the effect of coefficients, an intuitive understanding to Equation (16) and (17) is: the common goal of IB1ssl and IB 2 ssl is to learn the most informative part of raw dataX and remove the redundant information from it. Although IB1ssl and IB 2 ssl have consistent optimization objectives,
they have a minor difference at the minimization term: one is to minimize H(Z ′, Z ′′) and the other is to minimize H(Z ′|X) + H(Z ′′|X). It implies that IB1ssl replies more on the choice of predefined graph augmentation strategies, while IB2ssl is not limited to graph augmentation strategies. In a summary, if we aim to select the most suitable graph augmentation strategies, IB1ssl is more helpful; but if we want to avoid the bad impact of an improper choice of graph augmentation, IB2ssl is better than IB1ssl.

heading: A.4 DATASETS
text: For the node classification task, we utilize some commonly-used benchmarks in the experiments, including four citation networks (Cora, Citeseer, Pubmed, and ogbn-arxiv) for the transductive singlelabel node classification task, as well as the Protein-Protein Interaction dataset (PPI) for the inductive multi-label node classification task. In the citation networks, each node represents a paper whose features are topic types (Cora, Citeseer, and Pubmed) or are a vector obtained by averaging the embeddings of words in its title and abstract (ogbn-arxiv), and each edge represents the citation relationship between papers. In the PPI dataset, there are 24 graphs. Each node in the graphs of PPI denotes a protein with biological features, and each edge denotes the interaction between proteins. The details of the datasets are in Table 5.
For the graph classification task, we use four datasets including NCI1 (small molecules), PROTEINS (small molecules), MUTAG (bioinformatics), and IMDB-B (social networks). Their detailed information is shown in Table 6.

heading: Datasets # Nodes # Edges # Features # Classes

heading: A.5 EXPERIMENTAL SETUPS
text: Under the contrastive setting, we use data augmentation strategies, i.e., edge removing and node feature masking, to generate the new graphs G̃1 = (A′,X ′) and G̃2 = (A′′,X ′′). For the transductive task, we use a classical two-layer GCN model (Kipf & Welling (2016)), whose message passing form is Eq. (1), as the encoder of GLATE. For the inductive task, the encoder model is a two-layer GraphSAGE model (Hamilton et al. (2017)). Its message passing process can be formalized as:
Zki = σ ( W · CONCAT(Zk−1i ,AGGREGATE({Z k−1 j , j ∈ N (i)})) ) (18)
where Z0i = Xi, σ denotes the activation function, N (i) denotes the neighbors of node i, the aggregation function AGGREGATE(·) can employ arbitrary aggregators such as GCN, Mean, LSTM or Pool. In experiments, we consistently choose GCN as the aggregator for all methods.
To evaluate the performance of the GLATE, we use a logistic regression model with a one-vs-rest classifier framework for the downstream classification tasks. The size of both the hidden layer representations and the node representations are set to 256 for the transductive task and 512 for the inductive task. The temperature’s initial value τinitial is set as 0.8 and τ will decrease every 20 epochs until τlower = 0.2 in all the experiments of GLATE. The model runs 200 epochs and 100 epochs for the transductive and inductive tasks, respectively.

heading: A.6 MORE EXPERIMENTS

heading: A.6.1 THE GENERALITY OF DYNAMIC TEMPERATURE ESTIMATION
text: Considering the dominant role of InfoNCE objectives in GCL, we believe that deep insights into the temperature in InfoNCE objectives will have an important impact on the development of GCL. To validate our method’s generality, we conduct experiments for another GCL method GraphCL using the InfoNCE objective on four graph datasets. The results of the graph classification task are shown in Table 7 (“Improved GraphCL” denotes a new GraphCL method using dynamic temperature estimation). We can see that the dynamic temperature estimation indeed brings positive effects to GraphCL.

heading: A.6.2 TIME COMPLEXITY ANALYSIS
text: Because τ decreases every 20 epochs, compared with GRACE, the extra time cost for computing a new τ ’s value is minor. When computing the full negatives, GLATE’s time complexity is quadratic in the size of the input graph per epoch (O(N2)). When computing the sampled negatives (we denote the number of sampled negatives as k), GLATE’s time complexity is linear in the size of the input graph per epoch (O(kN )).
We also record the total running time of the two most representative baselines and GLATE in Table 8. We run all experiments on NVIDIA GeForce GTX 1080 Ti. As GLATE relies on less training epochs to achieve competitive results than GRACE and BGRL, it spends less training time than the other baselines.

heading: Methods Amazon-Computers Amazon-Photo Coauthor-CS Coauthor-Physics

heading: A.6.3 EXPERIMENTAL RESULTS ON THE OTHER GRAPH DATASETS.
text: We evaluate GLATE’s performance on the datasets of Amazon-Computers, Amazon-Photo, Coauthor-CS, and Coauthor-Physics, which are also used in GCA. The graphs in these datasets are larger than those in Cora and Citeseer. For example, there are totally 13,752 nodes and 245,861 edges in the graph of Amazon-Computers; there are totally 34,493 nodes and 247,962 edges in the graph of Coauthor-Physics. The experimental results are shown in Table 9. We can see that GLATE outperforms GRACE and is competitive with BGRL on the four datasets.

heading: A.6.4 SENSITIVITY ANALYSIS
text: We analyze the effects of different hyperparameters p1 and p2 (p1, p2 are the probability of removing edges and masking features) for GLATE’s performance. We plot the accuracy score’s results on the datasets of Cora, Citeseer, and Pubmed in Figure 5. Although GLATE’s best results occur at different locations (e.g., p1 = p2 = 0.8 for Citeseer, p1 = 0.1 and p2 = 0.4 for Pubmed) in different datasets, we note that the average of the difference between peak and valley performance values is less than 9 percent. It indicates that GLATE is not very sensitive to the two hyperparameters. Therefore, GLATE is robust to p1 and p2.
Moreover, we evaluate GLATE’s performance using different selections of τinitial under different initial contrastive views. The results on Cora are shown in Table 10 (p1 and p2 denote the rate of removing edges and masking features, respectively; the number in square brackets is the selected value of τinitial). We can see that using high τinitial (such as 0.6, 0.8, and 1.0) is better when p1 and p2 are large (top left corner), while using low τinitial is more suitable to the opposite situation (bottom right corner). As the lower-bound of τ is fixed as 0.2, we think it is reasonable because a wider adjustable range of temperatures is helpful to GLATE to learn from more noisy data.

Review summaries:
This paper investigates a crucial problem, i.e., how to generate good node representations under the InfoNCE contrastive loss in graphs. By studying the feature of loss function’s gradients, it finds that the dynamic temperature change is beneficial to learn uniform node representations. The proposed method GLATE is simple but effective. The analysis of the connection between GLATE and the information bottleneck principle helps us understand why GLATE is so effective. The presentation of this paper is clear and well-organized. For example, Figure 1 makes us easy to understand the problem generated by the static temperature and the significance of this work. The experiments are laid out in detail. The results on the tasks of transductive and inductive learning show GLATE’s advantage over the SOTA graph contrastive learning algorithms.

The authors explore the role of the temperature in the loss function for graph contrastive learning. They argue that global uniformity and local separation are both necessary to the learning quality and this can be controlled by the temperature. Thus, they develop a simple but effective algorithm GLATE to dynamically adjust the temperature value in the training phase. Experiments are also conducted to demonstrate the effectiveness of the method.