Rethinking Temperature in Graph Contrastive Learning

Abstract: Due to not relying on the rare human-labeled information, self-supervised learning, especially contrastive learning, attracted much attention from researchers. It has begun to show its strong advantages on both IID data (independent and identically distributed data, such as images and texts) and Non-IID data (such as nodes in graphs). Recently, researchers begin to explore the interpretability of contrastive learning and have proposed some metrics for measuring the learned representations' qualities of IID data, such as alignment, uniformity, and semantic closeness. It is very important to understand the relationships among node representations, which is helpful to design algorithms with stronger interpretability. However, few studies focus on evaluating good node representations in graph contrastive learning. In this paper, we investigate and discuss what a good representation should be for a general loss (InfoNCE) in graph contrastive learning. By theoretical analysis, we argue that global uniformity and local separation are both necessary to the learning quality. We find that the two new metrics can be regulated by the temperature coefficient in InfoNCE loss. Based on this characteristic, we develop a simple but effective algorithm GLATE to dynamically adjust the temperature value in the training phase. GLATE outperforms the state-of-the-art graph contrastive learning algorithms 2.8 and 0.9 percent on average under the transductive and inductive learning tasks, respectively. The code is available at: https://github.com/anonymousICLR22/GLATE.

Reviews and author responses:

 Dear Reviewer eDkW, thank you for your response. Our reply is as follows:  
__1. GraphCL actually uses inter-view negative examples, and the role of analyzing temperature in the loss function in GRACE can also be applied to the graph-level loss function.__ In Equation (3) of [1], we find that GraphCL uses the inter-view negative examples: $z\_{n,i}$ and $z\_{n',j}$ (note that $i$ and $j$ represent different graph views, i.e., $\hat{\mathcal{G}\_i}$ and $\hat{\mathcal{G}\_j}$). In this situation, the role of analyzing temperature in the loss function in GRACE is still applicable. We give a brief explanation: When applying GLATE for the graph-level task, the contrastive loss function is:
$
\mathcal{L}^{1,2}\_{i}=
-\mathrm{log}\frac{\mathrm{exp}(\frac{1}{\tau }\cdot \mathrm{S}(i', i''))}{\mathrm{exp}(\frac{1}{\tau }\cdot \mathrm{S}(i', i''))+
\sum_{k\neq i}\mathrm{exp}(\frac{1}{\tau }\cdot\mathrm{S}(i', k''))}.
$
Then the gradient results to positive examples and negative examples are respectively:
$
\displaystyle \nabla_{\mathrm{S}(i', i'')}\mathcal{L}^{1,2}\_i=-\frac{\sum_{k\neq i}\mathrm{E}\_{i', k''}}{\tau \cdot (\mathrm{E}\_{i', i''}+\sum_{k\neq i}\mathrm{E}\_{i', k''})}
$ and 
$
\displaystyle \nabla_{\mathrm{S}(i', j'')}\mathcal{L}^{1,2}\_i=-\frac{\mathrm{E}\_{i', j''}}{\tau \cdot (\mathrm{E}\_{i', i''}+\sum_{k\neq i}\mathrm{E}\_{i', k''})}.
$
The rate between the above gradients is:
$
\mathrm{R}(i',j'';i',i'')=\frac{\nabla\_{\mathrm{S}(i', j'')} \mathcal{L}^{1,2}_i}{\nabla\_{\mathrm{S}(i', i'')}\mathcal{L}^{1,2}\_i}=\frac{\mathrm{E}\_{i', j''}}{\sum\_{k\neq i}\mathrm{E}\_{i', k''}+\sum\_{k\neq i}\mathrm{E}\_{i', k'}}.
$
This rate obeys the Boltzmann distribution, so we can give the same conclusion of temperature's effects (i.e., high $\tau$ is helpful to the embedding’s global uniformity and low $\tau$ is helpful to the embedding’s local separation ) as that in the node-level task.  
__2. When integrating dynamic temperature estimation into GraphCL (called "Improved GraphCL"), we do not include positive examples in the denominator of the contrastive loss.__ In our previous response, we reported the results of two groups of different experimental settings (called "Improved GraphCL" and "GLATE") under the graph classification task. In the method of "GLATE", we include positive examples in the denominator of the contrastive loss to ensure the consistency between the node-level task and graph-level task. However, in the method of "Improved GraphCL", we follow the original GraphCL's design and do not include positive examples (see Equation (3) in [1]). We calculate the uniformity degree of graphs in the minibatch and apply it for updating the current state's temperature. Here we show the main difference between the code of GraphCL and that of "Improved GraphCL":
```python
def loss_cal(self, x, x_aug):
    batch_size, _ = x.size()
    x_abs = x.norm(dim=1)
    x_aug_abs = x_aug.norm(dim=1)
    sim_matrix = torch.einsum('ik,jk->ij', x, x_aug) / torch.einsum('i,j->ij', x_abs, x_aug_abs)
    sim_matrix = torch.exp(sim_matrix / 0.2)              # GraphCL
    pos_sim = sim_matrix[range(batch_size), range(batch_size)]
    loss = pos_sim / (sim_matrix.sum(dim=1) - pos_sim) 
    loss = - torch.log(loss).mean()
    return loss
```

```python
def loss_cal(self, x, x_aug):
    self.tau1 = self.momentum(self.tau1, x)           # using momentum to update temperature
    batch_size, _ = x.size()
    x_abs = x.norm(dim=1)
    x_aug_abs = x_aug.norm(dim=1)
    sim_matrix = torch.einsum('ik,jk->ij', x, x_aug) / torch.einsum('i,j->ij', x_abs, x_aug_abs)         
    sim_matrix = torch.exp(sim_matrix / self.tau1)       # Improved GraphCL
    pos_sim = sim_matrix[range(batch_size), range(batch_size)]
    loss = pos_sim / (sim_matrix.sum(dim=1) - pos_sim)              
    loss = - torch.log(loss).mean()
    return loss
```


[1] You Y, Chen T, Sui Y, et al. Graph contrastive learning with augmentations. NeurIPS, 2020.

 Thanks to the author's response.

However, with Reviewer xNkv, I still have doubts about the application of graph-level. The experiments done by the authors can only show that the model structure using GLATE can be used for graph classification problems, but that leaves out the premise of inter and intra-view, so how is this analyzed for temperature? As far as I understand personally, the role of analyzing temperature in the loss function in GRACE cannot be applied to the graph-level loss function because you cannot find intre and intra-view. 

I hope the author can explain in detail how dynamic temperature estimation is integrated into GraphCL as you reported [here](https://openreview.net/forum?id=vnOHGQY4FP1&noteId=gT8Z8vFHpT1)?

 We would like to thank you for your response and suggestion. Just a kindly reminder: Whether has our response addressed your concerns? We hope to receive your valuable comments.  
Sincerely,  
Authors

 We would like to thank you for your response and suggestion. Just a kindly reminder: Whether has our response addressed your concerns? We hope to receive your valuable comments.  
Sincerely,  
Authors

 We would like to thank you for your response and suggestion. Just a kindly reminder: Whether has our response addressed your concerns? We hope to receive your valuable comments.  
Sincerely,  
Authors

 We sincerely appreciate all reviewers for their hard work and helpful comments. We would like to address all reviewers’ main concerns in the corresponding responses. We also updated our manuscript according to the comments.
The change we made mainly includes (highlighted in blue in the revised manuscript):
1. We add the background information of contrastive objectives, and give more formal definitions of global uniformity and local separation in Equation (9) and Equation (10).
2. We add more descriptions about the importance of InfoNCE objectives (Section 2 and 5) and experiments on GraphCL (Appendix A.6.1).
3. We add the experiments on larger datasets such as ogbn-arxiv (Section 4.1.3), Amazon (Appendix A.6.3), and Coauthor (Appendix A.6.3)
4. We explain the difference between our work and one recent work (https://arxiv.org/abs/2106.05819) in Appendix A.3.
5. We add more descriptions about why we use full negatives, not sampling negatives in Section 3.2.
6. We add the experiments on the graph classification task in Section 4.2.
7. We improve the description information about Figure (1).
8. We add more references (e.g. [1], [2], [3], [4], and [5]), which are related to our work in Section 3.3 and Reference.
9. We add the experiments on temperature selection in Appendix A.6.4.
10. We add the discussion of time complexity and experimental results of time costs in Appendix A.6.2.

[1] Erlin Pan and Zhao Kang. Multi-view contrastive graph clustering. NeurIPS 2021.  
[2] Dongkuan Xu,Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang. InfoGCL: Information-aware
graph contrastive learning. NeurIPS 2021.  
[3] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph
contrastive learning with augmentations. NeurIPS 2020.  
[4] Yanqiao Zhu, Yichen Xu, Hejie Cui, Carl Yang, Qiang Liu, and Shu Wu. Structure-aware hard
negative mining for heterogeneous graph contrastive learning. DLG@KDD 2021.  
[5] Yanqiao Zhu, Yichen Xu, Qiang Liu, and ShuWu. An empirical study of graph contrastive learning.
NeurIPS 2021 (Datasets and Benchmarks).

This paper investigates a crucial problem, i.e., how to generate good node representations under the InfoNCE contrastive loss in graphs. By studying the feature of loss function’s gradients, it finds that the dynamic temperature change is beneficial to learn uniform node representations. The proposed method GLATE is simple but effective. The analysis of the connection between GLATE and the information bottleneck principle helps us understand why GLATE is so effective. The presentation of this paper is clear and well-organized. For example, Figure 1 makes us easy to understand the problem generated by the static temperature and the significance of this work. The experiments are laid out in detail. The results on the tasks of transductive and inductive learning show GLATE’s advantage over the SOTA graph contrastive learning algorithms. Pros:
1. An interesting and valuable problem. Understanding the contrastive representations is helpful to improve the current contrastive learning framework and promote the development of contrastive learning. Although some works have explored what is a good image representation and proposed the golden rule of alignment and uniformity, there is a lack of studies about the quality of node contrastive representation in a graph.
2. Simple, provable, and effective algorithm design. Inspired by Momentum, the authors dynamically adjust the temperature’s value along with the representation’s uniformity degree in the training phase. This solution is easy to implement and its performance on the downstream task is pleasing.
3. The experiment’s design is rigorous and its results are persuasive. For instance, in the inductive learning task, the comparison between baselines and GLATE is fair due to using the same choice (GraphSAGE-GCN) for the encoder network.

Suggestions:
1. Lacking the discussion of time complexity or experimental results of time costs. I notice that the authors have described a specific experimental setting in the appendix (A.2 EXPERIMENTAL SETUPS), but whether the setting of changing temperature every 20 epochs is reasonable and why the number is 20? I think authors should discuss the effect of this setting and show the comparison of time costs between GLATE and baselines.
2. It is interesting to see whether it is necessary to use different initial temperature values for different graph datasets. For different graph datasets, their initial uniformity degrees are different, so I think it will be better to design a task-dependent GALTE, not a task-agnostic GLATE. This paper investigates an interesting and valuable problem, solve it via  simple, provable, and effective algorithm. And the experiments are sufficient.

 Thank you for your constructive comments. We appreciate that you took the time to comment on our paper. We will submit the revised manuscript soon after. We respond to each comment below.

__1. Question__: The global uniformity and local separation are not clearly defined. It is better to provide a background of dissecting contrastive objectives and then proceed to the problem of the existing formulation.

_Response_: In Section 1 of the submitted manuscript, we mention that ``we propose two new metrics: global uniformity and local separation (Eq. (8) and Eq. (12))''. The formal definition of global uniformity is Equation (8) and the formal definition of local separation is Equation (12).  As you suggest, we will add more background information about different types of contrastive objectives (including JSD objectives, TM objectives, BL objectives, etc) and highlight the definitions of global uniformity and local separation in the revised manuscript.

__2. Question__: How does the analysis generalize to other graph contrastive algorithms, e.g., GraphCL that uses InfoNCE objectives, DGI that uses JSD objectives, and BGRL that uses bootstrapping objectives?

_Response_: 
* First, the recent empirical studies have verified that InfoNCE objectives generally achieve better performance compared with the other graph contrastive objectives such as JSD and TM (Observation 5, https://arxiv.org/abs/2109.01116). Considering the dominant role of InfoNCE objectives in GCL, we believe that deep insights into the temperature in InfoNCE objectives will have an important impact on the development of GCL.
* Second, we conduct experiments for another GCL method GraphCL using the InfoNCE objective on four graph datasets. With the same experimental setting, the results under the graph classification task are shown as below ("Raw GraphCL" and "Improved GraphCL" denote the GraphCL method without and with applying dynamic temperature estimation, respectively). We conclude that the dynamic temperature estimation indeed brings positive effects to GraphCL.

| Methods      | NCI1          | PROTEINS        | MUTAG           | IMDB-B          |
|------------------|-------------------|---------------------|---------------------|---------------------|
| Raw GraphCL      | 77.87±0.41        | 74.39±0.45          | 86.80±1.34          | 71.14±0.44          |
| Improved GraphCL | __78.22±0.59__ | __74.58±0.66__ | __89.27±0.15__ | __71.20±0.35__ |

__3. Question__: Evaluation on large-scale datasets such as open graph benchmarks is needed.

_Response_: As most of the baselines (DGI, GMI, MVGRL, and GRACE) use Cora, Citeseer, and Pubmed as experimental datasets, we also choose them to test GLATE's performance. Like GRACE, GLATE can easily scale to larger datasets by subsampling negatives. As you suggest, we evaluate GLATE's performance on a dataset from open graph benchmarks (ogbn-arxiv). 
The ogbn-arxiv dataset is a directed graph, representing the citation network between all Computer Science (CS) arXiv papers indexed by Microsoft Academic Graph (MAG).
There are 169,343 nodes and 1,166,243 edges in ogbn-arxiv. In GRACE and GLATE, we subsample 2048 nodes as each anchor node's negatives. The results are shown as below (note that the reported results of baselines are from https://arxiv.org/pdf/2102.06514.pdf).

|  Methods | Validation of obgn-arxiv |  Test of obgn-arxiv       |
|------------|-----------------------------|---------------------------|                  
| DGI                             | 71.26±0.11                                       | 70.34±0.16                   |
| GRACE                           | 72.61±0.15                                       | 71.51±0.11                   |
| BGRL                            | 72.53±0.09                                       | 71.64±0.12                   |
| GLATE                           | __72.80±0.02__                              | __72.13±0.05__          |

Moreover, we evaluate GLATE's performance on larger datasets Amazon and Coauthor, which are also used in GCA. For example, in the graph of Amazon-Computers, there are totally 13,752 nodes and 245,861 edges; in the graph of Coauthor-Physics, there are totally 34,493 nodes and 247,962 edges. The experimental results are shown as below. We can see that GLATE outperforms GRACE and is competitive with BGRL on the four datasets.


 __5. Question__: One particular concern with the presented formulation is that, unlike visual data, negative samples involved in graph contrastive learning are not aligned with their semantic relationship (Figure 1, https://arxiv.org/abs/2110.02027), partially due to the neighborhood smoothing operation of graph neural networks. That being said, the more closeness to negative samples (in other words, more difficult negative samples) will lead to the wrong selection of false negative samples, which are actually positive examples. Therefore, I in person believe that simply adjusting the temperature parameter is not sufficient to account for the balance of global uniformity and local separation.

_Response_: 
* First, the provided material (Figure 1, https://arxiv.org/abs/2110.02027) gives an important conclusion, i.e., negative sampling strategy is suitable to the image domain, but not to the graph domain. Our work does not contradict it and we do not use any truncated negative sampling strategy in this work (in other words, we retain all negative examples for the used datasets). We have illustrated this problem in Section 3.2 of the submitted manuscript: "The truncated strategy of hard negative sampling is rational in the image domain, .... So the truncated strategy is not suitable for the graph".
* Second, as you said, more hard negatives will lead to more wrong selection of false negative samples. So the model's tolerance to false negatives becomes very necessary. Based on the gradient analysis in Section 3.1, we think that the high temperature can provide good tolerance to false negatives. So, in GLATE, the temperature value is set to be high in the early stage. The temperature value gradually becomes lower in the training process because we do not want the node embeddings of the same class to be close when calculating their distance. If not, it will lead to the local collapse problem. The low temperature can make the distance of local embedding be further apart. Therefore, we use dynamic temperature estimation in GLATE.

__6. Question__: Property 2 seems to be off-context to the main text.

_Response_: We use Property 2 to prove Proposition 2, but we move the detailed proof to Appendix due to the page's limitation. We will move Property 2 to Appendix in the revised manuscript to ensure the main body of our paper to be self-contained.

__7. Question__: "Interpretable" contrastive representations in Section 2 is not accurate.

_Response_: Here we use "Interpretable Contrastive Representation" to represent some works that investigate the quality of contrastive representations. We plan to adjust the third heading and content in Section 2 of the revised manuscript. More background information about the hard negative sampling will be included in Section 2.

 |  | $p_{1}=0.9$ | $p_{1}=0.8$ | $p_{1}=0.6$ | $p_{1}=0.4$ | $p_{1}=0.2$ |
| -|-|-|-|-|-|
|  $p_{2}=0.9$ | 82.26 ($1.0$)| 83.35 ($1.0$)| 84.27 ($1.0$) | 83.58 ($1.0$) | 83.19 ($1.0$) |
|  $p_{2}=0.9$ | 83.35 ($0.8$)  | __84.02 ($\mathbf{0.8}$)__ | 82.75 ($0.8$)  | 84.46 ($0.8$)| 83.95 ($0.8$)|
| $p_{2}=0.9$  | __84.18 ($\mathbf{0.6}$)__| 82.47 ($0.6$) | __84.66 ($\mathbf{0.6}$)__ | 80.35 ($0.6$)| 83.26 ($0.6$)|
|  $p_{2}=0.9$ | 83.51 ($0.4$) | 83.83 ($0.4$) | 83.12 ($0.4$)| 82.40 ($0.4$)  | __84.92 ($\mathbf{0.4}$)__ |
| $p_{2}=0.9$  | 82.50 ($0.2$)  | 83.88 ($0.2$) | 82.89 ($0.2$)   | __85.33 ($\mathbf{0.2}$)__ | 84.06 ($0.2$)   |
|   |   |   |   |   |   |
| $p_{2}=0.8$  | __84.52 ($\mathbf{1.0}$)__     | 83.93 ($1.0$) | __84.92 ($\mathbf{1.0}$)__ | 81.85 ($1.0$)   | 83.95 ($1.0$)   |
| $p_{2}=0.8$  | 82.73 ($0.8$)  | __$\mathbf{\underline{86.07}}$ ($\mathbf{\underline{0.8}}$)__ | 82.03 ($0.8$)   | 81.90 ($0.8$)   | 83.37 ($0.8$)   |
| $p_{2}=0.8$  | 83.46 ($0.6$)  | 83.79 ($0.6$) | 81.20 ($0.6$)   | 82.68 ($0.6$)   | 82.54 ($0.6$)   |
|  $p_{2}=0.8$ | 82.20 ($0.4$)  | 82.75 ($0.4$) | 82.20 ($0.4$)   | 83.69 ($0.4$)   | 83.51 ($0.4$)   |
| $p_{2}=0.8$  | 83.90 ($0.2$)  | 84.32 ($0.2$) | 82.73 ($0.2$)   | __84.55 ($\mathbf{0.2}$)__ | __84.39 ($\mathbf{0.2}$)__ |
|   |   |   |   |   |   |
| $p_{2}=0.6$  | 83.39 ($1.0$)  | 83.74 ($1.0$) | 84.11 ($1.0$)   | 82.52 ($1.0$)   | 81.53 ($1.0$)   |
| $p_{2}=0.6$  | __84.36 ($\mathbf{0.8}$)__     | 82.52 ($0.8$) | 82.40 ($0.8$)   | 83.00 ($0.8$)   | 82.73 ($0.8$)   |
|  $p_{2}=0.6$ | 83.97 ($0.6$)  | 83.33 ($0.6$) | 84.13 ($0.6$)   | 82.84 ($0.6$)   | 82.29 ($0.6$)   |
| $p_{2}=0.6$  | 83.69 ($0.4$)  | __85.40 ($\mathbf{0.4}$)__  | __84.52 ($\mathbf{0.4}$)__ | __84.73 ($\mathbf{0.4}$)__ | 84.18 ($0.4$)   |
|  $p_{2}=0.6$ | 83.19 ($0.2$)  | 83.99 ($0.2$) | 83.00 ($0.2$)   | 82.66 ($0.2$)   | __84.39 ($\mathbf{0.2}$)__ |
|   |   |   |   |   |   |
|  $p_{2}=0.4$ | 83.33 ($1.0$)  | __84.89 ($\mathbf{1.0}$)__ | __85.06 ($\mathbf{1.0}$)__ | 84.52 ($1.0$)   | 82.66 ($1.0$)   |
|  $p_{2}=0.4$ | 82.63 ($0.8$)  | 83.74 ($0.8$) | 84.27 ($0.8$)   | 84.11 ($0.8$)   | 83.72 ($0.8$)   |
| $p_{2}=0.4$  | 83.51 ($0.6$)  | 84.52 ($0.6$) | 82.52 ($0.6$)   | 84.48 ($0.6$)   | 84.78 ($0.6$)   |
| $p_{2}=0.4$  | 84.41 ($0.4$)  | 84.29 ($0.4$) | 81.41 ($0.4$)   | __84.70 ($\mathbf{0.4}$)__ | __85.18 ($\mathbf{0.4}$)__ |
| $p_{2}=0.4$  | 82.84 ($0.2$)  | 84.78 ($0.2$) | 81.76 ($0.2$)   | 84.09 ($0.2$)   | 83.42 ($0.2$)   |
|   |   |   |   |   |   |
| $p_{2}=0.2$  | 83.81 ($1.0$)  | 80.51 ($1.0$) | 83.38 ($1.0$)   | 82.33 ($1.0$)   | 83.16 ($1.0$)   |
| $p_{2}=0.2$  | 82.80 ($0.8$)  | 84.20 ($0.8$) | __84.80 ($\mathbf{0.8}$)__ | 83.69 ($0.8$)   | 82.38 ($0.8$)   |
|  $p_{2}=0.2$ | 82.75 ($0.6$)  | 82.86 ($0.6$) | 84.52 ($0.6$)   | 83.97 ($0.6$)   | 82.66 ($0.6$)   |
|  $p_{2}=0.2$ | 83.76 ($0.4$)  | __85.40 ($\mathbf{0.4}$)__    | 82.39 ($0.4$)   | 83.19 ($0.4$)   | __84.27 ($\mathbf{0.4}$)__ |
|  $p_{2}=0.2$ | __84.46 ($\mathbf{0.2}$)__     | 82.31 ($0.2$) | 81.72 ($0.2$)   | __85.08 ($\mathbf{0.2}$)__ | 83.53 ($0.2$) |

__3. Question__: It is suggested that more content be added to explain and analyze why the truncated strategy cannot be applied to graph contrastive learning.

_Response_: Thank you for your constructive comment. After some thinking, in our humble opinion, the negative distribution of the truncated strategy in the graph is not the same as that in the image (Figure1, https://arxiv.org/abs/2110.02027). In the graph, most of the negatives being highly similar to the anchor example are positives actually (i.e., sampling bias). In addition, a nice negative sampling should be sub-linearly correlated to positive sampling: $p\_{neg}(i|j)\propto {p\_{pos}(i|j)}^{\alpha },\ \alpha \in (0,1)$, but the real $p\_{pos}$ is unknown and its approximation is hard to define (https://dl.acm.org/doi/abs/10.1145/3394486.3403218). So we retain all negatives in GLATE.
We will add the above-mentioned explanation about why the truncated strategy cannot be applied to graph contrastive learning in the revised manuscript.

__4. Question__: If possible, it is recommended to add results on larger datasets, such as amazon and coauthor datasets also used in GCA.

_Response_: We run GLATE on Amazon, Coauthor, and ogbn-arxiv datasets. The statistics of the three datasets and the experimental results are shown as below (note that the reported results of baselines are from https://arxiv.org/abs/2102.06514). On the whole, GLATE shows a good performance on these larger datasets.

 Thank you for your constructive comments. We appreciate that you took the time to comment on our paper. We will submit the revised manuscript soon after. We respond to each comment below.

__1. Question__: Lacking the discussion of time complexity of time costs.

_Response_: Because $\tau$ decreases every 20 epochs, compared with GRACE, the extra time cost for computing a new $\tau$'s value is minor. When computing the full negatives, GLATE's time complexity is quadratic in the size of the input graph per epoch (O($N^2$)). When computing the sampled negatives (we denote the number of sampled negatives as $k$), GLATE's time complexity is linear in the size of the input graph per epoch (O($kN$)).

We also provide the total running time of some representative baselines and GLATE in the following table (we run all experiments on NVIDIA GeForce GTX 1080 Ti). As GLATE relies on __less training epochs__ to achieve competitive results than GRACE and BGRL, it spends less training time than the other baselines.

| Methods| Cora | Citeseer | Pubmed   |
|-|-|-|-|
| GRACE | 6.7 seconds  | 11.2 seconds | 207.0 seconds  |
| BGRL | 6.3 seconds| 8.1 seconds | 149.7 seconds  |
| GLATE  | __3.9 seconds__| __5.9 seconds__ |__28.6 seconds__|

__2. Question__: For different graph datasets, their initial uniformity degrees are different, so I think it will be better to design a task-dependent GLATE, not a task-agnostic GLATE.

_Response_: In our current work, the empirical setting of $\tau_{initial}=0.8$ is a relatively good choice. Nevertheless, as you suggest, we will add the consideration of the graph dataset's features for $\tau_{initial}$ and design a task-dependent GLATE in the future. Here we show some experimental results of setting different $\tau_{initial}$ for different graph datasets to run GLATE. From these results, we conclude that it is a relatively good choice to empirically set $\tau_{initial}$ to be 0.8.

|  Methods | Cora |Citeseer |Pubmed   |
|-|-|-|-|
| GLATE ($\tau_{initial}=0.9$)| 83.38±0.12| 72.95±0.21 | 86.42±0.06  |
| GLATE ($\tau_{initial}=0.8$)    | __84.80±0.33__             | 73.40±0.19          | __87.29±0.12__ |
| GLATE ($\tau_{initial}=0.6$)    | 84.52±0.11                       | __73.42±0.65__ | 85.26±0.19          |
| GLATE ($\tau_{initial}=0.4$)    | 82.39±0.64                       | 71.06±0.13          | 84.95±0.07          |
| GLATE ($\tau_{initial}=0.2$)    | 81.72±0.15                       | 71.09±0.66          | 83.79±0.03          |

 Thank you for your constructive comments. We appreciate that you took the time to comment on our paper. We will submit the revised manuscript soon after. We respond to each comment below.

__1. Question__: One fixed $\tau$ already outperforms all baselines. In Table 1, frozen temperature ($\tau$ = $\tau_{initial}$ = 0.8) 84.32±0.15 67.83±0.22 85.23±0.08. 

_Response_: 
* As shown in Table 1, the results of frozen temperature ($\tau$ = $\tau_{initial}$ = 0.8) is: 84.32±0.15 67.83±0.22 85.23±0.08. Note that there is not fixed $\tau$ which already outperforms all baselines because frozen temperature ($\tau=\tau_{initial}=0.8$) actually performs better than baselines __only on Cora dataset__ and worse than some baselines (e.g., GRACE and GCA) on the other datasets. 
* Then the question is: why does the setting of $\tau=0.8$ outperforms better than GRACE on Cora? We think that different datasets have different requirements for the global uniformity of the learned representations. $\tau=0.4$ in GRACE is not an optimal choice for Cora and a higher $\tau$ is helpful to learn better representations for the task of node classification.

__2. Question__: The results(86.8±0.5) reported by MVGRL (which was also reproduced by some other papers) on Cora is much more than the authors’ reports(83.50±0.40).

_Response_: In experiments, we cannot reproduce the result of 86.8±0.5 with the authors' code (https://github.com/kavehhassani/mvgrl). Many users cannot also reproduce the result of 86.8±0.5, which can be seen in https://github.com/kavehhassani/mvgrl/issues/7. In some published papers, the reported result of MVGRL on Cora is no more than 83.6 (e.g., https://www.ijcai.org/proceedings/2021/0204.pdf).

__3. Question__: It is suggested to conduct experiments on the graph classification task on MUTAG.

_Response_: We run GLATE on some datasets for the graph classification task including MUTAG. The results are shown as below (note that the reported results of baselines are from https://arxiv.org/abs/2010.13902). We can see that GLATE has a competitive performance compared with other baselines.

| Methods | NCI1     | PROTEINS | MUTAG    | IMDB-B   |
|------------|----------|--------------|------------|------------|
| node2vec                        | 54.89±1.61                       | 57.49±3.57          | 72.63±10.20         | -                 |
| sub2vec                         | 52.84±1.47                       | 53.03±5.55          | 61.05±15.80         | 55.26±1.54        |
| graph2vec                       | 73.22±1.81                       | 73.30±2.05          | 83.15±9.25          | 71.10±0.54        |
| InfoGraph                       | 76.20±1.06                       | 74.44±0.31          | 89.01±1.13          | __73.03±0.87__ |
| GraphCL                         | 77.87±0.41                       | 74.39±0.45          | 86.80±1.34          | 71.14±0.44        |
| GLATE                           | __78.03±0.61__              | __74.68±0.21__ | __90.88±0.33__ | 71.60±0.85        |

__4. Question__: The symbolic meaning in Figure 1.

_Response_: We will revise the sentence for the symbolic meaning as "We use the shape and color to represent its class and embedding, respectively". In Figure 1, we use the point's shape to represent the node's class and the point's color to represent the node's embedding. The nodes with different colors are located at different positions of the unit hypersphere.

__5. Question__: The explanation about Formula (9) is not clear.

_Response_: We apologize for omitting some references here. The details of the information bottleneck principle for self-supervised learning (Equation (9)) come from the other works (such as Equation (13) in https://arxiv.org/abs/2106.12484 and Equation (5) in https://arxiv.org/abs/2103.03230). We will give a brief explanation to Equation (9) and provide the related reference for it in the revised manuscript.

__6. Question__: Some typos, e.g., "it decrease faster later" should be modified to be "it decreases faster later".

_Response_: Thanks very much! We will correct all typos in the revised manuscript.

__7. Question__: There are some newly accepted papers on graph CL by NeurIPS 21, they are suggested to be included in the related work, such as:...

_Response_: The papers given in the comment are indeed relevant to our work. In addition, we find some other important papers these days (e.g., Yuning You, Tianlong Chen, Yongduo Su, Ting Chen, ZhangyangWang, Yang Shen, Graph Contrastive Learning with Augmentations, NeurIPS 20; Yanqiao Zhu, Yichen Xu, Hejie Cui, Carl Yang, Qiang Liu, Shu Wu, Structure-Aware Hard Negative Mining for Heterogeneous Graph Contrastive Learning, KDD 21) are relevant to our work. We will include the papers you mentioned and the others in the related work of the revised manuscript. 


 | Datasets     | \#Nodes | \#Edges | \#Features | \#Classes |
|------------------|-------------|-------------|----------------|---------------|
| Amazon-Computers | 13,752      | 245,861     | 767            | 10            |
| Amazon-Photo     | 7,650       | 119,081     | 745            | 8             |
| Coauthor-CS      | 18,333      | 81,894      | 6,805          | 15            |
| Coauthor-Physics | 34,493      | 247,962     | 8,415          | 5             |
| ogbn-arxiv       | 169,343     | 1,166,243   | 128            | 40            |

| Methods | Validation of obgn-arxiv | Test of obgn-arxiv    |
|---------|--------------------------|-----------------------|
| DGI        | 71.26±0.11  | 70.34±0.16          |
| GRACE      | 72.61±0.15  | 71.51±0.11          |70.71±0.03          |
| BGRL       | 72.53±0.09  | 71.64±0.12          |
| GLATE      | __72.80±0.02__           | __72.13±0.05__ |


|  Methods |  Amazon-Computers |Amazon-Photo     | Coauthor-CS     | Coauthor-Physics |
|----------|-------------------|-----------------|-----------------|------------------|
| DGI        | 83.95±0.47          | 91.61±0.22          | 92.15±0.63        | 94.51±0.52          |
| GMI        | 82.21±0.31          | 90.68±0.17          | OOM               | OOM |
| MVGRL      | 87.52±0.11          | 91.74±0.07          | 92.11±0.12        | 95.33±0.03          |
| GRACE      | 87.46±0.22          | 92.15±0.24          | 92.93±0.01        | 95.26±0.02          |
| GCA        | 88.94±0.15          | 92.53±0.16          | 93.10±0.01        | 95.73±0.03          |
| BGRL       | 89.68±0.31          | 92.87±0.27          | __93.21±0.18__ | 95.56±0.12          |
| GLATE      | __89.96±0.20__| __93.12±0.14__ | 93.01±0.27        | __95.97±0.08__ |

 Thank you for your constructive comments. We appreciate that you took the time to comment on our paper. We will submit the revised manuscript soon after. We respond to each comment below.

__1. Question__: The exploration of temperature in this paper is limited to the node level, which is slightly inadequate.

_Response_: Thank you for your inspiring comment. GLATE can be easily applied for graph-level tasks such as the graph classification task. We report its experimental results on four graph-level datasets as follows (note that the reported results of baselines are from https://arxiv.org/abs/2010.13902). We can see that GLATE has a competitive performance compared with other baselines.

| Methods | NCI1     | PROTEINS | MUTAG    | IMDB-B |
|---------|----------|----------|----------|--------|
| node2vec   | 54.89±1.61  | 57.49±3.57          | 72.63±10.20         | - |
| sub2vec    | 52.84±1.47  | 53.03±5.55          | 61.05±15.80         | 55.26±1.54        |
| graph2vec  | 73.22±1.81  | 73.30±2.05          | 83.15±9.25          | 71.10±0.54        |
| InfoGraph  | 76.20±1.06  | 74.44±0.31          | 89.01±1.13          | __73.03±0.87__ |
| GraphCL    | 77.87±0.41  | 74.39±0.45          | 86.80±1.34          | 71.14±0.44        |
| GLATE      | __78.03±0.61__              | __74.68±0.21__ | __90.88±0.33__ | 71.60±0.85        |


__2. Question__: More analysis of changes in the initial contrastive views, such as the effect of changing the dropping rate on temperature selection.

_Response_: The edge removing rate and feature masking rate decide the initial contrastive views, so as you suggest, we evaluate GLATE's performance using different selections of $\tau\_{initial}$ under different initial contrastive views. The results on Cora are shown as below ($p\_{1}$ and $p\_{2}$ denote the rate of removing edges and masking features, respectively; the number in square brackets is the selected value of $\tau\_{initial}$). We can see that using high $\tau\_{initial}$ (such as 0.6, 0.8, and 1.0) is better when $p\_1$ and $p\_2$ are large (top left corner), while using low $\tau_{initial}$ is more suitable to the opposite situation (bottom right corner). As the lower-bound of $\tau$ is fixed as 0.2, we think it is reasonable because a wider adjustable range of temperatures is helpful to GLATE to learn from more noisy data.

 
| Methods | Amazon-Computers | Amazon-Photo              | Coauthor-CS               | Coauthor-Physics |
|---------------------------------|-------|--|--|-------------------------------|
| DGI                             | 83.95±0.47                               | 91.61±0.22          | 92.15±0.63        | 94.51±0.52          |
| GMI                             | 82.21±0.31                               | 90.68±0.17          | OOM               | OOM                 |
| MVGRL                           | 87.52±0.11                               | 91.74±0.07          | 92.11±0.12        | 95.33±0.03          |
| GRACE                           | 87.46±0.22                               | 92.15±0.24          | 92.93±0.01        | 95.26±0.02          |
| GCA                             | 88.94±0.15                               | 92.53±0.16          | 93.10±0.01        | 95.73±0.03          |
| BGRL                            | 89.68±0.31                               | 92.87±0.27          | __93.21±0.18__ | 95.56±0.12          |
| GLATE                           | __89.96±0.20__                      | __93.12±0.14__ | 93.01±0.27        | __95.97±0.08__ |


__4. Question__: The information bottleneck term counts for noise in node features only. What if the structure is perturbed as well? How does the presented analysis differ from one recent work (https://arxiv.org/abs/2106.05819)?

_Response_: 
* First, in our work, we view edges (the graph structure information) as message passing's paths, and the main goal of contrastive learning is to obtain each node's representation. The same practice can be seen in Equation (13) in https://arxiv.org/abs/2106.12484. For the other goals, such as aiming to obtain the edge's embedding, Equation (9) will change.
* Second, the difference between our presented analysis and another work (https://arxiv.org/abs/2106.05819) is __the negative terms in the information bottleneck__ when ignoring the impact of the coefficients. We analyze the information bottleneck principles of https://arxiv.org/abs/2106.05819 (denoted as $\mathcal{IB}\_{ssl}^1$) and our paper (denoted as $\mathcal{IB}\_{ssl}^2$) in detail and give the proof below.
For the task of learning node representations $Z'$ and $Z''$, Equation (5) in https://arxiv.org/abs/2106.05819 can be represented as: $\mathcal{IB}\_{ssl}^1 = I(Z', Z'') = H(Z') - H(Z' | Z'')$.
As $H(Z' | Z'') = H(Z', Z'') - H(Z'')$, then
$$
\mathcal{IB}\_{ssl}^1 = H(Z') + H(Z'') -H(Z', Z'').\ \ \ \ \ (\mathrm{I})
$$
While, in our paper (Equation (11)): $\mathcal{IB}\_{ssl}^2 = \lambda H(Z') - H(Z'|X)\quad(\lambda \in \left ( 0, 1\right ))$.
Because $Z''$ has the same property as $Z'$, we have:
$$
\mathcal{IB}\_{ssl}^2 = \frac{\lambda}{2}* (H(Z') + H(Z'')) - \frac{1}{2}*(H(Z'|X) + H(Z''|X)).\ \ \ \ \ (\mathrm{II})
$$
Regardless of the effect of coefficients, an intuitive understanding to Equation (I) and (II) is: the common goal of $\mathcal{IB}\_{ssl}^1$ and $\mathcal{IB}\_{ssl}^2$ is to learn the most informative part of raw data $X$ and remove the redundant information from it. Although $\mathcal{IB}\_{ssl}^1$ and $\mathcal{IB}\_{ssl}^2$ have consistent optimization objectives, they have a minor difference at the minimization term: one is to minimize $H(Z', Z'')$ and the other is to minimize $H(Z'|X) + H(Z''|X)$. It implies that $\mathcal{IB}\_{ssl}^1$ replies more on the choice of pre-defined graph augmentation strategies, while $\mathcal{IB}\_{ssl}^2$ is not limited to graph augmentation strategies. In a summary, if we aim to select the most suitable graph augmentation strategies, $\mathcal{IB}\_{ssl}^1$ is more helpful; but if we want to avoid the bad impact of an improper choice of graph augmentation, $\mathcal{IB}\_{ssl}^2$ is better than $\mathcal{IB}\_{ssl}^1$.

The authors explore the role of the temperature in the loss function for graph contrastive learning. They argue that global uniformity and local separation are both necessary to the learning quality and this can be controlled by the temperature. Thus, they develop a simple but effective algorithm GLATE to dynamically adjust the temperature value in the training phase. Experiments are also conducted to demonstrate the effectiveness of the method. Strengths: The problem faced by the paper is interesting and timely and the proposed approach seems reasonable. The article is well written, the method is clearly described, and the overall quality is good. The authors also provide the source code to facilitate experimental replication.

Weakness: 
1. The contrastive loss function used by the authors is not universally applicable to various graph tasks. In Section 3.1, the general loss function proposed by GRACE/GCA for both inter-view and intra-view is used for node classification tasks. The exploration of temperature in this paper is limited to the node level, which is slightly inadequate.
2. Changes in the initial contrastive views can also affect the balance between the different views. I would like to have an analysis of this, such as the effect of changing the dropping rate on temperature selection.

Suggestions:
1. It is suggested that more content be added to explain and analyze why the truncated strategy cannot be applied to graph contrastive learning. This will help in the understanding of dynamic temperature estimation.
2. I am well aware that the current GCL methods cannot be applied to large-scale graph datasets, such as the OGB series, due to the limitations that multiple encoders require a lot of memory. If possible, it is recommended to add results on larger datasets, such as amazon and coauthor datasets also used in GCA. The overall quality of the paper is good, with a clear narrative. The problems addressed are also much needed. The methodology is straightforward and clear, and there is also theoretical guarantee. So I recommend it for acceptance.

This paper proposes the importance of dynamically changing the temperature in the contrastive loss with a momentum style that will boost the graph contrastive learning. It also provides theoretical analysis on the reason to dynamically adapt the temperature. My major concern is that the results are not very trustable. For example, one fixed tau already outperforms all baselines. In table 1, Frozen temperature (τ = τinitial = 0.8) 84.32±0.15 67.83±0.22 85.23±0.08. This method actually is the typical InfoNCE CL on the graph. How it can easily outperform all baselines? some baselines even adopted sophisticated techniques. At the same time, the results(86.8 ± 0.5) reported by MVGRL [27](which was also reproduced by some other papers) on Cora is much more than the authors’ reports(83.50±0.40).  This makes me highly suspect the superiority of the proposed approach. It is possible the improved performance is only marginal. BTW, it is also suggested to conduct experiments on the graph classification task on MUTAG etc.


In figure 1, “we use the shape and color to represent its class and embedding”. Basically, what does the embedding mean? Does each point correspond to one node sample in the graph? If that is true, color means the label, what “shape” is used to denote?



Formula (9) is not clear. IB_{ssl} = I(Z 0 , X) − βI(Z 0 , X0 ). I know the authors want to express the case for SSL. But Information bottleneck is used for compression with the form IB= I(Z, X)- βI(Z , Y). Here the term is Y, not the augmentations. The authors directly claim such a formula without any proof or citation that also used this formula. 


Minor:

Typos:

it decrease faster later==>it decreases faster later
“Raw features” only uses the node==>“Raw features” only use the node
make the model optimizes consistently==>make the model optimize consistently

besides, there are some newly accepted papers on graph CL by NeurIPS 21, they are suggested to be included in the related work, such as:

Erlin Pan, Zhao Kang, Multi-view Contrastive Graph Clustering,  NeurIPS 21
Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, Xiang Zhang InfoGCL: Information-Aware Graph Contrastive Learning,  NeurIPS 21
Yanqiao Zhu, Yichen Xu, Qiang Liu, Shu Wu, An Empirical Study of Graph Contrastive Learning,  NeurIPS 21 Basically, the idea is novel and the theoretical analysis is also reasonable.  My major concern is focusing on the experiments part. Basically, the baselines reported by the authors are much smaller than those reported in the original papers. This makes me suspect if the contribution is only marginal in boosting the results.

This paper studies the temperature parameter of graph contrastive learning models. Especially, two terms, global uniformity and local separation are proposed to disentangle the contrastive objective and a GLATE algorithm is proposed to adaptively adjust this parameter. # Strengths
* The temperature parameter is important for understanding the behavior of contrastive objectives for graph-structured data.

# Weaknesses
* This paper is hard to follow. For example, the authors never clearly define global uniformity and local separation. It is suggested that, for better readability, the authors shall provide a background of dissecting contrastive objectives and then proceed to the problem of existing formulation.
* It feels to me the presented analysis is limited to GRACE that leverages the InfoNCE loss. How does the analysis generalize to other graph contrastive algorithms, e.g., GraphCL that excludes inter-view negative samples, DGI that uses JSD objectives, bootstrapping losses that eschew the need of negative samples?
* The transductive experiments are conducted on three small datasets, which are not convincing. Evaluation on large-scale datasets such as open graph benchmarks is needed.
* The analysis given in Section 3.3 seems to be insufficient. What is not clear to me is that the information bottleneck term counts for noise in node features only. What if the structure is perturbed as well? How does the presented analysis differ from one recent work (https://arxiv.org/abs/2106.05819)?
* One particular concern with the presented formulation is that, unlike visual data, negative samples involved in graph contrastive learning are not aligned with their semantic relationship (Figure 1, https://arxiv.org/abs/2110.02027), partially due to the neighborhood smoothing operation of graph neural networks. That being said, the more closeness to negative samples (in other words, more difficulty negative samples) will lead to wrong selection of false negative samples (i.e. true positives). Therefore, I in person believe that simply adjusting the temperature parameter is not sufficient to account for the balance of global uniformity and local separation. 

## Minor problems
* Property 2 seems to be off-context to the main text.
* "Interpretable" contrastive representations in Section 2 is not accurate.
 Overall, I appreciate the deep insights on the temperature parameter. However, many serious problems exist in the current version. Therefore, I recommend rejection.

Meta-review: The paper investigates the effect of temperature in the loss function for graph contrastive learning and proposes a novel method to dynamically adjust it during learning (GLATE). The paper is concerned with an interesting problem, is timely, and relevant for the ICLR community. 

After author response, reviewers did not come to a full agreement on the paper, with two reviewers indicating (weak) reject and two reviewers indicating (weak) accept. Reviewers highlighted the potential impact of improvements in graph contrastive learning as well as the theoretical analysis as strengths of the paper. However, reviewers raised also concerns regarding scope, novelty of the contributions, and clarity of presentation (method, evaluation, etc.). While the authors' response addressed some concerns regarding aspects of the experimental evaluation, it did not change the overall evaluation of reviewers. 

Taking author response and reviewer feedback into account, I narrowly agree that the manuscript is not ready yet for acceptance at ICLR due to the aforementioned concerns. However, I encourage the authors to revise and resubmit their work based on the feedback of this reviewing round.

Decision: Reject