Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Contrastive learning objectives are a popular and effective method for self-supervised learning. Most variations of the contrastive objective have a static temperature hyperparameter $\tau$. Previous work suggests that large values of $\tau$ lead to improved group-wise discrimination while small values lead to improved instance-wise discrimination. The authors study the effect of varying this temperature parameter in 'long-tail' settings with imbalanced data. They posit that long-tail class performance benefits from instance-wise discrimination ability and run initial 'coarse supervision' experiments to test this claim. Since this method requires (weak) labels, the authors also propose a dynamic temperature schedule that can be used in unsupervised settings. In a series of ablation experiments, they find that a cosine schedule that gradually switches between the two phases leads to the best performance. 
strength_and_weaknesses: Strengths:
- The topic is interesting and offers a promising direction for deepening our theoretical understanding of contrastive learning.
- The work is clear and easy to follow with the main theoretical claim explicitly laid out and experiments set-up to test it directly.
- The experiments are conducted with a variety of models and datasets. 


Weaknesses:
- It is unclear to me whether the cosine schedule is actually the optimal one. While it clearly outperforms the rapidly changing schedules like the step function, how does it compare, for example, to a linear (oscillating) schedule?
- The marginal performance gain from MoCo to MoCo+TS does not seem significantly higher in the imb 150 vs imb 50 case. It isn't entirely clear whether the results support the hypothesis posited initially by the authors, otherwise one might expect the marginal gain to be higher in the more imbalanced case. It would be useful to compare marginal performance gain on CIFAR10-LT and on CIFAR10 to confirm that the improvement from the temperature schedule is actually related to the data imbalance. 
- There have been a few recent papers studying temperature in contrastive learning. In particular, how does your dynamic temperature scaling compare to the method in 'Dynamic Temperature Scaling in Contrastive Self-supervised Learning for Sensor-based Human Activity Recognition' (Khaertdinov et al., 2022)?

***Update after author response***
The authors have addressed these weaknesses to my satisfaction.  

clarity,_quality,_novelty_and_reproducibility: - The paper is clearly written. I especially appreciate that the main theoretical claim is explicitly laid out and tested. 
- I have some small concerns about novelty, as described above. 
- It seems like all relevant details are provided for reproducing the main results.
summary_of_the_review: I think the work is interesting and important, but have some concerns detailed in the Strengths and Weaknesses section. 
I would be likely to raise my score and recommend acceptance if these concerns were addressed. 

***Update after author response***
The authors have addressed these weaknesses to my satisfaction. I have raised my score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the long-tailed learning problem by varying softmax temperature during the course of training. The authors argued that different magnitudes of the temperature value actually induce different learning preferences, which makes it possible to gradually switch the learning objective by carefully setting up a temperature scheduler. The proposed method has been evaluted on serveral long-tailed learning benchmarks, demonstrated its effectiveness.
strength_and_weaknesses: Strength:
- The proposed method is interesting and is of a unique view.
- The proposed method requires no computation overhead, which is a tempting merit.

Weaknesses:
- The evaluation is not comprehensive. Only contrastive learning based methods are compared against.
- The improvement is quite limited compared with other long-tailed learning methods.
clarity,_quality,_novelty_and_reproducibility: The overall quality is good, which is easy to follow and clearly written. Also the proposed method is novel.
summary_of_the_review: An interesting method, yet with limited empirical improvments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper explores self-supervised contrastive learning for long-tail data. It presents an empirical study of the impact of the temperature parameter on the head and tail categories. Based on the study, the paper proposes a simple yet effective solution to improve the imbalance pretraining which involves a dynamic temperature schedule. Compared to the vanilla contrastive learning and the SDCLR baseline, the proposed method demonstrates better performance on three long-tail benchmarks: CIFAR10-LT, CIFAR100-LT, ImageNet100-LT.
strength_and_weaknesses: Strength

**[S1]** Topic: the paper investigates self-supervised learning for long-tail data. This is one of the first steps toward applying self-supervised approaches to uncurated data, which is of great interest to the community;

**[S2]** Insight: the paper provides an empirical analysis of the impact of the temperature parameter on the head and tail categories;

**[S3]** Technology: the proposed dynamic temperature scheduling is technically sound;

**[S4]** Performance: compared to the vanilla contrastive learning and the SDCLR baseline, the proposed method demonstrates better performance on three long-tail benchmarks: CIFAR10-LT, CIFAR100-LT, ImageNet100-LT.

**[S5]** Presentation: the writing is clear.

---

Weaknesses

**[W1]** Interpretation:

i) I find it difficult to interpret the results of TS. The study in Sec.3.3 and Fig. 3 seems to indicate that TS achieves a trade-off between optimizing the performances of the head or tail classes: as shown in Fig.3, TS slightly reduces the performance of the head classes but significantly improves the performance of the tail classes. On the other hand, results in table 7/8/9 show that TS consistently improves the performance of both the head and tail classes. The improvements on the head and tail classes are also comparable: +2.1%/+2.4%  kNN@1 for the head/tail classes of ImageNet100-LT (table 9). It looks like TS is more than a simple trade-off?

ii) it is also unclear to me why decreasing the temperature during the training is the optimal scheduling solution (table 4 and 5). It looks like TS learns the head classes better at the early stage of the training then progressively prioritizes the tail classes. I wonder if TS may hurt the performance of the head classes at the later stage of the training. I would also be curious why alternative schedules perform worse than TS.

Overall, I’m happy to learn about the good performance of TS but the interpretation of TS deserves more investigation than what is presented in the current paper.

**[W2]** Evaluation. The results of SDCLR in table 3 look different from what were reported in the [SDCLR](https://arxiv.org/abs/2106.02990) paper. I wonder if these results are reproduced? If this is the case, I wonder if there is any difference in the experimental setting compared to the SDCLR paper. For example, the SDCLR paper reported results with 500 epochs pretraining while TS is trained with 800 epochs (Sec. 4.1), I wonder if different methods shown in table 3 share the same experimental configurations

**[W3]** The analysis in Sec. 3.2 and 3.3 is limited to contrastive learning. It is not clear to me if TS could be beneficial to non-contrastive SSL methods that use a temperature parameter, e.g. SwAV, DINO, etc.

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The proposed method is not new but effective. The authors are suggested to include some clarifications on the experimental settings, as discussed in [W2].
summary_of_the_review: The paper provides an empirical analysis of the impact of the temperature parameters on self-supervised contrastive long-tail pretraining. A simple yet effective temperature scheduling solution is also proposed and demonstrates superior performance on three long-tail benchmarks. In general, I'm happy to learn about the good performance of this simple solution but would prefer some clarifications on the evaluation ([W2]). The paper could be stronger if it includes further investigations to interpret the results of TS ([W1]) and analysis of non-contrastive self-supervised methods ([W3]).

***
After rebuttal:

My concerns are addressed in the rebuttal. I thereby raise the score to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper attempts to improve unsupervised representation learning for long tail data. In this regards, the author propose a very simple method of varying softmax temperature during the course of training. It is argued that different temperature regime value induce different learning focus: "group-wise discrimination" vs "instance discrimination". The proposed method has been evaluated on several long-tailed learning benchmarks, demonstrated its effectiveness. We thank the authors and reviewers for actively engaging in discussion and taking steps towards improving the paper including for providing additional experiments and improving writing clarity. All the reviewers appreciate the simplicity and effectiveness of the proposed method.
justification_for_why_not_higher_score: - Proposed method is not totally novel. Prior literature already studies that different softmax temperature value induce different learning focus, e.g. https://arxiv.org/abs/2007.07314, https://arxiv.org/abs/2103.01550, etc.
- Proposed method only brings improvement in the unsupervised setting and not on supervised setting, thereby limiting the audience of interest.
justification_for_why_not_lower_score: The simplicity and effectiveness of the proposed method for unsupervised representation learning on long tail data as appreciated by all the reviewers.
Decision: Accept: poster