Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors propose a new loss function to learn feature representations for image datasets that are class-imbalanced. The loss function is a simple yet effective tweak on an existing supervised contrastive loss work. A number of empirical tests are performed on long-tailed datasets showing the benefits of the proposed loss in beating state of the art methods.  Some specific questions are listed below:

1. Is Figure 1 hypothetical ("desired outcome") or real (based on actual observations)? 

2. Minor style comment: please don't call your own contributions "important" :-) - that is for others to decide.

3. Eqn. (3) is not clear at all - please provide an intuitive explanation and motivation. It seems to appear out of thin air.

4. Is there a tradeoff between accuracy and balancedness? Figure 2 seems to suggest so. For example, if we did not train CE loss to maximal accuracy, would be automatically get the balancedness property? Was this tested? How about a combined loss CE + CL to get the best of both worlds? And similarly, how about CE + KCL?

5. Table 1 the delta for VOC seems to be computed wrongly.

6. Figure 7: it's strange that there are no numbers on the y-axis.

7. What is the ImageNet accuracy of KCL?
review_title: Interesting paper with some improvements possible

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Overview:** The paper presents experiments showing that the contrastive learning losses produce better embeddings or feature spaces than those produced by using binary cross-entropy losses. The experiments show that embeddings learned using contrastive learning losses seem to favor long-tailed learning tasks, out-of-distribution tasks, and object detection. The paper also presents an extension of the contrastive loss to improve the embeddings. The experiments in the paper  use common and recent long-tail datasets as well as datasets for object detection and out-of-distribution tasks. 

**Pros**:
*Interesting problem and approach*. I think the paper tackles a hard and important problem, i.e., learning from a long-tailed dataset. Overall, I think that learning a feature space improving the learning from these imbalanced datasets is an interesting idea.

*Clarity of the paper*. Overall the clarity of the paper is good. The motivation is clear and the narrative is clear overall. However, I think the clarity in the experiments is insufficient, see below.

**Cons**:
*Insufficient clarity in the experiments*. I have several concerns with the experiments:
1. The *balancedness* metric in Eq.(3) may not be a robust metric for measure performance. The reason I am not convinced about this metric is that if the accuracies of the classifier are low but equal, then the metric will say that the *balancedness* is good. I think a good metric for a classifier learning from an imbalanced dataset is one that indicates if the overall accuracy is high, maintains the many-shot the classification accuracy high, and increases the accuracy of the classes in the tail. I think this metric does not indicate if the overall accuracy is high.

2. I am not convinced about how classifiers are trained in experiments in Sec. 3.2. The paper trains and tests using a balanced set after learning a feature space. To my understanding, the challenge of learning from a long-tailed dataset is to test whether a classifier can generalize well for classes with few training samples while maintaining a good performance on classes with more training examples. Thus, by training a linear classifier with a balanced set using the learned feature space does not really comply with learning from an imbalanced dataset. I think if the experiments would've been stronger if they included results of a trained linear classifier on the learned embedding and still showing good results, then I would be more convinced about the impact of a contrastive loss. From the practical point of view, what matters is the classifier performance. In practice, it is challenging to have a balanced dataset as the paper used. The main question is about the performance when training a linear classifier from a long-tailed dataset using the learned representation.

3. Datasets derived from ImageNet-LT. While I value the goal of using different datasets varying the imbalance in a dataset, I am not convinced that ImageNet-LT is the dataset to use. The reason is that ImageNet-LT is a synthetic long-tailed dataset. In fact, while the dataset shows imbalance, it does not necessarily follow a power-law distribution. I think the generation of these datasets in the experiments should be done using a power-law distribution. From the text, it is unclear how the datasets from ImageNet-LT were generated for the experiments.


Minor concerns:
*Plots lack information*. What is the y-scale in Fig. 2? The figure is missing y-scale information and it is hard to interpret the gap in accuracy  between CE and CL in the left plot. Same comment for Fig 4, what is the scale in y-axis?
review_title: Interesting Set of Experiments but Insufficient Clarity and Evaluations

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

- This paper made a key observation that the self-supervised contrastive learning methods perform stably well even when the datasets are heavily imbalanced. As for the method this paper proposes a classed-balanced version of supervised contrastive loss (Khosla et al., 2020). Extensive experiments demonstrate its superiority on multiple recognition tasks.

Pros:
- The paper is well written with nice figures.
- The paper conducted experiments on extensions other than image classification, which provides some useful insights.

Cons:
- The advantage of using self-supervised learning to learn a representation to combat label bias issue in imbalanced problems has been discovered and validated by a previous work [1]. I understand that [1] was accepted just a few days before the DDL of ICLR. Unfortunately I still have to devalue the contribution of this paper as it seems to me that the major contribution of this paper relies on the finding that SSL can help to alleviate the issue of label bias. In terms of large-scale experiments on ImageNet-LT and iNatualist, it seems that this paper has no advantage over [1].
- It seems the runtime of the proposed method is much worse than supervised based methods.

Additional Questions and Concerns:
- Why does the author use cosine lr schedular?
- How to interpret accuracys on imagenet in fig. 4? Why is there only 10 classes?

[1] Yang, Yuzhe, and Zhi Xu. "Rethinking the Value of Labels for Improving Class-Imbalanced Learning." NeurIPS 2020

----
post-rebuttal update

I appreciate the discussions between the authors. I plan to keep my original score, for the reason that, at least in my point of view, the difference of the two methods is subtle and it is not clear whether the subtle difference results in drastic improvement.
review_title: nice paper but the key discovery was published somewhere else

Meta-review: 
 This paper studies the difference between cross-entropy and contrastive learning losses in the feature representations that they learn, specifically looking at class-imbalanced datasets. The authors show that contrastive losses result in a more "balanced" representation, as measured by the balance of accuracy across the classes when a linear classifier is learned mapping from the feature representation to the class labels. They also show that empirically this tends to result in better generalization to downstream tasks. Inspired by this, they devise a simple modification of the prior supervised contrastive loss method and show that it can improve performance on ImageNet-LT and even generalization performance when trained on balanced datasets and applied to downstream tasks. 

  The reviewers identified several weaknesses, including some clarity issues (R1), limitations of how balancedness is measured and lack of theoretical/statistical rigor in terms of the resulting claims (R2), and differences with respect to concurrent work (R4). A lengthy discussion occurred between reviewers and authors, as well as input from a co-author of the concurrent work. In the end, the reviewers were not fully satisfied both in terms of the balancedness measure and relationship to the concurrent work. 

  Overall, despite this and the valid limitations of the work, I recommend accepting this paper as I believe the contributions outweigh the limitations, and that the findings would be interesting to the community. First, the paper provides some interesting analysis of balancedness and differences across these two loss functions, as well as connections to generalization, which even the concurrent work does not provide. The resulting method, while being a simple modification of the supervised contrastive loss work, is effective both for long-tailed datasets and generalization to downstream tasks (even when trained in a balanced manner) which is nice. In the end, we should not use [3] to reject this paper since it was accepted right before the ICLR deadline. 

  However, I **strongly** recommend that the paper address the valid limitations mentioned in the discussions. Specifically: 
  1) While I agree that [3] is concurrent work, this paper should none-the-less tone down its claims of being the first in exploring balance for the camera-ready version and clearly address differences between this paper and that one (even if mentioned as concurrent work). It is important to give credit when it is due, and while I think [3] is a different perspective it should be mentioned. Further, the claim that their methodology is not correct is highly arguable, so this should not be mentioned; rather the differences in perspectives and what each paper shows should be emphasized. Even without [3], self-supervised pre-training (initialization) should arguably be included as a baseline given that it is the logical first choice for incorporating self-supervised learning.

  2) Like R2, I do not believe the balancedness metric shows uniformity of the feature space. This would have to be shown through methods such as t-SNE or in some other way. Being linearly separable in a balanced way across classes (which is what you showed) is not sufficient to show that feature space "uniformity". One can draw many feature space distributions that do not have the intuitive meaning of this (which isn't precisely defined by the authors) but still be linearly separable. I recommend authors remove this type of characterization (unless they can define/show it) and instead include a discussion of the limitations of the current methodology for measuring balancedness. Figure 1 should also emphasize that it is notional (not from real data). 
Decision: Accept (Poster)