Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to prove that there is a bottleneck in feature learning for long-tailed classification and data augmentation can help relieve the issues in long-tail feature space. Three major experiments were done to prove that feature space 1) is more biased than balanced feature space, 2)  is more disused and less compact than balanced feature space, and 3) less localized in terms of feature centroids. And data augmentation can help alleviate all three issues. 
main_review: The weakness: 
1) The second objective of this paper is to discuss "why data augmentation helps in representation learning". However, in the paper, only positive effects from data augmentation were shown, the reasons and mechanisms were not fully discussed. 

2) The overall paper is based on the unserious term "good enough". What is this term defined? How good is good enough? Good enough in terms of what? Generalization and robustness compared to full balanced data sets or in terms of knowledge transfer? If it is the first one, of course, long-tailed representations are less generalized and robust compared to balanced representations. It is not a new idea and is already discussed in [1]. If it is the second one, then the later experiments don't make any sense. And I think when people say long-tail representations are "good enough" in studies like [2], it is more like it is good enough for long-tail learning rather than comparing it to balanced learning. 

3) All the experiments seem unfair to me. For example, D_LT are representations from long-tailed data sets, and D* are representations from balanced data sets. Balanced data sets always have much more training samples compared to corresponding long-tailed counterparts. How do you know these inferior results were not caused by the lack of training samples? 

4) In the "adding unseen samples" experiments (e.g., Fig 6, Fig 7, Fig 8), only results on D_LT were reported. I want to see results when unseen samples are added to D* as well. Only by doing this can you prove D* is less diffused and better localized. 

5) Fig 7 needs a more detailed legend. So many components don't have explanations. 

6) By looking at Figure 5, I don't see a significant difference between D_LT and D* in Cifar100-LT and ImageNet-LT.

[1] Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., & Yu, S. X. (2019). Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2537-2546).
[2] Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., & Kalantidis, Y. (2019). Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217.
summary_of_the_review: I think the intuition of this paper is not clear and the experiments are not persuasive.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper poses an interesting and important question - where are the bottlenecks in long-tailed classification. The authors use empirical experiments to show their observations: (1) representation is more critical than classifier, (2) data augmentation is helpful.  Three datasets (CIFAR-10 LT, CIFAR-100 LT and ImageNet-LT) are employed to work with ResNet-32 and ResNet-10 models to demonstrate their observations.  
main_review: Strength: 
1. The topic is interesting and the papers pose some unique observations after extensive empirical analysis and experiments
2. The paper defines several simple mathematical and statistical metrics to measure the differences between representations.

Weakness：
1. The posed questions are not well addressed
The paper shows some observations but did not provide a concrete and reasonable solution such that the long-tailed classification issue can be addressed. The useful insight from this paper is limited.  

2. The empirical observations are not solid and rigorous 
The paper only provides some simple metrics but did not explain why the metric is necessary and what's the high-level intuition. I did not get the motivation why the authors come up with these metrics to show the differences between representation. In addition, there is no rigorous mathematical proof or statistical analysis.

3. Lack of careful related work discussion 
The related work section is hard to follow and the authors did not explain their contributions and differences from existing works.  

4. The writing needs to be improved. 
Many typos result in additional difficulties to read.   The citation format is not consistent. For example,  Cui et al.. in section 3 does not have a   year, and "He et al (2015)" followed by Resnet-32 should be "(He et al, 2015)".  The "google enough" and "Normal" in the abstract should be corrected. 



summary_of_the_review: An overall feeling is that the paper is an ongoing work and needs to be carefully written and improved. My recommendation is to reject it in the current form.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors study the long-tail dataset problem in order to determine the true bottleneck for the task. After performing many ablations and experiments on 3 benchmark datasets they establish that contrary to common belief the bottleneck is in data representation rather than the classifier itself.
main_review: I really enjoyed reading the paper. It has a very clear direction from the beginning with good experiments to back it up. The writing is clear as well. I believe long-tailed classification is an interesting problem with clear real-world applications, so studying it in-depth is necessary for the community.

Overall, I don't see any major drawbacks or shortcomings as the experiments and ablations combined with the analysis are solid. That said I have few questions:

The difference in representation between D* and D_{LT} is clearly visible. However, apart from difference in the shape of distribution (long-tail vs balanced) there is also difference in the amount of data between those two which might play an important role, especially when considering learned representations. It would be good to see what is the difference as well between two datasets that have equal number of examples, but different distributions: normal ad LT. Since otherwise the authors conclusion might raise a question.

Additionally,  Yin et al. [1] performed a related analysis of classifier magnitude difference which was depending on the distribution of classes. The authors analysis seems deeper here, however it would be good to address any similarities/differences. On top of that, few methods [2, 3] used adversarial examples in order to modify the learned representations instead of the classifiers in LT task. It would be good to see what the authors think about such direction and what impact on the feature space and measured statistics it would have.

And finally, apart from the analysis, what are the conclusions here for the future researchers - any thoughts on proposed directions/approaches that could originate from the performed analysis?

[1] Yin, Xi, et al. "Feature transfer learning for face recognition with under-represented data." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[2] Kim, Jaehyung, Jongheon Jeong, and Jinwoo Shin. "M2m: Imbalanced classification via major-to-minor translation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[3] Kozerawski, Jedrzej, et al. "BLT: Balancing Long-Tailed Datasets with Adversarially-Perturbed Images." Proceedings of the Asian Conference on Computer Vision. 2020.
summary_of_the_review: Good analytical paper on interesting subject of long-tailed classification. It would be good to see authors thoughts on impact of the amount of data in training, impact of adversarial augmentations, and proposed directions stemming from the analysis.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper seeks to study what is the bottleneck in long-tailed learning. Based on extensive experiments, the authors propose that representation learning is the bottleneck in long-tailed classification. Also, this paper analyzes representation learning from the perspectives of intra-class compactness and inter-class separation, as well as the influence of data mixup on long-tailed representation learning.
main_review: Positive points:
1. This paper seeks to empirically investigate the importance of representation learning, which may provide a new understanding of deep long-tailed learning to the community.


2. This work shows the effectiveness of intra-class compactness and inter-class separation on long-tailed representation learning.  


3. This work also analyzes the influence of data augmentation on long-tailed representation learning, which provides a better understanding of data augmentation in deep long-tailed learning. 

Negative points:
1. This paper mentioned that "a commonly held belief in deep long-tailed classification is that performance bottleneck is the classification head atop the representation learner". However, such a belief may not be common. Note that many recent long-tailed learning studies focus on improving representation learning [1], e.g., KCL [2], Hybrid [3], PaCo[4] and DRO-LT [5].  Moreover, in the conclusion, this paper stated that "the results suggest that the primary problem in long-tailed classification may in fact be the few-shot learning problem on the tail classes, rather than the imbalance problem". However, this argument is too strong and the obtained results cannot support it. It would be better if the authors had written all the arguments more rigorously and verified them more completely.  


2. As mentioned by the above question, there are many representation learning-based long-tailed studies, e.g., [1-5]. Therefore, it would be better if authors can review mire representation learning based long-tailed learning methods in related work.


3. The vital problem in this paper is the used balanced set: i.e., CIFAR-10/100 and ImageNet-1K. Please note that the data number of  CIFAR-10/100 or ImageNet-1K is much more than their long-tailed variants, i.e., CIFAR-LT and ImageNet-LT. Considering using more training samples will lead to significant improvement in representation learning and model performance, most empirical comparisons in this paper (especially Table 1 and Figure 2) are unfair and the corresponding arguments are unpersuasive. The experiments would have been more persuasive, if the balanced training set is a variant of the long-tailed training set with a similar total data number but each class has the same/similar data number, like [1,2]. For example, a balanced set of ImageNet-LT can be obtained at https://github.com/Vanint/Awesome-LongTailed-Learning/tree/main/resources/data_txt/ImageNet_LT. 


4. Please discuss more of Figure 5. On ImageNet-LT and CIFAR100-LT, the variance trends of long-tailed representations and ideal representations are quite consistent. Such observations seem different from the conclusion of Sec 2.1. Since I am confused about the results, I guess other readers may also do so. Therefore, I suggest the authors explain them more. 




Minor suggestions:
1. Figure 3 is not clear enough. It would be better if the authors can explain it more in the captions. Moreover, what are "degrees" in Fig.6-8? Please make them more clear.

2. In Lines 4-5 of page 2, ImageNet-LT appears twice. In Line 1 of page 6, there should be a "full stop" before In practice. 


References:

[1] Deep long-tailed learning: A survey. ArXiv, 2021.

[2] Exploring balanced feature spaces for representation learning. In ICLR, 2021.

[3] Contrastive learning based hybrid networks for long-tailed image classification. In CVPR, 2021.

[4] Parametric contrastive learning. In ICCV, 2021.

[5] Distributional robustness loss for long-tail learning. In ICCV, 2021.


summary_of_the_review: Overall, I like the goal of this paper, i.e., analyzing the bottleneck of long-tailed learning. However, I cannot champion this paper since the data number of the used balanced set is much larger than the long-tailed set, which makes the empirical comparisons unfair and the corresponding finding unpersuasive. Moreover, the arguments in this paper should be written more rigorously. I am glad to see the response of the authors.
review_title: null

Meta-review: This paper investigates the role of representation learning when the distribution over the feature space has a long tail. The main motivation is to determine how much of the overall learning, in this case, is bottlenecked specifically by representation learning. The main findings are that vanilla learning gives brittle long-tailed representations, harming overall performance. The paper suggests a form of data augmentation to remedy this. Reviewers acknowledge that this investigation is worthwhile. However, many concerns were raised as to whether experiments support the drawn conclusions. A more principled approach to the data augmentation methodology is also needed. The authors address some of these, providing further experiments, but these were not enough to sway reviewers. Since results are fundamentally empirical in nature, this shortcoming indicates that the paper is not ready to share with the community just yet. Stronger experiments with clearer evidence are needed to fully support the thesis of the work.
Decision: Reject