Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper considers the problem of long-tailed image classification, where the class frequencies during (supervised) training of an image classifier are heavily skewed, so that the classifier underfits on under-represented classes. Different known and novel sampling schemes during training as well as post-training procedures to restore the class balance after training are studied. The overall best strategy turns out to be naive training on the skewed training set, and post-hoc rebalancing only of the classification stage. The paper presents various ablation studies and comparisons with related methods on the ImageNet-LT, Places-LT, and iNaturalist data sets, achieving state-of-the-art performance.

The paper is well-written and gives a nice overview on related work and in particular reweighted sampling schemes. The proposed methods and variations appear to be simple, yet very effective, and the insight that decoupling representation and classifier learning performs well on long-tailed classification seems novel. The experiments are mostly thorough and detailed. Here are some more detailed comments:

- My main concern is the selection strategy of \tau in \tau-normalized classification. The authors merely specify that they choose it in the interval (0,1). How is this tuned exactly? Per data set or the same for all data sets? On a validation set? It would be great to provide more details and guidelines for practitioners. Also, in Fig. 2 left, what is the \tau used?

- It would be interesting to see whether the performance can be improved by training a shallow MLP rather than just retraining the weights of the linear classifier W.

- Retraining a linear classifier on a fixed representation can be brittle, at least this can be observed in the context of unsupervised representation learning. The authors should add details about the exact schedules, batch size etc. used for retraining the linear classifier in cRT.

Overall I like the paper, but it is important to add more detail, in particular about the choice of of \tau.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper tries to handle the class imbalance problem by decoupling the learning process into representation learning and classification, in contrast to the current methods that jointly learn both of them. They comprehensively study several sampling methods for representation learning and different strategies for classification. They find that instance-balanced sampling gives the best representation, and simply adjusting the classifier will equip the model with long-tailed recognition ability. They achieve start of art on long-tailed data (ImageNet-LT, Places-LT and iNaturalist).

In general, this is paper is an interesting paper. The author propose that instance-balanced sampling already learns the best and most generalizable representations, which is out of common expectation. They perform extensive experiment to illustrate their points.

--Writing:
This paper is well written in English and is well structured. And there are two typos. One is in the second row of page 3, "… a more continuous decrease [in in] class labels …" and the other one is in the first paragraph of section 5.4, "… report state-of-art results [on on] three common long-tailed benchmarks …". 

--Introduction and review:
The authors do a comprehensive literature review, listing the main directions for solving the long-tailed recognition problem. They emphasis that these methods all jointly learn the representations and classifiers, which "make it unclear how the long-tailed recognition ability is achieved-is it from learning a better representation or by handling the data imbalance better via shifting classifier decision boundaries". This motivate them to decouple representation learning and classification.

--Experiment:
Since this paper decouples the representation learning and classification to "make it clear" whether the long-tailed recognition ability is achieved from better representation or more balanced classifier, I recommend that authors show us some visualization of the feature map besides number on performance. Because I am confused and difficult to image what "better representation" actually looks like. 

The authors conduct experiment with ResNeXt-{10,50,101,151}, and mainly use ResNeXt-50 for analysis. Will other networks get similar results as that of ResNeXt-50 shown in Figure 1?

When showing the results, like Figure 1, 2 and Table 2, it would be better to mention the parameters chosen for \tau-normalization and other methods.

Conclusion:
I tend to accept this paper since it is interesting and renews our understanding of the long-tailed recognition ability of neural network and sampling strategies. What's more, he experiment is comprehensive and rigorous.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes to tackle long-tailed classification by treating separately the representation learning and the creation of a classifier for test time. They evaluate their method on several standard long-tailed classification datasets like ImageNet-LT or Places-LT. 

Pros:
* Very well presented and clear
* Thorough experiments with baselines and comparisons with competitors 
* Novel and efficient approach of redesigning the classifier as a post-processing step after the representation training

Cons:
* I did not find any single value of the "temperature" coefficient that you use for the different datasets! According to Fig 2, it should be around 0.7 for ImageNet-LT but you should clearly specify the used values for all the datasets. For reproducibility. It is also important to know it as it has an impact on how useful is this approach in practice. Because if the coefficients are very different for all the datasets, then the method requires a validation set to find this hyperparameter.
* As middle point between NCM and cRT, you could also train a cosine classifier as done in the paper that you cite "Dynamic few-shot visual learning without forgetting" by Gidaris et al. There is pytorch code for it online. 


I am leaning towards acceptance as the method is clear, easy to implement, well studied through the experiments and has good results on standard benchmarks. The paper also provides interesting insights about long-tailed recognition in general like the effect of the different samplings with the proposed method.
review_title: Official Blind Review #3

Meta-review: This paper presents an approach for the long-tailed image classification, where the class frequencies during (supervised) training of an image classifier are heavily skewed, so that the classifier underfits on under-represented classes. The authors' responses to the reviews clarified most of their  concerns, although some reviewers pointed out that some of the details regarding experiments such as the construction of the validation set and the selection of balanced/imbalanced sets remain unclear. Overall, we believe this paper contains interesting observations to be shared.
Decision: Accept (Poster)