Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
This well-written paper re-visits the idea of logit adjustment to tackle long-tailed problems. The paper begins by setting up a statistical framework and use that to deliver two ways of realizing the logits adjustment effectively. They further prove the potential of such an approach by benchmarking it with several related baselines on both synthetic and natural long-tailed datasets.

###########################################################
+ves:
+ The paper motivates the proposed method very well, by exposing the cases of failures of certain existing approaches and addressing those shortcomings in the proposed method.
+ The explanation about how the proposed methods standout - that is post-hoc logit adjustment with respect to the weight normalization and logit adjusted loss with respect to the LDAM loss is clear and well written.
+ The proposed methods are very versatile as it can both be incorporated into the training, used after the training, and in combination with each other.
+ The experiments and the analysis are both comprehensive, and the paper has a nice technical depth to it.
+ The paper is very well-written.

###########################################################
Concerns/Potential Improvements:
- There is no justification as to why results on synthetic datasets are provided only for the imbalance ratio of 100 while it is a community norm to benchmark on a range of imbalance ratios (typically 100, 50, 20, 10, 1)
- Weight normalization has a term multiplied to the logits and the proposed post-hoc logit adjustment has a term added to the argmax of the logits. Therefore both the terms are independent of each other. It would be interesting to see how both of these methods work in conjunction. Does it improve the results or the cons of one just penalizes the pros of another?

###########################################################
Minor editorial/typo issues:
- For post-hoc logit adjustment, the paper provides a separate subsection named “COMPARISON TO POST-HOC WEIGHT NORMALISATON”. It would have been nice to see something similar for logit adjusted loss. The content is already there in the paper, and just has to be modified a bit to stand out separately.
- Near equation 9, the paper mentions that “(8) immediately suggests two means of optimizing for the balanced error” and goes on to provide the two methods. This was not very evident from (8). Some description to link would have been nice. 

POST-REBUTTAL:

I thank the authors for their response. I am happy with the responses to my concerns/questions, and retain my decision of Accept.
review_title: Review of AnonReviewer2

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: [Summary]

This paper provides a statistical framework for long-tail learning by revisiting the idea of logit adjustment based on the label frequencies. The proposed framework then yields two variant techniques that follow the paradigm of weight normalization or loss modification. Compared with the existing methods, the proposed methods are generalized and Fisher consistent for minimizing the balanced error. Finally, empirical results on four real-world datasets validate the effectiveness and statistical grounding of the proposed methods.

[Pros]
-	The idea of this paper is novel and interesting. The proposed framework is proved to be Fisher consistent for minimizing the balanced error and generalizes several recent methods for long-tail learning. Meanwhile, some insights about the logit adjustment technique are also revealed to help understanding.
-	The experiments are sufficient and supportive to validate the effectiveness and statistical grounding of the proposed methods.
-	The paper is well organized, which makes it easy to grasp the core idea.

[Cons]
-	From the left panel of Fig.2 it seems that the error bar of logit adjusted is even thinner than that of the Bayes predictor. It would be better to provide some explanation.
-	It seems that the proposed framework could work well with linear classifiers. Does it also applies to other classifiers such as cosine classifier?
-	Captions of tables should be put above the table contents.
-	Besides, there are some grammatical errors and typos to be corrected. Some are lists as follows:
  1)	Page 3, ‘an non-decreasing transform’ should be ‘a non-decreasing transform’;
  2)	Page 4, ‘which overcome’ should be ‘which overcomes’;
  3)	Page 5, ‘has negative score’ should be ‘has a negative score’.
  4)	Page 5, ‘another label with positive score’ should be ‘another label with a positive score’.

review_title: A novel statistical framework for long-tail learning that pursues Fisher consistent for minimizing the balanced error.

final_score: rating: 7: Good paper, accept
review_content: 
Summary:
This paper proposes an unifying statistical framework for imbalanced or long-tailed data, where the number of samples for some of the classes may be extremely small compared with other classes. Previous methods work empirically well, but was not consistent, meaning that even in the infinite sample limit, the minimiser doesn't result in a minimal balanced error. The paper proposes a framework based on logit adjustment in two ways: a post-hoc logit adjustment way and another way that injects the logit adjustment to the softmax cross-entropy loss function directly. The paper shows that they are both consistent. Experiments show that the proposed methods work better than previous methods.

Pros:
I feel the paper is very well organized and the story of the paper is clear.  The paper explains the issues of recent methods such as weight normalization and loss modification methods in long-tail learning, and propose a simple statistical formulation with consistency guarantee. It gives further theoretical insights for the binary classification case.

The synthetic data experiments are well designed, and it gives the reader a better understanding of the behavior of the proposed method. The proposed method matches the Bayes optimal decision boundary, while previous methods and naive ERM seem to be worse, demonstrating the characteristics of proposed and previous methods. The ablation study of the scaling parameter shows how a default value of 1 is already good enough in most cases, but can be tuned for further performance gains.

Cons:
The ImageNet-LT results were shown in the Appendix, but the proposed Post-hoc correction seems to be slightly worse than weight normalization. I feel readers will be more comfortable to have this in the main paper.

Other comments:
What does the gray-colored highlights in Table 2 mean? If it means the best performing method, then for CIFAR-10-LT, weight normalisation should be highlighted instead of logit adjustment loss.

It would be nicer to explain that "the proofs for the two theorems are in the supplementary" in page 6. Since the supplementary was in a separate PDF file, I didn't notice this when I first read this paper.

On page 8, it says "See Appendix D.4 for a plot on ImageNet-LT", but this should be D.2 instead.

*******************
After rebuttal period: thank you for answering my questions and for updating the paper. I still think it is a good paper and would like to keep my score.
review_title: Simple algorithm for long-tail learning with Fisher consistency

final_score: rating: 6: Marginally above acceptance threshold
review_content: This papers presents a general loss function for long-tail classification with several previous work as its special cases.

This is a well-written paper, and the results are impressive. The approach builds upon prior work and a general framework is presented. The proposed approaches are eveluated on several commonly used datasets and show some improvements. 

My one major technical concern are as follows:
1. The originality of this paper is not very high since the proposed framework and its components are not novel (there might be some minor novelty such as the fisher consistency property of the objective);
2. Regarding the post-hoc logit adjustment, I am supposing it is sensitive to $\pi_y^\tau$, which is not very much similar with weight normalization;
3. For the balanced error, I am interested in why it is supposed to a better performance measure, given that the test data distribution is uniform (as per datasets used in experiments);
4. In the experiments, e.g., Table 2, in my humble opinion, some of resfults for comparison methods are incorrect. Since prior work (including Weight normalisation, Adaptive, Equalised) report Top-1 classification error, instead of the balanced error. Hence, I guess that the comparison is not fair at all.

========== after reading the authors feedback =========

Thanks the authors for addressing my concerns and I am convinced that this work is very much different from prior literature. In addition, the evaluation metrics are correct in the studied problem setup. Based on that, I would like to raise my score from 5 to 6.
review_title: This papers presents a general loss function for long-tail classification with several previous work as its special cases.

Meta-review: This paper got 3 acceptance and 1 marginally below the threshold. After the rebuttal, the rating was raised to above the threshold. All the reviewers are positive about this submission. They agree that the method proposed in the submission is novel, the experiments are comprehensive and convincing. AC agrees and recommend acceptance. 

Decision: Accept (Spotlight)