Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: These are several concerns:
1. In the view of motivation, I don't think the motivation is strong enough and is convincing. Also, I don't think rewarding correct predictions but not penalizing incorrect ones is a reasonable way. In my opinion,  rewarding the correct predictions may be a good way, but penalizing the incorrect ones should also be important. 
2. In the view of experiments, though the authors add Table 7 in the appendix, which is the result for training 90 epochs, I still doubt why Eureka Loss does not work better than recent works when training 200 epochs (which is also a common setting recently). And it seems that using CE at the beginning of training is important, and +CB$^+$ works the best.  Moreover, In table 2, the results on "few" are especially not very good comparing with others, which makes it harder for me to believe that rewarding the high-likelihood area really matters a lot for tail classes. It seems that the experiment results are not strong enough to support the proposed opinion.
review_title: The motivation is not convincing enough.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The submission makes an intriguing claim that retaining focus on correctly predicted rare classes can improve performance for training with class-imbalanced datasets.

To illustrate this claim, the paper shows that one can find improvements at overall accuracy if a combination of the Focal Loss (which weights down examples with high predictive likelihood) and the cross-entropy loss is used such that the loss transitions from the Focal Loss to the CE for examples with predictive confidence above a threshold, for examples belonging to the top rarest classes. On long-tailed CIFAR-10, this produces a mild improvement at overall accuracy (around 0.5%) when the top 40% rarest classes receive this mixed loss. Further experiments with COCO-detection finds sparse improvements (around 0.4%) when applying the mixed loss to the tail classes. 

Based on the above findings, the paper argues for not weighting down confident predictions, especially if these belong to rare classes. However, perhaps these experiments are insufficient to arrive at such a conclusion? To ensure that the minor improvements in CIFAR-10 are in fact due to the claimed reasoning, one could also look at other combinations of the losses that do not conform to the claim. For example, apply the loss to the top k% most confident examples (without stratifying by rare classes), randomly select k% of images, etc. For COCO-detection, apply HFL to the head classes, and FL to the tail classes. Since improvements are so small, it would also be nice to see some standard deviation bars over multiple trials. Also, were the choices of Focal Loss hyper-parameters made to elicit their best performance? From Figure 2, it looks like it underperforms the cross-entropy loss.

The paper proposes a new loss meant to "reward the well-classified rare examples”. This augments the cross-entropy loss with a log(1-p_y) term scaled with a number that reflects the frequency of class y, such that rarer classes are scaled higher. 

Experiments have been conducted on 2 image classification datasets and 1 dialogue dataset. In all cases, the proposed loss appears to result in improvements over baselines. 

Some questions/comments about the experiments:
 - It appears that the proposed loss performs particularly well when combined with CB. Are the competing methods also similarly augmented? 
 - For Table 4, why is the Focal Loss only evaluated for 2 settings of gamma? Shouldn’t there be a hyper-parameter search and the best gamma used?
 - There are a lot of comparisons, with a lot of numbers being taken from past reported results. For all such comparisons, has it been ensured that the architectural and training details are fixed across comparisons? Otherwise the comparisons might not be fair, especially given that reported improvements are minor.
 - Especially when improvements are minor, it becomes important to look at aggregate numbers, so I’d suggest reporting standard deviations over multiple trials for all experiments.

Some typos:
“down applications” —> “downstream applications”
“a effective number” —> “an effective number”
“thus the likelihood” —> “so that the likelihood”
“deferred courage” —> “deferred encouragement"

Overall, the paper is clearly written and reports exhaustive experiments (with the caveats/questions above). While the motivating experiments in Section 2.2 are not very compelling, in part due to the very minor improvements, the key intuition that the classification of rare-class hard examples should be continued to be encouraged (so that their predictive confidence doesn’t drop as these examples are weighted down by some of the other methods) sounds interesting, although some of the phrasing about “rewarding well-classified examples” can be a bit awkward. My main concerns as of now are about experimental details, which are described above in the questions.

Post Rebuttal:
Thanks to the authors for responding. I'm still not sure if the experiments are particularly compelling. There appear to be differences amongst the baselines with regards to class balancing, and the motivating section is still weak; there are new experiments on a larger dataset, but now with a different loss (simplified EL) which is close enough to the proposed loss that this does not work very well as a motivation anymore. Apart from this, taking some of the comments from the other reviewers and the authors' responses into account, I am retaining my initial rating.
review_title: Intriguing submission; experiments might be improved

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper deals with learning imbalanced class distributions.  First, it empirically finds that the high-likelihood area for the rare classes benefits classification. Then, based on the findings, it proposes a new learning objective called Eureka Loss, which can be viewed as a combination of the frequency-based and likelihood-based methods to reward the classifier when examples belong to rare classes in the high-likelihood area are correctly predicted. Empirical results on two typical tasks (i.e. image classification and language generation tasks) illustrate its superiority compared with other baselines. 


###########################################################################################
pros:
1. Overall, it is well-written. 
2. It clearly discusses the existing two methods (i.e. frequency-based methods and likelihood-based methods). Furthermore, it highlights the limitation of likelihood-based methods that they neglect the correctly-predicted tail class examples.
3. The motivation for the design of the new learning objective(i.e., Eureka Loss) is based on the empirical finding that the high-likelihood area of the rare examples is important to improve the performance.

###########################################################################################
cons:
1. The finding is mainly on empirical observations, which may lack theoretical support. Why is the high-likelihood area of the rare examples is important for generalization?
2. For the experimental settings, e.g. iNaturalist 2018, the i.i.d. assumption does not hold for the training and test set.
3. For the experimental results, how to tune the hyperparameter of the Eureka Loss, in validation set or test set? Since the reason in 2, I guess the hyper-parameter selection becomes difficult.

Minor comments:
For the last subfigure in Figure 1, the ordinate value for the loss is negative, which is wrong.

review_title: The experimental setting needs to be clarified

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

- This paper made a finding that weighting up  correct predictions for rare class examples also can help to improve the performance of imbalanced classification. In light of this finding, it proposes the Eureka Loss to add additional gradients for examples belong to rare classes in the high-likelihood area when correctly predicted. Experiments on several large-scale benchmarks demonstrate its effectiveness.

Pros:
- The paper is clearly written and easy to follow.
- The experiments are thorough and demonstrate the effectiveness.

Cons:
- While the finding is quite interesting, I think the design of the proposed algorithm is quite arbitrary. It's not clear to me why the authors choose to add a term for rare classes rather than changing the weights directly. Why don't the authors use HFL in the end?
- Currently it seems that there lacks complementary theory/intuition that could explain why weighting up the already correctly classified rare examples help with the performance.

Additional Questions:
- Figure 4 seems quite interesting. It seems that the functionality of Eureka Loss is quite different from HFL. I could intuitively understand that the Eureka loss function would encourage the examples to have likelihood of either 1 or 0. Have the authors visually checked the examples with a likelihood of 0? Does that mean training on a carefully selected subset gives better performance?

----
post-rebuttal update 

I thank the authors for the responses. While I still think the idea is potentially interesting and original, I could not increase the score given the fact that this manuscript is naturally incremental without theoretical justifications.
review_title: interesting finding but lacking explanation/understanding

Meta-review: This submission got 1 reject and 3 marginally below the threshold. The concerns in the original reviews include (1) lack of theoretical justification. The motivation and claim are from empirical observation; (2) the performance improvement is minor compared with the existing methods; (3) some experiment settings and details are not explained clearly. Though the authors provide some additional experiments to the questions about the experiments, reviewers still keep their ratings. The rebuttal did not address their questions. AC has read the paper and all the reviews/discussions. AC has the same recommendation as the reviewers. The major concerns are (1) the theoretical justification is not clear. The additional explanation given by the authors in their rebuttal, i.e., the prediction becomes sharper and thus the model generalization ability can be improved, is not justified. (2) the experiments are not very convincing and can be further improved in the following two aspects: (1) the motivation experiments should be conducted in a consistent manner, instead of using simplified EL in some cases; (2) the effectiveness of EL should be more significant otherwise it is not clear whether the claim is true or not. At the current status of this submission, AC cannot recommend acceptance for the submission.
Decision: Reject