High-Likelihood Area Matters --- Rewarding Correct,Rare Predictions Under Imbalanced Distributions
Authors: guangxiang zhao
Lei Li
Xuancheng Ren
Xu Sun
Bin He
Venue:  (2021)
Keywords: classification
imbalance
long-tailed
likelihood
focal loss

Abstract: Learning from natural datasets poses significant challenges for traditional classification methods based on the cross-entropy objective due to imbalanced class distributions. It is intuitive to assume that the examples from rare classes are harder to learn so that the classifier is uncertain of the prediction, which establishes the low-likelihood area. Based on this, existing approaches drive the classifier actively to correctly predict those incorrect, rare examples. However, this assumption is one-sided and could be misleading. We find in practice that the high-likelihood area contains correct predictions for rare class examples and it plays a vital role in learning imbalanced class distributions. In light of this finding, we propose the Eureka Loss, which rewards the classifier when examples belong to rare classes in the high-likelihood area are correctly predicted. Experiments on the large-scale long-tailed iNaturalist 2018 classification dataset and the ImageNet-LT benchmark both validate the proposed approach. We further analyze the influence of the Eureka Loss in detail on diverse data distributions.

Under review as a conference paper at ICLR 2021
HIGH-LIKELIHOOD AREA MATTERS REWARDING CORRECT, RARE CLASS PREDICTIONS UNDER IMBAL- ANCED DISTRIBUTIONS
Anonymous authors Paper under double-bling review
ABSTRACT
Learning from natural datasets poses significant challenges for traditional clas- sification methods based on the cross-entropy objective due to imbalanced class distributions. is intuitive assume that the examples from rare classes are harder to learn so that the classifier is uncertain of the prediction, which establishes the low-likelihood area, Based on this, existing approaches drive the classifier actively to correctly predict those incorrect, rare examples. However, this assumption is one-sided and could be misleading. We find practice that the high-likelihood area contains correct predictions for rare class examples and it plays vital role in learning imbalanced class distributions. In light of this finding, we propose the Eureka Loss, which rewards the classifier when examples belong to rare classes in the high-likelihood area are correctly predicted. Experiments on the large-scale long-tailed iNaturalist 2018 classification dataset and the ImageNet-LT benchmark both validate the proposed approach. We further analyze the influence of the Eureka Loss in detail on diverse data distributions.
INTRODUCTION
Existing classification methods usually struggle real-world applications, where the class distri- butions are inherently imbalanced and long-tailed (Van Horn & Perona 2017 Buda et al. 2018 Liu et al., 2019 Gupta al., 2019), in which a few head classes occupy large probability mass while most tail (or rare) classes only possess few examples. The language generation task is vivid example of the long-tailed classification. In this case, word types are considered as the classes and the model predicts probabilities over the vocabulary. Common words such as the, of, and and are the head classes, while tailed classes are rare words like Gobbledygook, Scrumptious, and Agastopia. Conventional classifiers based deep neural networks require large number of training examples to generalize and have been found to under-peree peeder- on rare classes with a few training examples in
downstream applications (Va Horn & Perona 2017 Buda al. 2018 Cao al. 2019
is proposed that the traditional cross-entropy objective is unsuitable for learning imbalanced distributions since treats each instance and each class equivalently (Lin al. [017]] Tan al.) 2020). In contrast, the instances from tail classes should be paid more attention, indicated by two main approaches that have been recently investigated for class-imbalanced classification: the frequency- based methods and the likelihood-baaee directly adjust the weights of the instances in terms of their class frequeneies, so that the instances from the tail classes are learned with higher priority no matter whether they are correctly predicted or not. The latter (Lin et al., 2017 Zhu et al. 2018) instead penalize the inaceurate predictions more heavily, assuming that the well-classified instances, i.c., the instances in the high-likelihood area, factor inconsequentially in learning imbalanced distributions.
methods. The former (Cui et al. 2019 Cao et al.) 2019)
However, neither of these two approaches realistically depicts the likelihood landscape. In particular, the high-likelihood area, where the classifier makes the correct predictions for both common class examples and rare class ones, contributes significantly generalization. However, this area is not well-shaped, as illustrated in Figure[[ Specifically, the frequency- methods imply an impaired learning of common class examples that the principle part of the natural data, while the likelihood-

---

Under review as a conference paper at ICLR 2021
Croa Safrery
Orita Frimes
Cremn Enewa Carie Lons
Prezeandy Sieeed (tfand Pregamncy Sieed (Somman
Roced
6-
Starpen
Sharpen More
Botool
Softon
0.0
1.2
0.8 0.5 0.2 0.4 0.5 1.0
0.0
0.5
1-4
Figure 1: Conceptual illustration of approaches to learning unbalanced class distributions. For an instance in the training data, the frequency-based methods either sharpen soften the loss for all likelihoods according to its class frequency, while the likelihooddbased methods adjust the loss in the low- or high-likelihood area, respectively. The high-likelihood area relatively deprioritized in both cases. The proposed Eureka Loss progressively rewards the systems with higher bonus for higher-ikellihod. likelihood.
based methods ignore the into the underlying mechanism for predicting such examples.
rare class examples that can provide crucial insights
In this paper, we first demonstrate that existing practice of neglecting predictions the high- likelihood area is harmful to learning imbalanced class distributions. Furthermore, we find that simply mixing the cross-entropy loss and the Focal Loss (Lin et al. 2017) can induce substantially superior performance, which validates our motivation. In turn, predictions even further and design a novel objective called Eureka Loss. It progressively rewards the classifiers according to both the likelihood and the class frequency of example such that the system is encouraged to be more confident in the correct prediction of examples from rare classes. Experimental results on the image classification and the language generation tasks demonstrate that the Eureka Loss outperforms strong baselines in learning imbalanced class distributions.
propose elevate the importance of nigh-likelihood
Our contributions are twofold:
We challenge the common belief that learning for examples in low-likelihood area is more important for learning tail classes and reveal that the make important contribution to learning long-tailed class distributions.
rare class examples
We explore a new direction for learning imbalanced classification that focuses on rewarding correct predictions for tail classes examples, rather than penalizing incorrect ones. The proposed Eureka Loss rewards the classifier for its high- predictions progressively to the rarity of their class and achieves substantial improvements various problems with long-tain tailed distributions.
2 RELATED WORK
Frequency-bead based Data and Loss Re-balaneing Previous literature learning with long-taiie loigetailed distribution mainly focusing on re-balancing the data distribution and re-weighting the loss function.
The former is based on straightforward idea manually create a pseudo-balanced data distribution to ease the learning problem, including up-sampling for rare class examples (Chawla et al. 2002).. down-sammn sammling for head class examples (Drummond & Holte 2003) and a more concrete sampling strategy based on class frequency (Shen et al. (016).
As for the latter, recent studies propose to assign different weights to different classes, and the weights can be calculated according to the class distribution. For example, Khan et al.) (2018) design cost-sensitive loss for major and minor class examples. An intuitive method is to down-weight the loss of frequent classes, while up-weight the contribution of rare class examples. However, frequency is not suitable to be directly treated as the the weight since there exists overlap among samples. An advancing alternative loss CB (Cui al. 2019) proposes to calculate the effective number to substitute the frequency for loss re- weighigh.nng. However, since assigns lower weight to head classes in the maximum likelihood training (Cross Entropy objective), it seriously impairs the learning
2

---

Under review as a conference paper at ICLR 2021
of head classes. Moreover, CB requires delicate ppper-prraeeter tuning for every imbalanced distribution, leading to a lot of manul efforts. From the perspective of max mar-mn, a recent study LDAM (Cao et al. 2019) proposes to up-weight the loss of tail classes by class- dlas-diisributton based margin. Compared to the above methods, we choose to decrease the loss of tail classes by rewarding correct predictions rather than increasing the loss of tail classes through aggravated penalization.
Deferring the Frequency-based Class-balanced Training Recent studies find that deferring the class-balanced training helps learn high-quality representations (Liu al. 2019). and propose deferred Class-balanced training (deferred CB) (Cao et al.) 2019), which chooses to adopt Cross Entropy objective the beginning of training. Similarly, the Decoupling method (Kang et al., 2020) shows that the re- re-balancing strategies impair the quality of learned feature representations and demonstrate an improved performance learned with original data distribution, by training the model with Cross Entropy in the first phase and adopting class-balanced training in the second phase. This decoupling strategy can also be found in BBN (Zhou al., 2019), which includes both class imbalanced and balanced training, and the transition from the former to the latter is achieved through a curriculum learning schedule. These methods achieve state-of-the-art performance in long-tailed classification. To be comparable with these methods and to analyse whether Eureka Loss is complementary to this technique, we propose deferred Eureka Loss, in which rewarding for rare class prediction is introduced to encourage the model to learn rare patterns when learning stalled.
Likelihood-basee Loss Another dominant method for imbalanced classification is the likelihood- based method Focal Loss (FL) (Lin et al.) 2017), which proposes down- weight the contribution of examples in the high-likelihood area. However, we argue that it is harmful for learning tail classes and choose an opposite direction by highlighting the high-likelihood area with a steeper loss.
Transferring Representations Techniques for transferring information from sufficient head classes examples redeeseepre rare classes examples belong parallel successful direction in this field. They include MBJ (Liu et al., 2020). which utilizes external semantic feature memory and FSA (Chu al.) 2020). which decomposes feature in to class-specific and class-generic components. These latest transfer learning based studies are less related to our paper but they also obtain good improvements in
classification, so we add them into comparison in the experiments.
3 ROLE OF THE HIGH-LIKELIHOOD AREA
The existing approaches to the long-tailed classification independently consider the class frequency and the example likelihood. However, we show that this one-sided reflection is problematic when dealing with the tail class examples that can be confidently classified. The tail class examples can be easily classified by the classifier, and the head class examples can also be hard for the classifier to recognize. The difficulty of classification depends on the inherent characteristic of the classes, rather than the sample size of the class. For example, in species classification, the Portuguese man o' 'war may be rare class but can be casily classified due to its distinct features, compared to various kinds of moths which are common classes yet are hard to distinguish. However, the Trequency-based methods continuously drive the classifier to fit the rare class examples, especially when they are difficult to predict, which may lead to overfitting. On the other hand, the likelihood-based methods relax the concentration in the high-likelihood area, which contains the tail class examples that are not hard to predict and provide insights to generalization.
To verify our point of view, we analyze the problem by dissecting the influence of the high- high-likelihood area with respect to the class frequency and demonstrate that properly encouraging the learning of well-classified tail class examples induce substantial improvements, before which we first give a brief introduction of classification with long-taiin class distributions.
3.1 PREPARATION: CLASSIFICATION WITH -AIGET CLASS DISTRIBUTIONS
Let's consider the multi-class classification problem with the long-tailed class distribution. Given a class set C, denotes the number of different classes in C and m, is the number of examples of the class C,. For simplicity, we sort the class set C according to cardinal m, for C. such that Co is the class with the most examples and Cx-1 is the rarest class. Let p be n-dim probability vector predicted by a classifier model f fæ; 0) based on the input æ, where each element p; denotes the probability of the class C; and y is n-dim one-hot label vector with y being the ground-truth class.
3

---

Under review as a conference paper at ICLR 2021
The probability vector can be calculated as
p (f(())))
(I)
where is the normalizing function, e.g., softmax for multi-class classification. Typically, the parameters are estimated using maximum likelihood estimation (MLE), which is equivalent to using the Cross-Entropy Loss (CE) function, where the scalar y p can be regarded as the log)-likelihood:
(m. log ( == -E (a( log Pmodel = [D] Σ
(2)
For deep neural network-barc based classifiers, due to the non-linearity of the loss function, the problem typically solved by stochastic gradient descent, which requires the calculation of the gradient with respect to the parameters using the rulen the process of which is called back propagaopa:
OC ac 0p Oc do ( ())
00 8p 80
3p
00
We introduce the term likelihood gradient to denote ac/op, which modulates how the probability mass should be shifted and is a characteristic of the loss function instead of the classifier. For learning imbalanced class distributions, the common methods aim shape the likelihood gradient the rare classes are learned with priority, i.e., embodying sharper loss and a larger likelihood gradient.
Frequency-Base Based Methods Frequency- -based methods alter the likelihood gradient according to the class frequencies, which are irrelevant to how well individual examples are classified. A simple form using a n-dim weight vector w composed of the class weights based on their frequencies in the dataset to determine the importance of examples from each class:
L Wyy (y log
(4)
Note that when calculated as w;; = m//mm m, where m is the average of m, As we can see, the standard weight is taken the average of the class size, so that the classes with more examples are down-weighted and the classes with fewer examples are up-weighted. For a natural long-tailed distribution, the average is larger than the median, which suggests more classes are up- up-wweigted.. Advanced frequeney-basee methods try to obtain more meaningful measurement of the class size, e.g., the Blaas--allacee Loss (CB) proposed by Cui et al. (2019) utilizes an effective number 1 --mh/// /1 -1 for each class, where 3 E [0.9, 1) is a tunable class-balanced term.
1, it is identical to the cross-entropy objective. The weight vector is typically
Methods Different from the frequency- -base methods, likelihood- based meth- ods adjust the likelihood gradient based on the instance-level difficulty as predicted by the classifier such that the examples in the low-likelihood area are more focused training. For example, the well-known Focal Loss (FL.) with balanced factor a proposed b et al. (201)) takes the following form:
Likelihood-Base
(5) where 7 > O, which controls the convexness of the loss and higher y indicates adjustment that are more significant. Note that when Following previous works (Cao et al. 2019 Liu et al. 2019 Cui et al., 2019), the a is set to in multi-class classification, and Class- Balanced Focal Loss (FL+CB) can be viewed Focal Loss with uneven alpha for each class in the multi-clan setting. The key idea is to pay less attention to the well-classified examples and pay more attention to the badly-classified examples, because it is natural to assume the tail class examples are harder to learn and thus cannot be well-classified. However, such methods neglect the correctly-predicet tail class examples, the practice of which we show is not constructive to the learning of long tailed class distributions.
C= -=1 7t log
and Y 0, it is identical to the cross-entropy objective.
3.2 UNDERSTANDING THE INFLUENCE OF THE HIGHI-LIKELIIOOD AREA
To understand the influence of the likeliikel area, we first prepare variant of the Focal Loss, the Halted Focal Loss (HFL), such that the high-likelihood area is not deprioritized. The Halted Focal Loss reverts the Focal Loss to the Cross-Entropy Entropy Loss when the likelihood is high enough:
aa11 pby "y,)"r -y log.). if < 4 otherwise
f =
(6)
ay ag. [log + b).
4

---

Under review as a conference paper at ICLR 2021
Resuda Dee Notted Ford best
Method
AP
APso
APTs
CE
32.8
52.3
34.7
FL
33.8
52.5
35.9
HFL
34.0
52.7
36.2
FL (Head) HFL (Tail)
34.2
52.7
36.3
Table 1 Results of HFL the COCO de- tection dataset, AP denotes average precision. As we can see, increasing the importance of the high- high-likelihood area achieves better re- sults, and the main improvements come from the tail class examples.
Figure 2: Regaining focus on the high-likelihodd area for the classes benefits classification. Left: Illustration of HFL which reverts FL to CE in the high-likelihood area. Right: Applying HFL only the rare classes improves overall performance.
where Py is prediction probability of the correct label, b = a(1 - (1 ς) log y to ensure monotonicity and continuity, and & is the boundary between the low- and the high- high-likelihood area, which we set as 0.5, i.e., a likelihood higher than which definitely renders a correct prediction. This mixed loss is plotted in left of the Figure 2 which has the same likelihood gradient as the cross-entropy in the high-libe likeliiko area and remains the same as the Focal Loss in the low-likelihood area.
To decouple the effect of class frequency, we further explore to gradually transition from the Focal Loss to the Halted Focal Loss according to the class frequency of an example, e.g., from adopting the Halted Focal Loss only for the rarest class and the Focal Loss to other classes to adopting the Focal Loss only for the most common class and the Halted Focal Loss for the rest. Concretely, we set proportion / [0, 1] of classes to receive this loss and the remaining 1 proportion of classes adopt the original Focal Loss. The classes are ranked by inverse frequencies, such that the first class the rarest class.
We conduct experiments on long-tailed CIFAR-10 using the aforementioned protocol to examine the effect of the likelihokio area. The construction of the dataset is provided in Appendix C We run each configuration 5 times with different random initialization and report the average test performance. The results are shown the right of Figure 2
As we can see, compared to the original Focal Loss, the proposed adaptation achieves better per- formance, indicating regaining focus in the high-likelihood area beneficial. Nonetheless, the phenomenon can be also attributed to better learning of the common classes instead of the rare classes. Our analysis based on the class frequency resolves this concern because the Halted Focal Loss brings more improvements if only tailed classes are learned this way, e.g., applying to the top-4 rare classes achieve the best overall performance, which proves that there are rare class examples that reside in the high- licelhiid area and have negligibbe effect to generalization.
The importance of the high-likelihen detection dataset, where the classifier should determine whether the object appears in the image or not, The positive detection is the rare class since there are many false proposals. The experiment setting is in Appendix C AP5o and AP75 measure the precision under different levels of overlap between predictions and ground-truth. As shown in Table [ strengthening the high-likelihood area of the Focal Loss, especially for the rare class examples, obtains more accurate and confident prediction.
area of the rare examples is further validated on the COCO
4 EUREKA LOSS
We have shown that the nigh-likelihon area matters for long-tailed classification and in particular, the rare class examples in the area have pivotal contributions. Inspired by this finding, we propose to further enhance the importance of the high-likelihood area so that the likelihood gradient in the high-likelihood area can match even surpass that in the low-likelihood area. Moreover, the adjustment is inline with the frequency of the class so the rarer the class, the larger the likelihood gradient. Extending the adjustment term b in Eq. (6), propose the Eureka Loss (EL):
-y. -y-log log p- bonus-encouraggemen,
5

---

Under review as a conference paper at ICLR 2021
where -bonus-encoureneeee encouragement is intended to reward the well-classifedd rare examples. The bonus term depends on the example likelihood and the encouragement term depends on the class frequency. Different from the existing approaches that scale the Cross-Entropy Loss, punishing the incorrect predictions selectively, the proposed Eureka Loss deals with long-tailed classification from another perspective, rewarding the correct predictions progressively with their class frequencies.
Bonus indicates how well the system executes the task and is designed to be a function of the probability of the ground-truth class reward the model when it makes correct prediction. In particular, in light of the findings discussed in Section 3.2] we propose to increase the likelihood
gradient in the high- sigeliikelhood area and adopt the form of
bonus y log(1 p),
which ensures that the monotonicity of the likelihood gradient is consistent with that of the Cross- Entropy Loss, meaning that the classifier obtains more bonus when making highly-confident correct predictions. The design is against most existing studies that the high- nigh-likelihood area is given more focus than the low-likelihood area.
Encouragement implies the system realizes unusual achievements that should be encouraged. Since the unusual achievements in long-tailed classification should be correctly predicting rare class examples, we propose to reward the system based on the frequency of the example's 's class:
m
encouragement wy my
where my denotes the measurement of the frequency of the class y. The form is flexible and similar to the frequency-basee methods, and thus can be further extended based on the related studies. In our experiments, we use the effective number from Cui et al. (2019) as the measurement.
Compared to the existing frequency- -auseed and likelihood- -ased objective, our Eureka Loss takes the bonus term calibrate the attention to different likelihood landscapes and the encouragement term to inform the model with the class difficulty, composing more targeted yet comprehensive loss for learning imbalanced distributions.
5
EXPERIMENTS
We validate the proposed Eureka Loss on diverse loillt classification problems and analyze the characteristics of the Eureka Loss with insights into the learned models.
5.1 TASKS, DATASETS, AND TRAINING SETTINGS
Tasks and Datasets We conduct experiments on two image classification tasks and a dialogue generation task. iNaturalist 2018 is real-world dataset which embodies highly imbalanced class distribution of 8, 122 classes. Apart from the test performance, we also report the validation performance grouped by the class frequency and categorize the examples into three groups: many (classes with more than 100 examples), medium (classes with 20 to 100 examples), and few (classes with fewer than 20 examples). ImageNet-LT (Liu
2019) an artificially constructed long-tailed classification dataset based on ILSVRC 2012 of 1000 classes. ConvAl2 is a natural conversation dataset for evaluating dialogue system, where each word type can be treated class, i.e., 18,848 words (classes) in total, and have extremely imbalanced training and test datasets.
Evaluation Metric For the image classification tasks, we use the accuracy on 'All' data and subset of classes with Many'. 'Medium' and 'Few" examples i.e., the precision of the top-1 prediction. Since the test set of those tasks are balanced classes, we further propose to estimate the accuracy on the imbalanced class distribution that reflects natural performance in real-world scenarios. The natural accuracy is the linear interpolation of the accuracy on the balanced test set using the class frequencies from the training set. For the natural language generation task, we adopt the micro and macro F-scores from Zhang et al. 2018) between the generated and the reference sentences to check how well the systems participate in the conversation. We further adopt the 4-gram diversity to examine the rare phrases, since a well-known problem for dialogue tasks is that the model tends to generate common, dull and repetitive responses and thus cannot capture the diversity of natural

---

Under review as a conference paper at ICLR 2021
Method
All
Many
Medium
Few
All (Natural)
CE
64.3
74.1
63.9
59.8
71.7
CB
58.3
61.8
58.6
56.9
59.8
FL
62.9
72.9
64.1
58.8
70.7
FL + CB
59.6
45,5
61,8
60.5
46.7
Eureka Loss
68.5
70.8
69.3
66.7
70.6
CB
68.1
71.0
68.3
67.1
70.4
FL + CB
66.9
64.4
67.4
67.0
64.9
LDAM+CB! (Cao
2019
63.3
65.2
63.0
63.1
64.1
BBN (Zhou ct al.
2019
69.6
Decouplig-LWS
Kang al.
2020
69.5
71.0
68.8
69.5
Eureka Loss
69.9
73.3
69.3
69.6
73.0
FSA Chu et al.
2020
65.9
MBJ (Lu et al.
2020 5
68.6
68.0
69.8
68.5
Eureka Loss + CB
70.3
69.0
70.1
70.9
69.4
Table Results on iNaturalist 2018. denotes results from the corresponding paper and denotes deferred learning, where the base loss is applied at the beginning of training and the improved method adopted later. 3 denotes that it focuses on transferring representations and the method less related to our work. Best results are shown in bold. The proposed Eureka Loss achieves the best results in learning both common and rare classes.
language distributions. Since the test set of natural language generation task is naturally imbalanced, we do not need to estimate the natural performance.
For the detailed introduction tasks, datasets, and training settings, please refer to the appendix.
5.2 RESULTS
iNaturalist 2018 The results are reported in Table We tune the hyper-parametesd for our imple- mented baselines and report the averaged performance among runs at the best setting.
We compare Eureka Loss with frequency basee method Class-balanced Loss (CB), likelihood- based method Focal Loss (FL) and their combination FL+CB. As we can see from the first group in the table, neither FL nor CB achieves improvements over Cross Entropy (CE), but Eureka Loss outperforms CE by a large margin in terms of overall accuracy and accuracy for classes with few examples.
In contrary to the first group, considering only the accuracy the balanced test set, the two-stage version of frequency latter training phase includes deferred CB(denotes CBt in the table), LDAM + deferred CB, BBN and Decoupling-LWWs enjoy clear advantage over CE. In order to check whether Eureka Loss is additive with the deferred method and the bass-balanced training, we take deferred Eureka Loss and Eureka Loss + CB into comparison.
training which adopts the glass-balanced training only in the
The deferred Eureka Loss is motivated by an intuition that when training enters the bottleneck stage, Eureka Loss is introduced to reward rare classes to encourage the model to learn less common patterns, which may be helpful for learning. Compared with the original method. the deferred encouragement brings improvement on both balanced and imbalanced test distribution (+1.4 144 and +2.4 regarding All and All(Natural, respectively). Moreover, the class- maass-bllanced training still impairs the learning for common classes even under the deferred setting, which may cast into unfavorable natural performance in real applications, e.g., the accuracy on the 'Many' subset for CB and Decoupling- LWS is under-performs CE by 3.1, the results is that applying CB reduces the Natural accuracy by 1.3. But deferred Eureka Loss largely outperforms CE and these methods on both balanced and imbalanced test distributions. The reason may be that we do not impair the CE learning and the additional rewarding for rare classes is less harmful. Since Eureka Loss only introduces an additive term, it is flexible and can be combined with CB, the adoption results in best All accuracy of 70.3.
7

---

Under review as a conference paper at ICLR 2021
Method
All
All (Natural)
Method (Deferred)
All
All (Natural)
CE
44.6
63.7
OLTR (Liu 2019
46.5
FL
43.6
62.4
CB
49.2
60.5
CB
43.9
58.8
Decoupling- Kang al. 2020
49.9
59.8
FL+CB
31.1
25.0
Eureka Loss CB
50.4
62.2
Eureka Loss
48.4
63.8
Table Results on ImageNet-LT. and are defined similarly to Table consistent improvements against existing methods.
Eureka Loss demonstrates
Name
F-Score (Macro)
F-Score (Micro)
Diversity (4-gram)
Cross Entropy
1.17
16.9
36.5
Focal Loss (7
1.17
16.8
36.6
Focal Loss (7
1.13
16.7
37.3
Fureka Loss
1.32
17.2
40.4
Table 4: F-scores and 4-gram diversity on ConvA12. The proposed Eureka Loss achieves better performance than baseline methods and generates responses that are more diverse.
In all, adopting the Eureka Loss achieves balanced performance on both common and rare classes. Besides, we also outperform the latest representation transferring based methods including MBJ and FSA.
ImageNet-I.T Table demonstrates the results on ImageNet-I.T of various methods. For this artificial dataset, we first compare with the representative frequency-basee method CB and likelihood- based method Focal Loss (FL). As we see, the proposed method obtains significant improvement in the balanced lest set and also maintains the lead position in the virtual natural test set. For comparison with methods that defer the bassnbalanced training including deferred CB and Decouping- LWS, the Eureka Loss of corresponding modification also enjoys a comfortable margin and arguably excels in balancing the performance both of the common and the rare classes.
ConvAl2 Table shows that the proposal helps the prediction of rare words (+10% macro F-score) and thus improves the diversity of language generation (+10% 4-gram diversity). Since this dataset is extremely imbalanced, e.g., the imbalance ratio is over 200,000, the frequency-base extensive tuning work, which we thus omit from the comparison method Focal Loss, which is marginally better
methods require are not able reproduce
favorable results. Compared with the likelihood-baate that the original cross entropy loss, the Eureka Loss still obtains substantial improvements.
5.3 ANALYSIS
Effeet Distributions of Different Imbalance De- grees To analyze the effect on imbalanced distribu- tions of different degrees, construet several artificial datasets based on CIFAR-100 and control the size of the rarest class. The imbalance degree stands for the ratio of the class size of the most common class to that of the rarest class. Hence, the larger the degree, the more im- balanced the dataset. For example, if imbalance degree 100 for CIFAR-100, the most common class has 500 examples and the rarest class has examples. As shown in Table 5 the Eureka Loss consistently better than existing methods, especially for datasets that are imbalanced. For results on CIFAR-10, please refer to the appendix.
Method
ID-100
ID-50
ID-10 10
CE+RS CE+CB14 i
41.61
46.48
58.11
41.51
45.29
58.12
LDAM+CB
42.04
46.62
58.71
BBN
42.56
47.02
59.12
EL + CB1
43.19
48.48
59.31
Table 5: Results long-tailed CIFAR- 100 of different imbalance degrees (ID). denotes deferred learning: and de-
notes results taken from Zhou et al. (0099 and Cao al. (0199) respectively.
Varying Strength of Bonus To illustrate the impor- tance of the high-likelihood area in imbalanced classification within Eureka Loss, compare
8

---

Under review as a conference paper at ICLR 2021
Poact
Method
All
Many
Medium
Few
EL (3=0.99)
45.8
67.2
38.4
11.3
EL (f==0.999)
47.8
66.1
42.0
16.4
EL (3=0.9999)
50.0
67.1
44.7
19.8
Table 6: Varying strength of encourage- ment the dev set of ImageNet-LT. Higher & means higher strength.
Figure 3: Varying strength of bonus on long-tailed CIFAR-10. Higher power "/ indicates higher strength.
exponential form bonus called Power Bonus (PB) to the original bonus, which takes the power form of the probability vector of by factor 76:
PB(p) = y..pp"
(10)
where 76 is positive value ensure the monotonicity and can be tuned for different tasks. Besides, CE achieves a 71.4% accuracy and the likelihood bonus with deferred encouragement gets a 76. 1 % accuracy. Figure 3 demonstrates that bigger likelihood gradient in the high-likelln lighliikelhhod area brings more improvements, e.g., power-bonus with power of 4 is better than bonuses of small power.
Varying Strength of Encouragement The strength of encouragement is determined by both of the class frequency and the Syper--arameete 3 as we use the effective number of the class. As B controls the variance of the the effective number, e.g., when / = 0, the variance is O, meaning all of the classes receive equal encouragement, we control the strength of the encouragement towards tail classes by altering B. The results on the validation set of ImageNet-LT are shown in Table As we can see, higher / (more encouragement for tail classes), is connected to higher overall accuracy and better tail class performance, which again validates our motivation for encouraging correct rare class predictions.
Effect on Example-I.kellhood Likelihood The Eureka Loss re- wards the high-likelihood predictions especially for tail classes. It is interesting to see how the training dynamic is changed due to this preference. In order to understand the effect, we visualize the example likelihood grouped by target class frequencies after training in Figure 4 and Figure [5] (due to space limit, the complete comparison provided in the Appendix A), which from the val- idation set from iNaturalist 2018. As we can see, with
10000
1000
Mary
5000
unlium
the Eureka Loss, the examples the high-likelihood area are driven to the extreme. For example, considering the medium and the low frequency group, the "hard" ex- amples that may be inherently difficult to classify stand invariant, while for the examples that can be classified correctly, the system now treats them with more confi- dence. This dynamics translate into better accuracy unseen examples in the test set, hinting the importance of rare class examples in the high- high-likelihood area for the generalization of learning imbalanced class distributions.
5000
Fcw
Lisihood
Figure The visualization of test likeli- hood distribution for models trained with the Eureka Loss. The model classifies the rare class examples more decisively, compared to the existing methods.
6 CONCLUSIONS
In this paper, we examine the effect of the tions. We find that the existing practice of relatively diminishing the contribution of the examples in the high-likelihood area is actually harmful to the learning. We further show that the rare class examples in the high-likelihood area have pivotal contribution to model performance and should be focused on instead of being neglected. Motivated by this, we propose the Eureka Loss, which addi- tionally rewards the well-classified rare class examples. The results of the Eureka Loss in the image classification and natural language generation problems demonstrate the potential of reconsidering the role of the high- nigh-likelihood area. In-depth analysis also verifies the effectiveness of the investigated loss form and reveals the learning dynamics of different approaches long-tailed classification.
area on learning imbalanced class distribu-

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski, A systematic study of the class imbalance problem convolutional neural networks. Neural Networks, 106:249-22 259, 2018.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-awaar margin loss. CoRR, 1906.07 07.107711, 2019.
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16: 321-357, 2002.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling, Feature space augmentation for long-tailed data. CoRR, abs/2008.0360 03673, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Blass-balanced loss based on effective number of samples. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268-9277, 2019.
Chris Drummond and Robert Holte. C4.5, class imbalance, and cost sensitivity: Why under-sampling from beats oversampling. Proceedings of the (CML'03 Workshop on Learning from Imbalanced Datasets, 2003.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet hour. CoRR, abs/1706.00 706.02677,, 2017.
Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5351-5359, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-777 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9: 1735-1780, 1997.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-taile tailed recognition. In International Conference on Learning Representations, 2020.
Salman H. Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous Ahmed Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Trans. Neural Networks Learn. Syst., 29(8):3573-3587 2018.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In IEEE International Conference Computer Vision, pp. 2999-3007, 2017.
Jialun Liu, Jingwei Zhang, Wenhui Li, Chi Zhang, and Yifan Sun. Memory-based jitter: Improving visual recognition on long-tailed data with diversity in memory. CoRR, abs/2008.09809. 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong. and Stella X Yu. Large-scale long-tailed tailed recognition in an open world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In Proceedings the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1532-1543, 2014.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In Proceedings of the European Conference on Computer Vision, pp. 467-482, 2016.
10

---

Under review as a conference paper at ICLR 2021
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11659-111 11668, 2020.
Grant Van Horn and Pietro Perona. The devil in the tails: Fine-grained classification in the wild. CoRR, abs/1 709.01 709.01450, 2017.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: have a dog. do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: Bilateral-branch network with cumulative learning for long-tailed visual recognition. CoRR, abs/1912.02413,, 2019.
Xinge Zhu, Hui Zhou, Ceyuan Yang, Jianping Shi, and Dahua Lin. Penalizing top performers: Con- servative loss for semantic segmentation adaptation. In Proceedings of the European Conference on Computer Vision, pp. 568-583, 2018.
A VISUALIZATION OF EXAMPLE LIKELIHOOD
The complete comparison between the likelihood distribution for Eureka Loss, Cross-Entropy Loss, Focal Loss, trained with Eureka Loss gives high-confidence predictions for gold labels. Compared with the Cross-Entropy Loss, the Focal Loss diminish the contribution of high-likelihood examples, and the resulted model is unsure in the prediction of unseen examples. In particular, for the examples in the Few group, it almost produces no confident correct predictions. The Class-Balanced Loss, on the other hand, improves the confidence for the tail class examples but degrade the performance for the head class examples, which may imply potential issues regrading to natural performance in real-world applications. Besides, it is worth noting that the Decoupling-LWi obtains likelihood distribution similar to the -lass-Balanced Loss.
Loss are shown in Figure [ We see from the figure that the model
B FURTHER RESULTS AND ANALYSIS
B.1 RESULTS ON INATURALIST 2018 WITH TRAINING FOR 90 EPOCHS
It is found by existing work (Kang et al. 2020) that training much longer for the iNaturalist 2018 dataset can produce better scores and reflect the performance of the models more authentically. However,
previous studies conduct training for shorter time. To keep consistent with previous research in this field, we also train the models using Eureka Loss for 90 epochs and the results are shown in Table In this setting, eureka loss achieves better accuracy than the two-stage decoupling methods (Decoupling- LWS and BBN), the advantage is more profound under the Natural accuracy, for example, compared the Decoupling-LWo Compared to the one-state methods including Class -alsaced Loss (CB), Focal Loss (FL), Class- Balanced Focal Loss (FL+CB), LDAM, the model trained with the Eureka Loss is much more accurate on the test distribution.
the deferred Eureka Loss gains 3.8 Natural accuracy.
B.2 RESULTS ON LONG-TAILED CIFAR- 10
In the main text, we have reported the results of the Eureka Loss varying the class imbalance on the CIFAR- dataset. Here also perform comprehensive experiments on long-tailed CIFAR-10 10 and report top-1 precision on the balanced test set. The results are shown in the Table g When combined with Balancee Loss, Eureka Loss brings higher improvement terms of accuracy than Cross Cross-Entropy Loss and LDAM.
B.3 HYPER-PARAMETER OF THE FOCAL LOSS
In the paper, we report results for the Focal Loss with best hyper- parameters. For COCO detection, the hyper-parameter of a 0.25, 7 is the best setting reported in Table 1.b of the origin paper
11

---

Under review as a conference paper at ICLR 2021
10000
All
10000
1000
Many
Many
Modium
2500
2500
0
Eureka Loss
(b) Cross- Cross-Entropy Loss
Mary
Maduan
Fer
Fer
(c) Focal Loss
(d) Blass-Balanced Loss
(c) Decoupling-LWW
Helus
(f) Halted Focal Loss
(g) Halted Focal (oss(t=0.5)
(h) Simplified Fureka Loss
Figure 5: Example likelihood on the validation set of iNaturalist18 categorized by class frequencies into few, medium, many, and all.
(Lin al. 2017). For the other multi-class classification tasks, tune hyper-parameters of the Focal Loss on it. The accuracy for the Focal Loss with different hyper-parameter are listed in the Table") we set 7 for Focal Loss since it is consistently optimal in long-tailed image classification. For
ConvA12, 7 0.5 under-pen under-performs Cross Entropy, and neither 1 nor = 2 outperforms each other, so we report the Focal Loss of y and the Focal Loss of y in the Table
B.4 COMPLEMENTARY EXPERIMENT TO THE MOTIVATION EXPERIMENT
In Section we propose Halted Focal Loss(HFL) and compare it to Focal Loss(FL) to illustrate the potential of the high-likelihood area. However, its loss is steeper than Cross Entropy (CE). Moreover, Focal Loss does not beat CE in the setting multi-class classification. In order to bridge the gap between the possibly weak motivation experiment of Halted Focal Loss and the proposed method Eureka Loss. We propose simplified Eureka Loss:
y- log p.
if Py S y y log y [log(1 p) b). otherwise
(11)
Following (Cui et al, 2019 we omit the each class in the setting of mult--llass classification can be viewed as Class-balanced Focal I Loss (FL+CB) and FL&CB is compared individually.
a, since the Focal Loss with uneven 'alpha' for
12

---

Under review as a conference paper at ICLR 2021
Method
All
Many
Medium
Few
All (Natural)
CE
61.8
72.2
63.4
57.2
69.8
FL
60.2
70.4
61.5
56.0
68.3
CB
52.4
51.9
53.0
51.8
51.2
RS
56.7
59.0
56.9
55,9
56.5
FL+CB Cui et al.
2019
61.1
CE+CB
Cao et al.
2019
63.7
LDAM Cao et al
2019)
64.6
LDAM+CB (Cao et
2019)
68.0
LDAM+CB (hhou ct a.
2019
64.6
BBN (Zhou al.
2019
66.3
Decoupling-WW KKng et al.
2020
65.9
65.0
66.3
65.5
65.7
Eureka Loss
66.4
67.5
66.5
65.9
67.9
Eureka Loss
67.1
69.4
67.3
66.1
69.5
Table 7: Results on iNNaturliitt 2018 after training the model for 90 epochs. denotes results from the corresponding paper and i denotes that we use the base loss at the beginning of training and then adopt the method later. The proposed Eureka Loss achieves good results in learning both common and rare classes.
Method
ID-100
ID-50
ID-10
CE+RS &
75.61
79.81
87.38
CE+CB1
76.34
79.97
87.56
LDAM+CB1
77.03
81.03
88.16
BBN
79.82
82.18
88.32
EL + CB
77.95
82.00
88.35
Table 8: Results on long-tailed CIFAR-I0 data with different imbalance degrees. ID is short for imbalance degree. is defined similarly; denotes Zhou al. (2019): denotes Cao al. (2019).
where oo set to 0.5, and is log(1 sp) In the simplified Eureka Loss, the encouragement is removed, and to keep the low-likelihood area unchanged, the new bonus term starts rewarding the model from = y. As shown in Table TO, (FL((= also better than HFL and FL in terms accuracy of tail classes the large scale long-tailed classification dataset iNaturalist 2018, This result once again shows that high-likelihen play major role. But HFL is the combination of Focal Loss(in the low likelihood area) and Cross Entropy(in the high-likelihood area) and the performance is constrained. Unlike HFL, the loss of Simplified Eureka Loss is built on CE and the loss is much steeper than Cross Entropy in the high-likelihowe area, it outperforms Cross Entropy(CE)) and HFL in terms of all metrics, especially on the subset of tail classes. Eureka Loss reported in Table 2 is continuous version of simplified Eureka Loss with an additional encouragement for rare classes, similar to HFL((=0.55, this setting which rewards more for rare classes achieves the best overall performance.
area matters and ncar-correct predictions of rare classes
B.5 EUREKA Loss MITIGATES OVER-FITTING ON TAIL CLASSES
As shown in Figure 6 compared to Cross-Entrops Loss, Eureka Loss reduce the gap between the training accuracy and test accuracy from 33.0 to 28.6 on tail classes. Moreover, even though Class- balanced Loss achieves the highest training accuracy, its test accuracy is unexpectedly low. The difference performance between the seen examples and the unseen examples indicates the degree of over-fitting. The results are from the "Few" subset of the iNaturalist 2018 of training 90 epochs.
13

---

Under review as a conference paper at ICLR 2021
Y
CIFARIO
Imagenet-LT
iNaturalist201
0.5
21.2/10.8
8.3/7.1 43.8/0.3
9.2/3.2
1
70.8/0.6
60.2/0.3
2
70.3/0.6
43.6/7.9
59.5/0.2
Table 9: Mcan/(standard deviation) accuracy of the Focal Loss with different papper-prramterr ? in long-tailed image classification.
CIFARIO
INaturalList2011 Medium
method
Mean/Stdev
All
Many
Few
All(natural)
CE 71.4/0.5
64.3
74.1
65.9
59.8
71.7
FL
71.3/0.6
62.9
72.9
64.1
58.8
70.7
HFL
71,4/0.3
63.6
73.1
64.6
59.7
70.7
(FL((=0.5)
71.8/0.5
64.2
73.3
64.3
61.6
70.8
Simplified EL
71.8/0.9(+0.4)
66.3(+2.0)
75.1 (+1.0)
67.4(+1.6)
62.6(+2.8)
72.6(+1.1)
Table 10: Comparison between the Halted Focal Loss and the Simplified Eureka Loss on long-tailed Cifar-10 (imbalance ratio is 100) and iNaturalist 2018, the mean standard deviation (Stdev) for results on isstu1al is about 0.4.
C DETAILS OF EXPERIMENTAL SETTINGS
C.I DATASETS
There are six datasets used in this paper in total and an overview of the dataset statistics are de- montrated in Table and Figure 7 For image classification tasks, the iNaturalist 2018 dataset is the most imbalanced and has the most classes, which is most satisfactory for evaluating long-tailed classifications. For the language generation task, ConvA12 has an imbalance ratio of 277K, which, however, should be taken cautiously, since most of the tail classes are not covered in evaluation. The investigate
common practice to evaluate the learning on the imbalanced language distributions the diversity of the generated text. The 4-grams can be regarded as high-ordered classes and a 4-gram of four common words can also "rare class".
C.2 TRAINING SETTINGS
CIFAR-10 and CIFAR-100 For experiments on long-tailed CIFAR-I0 and CIFAR-100, the back- bone network is ResNet-32 (He et al. 2016). The model is optimized with SGD with a momentum of 0.9. The learning rate is 0.1 and the model is trained for 200 epochs with 128 examples per mini-batch. To stabilize the training. we adopt the warm-up strategy used by Goyal et al. (2017) in the first epochs. Following and again at the 1 180th epoch. For the results in Figure [ and Figure 3 long-tailed CIFAR-10 with an imbalance ratio of 10.
al. (2019). we decay the learning rate by 0.01 at the 160th epoch
we conduct experiments on
ImageNet Let-L For experiments on ImageNet-LT ILSVRC 2012, the base network is ResNext- 50 (He et al. 2016). The batch size is set to 512 to accelerate training. The initial learning rate 0.2 and we utilize a cosine learning rate scheduler.
iNaturalist 2018 Same as the experiments on ImageNet-L.T, we also follow the default setting in Kang al. (2020) for experiments iNaturalist. To be specific, adopt ResNet-50 model, use SGD with to train the model for 200 epochs with batch size 512 and a cosine learning rate schedule which gradually decays from 0.2 0.0. Results the valid set also reported subset of many
(> 100 samples), medium (20 100 samples), and few (< 20 samples), respectively.
ConvAI2 For the comversation generation task, utilize two-layer LSTM (Hochreiter & Schmid,
huber 1997) encoder-encodee architecture as our base network. The hidden size of both encoder and decoder is set to 1024. We optimize the model with SGD optimizer with momentum 0.9, the
14

---

Under review as a conference paper at ICLR 2021
Difference between the training and classes
103
45.8
33.0
33.4
Cross Entroey
Foss
Cass balanced Lass
Burdka Loss
Method
Figure 6: Illustration of the over-fitting phenomenon on tail classes, the number on the top of each bar is the difference between the training accuracy and the test accuracy.
Datasct
# Classes
Imbalance Ratio
COCO Detection
2
~1000
Long-tailed CIFAR-10
10
10-100
Long-tailed CIFAR- 100
100
10-100
Well Wwwherperyy
ImageNet-LT iNaturalist 2018
1000
256
8142
500
Figure 7: Word frequency distribution of ConvA12 dataset. For natural language generation tasks, each word type can be regarded as a class and most words appear scarcely in the data. If measured the same with long-tailed image classification tasks, the imbalance ratio is 277K.
Table 11: Data statistics of long-tailed image classifica- tion tasks. Imbalance Ratio denotes the ratio of the size of the most common class to that of the rarest class.
batch size is 64 and the learning rate is set 3. The embedding size is 256 and word vectors initialized with GloVe (Pennington et al.) 2014h. We select our final model until the performance on the validation is no longer improving after epochs.
COCO Dectection For experiments on COCO detection, we adopt the configuration of "RetinaNet- R-50-FPN-Ix" from the GitHub repository Detectron2 our default setting. In this setting, the one-stage detector of RetinaNet with the backbone ResNet50 is trained for 90k updates and the batch size is 8 images per batch.
For the image classification tasks, the default (3 is set to 0.9999 for all datasets. For the deferred version, we defer the adoption of Bureka Loss after training for 160 epochs and 180epochs on CIFAR 100 and iNaturalist 2018 respectively. As for the dialog generation task ConvA12, 3 is set to 0.999 and we start the encouragement after regularly training the model for 5 epochs.
We tune the 3 E {0.9, 0.99, 0.999, 0.9999) and y E (0.5.1.2) for the Class- Hlass-Balanced Loss (CB) and the Focal Loss (FL) respectively multi-class classification, and report the best results of these baselines. Following previous work (Cui et al. 2019), a is set to 1.0 for the Focal Loss(FL)), and the Class-Balances Focal Loss (FL+CB) multi-class classification tasks can be viewed as the origin Focal Loss with class-level weight in binary classification tasks.
The training costs are summarized in Table T2
15

---

Under review as a conference paper at ICLR 2021
Data
Infrastructure
Mem/GPU
Time
Epochs
Samples
Model
Long-taiie taigttilldd CIFAR-10
RTX 2080Ti
1.4G
0.3h
200
12.4K
ResNet32
Long-taie mailtailed CIFAR-100
RTX 2080Ti
1.4G
0.3h
200
10.8K
ResNet32
ImageNet-LT iNaturalist 2018
TITAN RTX
17G
8h
90
116K
ResNext50
RTX TITAN
15G
48h
200
438K
ResNet50
ConvA12 COCO Detection
RTX 2080Ti
9G
4h
131K
2-LLSTM
RTX 2080Ti
9G
8h
90K updates
118K
ResNet50
Table 12: Training costs of each task. Samples are dialogues in ConvA12 and images in image classification tasks
16


Reviews:
[{"review_id": "opqKNRDTtVc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n- This paper made a finding that weighting up  correct predictions for rare class examples also can help to improve the performance of imbalanced classification. In light of this finding, it proposes the Eureka Loss to add additional gradients for examples belong to rare classes in the high-likelihood area when correctly predicted. Experiments on several large-scale benchmarks demonstrate its effectiveness.\n\nPros:\n- The paper is clearly written and easy to follow.\n- The experiments are thorough and demonstrate the effectiveness.\n\nCons:\n- While the finding is quite interesting, I think the design of the proposed algorithm is quite arbitrary. It's not clear to me why the authors choose to add a term for rare classes rather than changing the weights directly. Why don't the authors use HFL in the end?\n- Currently it seems that there lacks complementary theory/intuition that could explain why weighting up the already correctly classified rare examples help with the performance.\n\nAdditional Questions:\n- Figure 4 seems quite interesting. It seems that the functionality of Eureka Loss is quite different from HFL. I could intuitively understand that the Eureka loss function would encourage the examples to have likelihood of either 1 or 0. Have the authors visually checked the examples with a likelihood of 0? Does that mean training on a carefully selected subset gives better performance?\n\n----\npost-rebuttal update \n\nI thank the authors for the responses. While I still think the idea is potentially interesting and original, I could not increase the score given the fact that this manuscript is naturally incremental without theoretical justifications.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QRQRZxS-51N", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission makes an intriguing claim that retaining focus on correctly predicted rare classes can improve performance for training with class-imbalanced datasets.\n\nTo illustrate this claim, the paper shows that one can find improvements at overall accuracy if a combination of the Focal Loss (which weights down examples with high predictive likelihood) and the cross-entropy loss is used such that the loss transitions from the Focal Loss to the CE for examples with predictive confidence above a threshold, for examples belonging to the top rarest classes. On long-tailed CIFAR-10, this produces a mild improvement at overall accuracy (around 0.5%) when the top 40% rarest classes receive this mixed loss. Further experiments with COCO-detection finds sparse improvements (around 0.4%) when applying the mixed loss to the tail classes. \n\nBased on the above findings, the paper argues for not weighting down confident predictions, especially if these belong to rare classes. However, perhaps these experiments are insufficient to arrive at such a conclusion? To ensure that the minor improvements in CIFAR-10 are in fact due to the claimed reasoning, one could also look at other combinations of the losses that do not conform to the claim. For example, apply the loss to the top k% most confident examples (without stratifying by rare classes), randomly select k% of images, etc. For COCO-detection, apply HFL to the head classes, and FL to the tail classes. Since improvements are so small, it would also be nice to see some standard deviation bars over multiple trials. Also, were the choices of Focal Loss hyper-parameters made to elicit their best performance? From Figure 2, it looks like it underperforms the cross-entropy loss.\n\nThe paper proposes a new loss meant to \"reward the well-classified rare examples”. This augments the cross-entropy loss with a log(1-p_y) term scaled with a number that reflects the frequency of class y, such that rarer classes are scaled higher. \n\nExperiments have been conducted on 2 image classification datasets and 1 dialogue dataset. In all cases, the proposed loss appears to result in improvements over baselines. \n\nSome questions/comments about the experiments:\n - It appears that the proposed loss performs particularly well when combined with CB. Are the competing methods also similarly augmented? \n - For Table 4, why is the Focal Loss only evaluated for 2 settings of gamma? Shouldn’t there be a hyper-parameter search and the best gamma used?\n - There are a lot of comparisons, with a lot of numbers being taken from past reported results. For all such comparisons, has it been ensured that the architectural and training details are fixed across comparisons? Otherwise the comparisons might not be fair, especially given that reported improvements are minor.\n - Especially when improvements are minor, it becomes important to look at aggregate numbers, so I’d suggest reporting standard deviations over multiple trials for all experiments.\n\nSome typos:\n“down applications” —> “downstream applications”\n“a effective number” —> “an effective number”\n“thus the likelihood” —> “so that the likelihood”\n“deferred courage” —> “deferred encouragement\"\n\nOverall, the paper is clearly written and reports exhaustive experiments (with the caveats/questions above). While the motivating experiments in Section 2.2 are not very compelling, in part due to the very minor improvements, the key intuition that the classification of rare-class hard examples should be continued to be encouraged (so that their predictive confidence doesn’t drop as these examples are weighted down by some of the other methods) sounds interesting, although some of the phrasing about “rewarding well-classified examples” can be a bit awkward. My main concerns as of now are about experimental details, which are described above in the questions.\n\nPost Rebuttal:\nThanks to the authors for responding. I'm still not sure if the experiments are particularly compelling. There appear to be differences amongst the baselines with regards to class balancing, and the motivating section is still weak; there are new experiments on a larger dataset, but now with a different loss (simplified EL) which is close enough to the proposed loss that this does not work very well as a motivation anymore. Apart from this, taking some of the comments from the other reviewers and the authors' responses into account, I am retaining my initial rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BEAS7F3O-4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "These are several concerns:\n1. In the view of motivation, I don't think the motivation is strong enough and is convincing. Also, I don't think rewarding correct predictions but not penalizing incorrect ones is a reasonable way. In my opinion,  rewarding the correct predictions may be a good way, but penalizing the incorrect ones should also be important. \n2. In the view of experiments, though the authors add Table 7 in the appendix, which is the result for training 90 epochs, I still doubt why Eureka Loss does not work better than recent works when training 200 epochs (which is also a common setting recently). And it seems that using CE at the beginning of training is important, and +CB$^+$ works the best.  Moreover, In table 2, the results on \"few\" are especially not very good comparing with others, which makes it harder for me to believe that rewarding the high-likelihood area really matters a lot for tail classes. It seems that the experiment results are not strong enough to support the proposed opinion.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3WcGM2zccmw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with learning imbalanced class distributions.  First, it empirically finds that the high-likelihood area for the rare classes benefits classification. Then, based on the findings, it proposes a new learning objective called Eureka Loss, which can be viewed as a combination of the frequency-based and likelihood-based methods to reward the classifier when examples belong to rare classes in the high-likelihood area are correctly predicted. Empirical results on two typical tasks (i.e. image classification and language generation tasks) illustrate its superiority compared with other baselines. \n\n\n###########################################################################################\npros:\n1. Overall, it is well-written. \n2. It clearly discusses the existing two methods (i.e. frequency-based methods and likelihood-based methods). Furthermore, it highlights the limitation of likelihood-based methods that they neglect the correctly-predicted tail class examples.\n3. The motivation for the design of the new learning objective(i.e., Eureka Loss) is based on the empirical finding that the high-likelihood area of the rare examples is important to improve the performance.\n\n###########################################################################################\ncons:\n1. The finding is mainly on empirical observations, which may lack theoretical support. Why is the high-likelihood area of the rare examples is important for generalization?\n2. For the experimental settings, e.g. iNaturalist 2018, the i.i.d. assumption does not hold for the training and test set.\n3. For the experimental results, how to tune the hyperparameter of the Eureka Loss, in validation set or test set? Since the reason in 2, I guess the hyper-parameter selection becomes difficult.\n\nMinor comments:\nFor the last subfigure in Figure 1, the ordinate value for the loss is negative, which is wrong.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission got 1 reject and 3 marginally below the threshold. The concerns in the original reviews include (1) lack of theoretical justification. The motivation and claim are from empirical observation; (2) the performance improvement is minor compared with the existing methods; (3) some experiment settings and details are not explained clearly. Though the authors provide some additional experiments to the questions about the experiments, reviewers still keep their ratings. The rebuttal did not address their questions. AC has read the paper and all the reviews/discussions. AC has the same recommendation as the reviewers. The major concerns are (1) the theoretical justification is not clear. The additional explanation given by the authors in their rebuttal, i.e., the prediction becomes sharper and thus the model generalization ability can be improved, is not justified. (2) the experiments are not very convincing and can be further improved in the following two aspects: (1) the motivation experiments should be conducted in a consistent manner, instead of using simplified EL in some cases; (2) the effectiveness of EL should be more significant otherwise it is not clear whether the claim is true or not. At the current status of this submission, AC cannot recommend acceptance for the submission.
Decision: Reject