High-Likelihood Area Matters --- Rewarding Correct,Rare Predictions Under Imbalanced Distributions

Learning from natural datasets poses significant challenges for traditional classification methods based on the cross-entropy objective due to imbalanced class distributions . It is intuitive to assume that the examples from rare classes are harder to learn so that the classifier is uncertain of the prediction , which establishes the low-likelihood area . Based on this , existing approaches drive the classifier actively to correctly predict those incorrect , rare examples . However , this assumption is one-sided and could be misleading . We find in practice that the high-likelihood area contains correct predictions for rare class examples and it plays a vital role in learning imbalanced class distributions . In light of this finding , we propose the Eureka Loss , which rewards the classifier when examples belong to rare classes in the high-likelihood area are correctly predicted . Experiments on the large-scale long-tailed iNaturalist 2018 classification dataset and the ImageNet-LT benchmark both validate the proposed approach . We further analyze the influence of the Eureka Loss in detail on diverse data distributions . 1 INTRODUCTION . Existing classification methods usually struggle in real-world applications , where the class distributions are inherently imbalanced and long-tailed ( Van Horn & Perona , 2017 ; Buda et al. , 2018 ; Liu et al. , 2019 ; Gupta et al. , 2019 ) , in which a few head classes occupy a large probability mass while most tail ( or rare ) classes only possess a few examples . The language generation task is a vivid example of the long-tailed classification . In this case , word types are considered as the classes and the model predicts probabilities over the vocabulary . Common words such as the , of , and and are the head classes , while tailed classes are rare words like Gobbledygook , Scrumptious , and Agastopia . Conventional classifiers based on deep neural networks require a large number of training examples to generalize and have been found to under-perform on rare classes with a few training examples in downstream applications ( Van Horn & Perona , 2017 ; Buda et al. , 2018 ; Cao et al. , 2019 ) . It is proposed that the traditional cross-entropy objective is unsuitable for learning imbalanced distributions since it treats each instance and each class equivalently ( Lin et al. , 2017 ; Tan et al. , 2020 ) . In contrast , the instances from tail classes should be paid more attention , indicated by two main approaches that have been recently investigated for class-imbalanced classification : the frequencybased methods and the likelihood-based methods . The former ( Cui et al. , 2019 ; Cao et al. , 2019 ) directly adjust the weights of the instances in terms of their class frequencies , so that the instances from the tail classes are learned with a higher priority no matter whether they are correctly predicted or not . The latter ( Lin et al. , 2017 ; Zhu et al. , 2018 ) instead penalize the inaccurate predictions more heavily , assuming that the well-classified instances , i.e. , the instances in the high-likelihood area , factor inconsequentially in learning imbalanced distributions . However , neither of these two approaches realistically depicts the likelihood landscape . In particular , the high-likelihood area , where the classifier makes the correct predictions for both common class examples and rare class ones , contributes significantly to generalization . However , this area is not well-shaped , as illustrated in Figure 1 . Specifically , the frequency-based methods imply an impaired learning of common class examples that are the principle part of the natural data , while the likelihood- based methods ignore the correctly-predicted rare class examples that can provide crucial insights into the underlying mechanism for predicting such examples . In this paper , we first demonstrate that existing practice of neglecting predictions in the high-likelihood area is harmful to learning imbalanced class distributions . Furthermore , we find that simply mixing the cross-entropy loss and the Focal Loss ( Lin et al. , 2017 ) can induce substantially superior performance , which validates our motivation . In turn , we propose to elevate the importance of high-likelihood predictions even further and design a novel objective called Eureka Loss . It progressively rewards the classifiers according to both the likelihood and the class frequency of an example such that the system is encouraged to be more confident in the correct prediction of examples from rare classes . Experimental results on the image classification and the language generation tasks demonstrate that the Eureka Loss outperforms strong baselines in learning imbalanced class distributions . Our contributions are twofold : • We challenge the common belief that learning for examples in low-likelihood area is more important for learning tail classes and reveal that the correctly-predicted rare class examples make important contribution to learning long-tailed class distributions . • We explore a new direction for learning imbalanced classification that focuses on rewarding correct predictions for tail classes examples , rather than penalizing incorrect ones . The proposed Eureka Loss rewards the classifier for its high-likelihood predictions progressively to the rarity of their class and achieves substantial improvements on various problems with long-tailed distributions . 2 RELATED WORK . Frequency-based Data and Loss Re-balancing Previous literature on learning with long-tailed distribution mainly focusing on re-balancing the data distribution and re-weighting the loss function . The former is based on a straightforward idea to manually create a pseudo-balanced data distribution to ease the learning problem , including up-sampling for rare class examples ( Chawla et al. , 2002 ) , down-sampling for head class examples ( Drummond & Holte , 2003 ) and a more concrete sampling strategy based on class frequency ( Shen et al. , 2016 ) . As for the latter , recent studies propose to assign different weights to different classes , and the weights can be calculated according to the class distribution . For example , Khan et al . ( 2018 ) design a cost-sensitive loss for major and minor class examples . An intuitive method is to down-weight the loss of frequent classes , while up-weight the contribution of rare class examples . However , frequency is not suitable to be directly treated as the the weight since there exists overlap among samples . An advancing alternative loss CB ( Cui et al. , 2019 ) proposes to calculate the effective number to substitute the frequency for loss re-weighting . However , since it assigns lower weight to head classes in the maximum likelihood training ( Cross Entropy objective ) , it seriously impairs the learning of head classes . Moreover , CB requires a delicate hyper-parameter tuning for every imbalanced distribution , leading to a lot of manul efforts . From the perspective of max-margin , a recent study LDAM ( Cao et al. , 2019 ) proposes to up-weight the loss of tail classes by a class-distribution based margin . Compared to the above methods , we choose to decrease the loss of tail classes by rewarding correct predictions rather than increasing the loss of tail classes through aggravated penalization . Deferring the Frequency-based Class-balanced Training Recent studies find that deferring the class-balanced training helps learn high-quality representations ( Liu et al. , 2019 ) , and propose deferred Class-balanced training ( deferred CB ) ( Cao et al. , 2019 ) , which chooses to adopt Cross Entropy objective at the beginning of training . Similarly , the Decoupling method ( Kang et al. , 2020 ) shows that the re-balancing strategies impair the quality of learned feature representations and demonstrate an improved performance learned with original data distribution , by training the model with Cross Entropy in the first phase and adopting class-balanced training in the second phase . This decoupling strategy can also be found in BBN ( Zhou et al. , 2019 ) , which includes both class imbalanced and balanced training , and the transition from the former to the latter is achieved through a curriculum learning schedule . These methods achieve state-of-the-art performance in long-tailed classification . To be comparable with these methods and to analyse whether Eureka Loss is complementary to this technique , we propose deferred Eureka Loss , in which rewarding for rare class prediction is introduced to encourage the model to learn rare patterns when learning is stalled . Likelihood-based Loss Another dominant method for imbalanced classification is the likelihoodbased method Focal Loss ( FL ) ( Lin et al. , 2017 ) , which proposes to down-weight the contribution of examples in the high-likelihood area . However , we argue that it is harmful for learning tail classes and choose an opposite direction by highlighting the high-likelihood area with a steeper loss . Transferring Representations Techniques for transferring information from sufficient head classes examples to under-represented rare classes examples belong to a parallel successful direction in this field . They include MBJ ( Liu et al. , 2020 ) , which utilizes external semantic feature memory and FSA ( Chu et al. , 2020 ) , which decomposes feature in to class-specific and class-generic components . These latest transfer learning based studies are less related to our paper but they also obtain good improvements in long-tailed classification , so we add them into comparison in the experiments . 3 ROLE OF THE HIGH-LIKELIHOOD AREA . The existing approaches to the long-tailed classification independently consider the class frequency and the example likelihood . However , we show that this one-sided reflection is problematic when dealing with the tail class examples that can be confidently classified . The tail class examples can be easily classified by the classifier , and the head class examples can also be hard for the classifier to recognize . The difficulty of classification depends on the inherent characteristic of the classes , rather than the sample size of the class . For example , in species classification , the Portuguese man o ’ war may be a rare class but can be easily classified due to its distinct features , compared to various kinds of moths which are common classes yet are hard to distinguish . However , the frequency-based methods continuously drive the classifier to fit the rare class examples , especially when they are difficult to predict , which may lead to overfitting . On the other hand , the likelihood-based methods relax the concentration in the high-likelihood area , which contains the tail class examples that are not hard to predict and provide insights to generalization . To verify our point of view , we analyze the problem by dissecting the influence of the high-likelihood area with respect to the class frequency and demonstrate that properly encouraging the learning of well-classified tail class examples induce substantial improvements , before which we first give a brief introduction of classification with long-tailed class distributions . 3.1 PREPARATION : CLASSIFICATION WITH LONG-TAILED CLASS DISTRIBUTIONS . Let ’ s consider the multi-class classification problem with the long-tailed class distribution . Given a class set C , n denotes the number of different classes in C and mi is the number of examples of the class Ci . For simplicity , we sort the class set C according to cardinal mi for Ci such that C0 is the class with the most examples and Cn−1 is the rarest class . Let p be a n-dim probability vector predicted by a classifier model f ( x ; θ ) based on the input x , where each element pi denotes the probability of the class Ci and y is a n-dim one-hot label vector with y being the ground-truth class . The probability vector can be calculated as p = σ ( f ( x ; θ ) ) , ( 1 ) where σ is the normalizing function , e.g. , softmax for multi-class classification . Typically , the parameters are estimated using maximum likelihood estimation ( MLE ) , which is equivalent to using the Cross-Entropy Loss ( CE ) function , where the scalar y · log p can be regarded as the ( log ) -likelihood : L = −E ( x , y ) ∈D log pmodel ( y|x ) = − 1 |D| ∑ ( x , y ) ∈D y · log p. ( 2 ) For deep neural network–based classifiers , due to the non-linearity of the loss function , the problem is typically solved by stochastic gradient descent , which requires the calculation of the gradient with respect to the parameters using the chain-rule , the process of which is called back-propagation : ∂L ∂θ = ∂L ∂p ∂p ∂θ = ∂L ∂p ∂σ ( f ( x ; θ ) ) ∂θ . ( 3 ) We introduce the term likelihood gradient to denote ∂L/∂p , which modulates how the probability mass should be shifted and is a characteristic of the loss function instead of the classifier . For learning imbalanced class distributions , the common methods aim to shape the likelihood gradient so the rare classes are learned with priority , i.e. , embodying a sharper loss and a larger likelihood gradient . Frequency-Based Methods Frequency-based methods alter the likelihood gradient according to the class frequencies , which are irrelevant to how well individual examples are classified . A simple form is using a n-dim weight vector w composed of the class weights based on their frequencies in the dataset to determine the importance of examples from each class : L = −wy · ( y · log p ) . ( 4 ) Note that whenw = 1 , it is identical to the cross-entropy objective . The weight vector is typically calculated as wi = m̄/mi , where m̄ is the average of mi . As we can see , the standard weight is taken as the average of the class size , so that the classes with more examples are down-weighted and the classes with fewer examples are up-weighted . For a natural long-tailed distribution , the average is larger than the median , which suggests more classes are up-weighted . Advanced frequency-based methods try to obtain a more meaningful measurement of the class size , e.g. , the Class-Balanced Loss ( CB ) proposed by Cui et al . ( 2019 ) utilizes an effective number 1−βmi/1−β for each class , where β ∈ [ 0.9 , 1 ) is a tunable class-balanced term . Likelihood-Based Methods Different from the frequency-based methods , likelihood-based methods adjust the likelihood gradient based on the instance-level difficulty as predicted by the classifier such that the examples in the low-likelihood area are more focused in training . For example , the well-known Focal Loss ( FL ) with a balanced factor α proposed by Lin et al . ( 2017 ) takes the following form : L = −α ( 1− py ) γf · ( y· log p ) , ( 5 ) where γf > 0 , which controls the convexness of the loss and higher γf indicates adjustment that are more significant . Note that when α = 1 and γf = 0 , it is identical to the cross-entropy objective . Following previous works ( Cao et al. , 2019 ; Liu et al. , 2019 ; Cui et al. , 2019 ) , the α is set to 1 in multi-class classification , and Class-Balanced Focal Loss ( FL+CB ) can be viewed Focal Loss with uneven alpha for each class in the multi-class setting . The key idea is to pay less attention to the well-classified examples and pay more attention to the badly-classified examples , because it is natural to assume the tail class examples are harder to learn and thus can not be well-classified . However , such methods neglect the correctly-predicted tail class examples , the practice of which we show is not constructive to the learning of long-tailed class distributions .

Summary: This paper deals with learning imbalanced class distributions.  First, it empirically finds that the high-likelihood area for the rare classes benefits classification. Then, based on the findings, it proposes a new learning objective called Eureka Loss, which can be viewed as a combination of the frequency-based and likelihood-based methods to reward the classifier when examples belong to rare classes in the high-likelihood area are correctly predicted. Empirical results on two typical tasks (i.e. image classification and language generation tasks) illustrate its superiority compared with other baselines.