Recall Loss for Imbalanced Image Classification and Semantic Segmentation

1 INTRODUCTION . Dataset imbalance is an important problem for many computer vision tasks such as semantic segmentation and image classification . In semantic segmentation , imbalance occurs as a result of natural occurrence and varying sizes of different classes . For example , in an outdoor driving segmentation dataset , light poles and pedestrians are considered minority classes compared to large classes such as building , sky , and road . These minority classes are often more important than large classes for safety reasons . In image classification , imbalance can occur as a result of data collection . Some classes are more difficult to obtain data for than others . For example , the inaturalist dataset ( Van Horn et al. , 2018 ) has collected images of over 8000 natural species . Since some species are rare , the dataset exhibits the notorious long-tail distribution . When presented with imbalanced datasets , the standard cross entropy loss often yields unsatisfactory results as the training process naturally biases towards large classes resulting in low accuracy and precision on small classes . Researchers have studied the imbalance problem for classification , detection , and segmentation extensively . Most prior research has been on designing balanced loss functions . We classify existing loss functions under three categories : region-based losses , statistics-balanced losses and performance-balanced losses . Region-based losses directly optimize region metrics ( e.g. , Jaccard index ( Rahman & Wang , 2016 ) ) and are mainly popular in medical segmentation applications ; Statistics-balanced losses ( e.g. , LDAM ( Cao et al. , 2019 ) , Class-Balanced ( CB ) loss ( Cui et al. , 2019 ) ) up/down weighs the contribution of a class based on its class margin or class size ; however , they tend to encourage excessive false positives in minority classes to improve mean accuracy especially in segmentation . A recent study in Zhou et al . ( 2020 ) also shows that the weighting undermines the generic representation learning capability of the feature extractors ; Performancebalanced losses ( e.g. , focal loss ( Lin et al. , 2017 ) ) use a certain performance indicator to weigh the loss of each class . As an example , focal loss assumes that the difficulty of a class is correlated with imbalance and can be reflected by the predicted confidence . However , it has not been very successful in other applications for dealing with imbalance as reported by Cui et al . ( 2019 ) . We investigate the reasons of failure in Appendix A.1 . Besides various losses , another thread focuses on training strategies to decouple classifier and representation learning in image classification such as the two-stage ( Kang et al. , 2020 ) and two-branch ( Zhou et al. , 2020 ) approaches . The decoupling approaches have shown state-of-the-art performance compared to other carefully designed losses . As studied by Zhou et al . ( 2020 ) , statistics-balanced losses might even negatively affect representation learning because they always upweigh a minority class and ignores many more examples from the large classes . We propose a novel performance-balanced loss using the recall metric to address the imbalance problem . The recall loss down/up weighs a class based on the training recall performance of that class . It is an example of hard class mining as supposed to the hard example mining strategy in the focal loss . Unlike the statistics-balanced losses , the recall loss dynamically changes its weights with training based on per-class recall performance ( see fig . 1 ( a ) ) . The dynamism is the key to overcome many drawbacks of the statistics-balanced losses . In our experiments , the CB loss improves accuracy at the expense of Intersection over Union ( IOU ) which considers false positives in semantic segmentation . However , our recall loss can effectively balance between precision and recall of each class , and hence , it improves accuracy but maintains a competitive IOU . Experiments on two benchmark semantic segmentation datasets demonstrate that the proposed recall loss shows significantly better performance than state-of-the-art loss functions used in prior works . We also show that while statistics-balanced losses negatively affect representation learning , the recall loss improves representation learning for imbalanced image classification and achieves state-of-the-art results with our simple decoupled network ( fig . 1 ( b ) , ( c ) ) on two common benchmarks . Specifically , we outperform previous state-of-the-art methods on Place-LT by 5.7 % and iNaturalist2018 by 1.1 % . Our main contributions are summarized below . • We introduce a novel loss function based on the metric recall . Recall loss weighs the standard cross entropy loss for each class with its instantaneous training recall performance . • The proposed recall loss learns a better semantic segmentation model that provides improved and balanced performance of accuracy and IOU . We demonstrate the loss on both synthetic and real semantic segmentation datasets . • The proposed loss also improves feature learning in image classification . We show state-of-the-art results on two common classification benchmarks with a simple decoupled network . 2 RELATED WORK . Imbalance in Image Classification . Various losses have been proposed to deal with imbalance or long-tail distributions in image classification . Cost-sensitive loss ( Khan et al. , 2017 ) proposes to iteratively optimize both the model parameters and also a cost-sensitive layer which is integrated into the cost function ( more in Appendix B ) . Lifted Loss ( Oh Song et al. , 2016 ) considers all positive and negative pairs in a mini-batch . Range loss ( Zhang et al. , 2017 ) pushes examples in the same class together while forcing different class centers away from each other . More complicated marginbased approaches , ( Dong et al. , 2018 ; Khan et al. , 2019 ; Hayat et al. , 2019 ) are discussed in the Appendix B. Class-Balanced Loss ( Cui et al. , 2019 ) motivates a weighted cross entropy loss with the concept of effective number of samples in each class . LDAM ( Cao et al. , 2019 ) also derives a weighted cross entropy loss based on margins between classes . However , DecoupleRC ( Kang et al. , 2020 ) pointed out that balanced losses might negatively affect the representation learning process ; hence , classifier learning and representation learning should be separated . OLTR ( Liu et al. , 2019 ) first learns a good representation and uses an attention mechanism to learn a balanced classifier . In the same spirit , DRW ( Cao et al. , 2019 ) uses a two-stage training , and BBN ( Zhou et al. , 2020 ) proposes a two-branch network with a custom training schedule . Both methods emphasize generic representation learning in the beginning and rebalancing the small classes at a later stage . However , both methods require extensive experiments for finding a good learning schedule . Drawing from the same idea , we design a Simple Decoupled Network ( SDN ) that uses two classification heads where one head is responsible for feature extractor training and the other for classifier training . Imbalance in Image Segmentation . In image segmentation , Dice and Jaccard indices ( Intersection over Union ) are commonly used as the evaluation metrics . However , the most common training criterion , cross entropy , does not directly optimize these metrics . In medical imaging , researchers proposed to optimize soft/surrogate version of these indices . SoftIOU ( Rahman & Wang , 2016 ) proposes to optimize a soft version of the Jaccard index ; Lovasz Softmax ( Berman et al. , 2018 ) also optimizes the Jaccard index based on the Lovasz convex extension ; SoftDice ( Sudre et al. , 2017 ) optimizes a soft version of the Dice index and similarly softTversky ( Salehi et al. , 2017 ) optimizes a soft Tversky index . Table 1 in Appendix 3.4 provides an overview of the different indices . However , concerns have been raised in Taghanaki et al . ( 2019 ) on whether these losses consider the tradeoff between false positives and false negatives . We show that they also tend to yield high mean accuracy at the expense of lower mean IOU , whereas our loss improves accuracy while maintaining a competitive IOU . Imbalance in Object Detection . Imbalance is also a problem in object detection where the foreground-background imbalance is extreme and undermines learning . Online Hard Example Mining ( OHEM ) ( Shrivastava et al. , 2016 ) proposes to find hard examples by ranking the losses and only keeping those with the highest losses . Seesaw Loss ( Wang et al. , 2020 ) proposes to dynamically weight the cross entropy loss with cumulative class ratios . Focal Loss ( FL ) ( Lin et al. , 2017 ) chooses to down weigh easy samples and emphasize hard samples by weighting each sample by 1 p where p is the predicted probability for the sample . The weight for each sample dynamically changes with training and the method never completely discards any samples . Focal loss is especially successful because it is easy to implement and proves effective in the binary foreground-background imbalance setting . We compare the proposed method with these losses on image classification and semantic segmentation . 3 RECALL LOSS . 3.1 MOTIVATION : FROM INVERSE FREQUENCY LOSS TO RECALL LOSS . To motivate our proposed loss , we first analyze the standard cross entropy loss . Let { xn , yn } 8n 2 { 1 , ... , N } , where xn 2 Rd , yn 2 { 1 , ... , C } denote the set of training data and corresponding labels . Let Pn denotes the predictive softmax-distribution over all classes for input xn and P in denotes the probability of the i-th class . The cross entropy loss used in multiclass classification is defined as : CE = NX n=1 log ( P ynn ) = CX c=1 X n : yn=c log ( P ynn ) = CX c=1 Nc log ( P c ) ( 1 ) where P c = ( Q n : yn=c P ynn ) 1/Nc denotes the geometric mean confidence of class c and Nc denotes the number of samples in class c. As shown in Eq . 1 , the conventional cross entropy optimizes the geometric mean confidence of each class weighted by the number of pixels in each class . When there is a significant class imbalance in the dataset , the loss function biases towards large classes as a result of larger Nc . One commonly used loss for imbalanced datasets is inverse frequency cross entropy loss ( Eigen & Fergus , 2015 ; Badrinarayanan et al. , 2017 ) which assigns more weight to the loss of minority classes . Let N denote the total number of pixels in the training set and Nc denotes the number of pixels belonging to class c 2 { 1 , .. , C } . The frequency of a class is calculated as freq ( c ) = Nc/N . We show that while the unweighted cross entropy loss optimizes the overall confidence , the loss weighted by inverse frequency optimizes mean confidence . If we use an inverse frequency weighting , the loss is rebalanced . Note we leave out the N in freq ( c ) as it is shared by all classes . InvCE = CX c=1 1 freq ( c ) Nc log ( P c ) = CX c=1 1 Nc Nc log ( P c ) = CX c=1 log ( P c ) ( 2 ) As shown in Eq . 2 , the weighted loss optimizes the geometric mean of accuracy directly . However , the inverse frequency loss might not be optimal in practice because it over-weighs the minority classes and introduces excessive false positives , i.e. , it sacrifices precision for recall . This problem is especially severe in semantic segmentation ( Chan et al. , 2019 ) . Applying the inverse frequency loss to segmentation increases recall for each class . However , the improvement comes at the cost of excessive false positives , especially for small classes . While the inverse frequency loss solves the problem of imbalance , it focuses only on improving one aspect of the problem in classification , i.e . the recall of each class . To solve this issue , we propose to weigh the inverse frequency loss in Eq . 2 with the false negative ( FNc ) counts for each class . The first insight is that the FNc is bounded by the total number of samples in a class and zero , i.e . Nc FNc 0 ( 3 ) By weighting the inverse frequency cross entropy loss in Eq . 2 by the false negative counts for each class , we obtain a moderate loss function which sits between the regular cross entropy loss and inverse frequency loss . We want to note that the idea of finding a middle ground between these two loss functions has been explored in different forms . For example , the BBN ( Zhou et al. , 2020 ) method explicitly uses an adaptor function that controls the contribution of the two losses . However , an obvious drawback is that the adaptor function needs to be extensively searched based on empirical evidence and intuition . RecallCE = CX c=1 FNc log ( P c ) = CX c=1 FNc Nc Nc log ( P c ) = CX c=1 FNc FNc + TPc Nc log ( P c ) ( 4 ) As Eq . 4 shows , the loss can be implemented as the regular cross entropy loss weighted by classwise false negative rate ( FNR ) . The second insight is that minority classes are most likely more difficult to classify with higher FNR and large classes with smaller FNR . Therefore , similar to inverse frequency loss , gradients of minority classes will be boosted and gradients of majority classes will be suppressed . However , unlike frequency weighting , the weighting will not be as extreme as motivated in Eq . 3 . In the next section , we will derive the final dynamic form and compare it to the other performance-balanced loss : the focal loss ( Lin et al. , 2017 ) .

Summary: A novel recall loss (RecallCE) that considers dynamically-changing class recalls is proposed in this paper to mitigate class imbalance in long-tailed recognition problems. The class recalls are estimated using either the current batch statistics or an exponential moving average, depending on the number of class (or class diversity) present in training batches. Relationships between RecallCE and existing widely-used loss functions are mathematically shown. RecallCE performs competitively with existing loss functions on semantic segmentation tasks and outperform them on image classification tasks.