Recall Loss for Imbalanced Image Classification and Semantic Segmentation
Authors: Junjiao Tian
Niluthpol Chowdhury Mithun
Zachary Seymour
Han-pang Chiu
Zsolt Kira
Venue:  (2021)
Keywords: Data Imbalance
Classification
Semantic Segmentation
Deep Learning

Abstract: Class imbalance is a fundamental problem in computer vision applications such as semantic segmentation and image classification. Specifically, uneven class distributions in a training dataset often result in unsatisfactory performance on under-represented classes. Many works have proposed to weigh the standard cross entropy loss function with pre-computed weights based on class statistics such as the number of samples and class margins. There are two major drawbacks to these methods: 1) constantly up-weighing minority classes can introduce excessive false positives especially in semantic segmentation; 2) many recent works discovered that pre-computed weights have adversarial effects on representation learning. In this regard, we propose a hard-class mining loss by reshaping the vanilla cross entropy loss such that it weights the loss for each class dynamically based on changing recall performance. We show mathematically that the novel recall loss changes gradually between the standard cross entropy loss and the well-known inverse frequency cross entropy loss and balances precision and accuracy. We first demonstrate that the proposed loss effectively balances precision and accuracy on semantic segmentation datasets, and leads to significant performance improvement over other state-of-the-art loss functions used in semantic segmentation, especially on shallow networks. On image classification, we design a simple two-head training strategy to show that the novel loss function improves representation learning on imbalanced datasets. We outperform the previously best performing method by $5.7\%$ on Place365-LT and by $1.1\%$ on iNaturalist.

Under review as a conference paper at ICLR 2021
RECALL LOSS FOR IMBALANCED IMAGE CLASSIFICA- TION AND SEMANTIC SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Class imbalance is a fundamental problem in computer vision applications such as semantic segmentation and image classification. Specifically, uneven class distri- butions in training dataset often result in unsatisfactory performance on under- represented classes. Many works have proposed to weigh the standard cross en- tropy loss function with pre-computed weights based on class statisties such as the number of samples and class margins. There are two major drawbacks to these methods: I) constantly up up-weighing minority classes can introduce exces- sive false positives especially in semantic segmentation; 2) many recent works discovered that pre-computed weights have adversarial effects on representation learning. In this regard, we propose hard-class mining loss by reshaping the vanilla cross entropy loss such that it weights the loss for each class dynami- cally based on changing recall performance. We show mathematically that the novel recall loss changes gradually between the standard cross entropy loss and the well-known inverse frequency cross entropy loss and balances precision and accuracy. We first demonstrate that the proposed loss effectively balances preci- sion and accuracy on semantic segmentation datasets, and leads significant per- formance improvement over other state-of of-the-art loss functions used in semantic segmentation, especially on shallow networks. On image classification, we design simple two-head training strategy to show that the novel loss function improves representation learning on imbalanced datasets. We outperform the previously best performing method by 5.7% on Place365-LT and by 1.1% iNaturalist.
INTRODUCTION
Dataset imbalance is an important problem for many computer vision tasks such as semantic seg- mentation and image classification. semantic segmentation, imbalance occurs as a result of natural occurrence and varying sizes of different classes. For example, in an outdoor driving segmentation dataset, light poles and pedestrians are considered minority classes compared to large classes such as building, sky, and road. These minority classes are often more important than large classes for safety reasons. In image classification, imbalance can occur as result of data collection. Some classes are more difficult obtain data for than others. For example, the inaturalist dataset (Van Horn al.) 2018) has collected images of over 8000 natural species. Since some species are rare, the dataset exhibits the notorious long-tail distribution. When presented with imbalanced datasets, the standard cross entropy loss often yields unsatisfactory results as the training process naturally biases towards large classes resulting in low accuracy and precision on small classes.
Researchers have studied the imbalance problem for classification, detection, and segmentation ex- tensively. Most prior research has been on designing balanced loss functions. We classify ex- isting loss functions under three categories: region-based losses, carristiss-allnncee losses and performance-balance losses. Region-based losses directly optimize region metries (e.g.. Jac- card index (Rahman & Wang 2016)) and are mainly popular in medical segmentation applications; Saatittcs-balancee losses (e.g.. LDAM (Cao et al. 2019),
mass-Balanced (CB) loss (Cui et al.) 2019) up/down weighs the contribution of class based on its class margin or class size; how- ever, they tend to encourage excessive false positives in minority classes to improve mean accuracy especially in segmentation. A recent study in Zhou et al. (020) also shows that the weighting undermines the generic representation learning capability of the feature extractors; Performance- balanced losses (e.g., focal loss (Lin et al. 2017)) certain performance indicator to weigh

---

Under review as a conference paper at ICLR 2021
Training
Inference
CE Loss
Weight
Recoll OF Lows
Her teration
admwkiigg
speanectt)
boe inn
hall
petsaniei PN
Her Werr 10
Foature Learning Classifier Learning
Single Branch Inference (c)
(b)
Figure 1: (a) We show normalized recall weights over iterations for recall loss. The horizontal axis arranged in descending order in terms of pixel percentage. % indicates class pixel percentage. The recall weights change dynamically according to the performance metric recall. (b) We design a Sim- ple Decoupled Network (SDN) to decouple representation and classifier learning. (c) In inference, only one branch is used in SDN.
the loss of each class. As example, focal loss assumes that the difficulty of class is correlated with imbalance and can reflected by the predicted confidence. However, it has not been very successful in other applications for dealing with imbalance reported by Cui et al. (219). We investigate the reasons of failure in Appendix A. Besides various losses, another thread focuses on training strategies decouple classifier and representation learning in image classification such as the two-stage (Kang et al. 2020) and two-branch (hou et al. 2020) approaches. The decoupling approaches have shown state-of-the-art performance compared to other carefully designed losses. As studied by Zhou et al. (2020),
losses might even negatively affect representa- tion learning because they alwayss upweigh minority class and ignores many more examples from the large classes.
We propose a novel iefformanne-ballnneee loss using the recall metric to address the imbalance problem. The recall loss down/up weighs class based on the training recall performance of that class. an example of hard class mining as supposed to the hard example mining strategy in the focal loss. Unlike the statistics-balance losses, the recall loss dynamically changes its weights with training based on per-class recall performance (see fig. Ia)). The dynamism the key to overcome many drawbacks of the racy at the expense of Intersection over Union (IOU) which considers false positives semantic segmentation. However, recall loss effectively balance between precision and recall of each class, and hence, improves accuracy but maintains a competitive IOU. Experiments on two bench- mark semantic segmentation datasets demonstrate that the proposed recall loss shows significantly better performance than state-of-the-ee loss functions used in prior works. We also show that while statistics-balaneed losses negatively affect representation learning, the recall loss improves repre- sentation learning for imbalanced image classification and achieves state-of ofate--o-thh-atee art results with our simple decoupled network (fig. (c) on two common benchmarks. Specifically, we outperform previous state-of-the-an methods Place-LT by 5.7% and iNaturalist2018 by 1.1%.
losses. In our experiments, the CB loss improves accu-
Our main contributions are summarized below.
We introduce novel loss function based the metric recall. Recall loss weighs the standard cross entropy loss for each class with its instantancous training recall performance.
The proposed recall loss learns a better semantic segmentation model that provides improved and balanced performance of accuracy and IOU. We demonstrate the loss on both synthetic and real semantic segmentation datasets.
The proposed loss also improves feature learning in image classification. We show state-of-the-art results on two common classification benchmarks with a simple decoupled network.
2 RELATED WORK
Imbalance in Image Classification. Various losses have been proposed to deal with imbalance or long-tail distributions in image classification. Cost-sensitive loss (Khan al. 2017) proposes to
2

---

Under review as a conference paper at ICLR 2021
iteratively optimize both the model parameters and also cost- sest-sen layer which is integrated into the cost function (more in Appendix B Lifted Loss (Oh Song et al. 2016) considers all positive and negative pairs in mini-batch. Range loss (Zhagg et al. 2017) 7) pushes examples in the same class together while forcing different class centers away from each other. More complicated margin- based approaches, (Dong et al., 2018 Khan et al. 2019 Hayat al., 2019) discussed in the Appendix B Class-Balanced Loss (Cui et al.] 2019) motivates a weighted cross entropy loss with the concept of effective number of samples in each class. LDAM (Cao et al., 2019) also derives weighted cross entropy loss based on margins between classes. However, DecoupleRC (Kang et al., 2020) pointed out that balanced losses might negatively affect the representation learning process; hence, classifier learning and representation learning should be separated. OLTR (Liu et al. 2019) first learns good representation and uses an attention mechanism to learn balanced classifier. In the same spirit, DRW (Cao et al. 2019) uses two-stage training, and BBN Zhou et al. 2020) proposes a two-branch network with a custom training schedule. Both methods emphasize generic representation learning in the beginning and rebalancing the small classes at later stage. However, both methods require extensive experiments for finding a good learning schedule. Drawing from the same idea, we design Simple Decoupled Network (SDN) that one head is responsible for feature extractor training and the other for classifier training.
two classification heads where
Imbalance in Image Segmentation. In image segmentation, Dice and Jaccard indices (Intersection over Union) are commonly used as the evaluation metrics. However, the most common training criterion, cross entropy, does not directly optimize these metrics. In medical imaging, researchers proposed to optimize soft/surrogate version of these indices. SoftIOU (Rahman & Wang 2016) proposes to optimize a soft version of the Jaccard index; Lovasz Softmax (Berman et al. 2018) also optimizes the Jaccard index based on the Lovasz convex extension; SoftDice (Sudre et al.) (017)) optimizes soft version of the Dice index and similarly softsversk (Salehi et al. 2017) optimizes soft Tversky index. Table in Appendix 3.4 provides an overview of the different indices. However, concerns have been raised in Taghanaki et al. (019)) on whether these losses consider the trade- off between false positives and false negatives. We show that they also tend to yield high mean accuracy at the expense of lower mean IOU, whereas our loss improves accuracy while maintaining competitive IOU.
Imbalance in Object Detection. Imbalance is also a problem in object detection where the imbalance is extreme and undermines learning. Online Hard Example Min- ing (OHEM) (Shrivastava et al., 2016) proposes to find hard examples by ranking the losses and only keeping those with the highest losses. Seesaw Loss (Wang et al. 2020 proposes dynami- cally weight the cross entropy loss with cumulative class ratios. Focal Loss (FL.) (Lin et al. 2017)
chooses down weigh easy samples and emphasize hard samples by weighting each sample by 1-p where p is the predicted probability for the sample. The weight for each sample dynamically changes with training and the method never completely discards any samples. Focal loss is especially suc- cessful because it is easy to implement and proves effective in the binary foreground backkuuud imbalance setting. We compare the proposed method with these losses on image classification and semantic segmentation.
3 RECALL LOSS
3.1 MOTIVATION: FROM INVERSE FREQUENCY LOSS TO RECALL LOSS
To motivate our proposed loss, we first analyze the standard cross entropy loss. Let {In, Un Vnn E {1....,NY N). where n, E Rd yn E {1...., C'} denote the set of training data and corresponding labels. Let P. denotes the predictive probability of the i-th class. The cross entropy loss used in multiclass classification is defined as:
over all classes for input v, and P denotes the
CE Σ log(P!"")
Σ
Σ 12
log(P
Σ
No log(Pc)
where pc_ = (II ney. PH!. I/N2 denotes the geometric mean confidence of class e and N₂ denotes the number of samples in class c. As shown in Eq. [ the conventional cross entropy optimizes the geometric mean confidence of each class weighted by the number of pixels in each class. When there is a significant class imbalance in the dataset, the loss function biases towards large classes as result of larger Ne.
3

---

Under review as a conference paper at ICLR 2021
One commonly used loss for imbalanced datasets is inverse frequency cross entropy loss (Eigen
& Fergus 2015 Badrinarayanan et al., 2017) which assigns more weight to the loss of minority classes. Let N denote the total number of pixels in the training set and N, denotes the number of pixels belonging to class e E {1,-. C). The frequency of a class is calculated as freg(c) = Ne/N. We show that while the unweighted cross entropy loss optimizes the overall confidence, the loss weighted by inverse frequency optimizes mean confidence. If we use an inverse frequency weighting, the loss rebalanced. Note we leave out the N in freq(e) is shared by all classes.
InwCE = E freq(e) freq(e)
Er No log log
1
Σ
C
No No log((") No
1
log(P
As shown in Eq the weighted loss optimizes the geometric mean of accuracy directly. However, the inverse frequency loss might not be optimal in practice because it over-weighs the minority classes and introduces excessive false positives, i.e., it sacrifices precision for recall. This problem is especially severe in semantic segmentation (Chan et al. 2019). Applying the inverse frequency loss to segmentation increases recall for each class. However, the improvement comes at the cost of excessive false positives, especially for small classes.
While the inverse frequency loss solves the problem of imbalance, it focuses only on improving one aspect of the problem in classification. i.e. the recall of each class. To solve this issue, propose to weigh the inverse frequency loss in Eq. 2 with the false negative (F Ne) counts for each class. The first insight is that the FN, bounded by the total number of samples in class and zero, i.e.
N₂ FN, 2 0
By weighting the inverse frequency cross entropy loss in Eq. 2 by the false negative counts for each class, we obtain moderate loss function which sits between the regular cross entropy loss and inverse frequency loss. We want to note that the idea of finding middle ground between these two loss functions has been explored in different forms. For example, the BEN (Zhou et al.] 2020) method explicitly uses an adaptor function that controls the contribution of the two losses. However, an obvious drawback is that the adaptor function needs to be extensively searched based on empirical evidence and intuition.
RecallCEE = Σ PN, log FNe Ne
FN FN. + TP. No Nolog( Po))
N. log((")==
Σ
(4)
As Eq. 4 shows, the loss can be implemented the regular cross entropy loss weighted by class- wise false negative rate (FNR). The second insight is that minority classes are most likely more difficult to classify with higher FNR and large classes with smaller FNR. Therefore, similar to inverse frequency loss, gradients of minority classes will be boosted and gradients of majority classes will be suppressed. However, unlike frequency weighting. the weighting will not be as extreme as motivated in Eq. 3 In the next section, we will derive the final dynamic form and compare it to the other performance-balancee loss: the focal loss (Lin et al. 2017
3.2 MOTIVATION: FROM FOCAL LOSS TO RECALL LOSS
The previous section proposed to weigh cross entropy with the false negative rate of each class. Unlike frequency and decision margin (Cao et al., 2019) which are characteristics of the dataset, FNR is metric of a model's 's performance. As continue to update the model's parameters, FNR changes. Therefore, the weights for each class change dynamically to reflect a model's instantaneous performance. We rewrite Eq.
4 and introduce subscript to denote the time dependency.
C
TPG.t RecallCE= - D( FNo, + TPP.
N logg((p
Σ
Σ
(1- (1- Re,)) log(Pn,)) (5)
c=1 m.y.
where Re,s is the recall for class at optimization step t. n yi = c denotes all samples such that the ground truth label y; is class
The other cittormmnnc-ballaneee loss is focal loss (Ln et al. 2017 It is developed originally for mackground-foregroumr imbalance in object detection. The loss function weighs the cross entropy
4

---

Under review as a conference paper at ICLR 2021
(S.. Pc)
Precision(Ge," Pc)
DioeiGe Pc)
Jacrard(Ge, Pc)
Pl Pa)
ThershylGe Pc) (8.
Set Rep.
S.O.P
16.0%
9+
P4
Boolean Rep.
Table 1: Region Metrics: We show both set representation and Boolean representation. TP, FN and FP stand for true positive, false negative and false positive respectively. The subscript e means that the metrics are calculated for each class.
loss each sample by 1-p where i the predicted probability/confidencc Intuitively, hard samples will have low confidence and therefore high weight. It can be thought of as hard-example mining loss. To see recall loss's 'ss' resemblance to focal loss, we need to need to rewrite it slightly.
N E - -1-pp log = Σ
FocalC
Σ
(n-P),)) ( log Pr.e
(6)
n=1
c=1 vargi
where Pig is predicted probability of class yh for sample n at time t: 7 is scalar hyperparameter.
Focal loss has been very popular loss function for imbalance in detection. is appealing because dynamically adjusts the weight for each sample depending on the difficulty of the sample and model performance. However, the focal loss is not specifically effective against imbalanced classification problems as reflected by poor performance reported by many papers (Cao et al. 2019 Cui et al. 2019). By examining the similarity between Eq. 5 and Eq. 5 we argue that the proposed recall loss function can be seen class-wise focal loss with = I and the per-class metric R.., replacing
per-sample probability P.., The next section will discuss how to estimate the recall for each class.
3.3 RECALL ESTIMATION
The recall loss is designed to reflect the instantaneous training performance of model on the cur- rent input data. A straightforward way is to estimate the recall based on the current batch statistics, ie., count false positives for each class from an entire batch. This method provides reliable esti- mation of the model's current performance if there is a sufficient number of samples for each class in the batch. Intuitively for classification, batch recall is good estimation if the number of classes not much larger than the batch size. For semantic segmentation, batch recall is almost always reliable since each image contains hundreds of pixels for each class. For subsequent segmentation experiments, we use the batch recall loss where the batch recall is calculated as follows:
TPc.
Re, TP..t + FNe.
For classification, estimating recall is problematic for a large number of classes. For example, the iNaturalist2018 dataset has 8, 1.42 classes. For batch size of 128, is difficult sample sufficient data for any class. To mitigate the problem, we use the Exponential Moving Average (EMA) to estimate the recall and calculate the EMA recall loss.
Rott aRe, + (1 Ro,R,,111
(8)
3.4 ANALYSIS: RECALL, PRECISION, DICE, JACCARD AND TVESKY
Why do we not use other metrics such as Fl, Dice, Jaccard and Tvesky as the weights? Following previous convention, let G. and Pe denote the set of ground truth (positive) samples and predicted samples for class c. Let FP.. TN denote the set of false positive and true negative samples respec- tively for class c, and other terms are defined similarly. Recall is different from the other metrics in that it does not have false positive in the denominator and this distinction makes it ideal for weight- ing entropy loss (and others not) as shown in table T Referring back to Eq. where recall loss is defined as weighted cross entropy by 1 - R., replacing recall by any other metrics above would result in FP appearing in the numerator of the weights. For example, a hypothetical precision loss can be defined as following. PrecisionLoss
FPo PPa. TP
Σ
Σ
log(pa.)
no =C
5

---

Under review as a conference paper at ICLR 2021
This formulation will introduce unexpected behavior. A large false positive count in class will result in a large weight, which further encourages false detection for that class. This will cause the number of false positives to explode. From a different perspective, because in cross entropy loss always penalize the ground truth samples i E Ge (i yi = c} for a class c, a proper weighting should be proportional to F 2.2 Ge but not P& & ge which does not belong the set of ground truth samples. The same analysis can be applied to other metrics involving false positives.
3.5 RECALL LOSS AS A FEATURE LEARNING LOSS FOR IMBALANCED CLASSIFICATION
Recent works (Kang et al., 2020 on the imbalanced classification problem proposed to separate representation learning and classifier learning. Intuitively, the final classifier layer is negatively affected by highly imbalanced data distributions while the convolutional neural network backbone is not as affected. In other words, representation learning benefits from all the data regardless of their class membership. It has been shown in Kang et al. (020)) that weighted losses do not produce large improvement because they can negatively affect representation learning. While we need to be careful when introducing weighted losses to train the feature extractor, some previous works (Cao et al. 2019 Zhou al. 2020) showed that it can still be beneficial to carefully "fine-tune" CNNs with balancing techniques towards the end of a training cycle when the learning rate has been annealed. We propose use recall loss a feature learning loss to replace the standard cross entropy. We experimentally show that recall loss is better suited loss for representation learning because it considers imbalance in datasets while dynamically adjusting the weights to not bias towards any class. To apply recall loss to classification, we introduce Simple Decoupled Network (SDN) to decouple representation and classifier learning (fig. (b).)).....
Let fo denote the feature extractor and {go, hof denote two classifier heads on top of the feature extractor. Generally speaking, So is parameterized by a CNN and hop are two separate fully connected networks. Similar to previous works (Kang et al. 2020). (Zhou et al., 2020), we design a simple decoupled network with two classification heads and shared feature backbone as shown in fig. (b). More specifically, the loss from the head go backpropagates to the feature extractor fe while the loss from the head hg is stopped. The go head is trained with recall loss and the he head is trained with the CB loss (Cui et al. 2019). In other words, recall loss trains the feature extractor while the CB loss does not. We only use the ha head in inference. Therefore, this simple modification during training does not introduce any additional change to inference. The proposed method simplifies BBN (Zhou et al.) (020)) in two ways. 1) Only one loss function affects the backbone. Therefore, there is no need for hand-tuning an adaptor function for controlling two losses. 2) We only use one head for inference and discard the other. This simple design proves to be effective in our experiments.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Datasets. We evaluate our recall loss on two popular large-scale outdoor semantic segmentation datasets, Synthia Ros et al. 2016 and Cityscapes (Cordis al, 2016). Synthia photorealistic synthetic dataset with different seasons, weather, and lighting conditions. Specifically, we use the Synthia-sequence Summer split for training (4400), validation (624), and testing (1272). Cityscapes consists of real photos of urban street scenes in several cities in Europe. It has 5000 annotated images for training and another 5000 for evaluation. We further show that recall loss is beneficial for feature learning in image classification on two large-scale long-tailed datasets including Place- LT (Liu et al. 2019) and (Naturvspacealist201 (Van Horn et al. 2018p. Place-I.T has 365 classes and long-tailed class distribution. It is created by sampling from the original balanced dataset (Zhou) ctt al., 2017) following a Pareto distribution. iNaturalist2018 is a long-tailed image collection of natural species of 8142 classes. Please refer to Appendix A.2 for details implementation.
Evaluation Metrics. We report mean accuracy and mean I0U for semantic segmentation exper- iments and overall accuracy for image classification following previous works (Cao et al. 2019 Zhou et al. 2020) Liu et al. 201%; Kang et al.) 2020) on these datasts. We note that both mean accuracy and mean IOU important metrics for semantic segmentation. While good mean IOU

---

Under review as a conference paper at ICLR 2021
ICE CS-CE Recal
CE CE IN Recell
100
roud
Wegtt
(a)
(b)
(c)
Figure 2: (a) Mean IOU per class on Synthia dataset with Resnet18. (b) Mean Accuracy per class on Synthia dataset with Resnet18. (c) Normalized Re weights over time for recall loss. The weight for the bike class decreased over time indicating improved performance.
Ground
Cross-Entropy
Recall
JH
Figure 3: Visualization of Segmentation results on Synthia with Resenet18. Recall loss encourages models to predict more small classes such as poles and pedestrians. Compared to the cross-entropy trained model, the recall loss trained model is able to output finer details especially for small classes.
indicates balanced performance of precision and recall, mean accuracy is an indicator of the de- tection rate of each class, which important for safety-critical applications such as self-driving.
4.2 SEMANTIC SEGMENTATION EXPERIMENTS
For semantic segmentation experiments, we compare our method with two region-based losses, SoftIOU (Rahman & Wang 2016 and Lovasz softamax (Berman et al.) 2018). Both of these losses aim to minimize soft/surogate version of the metric IOU. As analyzed by Eclbode et al. 2020)) the softDice loss (Sudre al. 2017) and softTversky loss (Salchi et al. 2017) are similar to the two chosen losses. It is representative to compare to two of them. We also compare with state-of- the-art losses for imbalanced image classification and detection such as CB-CE (Cui et al.. 2019 Focal loss (Lin et al., 2017) and Online Hard Example Mining (OHEM) loss (Shrivastava et al.) 2016 We keep top 70% samples in OHEM. While there are other techniques for imbalanced image classification, they require changes to learning schedules and architectures. A direct adaptation of them for semantic segmentation is not trivial. We compare with them directly on image classification datasets.
CE
Soh1OU
Locase
OHEM
CB-CE
Focal
Recall
CE
SufIOU
Lowisz
OHEM
CB-CE
Focal
Recall
mIOu mACC
90.78
$7.81
84.78
91.03
85.34
95.78
(a) DeepLab Resnet18 performance Synthia
(b) DeepLab Resnet101 performance Synthia
Table 2: We compare the recall loss with other losses on the Synthia dataset using two different backbone networks. Recall loss improves accuracy significantly while offers competitive IOU.
7

---

Under review as a conference paper at ICLR 2021
CB
SatIOU
Lovasz
OHEM
CB-CE
Focal
Recall
CE
SontOU
Lonasz
OHEM
CB-CE
Focal
Recall
mlOU
60.66
60. 19
55,70
50.00
36.37
62.11
60.99
mIOU
70.12
65.19
19.83
56,69
58.59
67.11
70.51
mACC
72.24
73.51
80.65
86,42
(a) DeepLab Resnet18 performance on Cityscapes (b) DeepLab Resnet101 performance on Cityscapes
Table 3: We compare recall loss with other losses on the Cityscapes dataset with real images using different backbones. Recall loss yields the best accuracy and maintains a competitive IOU, This demonstrates its robustness to label noise in real datasets.
0.1
0.3
0.5
0.7
0.9
0.001
0.01
0.1
0.2
0.3
Resnet152
40.01
39.43
40.01
39,56
39.81
Resnet50 67.03
66.70
66.57
66.20
66.01
(a) Accuracy with different Place-LT
(b) Accuracy with different on Naturaaist20188.
Table 4: To study sensitivity of EMA on different datasets, we experimented with different a values and report the validation accuracy. We use a 0.9 and 0.001 for Place365 and iNaturatist0018 respectively in subsequent comparison with other methods.
Synthia. We first present results on the synthetic Synthia segmentation dataset in table 2 Soft- IOU, Lovasz, CB-CE all improved the mean aceuracy compared to the baseline cross entropy loss. However, the improvement comes the standard cross entropy loss. This is consistent with our statement that hard-example weighting in focal loss is not effective against multi-class imbalance. OHEM performs worse on both met- rics. We think this is because OHEM completely discards 30% training samples at each iteration and this negatively affects feature learning. On the other hand, the proposed recall loss improves the mean accuracy metric and maintains good mean IOU. This validates our analysis that the re- call loss can balance between precision and recall. The same trend is observed for both shallow backbone, resnet18, 18, and deep backbone 101.et101. However, we note that the effectiveness of the recall loss is more obvious on less powerful feature backbone. We hypothesize that less powerful backbones are more likely to spend its limited representation capability on large class and thus bias towards them. This is important because in many applications, hardware limits the deployment of computation-heavy backbones and we need to attend to small classes for safety relying on shal- low feature extractors. In fig. - we show per-class accuracy and IOU performance on three losses including the recall loss. We observe that both CB-CE and recall loss improve accuracy on small classes significantly. However, CB-CE deteriorates IOU for those classes significantly while recall loss maintains competitive performance because CB-CE uses fixed weighting which always emphasizes small classes. This observation supports our claim that recall loss balances recall and precision because of its dynamic adaptability performance. In the fig. e).. show the 1 Re, weights on resenet18 8 Synthia experiment. We observe that the weight for small classes such as bike decreases over time. This indicates that the performance of the bike class has increased. We further provide a visual comparison between a model trained with the ross-entropy loss and the proposed recall loss in fig. Our method provides more fine details on small classes which are often suppressed in traditional cross entropy training.
cost of lower mean IOU. Focal loss performs similarly to
Cityscapes. We further present results on the Cityscapes segmentation dataset with real images. As shown in B softIOU, and focal loss perform similarly to the standard cross entropy loss while Lovasz, OHEM, and CB-CE are consistently worse on both of the resnet backbones. Compared to performance Synthia, this shows that some of the losses dataset. Recall loss again outperforms other losses by improving mean accuracy and maintaining good mean IOU. In other words, recall loss improves the detection rate of small classes such as pedestrians, light poles on the road and maintains a good precision. This demonstrates the effective- ness of recall loss both synthetic and real datasets outdoor driving segmentation datasets.
not robust to label noise in real
4.3 IMAGE CLASSIFICATION EXPERIMENTS
For classification experiments, we compare on two popular imbalanced datasets, Place-LT [Lu et al. 2019) and iNaturalist2018 (Van Hom et al. 2018 We compare to the methods that have achieved State-of-the-arn performance on either dataset. We specifically introduce a baseline model, SDN-CE, which replaces recall loss with standard cross entropy loss to train the feature backbone.
8

---

Under review as a conference paper at ICLR 2021
Lifted Lossi
Focal Loss
Range Loss f OLTR
tau-norm
LWSE
(CB)
SDN (recall)
All
30.6
35,20
34.60
35.10
35.9
37.9
37.6
39.3
39.8
Many 46.7
41.1
41.1
41.1
44.7
37.8
40.6
43.0
43.3
Medium
30.1
35.4
34.8
35.4
37.0
40.7
39.1
40.3
41.0
Few
11.9
24.0
22.4
23.2
25.3
31.8
28.6
31.1
31.2
Table 5: Test Accuracy on Place-LT. i denotes results from Liu et al. (2019) and ğ denotes results from Kang et al. (2020). We follow previous works to report performance on many (more than 100 images), medium (20 to 100 images) and few-shot (fewer than 20 images). Our method yields overall best performance. We highlight the Best and Second Best baseline method.
CB
Focal
Focal-DRW
LDAM
LDAM-DRW
BBN
tau-normi
I.Wt
SDN (CF)
DN (Recall)
Resnet50
62.00
61.67
65.62
60.38
64.58
66.29
65.60
65.90
66.13
67.03
Table 6: Validation Accuracy on Naturallist0188 i denotes results from Zhou et al. (020 and p
denotes results from Kang et al. (2020).
Place-LT. Following previous works (Kang et al. 2020). Liu et al.
2019p, compare to the Lifted
loss (Oh Song et al. 2016). Focal loss (Lin et al. 2017). ,. Range Loss
Zhang et al.] 2017). OLTR (Liu 2020). We note that Kang et al. experimented with many variants and tau-norm is the best performing one on this dataset.
al.
2019), tau-norm (Kang et al. 2020) and LWS (Kang al.
(2020)
shows that the recall loss outperforms other loss including SDN-CE. The result is two- First, the strong performance of the baseline, SDN-CE agrees with the finding in Kang et al. 2020 that imbalance affects the classifier more than the backbone and a simple decoupling trick can outperform carefully designed losses. Second, the result validates our claim that the recall loss, SDN-recall, is a more suitable feature loss for imbalanced datasets when comparing to SDN-CE. Note that we use the EMA version of the recall loss. The table 4(a) shows the results of SDN-recall with different a on a validation set. We can conclude that the sensitivity of a is low on the Place-LT dataset. Specifically, the ratio of the number of classes to batch size is 365: 128 in this case.
Table
fold.
iNaturalist2018. On iNaturalist we compare to LDAM-DRW (Cao et al. 2019 BBN Zhou et al. 2020). tau-norm (Kang et al. 2020) and LWS (Kng et al. 2020 This dataset is the most chal- lenging due to its extremely large number of classes. This presents a specific challenge to recall loss since the effectiveness of recall loss depends on reliable estimation of the training recall for each class. Consequently, it motivated us to propose the exponential-movin average update rules. Table shows the results of SDN-recall, SDN-CE, and all compared methods. SDN-recall out- performs all other methods including SDN-CE. It is worth mentioning that both LDAM-DRW and BBN proposed finetune the feature extractor with a balanced loss using two-stage and two- branch strategy respectively. Recall loss trains feature backbone in an end-to-end manner and outperforms other techniques that require careful hyperparameter tuning or modification to the ar- chitectures. Table [(b shows the sensitivity of a on this dataset. As the number of classes (8,) is much larger than the batch size (128), a small a is critical to provide a reliable recall estimation. We also present the training curves with different in the Appendix A.3 We observe that smaller a yields lower training loss.
5 CONCLUSION
In this paper, we proposed a novel loss function based on the metric recall. The loss function uses hard-class mining strategy to improve model performance on imbalanced datasets. Specifically, the recall loss weighs examples in class based on its instantancous recall performance during train- ing, and the weights change dynamically to reflect relative change in performance among classes. Experimentally, we demonstrated several advantages of the loss: 1) Recall loss improves accuracy while maintains a competitive IOU performance in semantic segmentation. Most notably, recall loss improves both accuracy and precision significantly in small networks, which possesses limited representation power and is more prone to biased performance due to data imbalance. 2) Recall loss works on both synthetic and real data and is robust to label noise present in real datasets. 3) The EMA version of recall loss is able to handle an extremely large numbers of classes and provides a stable improvement on representation learning. 4) Recall loss facilitates representation learning in image classification. Using simple decoupled training strategy and the recall loss, we outperform more complicated methods on common imbalance learning benchmarks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12):2481-2495) 2017.
Maxim Berman, Amal Rannen Triki, and Matthew Blaschko. The lovász-softman loss: tractable surrogate for the optimization of the intersection-over-miion measure in neural networks. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4413-4421, 2018.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with bobol-distributio-awwaaaa margin loss. In Advances in Neural Information Processing Systems, pp.
2019.
Robin Chan, Matthias Rottmann, Fabian Hüger, Peter Schlicht, and Hanno Gottschalk. Applica- tion of decision rules for handling class imbalance in semantic segmentation. arXiv preprint urXiv:/901.088 2019.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 0tft::/706.055888 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268-9277 2019.
Qi Dong, Shaogang Gong, and Xiatian Zhu. Imbalanced deep learning by minority class incremental rectification. IEEE transactions on pattern analysis and machine intelligence, 41(6))1337-1381, 2018.
Tom Eelbode, Jeroen Bertels, Maxim Berman, Dirk Vandermeulen, Frederik Maes, Raf Bisschops, and Matthew B Blaschko. Optimization for medical image segmentation: Theory and practice when evaluating with dice score or jaccard index. IEEE Transactions on Medical Imaging, 2020.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE international conference on computer vision, pp. 2650-2658 2015.
Munawar Hayat, Salman Khan, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Gaussian affinity for max-margin class imbalanced learning. In Proceedings of the IEEE International Conference Computer Vision, pp. 6469-6479, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217. 2020.
Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 103-112, 2019.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions neural networks and learning systems, 29(8):3573-358) 2017.
10

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition an open world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Md Atiqur Rahman and Yang Wang. Optimizing intersection-over-niie in deep neural networks for image segmentation. In International symposium on visual computing, pp. 234-244. Springer, 2016.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3234- 3243, 2016.
Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour. Tversky loss function for image segmentation using 3d fully convolutional deep networks. In International Workshop on Machine Learning Medical Imaging, pp. 379-387. Springer, 2017.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 761-769, 2016.
Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised dice overlap deep learning loss function for highly unbalanced segmentations. In Deep learning in medical image analysis and multimodal learning for clinical decision support, pp- 240-248. Springer, 2017.
Saeid Asgari Taghanaki, Yefeng Zheng. S Kevin Zhou, Bogdan Georgescu, Puneet Sharma, Daguang Xu, Dorin Comaniciu, and Ghassan Hamarneh. Combo loss: Handling input and out- put imbalance in multi-organ segmentation. Computerized Medical Imaging and Graphics, 75: 24-33, 2019.
Grant Van Horn, Oisin Mae Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8778. 2018.
Jaqqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Scesaw loss for long-tailes instance segmentation. arXiv preprint 2008 10032, 2020.
Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. Proceedings of the IEEE International Conference Computer Vision, pp. 5400-54188 2017.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on patlern analysis and machine intelligence, 4066)):452-1464 2017.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral bilateral-branch network with cumulative learning for long-tailed visual recognition. CVPR, 2020.
11


Reviews:
[{"review_id": "ofhy7aK1q2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new loss based on the Recall metric to deal with imbalance problems in several visual recognition tasks (i.e., classification and segmentation). Authors show, in several public benchmarks, that the proposed recall loss outperforms other losses in the tasks of classification and segmentation. Please find below my comments.\n\nStrengths.\n\n-The paper is easy to follow.\n-Proposing new losses to deal with imbalance problems is an interesting venue.\n-Results show that the proposed loss outperforms the other losses compared in the paper.\n\nWeaknesses.\n\n-The methodological contribution is marginal/incremental. The proposed loss is a straightforward extension of prior losses. \n\n-More fundamentally, I wonder about the motivation of this loss wrt to the use of other metrics as losses. For example, the F1-score of a class is given by the harmonic mean of precision and recall, combining both. Why a recall loss is better than an F1-based loss? For example, the paper in [1] (not included in related work), propose a novel balance function balancing these two terms in the context of highly unbalanced segmentation. Related to this, if model predictions obtain low recall and low precision for a given class, that class is poorly handled by the model, which again motivates the use of something beyond just the recall as objective function.\n\n-There exist missing literature in the losses to deal with imbalance. Particularly, recent works (e.g., [2]) have introduced boundary-based losses for segmentation, which deal with the problems of region-based losses (e.g., large gradients or issues to weight classes bases on their frequency). Authors should discuss these papers, as well as include them in the results.\n\n-I also believe the experiments are poorly conducted. Authors resort to mean-IoU and mean-Acc for segmentation, and accuracy for classification. Nevertheless, F1-score is a popular choice on imbalance problems. Authors should reconsider other metrics to better show the impact of the proposed loss. Furthermore, in both classification and segmentation, the results are averaged over all the classes, when comparing to prior work (not just CE and CBCE). This makes that one does not know which is the impact of the proposed loss in the under-represented classes. Is the improvement due to the model improves the performance over all the classes? Or is it actually coming because the model better handles minority classes? With the current results, it is not possible to evaluate the contribution of this loss in imbalance problems. \n\n-Related to my previous comment, authors only show the per-class mIOU and Acc on the segmentation task, and comparing to CE and CB-CE (Fig. 2a and 2b). Looking at the Fig 2a, which is the metric typically employed to evaluate the segmentation task on these datasets, we can observe that the proposed loss only improves one class over the standard CE (i.e., bike), falling behind the standard CE in all the rest. These results, together with the values reported in Tables 1 and 2 (where CE outperforms the proposed loss in 1 case, and in the other 3 cases the improvement over CE is marginal), make me wonder about the usability of the proposed loss. \n\n[1] Hashemi et al. Assymmetric similarity loss function to balance precision and recall in highly unbalanced deep medical image segmentation IEEE 2019.\n[2] Kervadec et al. Boundary loss for highly unbalanced segmentation. MIDL 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ekrYNNYzxQd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:    \n\n1- The problem of class imbalance issue studied in this work is important and necessary for general image analysis tasks.  \n    \n2- On the image classification tasks for Place-LT and iNaturalist2018, the proposed method achieves state-of-the-art performance. This further demonstrates the effectiveness of the proposed method. \n  \n3- On the image segmentation tasks, the proposed method achieves the best mean accuracies and competitive mean IoU in comparison with several typical loss functions.  \n   \n4- The overall paper is clearly presented and easy to follow. The motivation introduced in Section 3 provides the readers with a detailed illustration of the importance of the method. The clearly described motivations induce the readers to better and more clearly understand the overall work. \n    \n    \nWeakness:    \n  \n1- The first concern of the proposed method is unstable performance. For the image segmentation tasks, the overall performance of the proposed method is not stable and robust. For example, the method sometimes achieves worse mIoU in comparison with Focal loss (as shown in Table 1-b and Table 2-a). Although the proposed method always achieves the best mACC metric, the less competitive performance under mIoU score still limits the overall performance of the proposed method. \n \n2- The second concern is the lack of comparison with the state-of-the-art methods on the semantic segmentation tasks. Although the proposed method is demonstrated to be effective compare with recent loss functions, it is not convinced without a competitive performance compared with recent state-of-the-art semantic segmentation methods.  \n   \n  \n3- There is a lack of visualization comparison results among each method. As a paper based on image analysis, the visualization comparison experiment is also important and necessary. This could be easily addressed but should be included nonetheless.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NU2rOj6T8uy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A novel recall loss (RecallCE) that considers dynamically-changing class recalls is proposed in this paper to mitigate class imbalance in long-tailed recognition problems. The class recalls are estimated using either the current batch statistics or an exponential moving average, depending on the number of class (or class diversity) present in training batches. Relationships between RecallCE and existing widely-used loss functions are mathematically shown. RecallCE performs competitively with existing loss functions on semantic segmentation tasks and outperform them on image classification tasks.\n\nPaper strengths\n- The paper does a good job introducing the proposed loss by gradually progressing from conventional CE to InvCE, and from InvCE to RecallCE. And it also relates RecallCE to Focal loss. This provides a clear picture to readers and allows readers to understand RecallCE more intuitively. The writing is generally good.\n- RecallCE requires almost no hyperparameters for tuning. For a training batch that includes all the classes (e.g., Cityscapes), RecallCE has no hyperparameter. Otherwise, RecallCE only requires $\\alpha$ for EMA which has been shown to robust against a good range of values in Table 3.\n- The experimental results are pretty good, except on Synthia dataset where it does not always outperform other methods.\n\nPaper weaknesses\n- The choice of using/not using SDN seems arbitrary and is not motivated by anything. In the experiments, image classification tasks use SDN while semantic segmentation does not seem to be using SDN (there is no explicit statement indicating whether semantic segmentation uses SDN). Semantic segmentation is a pixel wise classification task which is still essentially a classification task that should benefit from SDN.\n- There are no indications on how the hyperparameters for other loss functions are tuned (or if they use only default hyperparameters) and how they compare with RecallCE if the hyperparameters are well-tuned.\n- One closely-related prior paper is not cited/discussed or compared with. This prior paper introduces Seesaw loss which dynamically adjusts class weights based on accumulated/seen training samples. While this is different from RecallCE, there is some degree of similarity. It was made available on arXiv some time before ICLR deadline.\n> - Seesaw Loss for Long-Tailed Instance Segmentation (arXiv. Aug23) https://arxiv.org/abs/2008.10032\n- LVIS instance segmentation dataset commonly used for evaluating loss functions designed for long-tailed recognition is not considered in this paper. Due to a large number of classes in LVIS, it is a more challenging and convincing task (since the long-tailed class distribution problem is usually more serious with a large number of classes) than Synthia and Cityscapes semantic segmentation datasets which only have relatively few classes.\n- Code is not provided in the submission for reproducibility and there is no promise of code release.\n\nMinor comments\n- $L$ in Equation 4 is not consistent with the naming of other loss functions ($CE, InvCE, FocalCE, RecallCE$).\n- It would be good to explain what $n:y_n=c$ stands for. Not all readers can understand this.\n\nOverall, this paper is good but the authors need to address some of the weaknesses to make it more convincing.\n\n=== post-rebuttal comments ==\n\nI maintain my pre-rebuttal rating. Although the authors address my concerns reasonably well in the rebuttal, they are mostly not reflected in the paper revision (particularly on the choice of using SDN, which is quite an important piece of information for readers). There were no attempts to make a comparison with Seesaw loss, evaluating on LVIS, and releasing code anonymously (as other ICLR submissions did).", "extra_scores": {}, "extra_text": {}}, {"review_id": "K5RMxHsyLb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a learning approach to tackle class-imbalance, resulting in a good performance on frequent as well as infrequent classes. The main idea is to use the recall performance of class to adaptively weight it, such that the performance is balanced across the categories in a dataset. The paper is overall presents a strong set of ideas, is clearly presented with several performance improvements. Especially, I believe this work to be significant for segmentation literature where not many papers have shown good improvements in terms of imbalanced learning.\n\nPros: \n\n+ The loss can dynamically adapt during the training process based on the recall rates.\n\n+ Exponential moving average is used for a feasible computation of recall with a large number of classes. \n\n+ The paper is clearly written and the loss formulations seem sound. \n\n+ Good performance is demonstrated on classification and segmentation tasks. \n\n+ Some of the drawbacks of statistics-balanced losses can be avoided with the use of proposed loss, e.g., reduction in the false positives compared to CB loss. \n\n+ Good analysis is provided with other metrics and ablation studies. \n\n+ The proposed loss is claimed to improve feature learning capability in image classification.  This is demonstrated via a Simple Decoupled Network (SDN)\n\nCons: \n\n- The overall approach seems to be heavily inspired by BBN (Zhou et al., CVPR'20), especially the SDN part and the fact that a coupling adaptor function is replaced with the recall measure. This is not necessarily a negative thing in itself, however, it is not clear to me if the benefit for representation learning is due to the decoupled design or the proposed recall loss. Can the authors comment on this? Also, given the similarity with BBN, I expected a more thorough comparison with Zhou et al, they are compared against only in Table 5.  \n\n- The idea to use performance rates (e.g., recall) in class-imbalanced learning is not entirely new. For example, [a] dynamically updates reweighting costs based on the error rates in the confusion matrix (check Eq. 13). It will be appropriate to cite and discuss this paper. \n\n- Discussing differences with some related loss functions in the imbalanced learning literature will improve Sec. 2. Examples include class rectification loss [b], cost-sensitive loss [c], range loss [d] and [e] affinity loss.\n\n- There is a typo in Table 5, BNN should be BBN.\n\nRefs:\n\n[a] \"Cost-sensitive learning of deep feature representations from imbalanced data.\" IEEE transactions on neural networks and learning systems, 2017. \n\n[b] \"Imbalanced Deep Learning by Minority Class Incremental Rectification.\" IEEE transactions on pattern analysis and machine intelligence, 2019.\n\n[c] \"Striking the Right Balance with Uncertainty\", IEEE conference on computer vision and pattern recognition, 2019.\n\n[d] \"Range loss for deep face recognition with long-tailed training data.\" International conference on computer vision, 2017.\n\n[e] “Gaussian Affinity for Max-margin Class Imbalanced Learning,” International conference on computer vision, 2019.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the recall loss for dealing with imbalance training contexts. The authors propose to perform a class-wise weighting of examples based on the instantaneous recall performance during training. 

The reviewers like the clarity of the presentation, but raise several concerns regarding novelty of the approach, comparison to more baseline loss functions and state-of-the-art methods.

The AC carefully reads the paper and discussions. The AC appreciates the discussion with respect to competitive loss functions, e.g. focal loss or segmentation loss approximations (SoftDice or Lovasz), which clearly highlights some limitations in existing approaches. \
However, the AC  considers that the approach is essentially a new way to setup the compromise between precision and recall. In that respect, the claims in the paper and in discussion regarding the performances of the proposed method are often exaggerated. For example in segmentation, the method obtaining the best accuracy is CB-CE in 3 out of 4 experiments (on Syntia the mIOU improvement of the recall loss compared to CB-CE corresponds to about the same drop in mACC). This is also verified on Fig 2 where CB-CE outperforms the proposed method by a large margin on mean accuracy for small classes. For classification, the performance gains' compared to SDN (CE) are small. \
The AC thus considers that the paper in the current form falls short of the ICLR acceptance threshold. 
Decision: Reject