Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents a learning approach to tackle class-imbalance, resulting in a good performance on frequent as well as infrequent classes. The main idea is to use the recall performance of class to adaptively weight it, such that the performance is balanced across the categories in a dataset. The paper is overall presents a strong set of ideas, is clearly presented with several performance improvements. Especially, I believe this work to be significant for segmentation literature where not many papers have shown good improvements in terms of imbalanced learning.

Pros: 

+ The loss can dynamically adapt during the training process based on the recall rates.

+ Exponential moving average is used for a feasible computation of recall with a large number of classes. 

+ The paper is clearly written and the loss formulations seem sound. 

+ Good performance is demonstrated on classification and segmentation tasks. 

+ Some of the drawbacks of statistics-balanced losses can be avoided with the use of proposed loss, e.g., reduction in the false positives compared to CB loss. 

+ Good analysis is provided with other metrics and ablation studies. 

+ The proposed loss is claimed to improve feature learning capability in image classification.  This is demonstrated via a Simple Decoupled Network (SDN)

Cons: 

- The overall approach seems to be heavily inspired by BBN (Zhou et al., CVPR'20), especially the SDN part and the fact that a coupling adaptor function is replaced with the recall measure. This is not necessarily a negative thing in itself, however, it is not clear to me if the benefit for representation learning is due to the decoupled design or the proposed recall loss. Can the authors comment on this? Also, given the similarity with BBN, I expected a more thorough comparison with Zhou et al, they are compared against only in Table 5.  

- The idea to use performance rates (e.g., recall) in class-imbalanced learning is not entirely new. For example, [a] dynamically updates reweighting costs based on the error rates in the confusion matrix (check Eq. 13). It will be appropriate to cite and discuss this paper. 

- Discussing differences with some related loss functions in the imbalanced learning literature will improve Sec. 2. Examples include class rectification loss [b], cost-sensitive loss [c], range loss [d] and [e] affinity loss.

- There is a typo in Table 5, BNN should be BBN.

Refs:

[a] "Cost-sensitive learning of deep feature representations from imbalanced data." IEEE transactions on neural networks and learning systems, 2017. 

[b] "Imbalanced Deep Learning by Minority Class Incremental Rectification." IEEE transactions on pattern analysis and machine intelligence, 2019.

[c] "Striking the Right Balance with Uncertainty", IEEE conference on computer vision and pattern recognition, 2019.

[d] "Range loss for deep face recognition with long-tailed training data." International conference on computer vision, 2017.

[e] “Gaussian Affinity for Max-margin Class Imbalanced Learning,” International conference on computer vision, 2019.
review_title: A good work overall, some details can be improved

final_score: rating: 6: Marginally above acceptance threshold
review_content: A novel recall loss (RecallCE) that considers dynamically-changing class recalls is proposed in this paper to mitigate class imbalance in long-tailed recognition problems. The class recalls are estimated using either the current batch statistics or an exponential moving average, depending on the number of class (or class diversity) present in training batches. Relationships between RecallCE and existing widely-used loss functions are mathematically shown. RecallCE performs competitively with existing loss functions on semantic segmentation tasks and outperform them on image classification tasks.

Paper strengths
- The paper does a good job introducing the proposed loss by gradually progressing from conventional CE to InvCE, and from InvCE to RecallCE. And it also relates RecallCE to Focal loss. This provides a clear picture to readers and allows readers to understand RecallCE more intuitively. The writing is generally good.
- RecallCE requires almost no hyperparameters for tuning. For a training batch that includes all the classes (e.g., Cityscapes), RecallCE has no hyperparameter. Otherwise, RecallCE only requires $\alpha$ for EMA which has been shown to robust against a good range of values in Table 3.
- The experimental results are pretty good, except on Synthia dataset where it does not always outperform other methods.

Paper weaknesses
- The choice of using/not using SDN seems arbitrary and is not motivated by anything. In the experiments, image classification tasks use SDN while semantic segmentation does not seem to be using SDN (there is no explicit statement indicating whether semantic segmentation uses SDN). Semantic segmentation is a pixel wise classification task which is still essentially a classification task that should benefit from SDN.
- There are no indications on how the hyperparameters for other loss functions are tuned (or if they use only default hyperparameters) and how they compare with RecallCE if the hyperparameters are well-tuned.
- One closely-related prior paper is not cited/discussed or compared with. This prior paper introduces Seesaw loss which dynamically adjusts class weights based on accumulated/seen training samples. While this is different from RecallCE, there is some degree of similarity. It was made available on arXiv some time before ICLR deadline.
> - Seesaw Loss for Long-Tailed Instance Segmentation (arXiv. Aug23) https://arxiv.org/abs/2008.10032
- LVIS instance segmentation dataset commonly used for evaluating loss functions designed for long-tailed recognition is not considered in this paper. Due to a large number of classes in LVIS, it is a more challenging and convincing task (since the long-tailed class distribution problem is usually more serious with a large number of classes) than Synthia and Cityscapes semantic segmentation datasets which only have relatively few classes.
- Code is not provided in the submission for reproducibility and there is no promise of code release.

Minor comments
- $L$ in Equation 4 is not consistent with the naming of other loss functions ($CE, InvCE, FocalCE, RecallCE$).
- It would be good to explain what $n:y_n=c$ stands for. Not all readers can understand this.

Overall, this paper is good but the authors need to address some of the weaknesses to make it more convincing.

=== post-rebuttal comments ==

I maintain my pre-rebuttal rating. Although the authors address my concerns reasonably well in the rebuttal, they are mostly not reflected in the paper revision (particularly on the choice of using SDN, which is quite an important piece of information for readers). There were no attempts to make a comparison with Seesaw loss, evaluating on LVIS, and releasing code anonymously (as other ICLR submissions did).
review_title: Novel, intuitive, and clean idea with good results. Not so convincing in certain aspects.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new loss based on the Recall metric to deal with imbalance problems in several visual recognition tasks (i.e., classification and segmentation). Authors show, in several public benchmarks, that the proposed recall loss outperforms other losses in the tasks of classification and segmentation. Please find below my comments.

Strengths.

-The paper is easy to follow.
-Proposing new losses to deal with imbalance problems is an interesting venue.
-Results show that the proposed loss outperforms the other losses compared in the paper.

Weaknesses.

-The methodological contribution is marginal/incremental. The proposed loss is a straightforward extension of prior losses. 

-More fundamentally, I wonder about the motivation of this loss wrt to the use of other metrics as losses. For example, the F1-score of a class is given by the harmonic mean of precision and recall, combining both. Why a recall loss is better than an F1-based loss? For example, the paper in [1] (not included in related work), propose a novel balance function balancing these two terms in the context of highly unbalanced segmentation. Related to this, if model predictions obtain low recall and low precision for a given class, that class is poorly handled by the model, which again motivates the use of something beyond just the recall as objective function.

-There exist missing literature in the losses to deal with imbalance. Particularly, recent works (e.g., [2]) have introduced boundary-based losses for segmentation, which deal with the problems of region-based losses (e.g., large gradients or issues to weight classes bases on their frequency). Authors should discuss these papers, as well as include them in the results.

-I also believe the experiments are poorly conducted. Authors resort to mean-IoU and mean-Acc for segmentation, and accuracy for classification. Nevertheless, F1-score is a popular choice on imbalance problems. Authors should reconsider other metrics to better show the impact of the proposed loss. Furthermore, in both classification and segmentation, the results are averaged over all the classes, when comparing to prior work (not just CE and CBCE). This makes that one does not know which is the impact of the proposed loss in the under-represented classes. Is the improvement due to the model improves the performance over all the classes? Or is it actually coming because the model better handles minority classes? With the current results, it is not possible to evaluate the contribution of this loss in imbalance problems. 

-Related to my previous comment, authors only show the per-class mIOU and Acc on the segmentation task, and comparing to CE and CB-CE (Fig. 2a and 2b). Looking at the Fig 2a, which is the metric typically employed to evaluate the segmentation task on these datasets, we can observe that the proposed loss only improves one class over the standard CE (i.e., bike), falling behind the standard CE in all the rest. These results, together with the values reported in Tables 1 and 2 (where CE outperforms the proposed loss in 1 case, and in the other 3 cases the improvement over CE is marginal), make me wonder about the usability of the proposed loss. 

[1] Hashemi et al. Assymmetric similarity loss function to balance precision and recall in highly unbalanced deep medical image segmentation IEEE 2019.
[2] Kervadec et al. Boundary loss for highly unbalanced segmentation. MIDL 2019.
review_title: Review-Recall loss for imbalanced problems

final_score: rating: 5: Marginally below acceptance threshold
review_content: Strength:    

1- The problem of class imbalance issue studied in this work is important and necessary for general image analysis tasks.  
    
2- On the image classification tasks for Place-LT and iNaturalist2018, the proposed method achieves state-of-the-art performance. This further demonstrates the effectiveness of the proposed method. 
  
3- On the image segmentation tasks, the proposed method achieves the best mean accuracies and competitive mean IoU in comparison with several typical loss functions.  
   
4- The overall paper is clearly presented and easy to follow. The motivation introduced in Section 3 provides the readers with a detailed illustration of the importance of the method. The clearly described motivations induce the readers to better and more clearly understand the overall work. 
    
    
Weakness:    
  
1- The first concern of the proposed method is unstable performance. For the image segmentation tasks, the overall performance of the proposed method is not stable and robust. For example, the method sometimes achieves worse mIoU in comparison with Focal loss (as shown in Table 1-b and Table 2-a). Although the proposed method always achieves the best mACC metric, the less competitive performance under mIoU score still limits the overall performance of the proposed method. 
 
2- The second concern is the lack of comparison with the state-of-the-art methods on the semantic segmentation tasks. Although the proposed method is demonstrated to be effective compare with recent loss functions, it is not convinced without a competitive performance compared with recent state-of-the-art semantic segmentation methods.  
   
  
3- There is a lack of visualization comparison results among each method. As a paper based on image analysis, the visualization comparison experiment is also important and necessary. This could be easily addressed but should be included nonetheless.  

review_title: In this manuscript, the authors proposed a recall loss to solve the class-imbalance issue in the image classification and segmentation tasks. The recall loss assigns weights for each class in the traditional cross-entropy loss based on the training performance under the recall metric. 

Meta-review: This paper introduces the recall loss for dealing with imbalance training contexts. The authors propose to perform a class-wise weighting of examples based on the instantaneous recall performance during training. 

The reviewers like the clarity of the presentation, but raise several concerns regarding novelty of the approach, comparison to more baseline loss functions and state-of-the-art methods.

The AC carefully reads the paper and discussions. The AC appreciates the discussion with respect to competitive loss functions, e.g. focal loss or segmentation loss approximations (SoftDice or Lovasz), which clearly highlights some limitations in existing approaches. \
However, the AC  considers that the approach is essentially a new way to setup the compromise between precision and recall. In that respect, the claims in the paper and in discussion regarding the performances of the proposed method are often exaggerated. For example in segmentation, the method obtaining the best accuracy is CB-CE in 3 out of 4 experiments (on Syntia the mIOU improvement of the recall loss compared to CB-CE corresponds to about the same drop in mACC). This is also verified on Fig 2 where CB-CE outperforms the proposed method by a large margin on mean accuracy for small classes. For classification, the performance gains' compared to SDN (CE) are small. \
The AC thus considers that the paper in the current form falls short of the ICLR acceptance threshold. 
Decision: Reject