On Model Calibration for Long-Tailed Object Detection and Instance Segmentation

1 Introduction . Object detection and instance segmentation are the fundamental tasks in computer vision and have been approached from various perspectives over the past few decades [ 8 , 13 , 26 , 36 , 52 ] . With the recent advances in neural networks [ 1 , 4 , 10 , 18 , 29 , 30 , 32 , 37 , 40 , 41 , 43 ] , we have witnessed an unprecedented breakthrough in detecting and segmenting frequently seen objects such as people , cars , and TVs [ 14 , 15 , 21 , 31 , 70 ] . Yet , when it comes to detect rare , less commonly seen objects ( e.g. , walruses , pitchforks , seaplanes , etc . ) [ 12 , 51 ] , there is a drastic performance drop largely due to insufficient training samples [ 47 , 68 ] . How to overcome the “ long-tailed ” distribution of different object classes [ 69 ] has therefore attracted increasing attention lately [ 27 , 35 , 46 , 55 ] . To date , most existing works tackle this problem in the model training phase , e.g. , by developing algorithms , objectives , or model architectures to tackle the long-tailed distribution [ 12 , 20 , 27 , 46 , 48 , 53 , 55 , 57 , 58 ] . Wang et al . [ 55 ] investigated the widely used instance segmentation model Mask R-CNN [ 18 ] and found that the performance drop comes primarily from mis-classification of object proposals . Concretely , the model tends to give frequent classes higher confidence scores [ 7 ] , hence biasing the label assignment towards frequent classes . This observation suggests that techniques of class-imbalanced learning [ 2 , 6 , 16 , 42 ] can be applied to long-tailed detection and segmentation . Building upon the aforementioned observation , we take another route in the model inference phase by explicit post-processing calibration [ 2 , 22 , 23 , 33 , 61 ] , which adjusts a classifier ’ s confidence scores among classes , without changing its internal weights or architectures . Post-processing calibration is efficient and widely applicable since it requires no re-training of the classifier . Its effectiveness ∗Equal contributions 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . on multiple imbalanced classification benchmarks [ 22 , 59 ] may also translate to long-tailed object detection and instance segmentation . In this paper , we propose a simple post-processing calibration technique inspired by class-imbalanced learning [ 33 , 61 ] and show that it can significantly improve a pre-trained object detector ’ s performance on detecting both rare and common classes of objects . We note that our results are in sharp contrast to a couple of previous attempts on exploring post-processing calibration in object detection [ 7 , 27 ] , which reported poor performance and/or sensitivity to hyper-parameter tuning . We also note that the calibration techniques in [ 54 , 55 ] are implemented in the training phase and are not post-processing . Concretely , we apply post-processing calibration to the classification sub-network of a pre-trained object detector . Taking Faster R-CNN [ 43 ] and Mask R-CNN [ 18 ] for examples , they apply to each object proposal a ( C + 1 ) -way softmax classifier , where C is the number of foreground classes , and 1 is the background class . To prevent the scores from being biased toward frequent classes [ 7 , 55 ] , we re-scale the logit of every class according to its class size , e.g. , number of training images . Importantly , we leave the logit of the background class intact because ( a ) the background class has a drastically different meaning from object classes and ( b ) its value does not affect the ranking among different foreground classes . After adjusting the logits , we then re-compute the confidence scores ( with normalization across all classes , including the background ) to decide the label assignment for each object proposal2 ( see Figure 1 ) . We note that it is crucial to normalize the scores across all classes since it triggers re-ranking of the detection results within each class ( see Figure 3 ) , influencing the class-wise precision and recall . Instead of separately adjusting each class by a specific factor [ 7 ] , we follow [ 6 , 33 , 61 ] to set the factor as a function of the class size , leaving only one hyper-parameter to tune . We find that it is robust to use the training set to set this hyper-parameter , making our approach applicable to scenarios where collecting a held-out representative validation set is challenging . Our approach , named Normalized Calibration for long-tailed object detection and instance segmentation ( NORCAL ) , is model-agnostic as long as the detector has a softmax classifier or multiple binary sigmoid classifiers for the objects and the background . We validate NORCAL on the LVIS [ 12 ] dataset for both long-tailed object detection and instance segmentation . NORCAL can consistently improve not only baseline models ( e.g. , Faster R-CNN [ 43 ] or Mask R-CNN [ 18 ] ) but also many models that are dedicated to the long-tailed distribution . Hence , our best results notably advance the state of the art . Moreover , NORCAL can improve both the standard average precision ( AP ) and the category-independent APFixed metric [ 7 ] , implying that NORCAL does not trade frequent class predictions for rare classes but rather improve the proposal ranking within each class . Indeed , through a detailed analysis , we show that NORCAL can in general improve both the precision and recall for each class , making it appealing to almost any existing evaluation metrics . Overall , we view NORCAL a simple plug-and-play component to improve object detectors ’ performance during inference . 2 Related Work . Long-tailed detection and segmentation . Existing works on long-tailed object detection can roughly be categorized into re-sampling , cost-sensitive learning , and data augmentation . Re-sampling 2Popular evaluation protocols allow multiple labels per proposal if their confidence scores are high enough . methods change the long-tailed training distribution into a more balanced one by sampling data from rare classes more often [ 3 , 12 , 45 ] . Cost-sensitive learning aims at adjusting the loss of data instances according to their labels [ 19 , 46 , 48 , 53 ] . Building upon these , some methods perform twoor multi-staged training [ 20 , 22 , 27 , 42 , 55–57 , 67 ] , which first pre-train the models in a conventional way , using data from all or just the head classes ; the models are then fine-tuned on the entire long-tailed data using either re-sampling or cost-sensitive learning . Besides , another thread of works leverages data augmentation for the object instances of the tail classes to improve long-tailed object detection [ 9 , 39 , 65 , 66 ] . In contrast to all these previous works , we investigate post-processing calibration [ 22 , 23 , 33 , 50 , 61 ] to adjust the learned model in the testing phase , without modifying the training phase or modeling . Concretely , these methods adjust the predicted confident scores ( i.e. , the posterior over classes ) for each test instance , e.g. , by normalizing the classifier norms [ 22 ] or by scaling or reducing the logits according to class sizes [ 23 , 33 , 61 ] . Post-processing calibration is quite popular in imbalanced classification but not in long-tailed object detection . To our knowledge , only Li et al . [ 27 ] and Tang et al . [ 49 ] have studied this approach for object detection3 . Li et al . [ 27 ] applied classifier normalization [ 22 ] as a baseline but showed inferior results ; Tang et al . [ 49 ] developed causal inference calibration rules , which however require a corresponding de-confounded training step . Dave et al . [ 7 ] applied methods for calibrating model uncertainty , which are quite different from class-imbalanced learning ( see the next paragraph ) . In this paper , we demonstrate that existing calibration rules for class-imbalanced learning [ 23 , 33 , 61 ] can significantly improve long-tailed object detection , if paired with appropriate ways to deal with the background class and normalized the adjusted logits . We refer the reader to the supplementary material for a comprehensive survey and comparison of the literature . Calibration of model uncertainty . The calibration techniques we employ are different from the ones used for calibrating model uncertainty [ 11 , 24 , 25 , 34 , 38 , 63 , 64 ] : we aim to adjust the prediction across classes , while the latter adjusts the predicted probability to reflect the true correctness likelihood . Specifically for long-tailed object detection , Dave et al . [ 7 ] applied techniques for calibrating model uncertainty to each object class individually . Namely , a temperature factor or a set of binning grids ( i.e. , hyper-parameters ) has to be estimated for each of the hundreds of classes in the LVIS dataset , leaving the techniques sensitive to hyper-parameter tuning . Indeed , Dave et al . [ 7 ] showed that it is quite challenging to estimate those hyper-parameters for tail classes . In contrast , the techniques we apply have only a single hyper-parameter , which can be selected robustly from the training data . 3 Post-Processing Calibration for Long-Tailed Object Detection . In this section , we provide the background and notation for long-tailed object detection and instance segmentation , describe our approach Normalized Calibration ( NORCAL ) , and discuss its relation to existing post-processing calibration methods . 3.1 Background and Notation . Our tasks of interests are object detection and instance segmentation . Object detection focuses on detecting objects via bounding boxes while instance segmentation additionally requires precisely segmenting each object instance in an image . Both tasks involve classifying the object in each box/mask proposal region into one of the pre-defined classes . This classification component is what our proposed approach aims to improve . The most common object classification loss is the cross-entropy ( CE ) loss , LCE ( x , y ) = − C+1∑ c=1 y [ c ] × log ( p ( c|x ) ) , ( 1 ) where y ∈ { 0 , 1 } C+1 is the one-hot vector of the ground-truth class and p ( c|x ) is the predicted probability ( i.e. , confidence score ) of the proposal x belonging to the class c , which is of the form sc = p ( c|x ) = exp ( φc ( x ) ) ∑C c′=1 exp ( φc′ ( x ) ) + exp ( φC+1 ( x ) ) . ( 2 ) 3Calibration in [ 54 , 55 ] is in the training phase and is not post-processing . Here , φc is the logit for class c , which is usually realized by w > c fθ ( x ) : wc is the linear classifier associated with class c and fθ is the feature network . We use C +1 to denote the “ background ” class . During testing , a set of “ ( box/mask proposal , object class , confidence score ) ” tuples are generated for each image ; each proposal can be paired with multiple classes and appears in multiple tuples if the corresponding scores are high enough . The most common evaluation metric for these tuples is average precision ( AP ) , where they are compared against the ground-truths for each class4 . Concretely , the tuples with predicted class c will be gathered , sorted by their scores , and compared with the ground-truths for class c. Further , for popular benchmarks such as MSCOCO [ 28 ] and LVIS [ 12 ] , there is a cap K ( often set to 300 ) on the number of detected objects per image , which is enforced usually by including only the tuples with top K confidence scores . Such a cap makes sense in practice , since a scene seldom contains over 300 objects ; creating too many , likely noisy tuples can also be annoying to users ( e.g. , for a camera equipped with object detection ) . Long-tailed object detection and instance segmentation : problems and empirical evidence . Let Nc denote the number of training images of class c. A major challenge in long-tailed object detection is that Nc is imbalanced across classes , and the learned classifier using Eq . 1 is biased toward giving higher scores to the head classes ( whose Nc is larger ) [ 2 , 6 , 7 , 16 ] . For instance , in the longtailed object detection benchmark LVIS [ 12 ] whose classes are divided into frequent ( Nc > 100 ) , com- mon ( 100 ≥ Nc > 10 ) , and rare ( Nc ≤ 10 ) , the confidence scores of the rare classes are much smaller than the frequent classes during inference ( see Figure 2 ) . As a result , the top K tuples mostly belong to the frequent classes ; proposals of the rare classes are often mis-classified as frequent classes , which aligns with the observations by Wang et al . [ 55 ] .

Summary: In this paper, the authors study the problem of "calibration" of long-tail object detection models. To be specific, they take an existing pre-trained object detector and modify its logits according to the number of samples in each class to obtain better logits/scores for each class. They demonstrate on the long-tailed LVIS dataset with different models that this significantly and consistently improves the performance of methods (in terms of both precision and recall).