On Model Calibration for Long-Tailed Object Detection and Instance Segmentation

1 Introduction . Object detection and instance segmentation are the fundamental tasks in computer vision and have been approached from various perspectives over the past few decades [ 8 , 13 , 26 , 36 , 52 ] . With the recent advances in neural networks [ 1 , 4 , 10 , 18 , 29 , 30 , 32 , 37 , 40 , 41 , 43 ] , we have witnessed an unprecedented breakthrough in detecting and segmenting frequently seen objects such as people , cars , and TVs [ 14 , 15 , 21 , 31 , 70 ] . Yet , when it comes to detect rare , less commonly seen objects ( e.g. , walruses , pitchforks , seaplanes , etc . ) [ 12 , 51 ] , there is a drastic performance drop largely due to insufficient training samples [ 47 , 68 ] . How to overcome the “ long-tailed ” distribution of different object classes [ 69 ] has therefore attracted increasing attention lately [ 27 , 35 , 46 , 55 ] . To date , most existing works tackle this problem in the model training phase , e.g. , by developing algorithms , objectives , or model architectures to tackle the long-tailed distribution [ 12 , 20 , 27 , 46 , 48 , 53 , 55 , 57 , 58 ] . Wang et al . [ 55 ] investigated the widely used instance segmentation model Mask R-CNN [ 18 ] and found that the performance drop comes primarily from mis-classification of object proposals . Concretely , the model tends to give frequent classes higher confidence scores [ 7 ] , hence biasing the label assignment towards frequent classes . This observation suggests that techniques of class-imbalanced learning [ 2 , 6 , 16 , 42 ] can be applied to long-tailed detection and segmentation . Building upon the aforementioned observation , we take another route in the model inference phase by explicit post-processing calibration [ 2 , 22 , 23 , 33 , 61 ] , which adjusts a classifier ’ s confidence scores among classes , without changing its internal weights or architectures . Post-processing calibration is efficient and widely applicable since it requires no re-training of the classifier . Its effectiveness ∗Equal contributions 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . on multiple imbalanced classification benchmarks [ 22 , 59 ] may also translate to long-tailed object detection and instance segmentation . In this paper , we propose a simple post-processing calibration technique inspired by class-imbalanced learning [ 33 , 61 ] and show that it can significantly improve a pre-trained object detector ’ s performance on detecting both rare and common classes of objects . We note that our results are in sharp contrast to a couple of previous attempts on exploring post-processing calibration in object detection [ 7 , 27 ] , which reported poor performance and/or sensitivity to hyper-parameter tuning . We also note that the calibration techniques in [ 54 , 55 ] are implemented in the training phase and are not post-processing . Concretely , we apply post-processing calibration to the classification sub-network of a pre-trained object detector . Taking Faster R-CNN [ 43 ] and Mask R-CNN [ 18 ] for examples , they apply to each object proposal a ( C + 1 ) -way softmax classifier , where C is the number of foreground classes , and 1 is the background class . To prevent the scores from being biased toward frequent classes [ 7 , 55 ] , we re-scale the logit of every class according to its class size , e.g. , number of training images . Importantly , we leave the logit of the background class intact because ( a ) the background class has a drastically different meaning from object classes and ( b ) its value does not affect the ranking among different foreground classes . After adjusting the logits , we then re-compute the confidence scores ( with normalization across all classes , including the background ) to decide the label assignment for each object proposal2 ( see Figure 1 ) . We note that it is crucial to normalize the scores across all classes since it triggers re-ranking of the detection results within each class ( see Figure 3 ) , influencing the class-wise precision and recall . Instead of separately adjusting each class by a specific factor [ 7 ] , we follow [ 6 , 33 , 61 ] to set the factor as a function of the class size , leaving only one hyper-parameter to tune . We find that it is robust to use the training set to set this hyper-parameter , making our approach applicable to scenarios where collecting a held-out representative validation set is challenging . Our approach , named Normalized Calibration for long-tailed object detection and instance segmentation ( NORCAL ) , is model-agnostic as long as the detector has a softmax classifier or multiple binary sigmoid classifiers for the objects and the background . We validate NORCAL on the LVIS [ 12 ] dataset for both long-tailed object detection and instance segmentation . NORCAL can consistently improve not only baseline models ( e.g. , Faster R-CNN [ 43 ] or Mask R-CNN [ 18 ] ) but also many models that are dedicated to the long-tailed distribution . Hence , our best results notably advance the state of the art . Moreover , NORCAL can improve both the standard average precision ( AP ) and the category-independent APFixed metric [ 7 ] , implying that NORCAL does not trade frequent class predictions for rare classes but rather improve the proposal ranking within each class . Indeed , through a detailed analysis , we show that NORCAL can in general improve both the precision and recall for each class , making it appealing to almost any existing evaluation metrics . Overall , we view NORCAL a simple plug-and-play component to improve object detectors ’ performance during inference . 2 Related Work . Long-tailed detection and segmentation . Existing works on long-tailed object detection can roughly be categorized into re-sampling , cost-sensitive learning , and data augmentation . Re-sampling 2Popular evaluation protocols allow multiple labels per proposal if their confidence scores are high enough . methods change the long-tailed training distribution into a more balanced one by sampling data from rare classes more often [ 3 , 12 , 45 ] . Cost-sensitive learning aims at adjusting the loss of data instances according to their labels [ 19 , 46 , 48 , 53 ] . Building upon these , some methods perform twoor multi-staged training [ 20 , 22 , 27 , 42 , 55–57 , 67 ] , which first pre-train the models in a conventional way , using data from all or just the head classes ; the models are then fine-tuned on the entire long-tailed data using either re-sampling or cost-sensitive learning . Besides , another thread of works leverages data augmentation for the object instances of the tail classes to improve long-tailed object detection [ 9 , 39 , 65 , 66 ] . In contrast to all these previous works , we investigate post-processing calibration [ 22 , 23 , 33 , 50 , 61 ] to adjust the learned model in the testing phase , without modifying the training phase or modeling . Concretely , these methods adjust the predicted confident scores ( i.e. , the posterior over classes ) for each test instance , e.g. , by normalizing the classifier norms [ 22 ] or by scaling or reducing the logits according to class sizes [ 23 , 33 , 61 ] . Post-processing calibration is quite popular in imbalanced classification but not in long-tailed object detection . To our knowledge , only Li et al . [ 27 ] and Tang et al . [ 49 ] have studied this approach for object detection3 . Li et al . [ 27 ] applied classifier normalization [ 22 ] as a baseline but showed inferior results ; Tang et al . [ 49 ] developed causal inference calibration rules , which however require a corresponding de-confounded training step . Dave et al . [ 7 ] applied methods for calibrating model uncertainty , which are quite different from class-imbalanced learning ( see the next paragraph ) . In this paper , we demonstrate that existing calibration rules for class-imbalanced learning [ 23 , 33 , 61 ] can significantly improve long-tailed object detection , if paired with appropriate ways to deal with the background class and normalized the adjusted logits . We refer the reader to the supplementary material for a comprehensive survey and comparison of the literature . Calibration of model uncertainty . The calibration techniques we employ are different from the ones used for calibrating model uncertainty [ 11 , 24 , 25 , 34 , 38 , 63 , 64 ] : we aim to adjust the prediction across classes , while the latter adjusts the predicted probability to reflect the true correctness likelihood . Specifically for long-tailed object detection , Dave et al . [ 7 ] applied techniques for calibrating model uncertainty to each object class individually . Namely , a temperature factor or a set of binning grids ( i.e. , hyper-parameters ) has to be estimated for each of the hundreds of classes in the LVIS dataset , leaving the techniques sensitive to hyper-parameter tuning . Indeed , Dave et al . [ 7 ] showed that it is quite challenging to estimate those hyper-parameters for tail classes . In contrast , the techniques we apply have only a single hyper-parameter , which can be selected robustly from the training data . 3 Post-Processing Calibration for Long-Tailed Object Detection . In this section , we provide the background and notation for long-tailed object detection and instance segmentation , describe our approach Normalized Calibration ( NORCAL ) , and discuss its relation to existing post-processing calibration methods . 3.1 Background and Notation . Our tasks of interests are object detection and instance segmentation . Object detection focuses on detecting objects via bounding boxes while instance segmentation additionally requires precisely segmenting each object instance in an image . Both tasks involve classifying the object in each box/mask proposal region into one of the pre-defined classes . This classification component is what our proposed approach aims to improve . The most common object classification loss is the cross-entropy ( CE ) loss , LCE ( x , y ) = − C+1∑ c=1 y [ c ] × log ( p ( c|x ) ) , ( 1 ) where y ∈ { 0 , 1 } C+1 is the one-hot vector of the ground-truth class and p ( c|x ) is the predicted probability ( i.e. , confidence score ) of the proposal x belonging to the class c , which is of the form sc = p ( c|x ) = exp ( φc ( x ) ) ∑C c′=1 exp ( φc′ ( x ) ) + exp ( φC+1 ( x ) ) . ( 2 ) 3Calibration in [ 54 , 55 ] is in the training phase and is not post-processing . Here , φc is the logit for class c , which is usually realized by w > c fθ ( x ) : wc is the linear classifier associated with class c and fθ is the feature network . We use C +1 to denote the “ background ” class . During testing , a set of “ ( box/mask proposal , object class , confidence score ) ” tuples are generated for each image ; each proposal can be paired with multiple classes and appears in multiple tuples if the corresponding scores are high enough . The most common evaluation metric for these tuples is average precision ( AP ) , where they are compared against the ground-truths for each class4 . Concretely , the tuples with predicted class c will be gathered , sorted by their scores , and compared with the ground-truths for class c. Further , for popular benchmarks such as MSCOCO [ 28 ] and LVIS [ 12 ] , there is a cap K ( often set to 300 ) on the number of detected objects per image , which is enforced usually by including only the tuples with top K confidence scores . Such a cap makes sense in practice , since a scene seldom contains over 300 objects ; creating too many , likely noisy tuples can also be annoying to users ( e.g. , for a camera equipped with object detection ) . Long-tailed object detection and instance segmentation : problems and empirical evidence . Let Nc denote the number of training images of class c. A major challenge in long-tailed object detection is that Nc is imbalanced across classes , and the learned classifier using Eq . 1 is biased toward giving higher scores to the head classes ( whose Nc is larger ) [ 2 , 6 , 7 , 16 ] . For instance , in the longtailed object detection benchmark LVIS [ 12 ] whose classes are divided into frequent ( Nc > 100 ) , com- mon ( 100 ≥ Nc > 10 ) , and rare ( Nc ≤ 10 ) , the confidence scores of the rare classes are much smaller than the frequent classes during inference ( see Figure 2 ) . As a result , the top K tuples mostly belong to the frequent classes ; proposals of the rare classes are often mis-classified as frequent classes , which aligns with the observations by Wang et al . [ 55 ] .

Summary: This paper presents a normalized calibration method for long-tailed object detection and instance segmentation task. Experimental resutls show that separately treating the background class and normalizing the scores over classes for each proposal are keys to achieving superior performance. Experiments are conducted on the LVIS dataset with diverse evaluation sets. Results shows that consistent improvement can be achieved by the proposed scheme, and  extensive analysis and ablation studies are provided.