On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
Authors: Tai-Yu Pan
Cheng Zhang
YANDONG LI
Hexiang Hu
Dong Xuan
Soravit Changpinyo
Boqing Gong
Wei-Lun Chao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Long-tailed distribution
Object detection
Instance segmentation
Calibration

Abstract: Vanilla models for object detection and instance segmentation suffer from the heavy bias toward detecting frequent objects in the long-tailed setting. Existing methods address this issue mostly during training, e.g., by re-sampling or re-weighting. In this paper, we investigate a largely overlooked approach --- post-processing calibration of confidence scores. We propose NorCal, Normalized Calibration for long-tailed object detection and instance segmentation, a simple and straightforward recipe that reweighs the predicted scores of each class by its training sample size. We show that separately handling the background class and normalizing the scores over classes for each proposal are keys to achieving superior performance. On the LVIS dataset, NorCal can effectively improve nearly all the baseline models not only on rare classes but also on common and frequent classes.  Finally, we conduct extensive analysis and ablation studies to offer insights into various modeling choices and mechanisms of our approach. Our code is publicly available at https://github.com/tydpan/NorCal.

On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
Yui-Yu Pan
Cheng Zhang
Yandong Li Boqing Gong"
Hexiang Hu?
Dong Xuan
Soravit Changpinyo²
Wei-Lun Chao
'The Ohio State University
GGogle Research
Abstract
Vanilla models for object detection and instance segmentation suffer from the heavy bias toward detecting frequent objects in the long-tailed setting. Existing methods address this issue mostly during training, egg., by re-sampling or re- weighting. In this paper, investigate largely overlooked approach -asucchhpp post- processing calibration of confidence scores. We propose NORCAL, Normalized Calibration for long-tailed object detection and instance segmentation, simple and straightforward recipe that reweighs the predicted scores of each class by its training sample size. We show that separately handling the background class and normalizing the scores over classes for each proposal are keys to achieving superior performance. On the LVIS dataset, NORCAL can effectively improve nearly all the baseline models not only on rare classes but also on common and frequent classes. Finally, we conduct extensive analysis and ablation studies offer insights into various modeling choices and mechanisms of our approach. Our code is publicly available https: //gi thub. com/ ttdpptyydooooooooo
Introduction
Object detection and instance segmentation are the fundamental tasks in computer vision and have been approached from various perspectives over the past few decades [8, 13, 26, 36, 52]. With the recent advances in neural networks [1, 4, 10, 18, 29, 30, 32, 37, 40, 41, 43], we have witnessed an unprecedented breakthrough in detecting and segmenting frequently seen objects such as people, cars, and TVs [14, 15. 21, 31, 70]. Yet, when comes to detect rare, less commonly seen objects (e.g., walruses, pitchforks, seaplanes, etc.) [12, 51], there is a drastic performance drop largely due to insufficient training samples [47, 68]. How to overcome the "long-tailed" distribution of different object classes [69] has therefore attracted increasing attention lately [27, 35, 46. 55].
To date, most existing works tackle this problem in the model training phase, e.g., by developing algorithms, objectives, model architectures to tackle the long-tailed distribution [12, 20, 27, 46, 48, 53, 55, 57, 58]. Wang et al. [55] investigated the widely used instance segmentation model Mask R-CNN [18] and found that the performance drop comes primarily from mis- tiiissaassiiiittinn of object proposals. Concretely,
model tends to give frequent classes higher confidence scores [7], hence biasing the label assignment towards frequent classes. This observation suggests that techniques of class-imbalanced learning [2, 6, 16, 42] can be applied to long-tailed detection and segmentation.
Building upon the aforementioned observation, we take another route in the model inference phase by explicit post-procsssing calibration [2. 22. 23, 33, 61], which adjusts a classifier's is confidence scores among classes, without changing its internal weights or architectures. Post- Processin calibration efficient and widely applicable since requires no re-training of the classifier. Its effectiveness
Equal contributions
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Input Image
Long-Taine TangTailed Object Detection
Post-Processing Calibration
Classification
CNN
NORCAL
Regression
RPN
Mask
Truck
Sahool bus
Bulldojes
Truck
School bus
Bulldoser
Figure 1: Normalized Calibration (NORCAL). Object detection instance segmentation models (e.g. [18, 43)) trained with data from a long-tailed distribution tend output higher confidence scores for the head classes (e.g., "Truck") than for the tail ones (e.g., the true class label "Bulldozer"). NORCAL. investigates a simple but largely overlooked approach to correct this mistake - post-processing calibration of the classification scores after training and significantly improves nearly all the models we consider.
on multiple imbalanced classification benchmarks [22. 59] may also translate to long-tailes object detection and instance segmentation.
In this paper, propose simple post-processing calibration technique inspired by class-imbalanced learning [33, 61] and show that it can significantly improve a pre-trained object detector's 's performance on detecting both rare and common classes of objects. We note that results are in sharp contrast to couple of previous attempts on exploring post-processing calibration in object detection [7, 27J, which reported poor performance and/or sensitivity to hyper-parameter tuning. We also note that the calibration techniques in [54, 55] are implemented in the training phase and are not post-processing.
Concretely, we apply post-proeessing calibration the classification sub-network of pre-trained object detector. Taking Faster R-CNN [43] and Mask R-CNN [18] for examples, they apply to each object proposal (C + --wayy softmax classifier, where C is the number of foreground classes, and 1 is the background class. To prevent the scores from being biased toward frequent classes [7, 55), re-scale the logit of every class according its class size, e.g., number of training images. Importantly, we leave the logit of the background class intact because (a) the background class has a drastically different meaning from object classes and (b) its value does affect the ranking among different foreground classes. After adjusting the logits, we then re-compute the confidence scores (with normalization across all classes, including the background) decide the label assignment for each object proposal (see Figure 1). We note that is crucial to normalize the scores across all classes since it triggers re-ranking of the detection results within each class (see Figure 3), influencing the class-wisg wise precision and recall. Instead of separately adjusting each class by a specific factor [7], we follow [6, 33, 61] to set the factor as a function of the class size, leaving only one hyper-parameter to making our approach
tune. We find that it is robust to use the training set to set this applicable to scenarios where collecting held-out representative validation set challenging.
Our approach, named Normalized Calibration for long-tailed object detection and instance seg- mentation (NORCAL), model-agnostic long as the detector has softmax classifier multiple binary sigmoid classifiers for the objects and the background. We validate NORCAL on the LVIS [12] dataset for both long-tailed object detection and instance segmentation. NORCAL can consistently improve not only baseline models (e.g.. Faster R-CNN [43] or Mask R-CNN [18]) but also many models that are dedicated the long-tailes distribution. Hence, our best results notably advance the state of the art. Moreover, NORCAL can improve both the standard average precision (AP) and the integory-independen A metric [7], implying that NORCAL does not trade frequent class predictions for rare classes but rather improve the proposal ranking within each class. Indeed, through a detailed analysis, we show that NORCAL can general improve both the precision and recall for each class, making it appealing to almost any existing evaluation metrics. Overall, we view NORCAL a simple plug-and- -uu-anndplay component to improve object detectors' performance during inference.
2 Related Work
Long-tailed detection and segmentation. Existing works long-tailed object detection can roughly be categorized into re-sampling, cost-sensitive learning, and data augmentation. Re-sampling
PPopular evaluation protocols allow multiple labels per proposal if their confidence scores are high enough.

---

methods change the long-tailed tailed training distribution into more balanced one by sampling data from rare classes more often [3, 12, 45]. Cost-sensitive learning aims adjusting the loss of data instances according to their labels [19, 46, 48, 53]. Building upon these, some methods perform two- or multi-staged training [20, 22, 27, 42, 55-57, 67], which first pre-train the models in a conven- tional way, using data from all or just the head classes; the models are then fine-tuned on the entire long-tailed data using either re-sampling or cost-sensitive learning. Besides, another thread works leverages data augmentation for the object instances of the tail classes to improve long-tailed object detection [9, 39, 65, 66].
In contrast to all these previous works, we investigate post-proceseo calibration [22, 23, 33, 50, 61] to adjust the learned model in the testing phase, without modifying the training phase or modeling. Concretely, these methods adjust the predicted confident scores (i.e., the posterior over classes) for each test instance, e.g., by normalizing the classifier norms [22] or by scaling or reducing the logits according to class sizes [23, 33, 61]. Post-processing calibration is quite popular in imbalanced classification but not long-tailed object detection. To our knowledge, only Li et al. [27] and
Tang et al. [49] have studied this approach for object detection' Li et al. [27] applied classifier normalization [22] as a baseline but showed inferior results; Tang et al. [49] developed causal inference calibration rules, which however require corresponding de-confounded training step. Dave et al. [7] applied methods for calibrating model uncertainty, which are quite different from class-imb iaass-iebalaneed learning (see the next paragraph). In this paper, we demonstrate that existing calibration rules for caaasiimbalneeed learning [23, 33, 61] can significantly improve long-tailed object detection, if paired with appropriate ways deal with the background class and normalized the adjusted logits. We refer the reader to the supplementary material for a comprehensive survey and comparison of the literature.
Calibration of model uncertainty. The calibration techniques we employ are different from the ones used for calibrating model uncertainty [11, 24, 25, 34, 38, 63, 64]: we aim to adjust the prediction across classes, while the latter adjusts the predicted probability to reflect the true correctness likelihood. Specifically for long-tailed object detection, Dave et al. [7] applied techniques for calibrating model uncertainty to each object class individually. Namely, a temperature factor or a set of binning grids (i.e., ) has to be estimated for each of the hundreds of classes in the LVIS dataset, leaving the techniques sensitive to pyper-parameter tuning. Indeed, Dave et al. [7] showed that is quite challenging to estimate those paremeaereeeeess for tail classes. In contrast, the techniques apply have only a single
which can be selected robustly from the training data.
3 Post-Processing Calibration for Long-Taile Object Detection
In this section, we provide the background and notation for long-tailed object detection and instance segmentation, describe our approach Normalized Calibration (NORCaL), and discuss its relation to existing post- procesrocessing calibration methods.
3.1 Background and Notation
Our tasks of interests are object detection and instance segmentation. Object detection focuses on detecting objects via bounding boxes while instance segmentation additionally requires precisely segmenting each object instance in an image. Both tasks involve classifying the object in each box/mask proposal region into one of the pre-defined classes. This classification component is what our proposed approach aims to improve. The most common object classification loss is the cross-entropy (CE) loss,
C+1
Lar( v) y Σ vlcl x log (p(clx)),
where y E {0, 1j the one-hot vector of the ground-truth class and p(c[æ) is the predicted probability (i.e., contidence score) of the proposal æ belonging to the class e, which is of the form
cxp()e(x))
82 p(c())
(2)
Σ (x)( + exp(#c+1(x))
"Calibration in [54, 55] is in the training phase and is not post processing.
3

---

Here, de is the logit for class which is usually realized by w. fa():: associated with class and fo is the feature network. We use C' 1 denote the "background" class.
: we is the linear classifier
During testing, a set of "(box/mask proposal, ob- ject class, confidence score)" tuples are generated for each image; each proposal can be paired with
baseline
0.2
Fyther
1.0
multiple classes and appears in multiple tuples if the corresponding scores are high enough. The most
Average score
Cother
faster
common evaluation metric for these tuples aver-
age precision (AP), where they are compared against the ground -trnths for each class Concretely, the tuples with predicted class e will be gathered, sorted by their scores, and compared with the ground-truths for class c. Further, for popular benchmarks such as MSCOCO [28] and LVIS [12], there cap K (often set to 300) on the number of detected objects per image, which is enforced usually by including only the tuples with top K confidence scores. Such a cap makes sense in practice, since a scene seldom contains over 300 objects; creating too many, likely noisy tuples can also be annoying to users (e.g., for camera equipped with object detection).
0.1
0.4
0.2
0.0
0.0
C
Figure 2: The effect of long-tailed distributions LVTS v0.5 [12]. Left: We show the confidence scores of the top 300 tuples of each image by the baseline Faster R-CNN detector [43] w/o or w/ NORCAL. We average the scores for rare, com- mon, and frequent classes and then linearly scale these averaged scores such that the frequent class has score of 1. The baseline detector gives fre- quent objects higher scores, which can be allevi- ated by NORCAL. Right: For tuples of the rare class predicted by the baseline Faster R-CNN w/o NOR CAI, we further show average scores of them and the average highest scores from another rare, common, and frequent classes on the same proposals. The learned baseline detector tends to predict frequent classes for proposal.
Long-tailed object detection and instance segmen- tation: problems and empirical evidence. Let Nr denote the number of training images of class A major challenge in long-tailed object detec- tion is that N₂ is imbalanced across classes, and the learned classifier using Eq. 1 is biased toward giving higher scores to the head classes (whose No larger) [2. 6, 7, 16]. For instance, in the long- tailed object detection benchmark LVIS [12] whose classes divided into frequent (No 100), com-
mon (100 2 N: > 10), and rare (N₂ < 10), the confidence scores the rare classes are much smaller than the frequent classes during inference (see Figure 2). As result, the top K tuples mostly belong to the frequent classes; proposals of the rare classes are often mis-classified as frequent classes, which aligns with the observations by Wang et al. [55].
3.2 Normalized Calibration for Long-tailed Object Detection (NoRCAL)
Next, we describe the key components of the proposed NORCAL, including confidence score calibration and normalization. The former re-ranks the confidence scores across classes to overcome the bias that rare classes usually have lower confidence scores; the latter helps re-order the scores of detected tuples within each class for further improving the performance. Both the confidence score calibration and normalization are essential to the success of NORCAL.
Post-processing calibration and saggrouudd-ackkroundd decomposition. We explore applying simple post-calibration techniques from standard multi-way way classification [22, 59] to object detection and instance segmentation. The main idea is to scale down the logit of each class e by its size N₂ [33, 61]. In case, however, the background class poses unique challenge. First, No+1 is ill-defined since nearly all images contain backgrounds. Second, the background regions extracted during model training are drastically different from the foreground object proposals in terms of amounts and appearances. We thus propose to decompose Eq. 2 as follows,
Ee exp(dee(a)) E. exp(px exp(#c+1(z)) X exp(p.(x))
p(c[))
(3)
exp(
'The difference between AP for object detection and instance segmentation lies in the computation of the intersection union (loU): the former based boxes and the latter based masks.

---

6= 1 c= 2 == BG
<= c= c=3 BG
c= c= 2 == BG
Proposal A (cor* = 3): [0.00, 0.50,
0.50 0.10]
[0.00, 0.10, 0.13, 0.10] [0.00, 0.31, 0.38, 0.31]
1
Proposal B (Car= 1): [0.30, 0.00, 0.60. 0.10]
[0.30, 0.00, 0.15, 0.10]
0.55 0.00, 0.27, 0.18]
a1
a2
3g
Original
NoRCAL w/o score normalization
NORCAL w/ score normalization
Class frequency
predictions
Figure 3: NORCAL with score normalization can improve AP for head classes. Here assume there are three possible foreground classes, and show the ground-truth classes (i.e., eGr) and predictions for two object proposals. Bold and underlined numbers indicate the highest scored class for each proposal. The proposed calibration approach and score normalization can be organically coupled together to improve the ranking of personals/tuples for each class. See the text for details.
where the first term on the right-hand side predicts how likely is foreground (vs. background, i.e. class C' + 1) and the second term predicts how likely æ belongs to class c given that is foreground. Note that, the background logit Φc+1 1 (a: only appears in the first term and is compared to all the foreground classes as a whole. In other words, scaling or reducing it does not change the order of confidence scores among the object classes E {I,... CC. We thus choose keep Φ0+1 (x) intact. Please refer to Section 4 for a detailed analysis, including the effect of adjusting Φc+1 (z).
For the foreground object classes, inspired by Figure 2 and the studies in [7, 33, 61], we propose to scale down the exponential of the logit (x(x), Vc E
by a positive factor ac.
cxp()<(x))/a=
p(c()) Ee exp(de (x)) 0e + exp()c++
(4)
in which a, should monotonically increase with respect No such that the scores for head classes will be suppressed. We investigate a simple way to set a, inspired by [23, 61],
a = N2 7 2 0,
(5)
which has a single hyper-parameter y that controls the strength of dependency between a, and N Specifically, if = 0, we recover the original confidence scores in Eq. 2. We investigate other methods beyond Eq. and Eq. in Section 4.
Hyper-parameter tuning. Our approach only has a single hyper-parameter y to tune. We observe that we can tune y directly the training data", bypassing the need of held-out set which can be hard to collect due to the scarcity of examples for the tail classes. Dave et al. [7] also investigate this idea; however, the selected hyper-parametee from training data hurt the test results of rare classes. We attribute this to the fact that their methods have separate hyper-paane seper-parameters for each class, and that makes them hard tune.
The importance of normalization and its effect on AP. At first glance, our approach NORCAL seems to simply scale down the scores for head classes, and may unavoidably hurt their AP due to the decrease of detected tuples (hence the recall) within the cap. However, we point out that the normalization operation (i.e., sum to 1) in Eq. 4 can indeed improve AP for head classes normalization enables re- re-ordering the scores of tuples within each class.
Let us consider a three-class example (see Figure 3), in which e = 1 is a tail class, c 2 and c 3 head classes, and 4 is the background class. Suppose two proposals are found from an image: proposal A has scores [0.0, 0.4, 0.5, 0.1] and the true label cGr = 3; proposal B has scores [0.3, 0.0, 0.6, 0.1] and the true label CGT 1. Before calibration, proposal B is ranked higher than A for = 3, resulting in low AP. Let us assume
= 1 and a2 = the scores of object classes by these factors, proposal B will still be ranked higher than A for However, by applying Eq. 4. we get the new scores for proposal A as [0.0, 0.31, 0.38, 0.31] and for proposal as [0.55, 0.0, 0.27, 0.18) - proposal A is now ranked higher than B for c 3, leading to higher AP for this class. As will be seen in Section 4, such "re-ranking" property is the key to making NORCAL excel in AP for all classes as well as in other metrics like ApPixed [7].
4. If we simply divide
$ Unlike imbalanced classification in which the learned classifier ultimately achieves ~ 100% accuracy on the training data [61. 62] (so hyper-parc parameeer tuning using the training data becomes infeasible), long-taiie long-tailed object detector can hardly achieve 100% AP per class even on the training data.
5

---

3.3 Comparison to Existing Work
Li et al. [27] investigated classifier normalization [22] for post-processing calibration. They modified the calculation of Φ. from w fa) to w=
['well3" fo( (x), building upon the observation that the classifier weights of head classes tend to exhibit larger norms [22]. The results, however, were much worse than their proposed cost-sensitive method BaGS. They attributed the inferior result to the background class, and had combined two models, with or without classifier normalization, attempting to improve the accuracy. Our decomposition in Eq. suggests more straightforward way to handle the background class, Moreover, Ne provides better signal for calibration than || /2. according to [23, 61J. We provide more discussions and comparison results in the supplementary material.
3.4 Extension to Multiple Binary Sigmoid Classifiers
Many existing models for long-tailed object detection and instance segmentation are based multiple binary classifiers instead of the softmax classifier [19, 42, 46, 48, 56]. That is, 8c in Eq. becomes
1
exp(ox(x))
Se 1+ expepp-uu w. fo(æ)) 1 + (o)()))()))
(6)
(x)(
in which w. treats every class e' * and the background class together as the "negative" class. In other words, the background logit Φc+1 = (o
Eq. is not explicitly learned.
Our post-processing calibration approach can be extended to multiple binary classifiers as well. For example, Eq. 4 becomes
exp(de(az))/a
exp()x(x))/a
We note that solely calibrating the scores can re-rank the detected tuples across classes within each image such that rare and common objects, which initially have lower scores, could be included in the cap to largely increase the recall. Therefore, as will be shown in the experimental results, the improvement for multiple binary classifiers mainly comes from the rare and common objects.
However, one drawback of the score calibration alone is the infeasibility of normalization across classes; 8, does not necessarily sum to I, making it hard to re-order the scores of tuples within each class. Foreing the confidence scores across classes of each proposal to to 1 would inevitably turn many background patches into foreground proposals due to the lack of the background logit Φc+1-
Experiments
4.1 Setup
Dataset. We validate NORCAL on the LVIS vI dataset [12], benehmark dataset for large-vocabulary instance segmentation which has (00K/19.80) 100K/19.8K/19 8K categories, divided into three groups based on the number of training images per class: rare (1-10 images), common (11-100 images), and frequent (> 100 images). All results are reported on the validation set of LVIS vl. For comparisons to more existing works and different tasks, we also conduct detailed experiments and analyses on LVIS v0.5 [12], Objects365 [44], MSCOCO [28], and image classification datasets in the supplementary material.
images. There are 1,203
Evaluation metrics. We adopt the standard mean Average Precision (AP) [28] for evaluation. The cap over detected objects per image is set 300 (cf. Section 3.1). Following [12]. denote the mean AP for rare, common, and frequent categories by AP,, AP,. and AP,, respectively. We also report results with a complementary metric Fieed [7]. which replaces the cap over detected objects per image by cap over detected objects per class from the entire validation set. Namely, Fixed removes the competition of confidence scores among classes within an image, making itself integory-ideependeet. We follow [7] to set the per-class cap as 10, 000. Instead of Mask AP, we also report the results in Fixed [7] with Boundary IoU, following the standard evaluation metric in LVIS Challenge 20216, Meanwhile, we report AP", which assesses the AP for the bounding boxes produced by the instance segmentation models.
'nttps://www wrisdataset. .rg. pergcchallnng.... 2021.

---

Table I: Comparison of instance segmentation the validation set of LVIS v1. NORCAL provides solid improvement to existing models. with EQL, we see a slight drop on the frequent classes due to the infeasibility of score normalization across classes with multiple binary classifiers. *: models from [66]. f: models from [48].
Backbone
Method
NORCAL
AP
AP,
AP.
AP,
Ap
X
18.0
2.10
17.40
27.20
19.30
EQL [46]]
/
(+2.30) 20.90
(+3.90)
6,00
(+3.80)
21.20
(-00.0)
27.10
(+2.50)
21.80
X
22.10
11.90
20.20
29.00
22.20
cRT [22]
R-50 [17]
(+2.20) 24.30
(+3.50)
15,40
(+2.70)
22.90
(+0.70)
29.70
(+1.50)
23.70
22.58
12.30
21.28
28.55
23.25
RFS [12]*
(+2.65) 25.22
(+7.03)
19.33
(+2.88) 24.16
(+0.43) 28.98
(+2.83) 26.08
24.45
18.17
22.99
28.83
25.05
MosaicOS [66]
(+2.32) 26.76
(+5.69) 23.86
(+2.82)
82
(+0.27) 29.10
(+2.73) 27.77
X
24.82
15.18
23.71
30.31
25.45
RFS [12]*
(+2.43) 27.25
(+5.61) 20.79
(+2,74)
26.45
(+0.68)
30.99
(+2.60)
28.05
R-101 [17]
26.73
20.53
25.78
30.53
27.41
MosaicOS [66]
(+2.30) 29.03
(+5.85)
26.38
(+2.37) 28.15
(+0.66)
31.19
(+2.55)
29.96
26.67
17.60
25.58
31.89
27.35
RFS [12]*
(+1.25) 27.92
(+2.15) 19.75
(+1.61) 27.19
(+0.45) 32.34
(+1.49)
28.83
X-101 [60]
28.29
21.75
27.22
32.35
28.85
MosaicOS [66]
(+1.52) 29.81
(43.97)22.77
(+1.70) 28.92
(+0.24) 32.39
(+1.71) 30.56
Implementation details and variants. We apply NORCAL to post-calibrate several representative the released checkpoints from the corresponding papers. We
baseline models, for which focus on models that have a softmax classifter or multiple binary classifiers for assigning labels to proposals?. For NORCAL, (a) investigate different mechanisms by applying post-calibration to the classifier logits, exponentials, or probabilities (cf. Eq. 4); (b) we study different types of calibration factor de, using the class-dependent temperature (CDT) [61] presented in Eq. 5 or the effective number of samples (ENS) [6]; (c) we compare with or without score normalization. We tune the only hyper-parameter of NORCAL (i.e., in a.) on training data.
4.2 Main Results
NORCAL effectively improves baselines in diverse scenarios. We first apply NORCAL to represen-
tative baselines for instance segmentation: (1) Mask R-CNN [18] with feature pyramid networks [29], which is trained with repeated factor sampling (RFS), following the standard training procedure
(3) seampling/cost-senstteee -ensssive based methods that have a multi-class classifier, e.g., cRT [22];
in [12]; (2) re-sampling/cost-snnitte
based methods that have multiple binary classifiers, e.g., EQL [46]; (4) data augmentation based methods, e.g., a state-of-the-as the-art method MosaicOS [66]. Please see the supplementary material for a comparison with other existing methods.
Table provides our main results on LVIS vl. NORCAL achieves consistent gains on top of all the models of different backbone architectures. For instance, for RFS [12] with ResNet-50. the overall AP improves from 22.58% to 25. 25.22%. including ~ 7%/3% gains on AP, A.P/AAP AP, for rare/common objects. Importantly, note that NORCAL's improvement is almost all the evaluation metrics (columns), demonstrating a key strength of NORCAL that is not commonly seen in literature: achieving overall gains without sacrificing the AP, frequent classes. We attribute this the score normalization operation of NORCAL: unlike [7] which only re-ranks scores across categories, NORCAL further re-ranks the scores within each category. Indeed, the only performance drop in Table is on frequent classes for EQL, which is equipped with multiple binary classifiers such that score normalization across classes is infeasible (cf. Section 3.4). We provide more discussions in the ablation studies.
Comparison to existing post-calibration methods. We then compare our NORCAL to other post- calibration techniques. Specifically, we compare those in [7] on the LVIS vI instance segmentation task, including Histogram Binning [63], Bayesian binning into quantiles (BBQ) [34], Beta calibra- tion [24], isotonic regression [64], and Platt scaling [38]. We also compare to classifier normalization (r-normalized) [22, 27] on the LVIS v0.5 object detection task. All the prper--prameters for calibra- tion are tuned from the training data.
"Seerall existing methods (e.g., [27, 48, 53)) develop specific classification rules to which NORCAL cannot be directly applied.
7

---

Table 2: Comparison to other existing post- calibration methods. NORCAL outperforms methods studied in [7] and [27]. t: w/o
Table Ablation studies NORCAL with various mod- cling choices and mechanisms. We report results on LVIS vI instance segmentation. CAL: calibration mechanism. NOR: class score normalization. The best ones are in bold.
RFS [12].
Segmentation vl
AP
AP,
APe
APy
Gc
CAL.
NOR
AP
AP, APc
AP/
Baseline
exp(de(x=)
V
22.58
12.30
21.28
28.55
RFS [12]
22.58
12.30
21.28
28.55
HistBin [63]
21.82
11.28
20.31
28.13
exp(ác(x)/aa))
x 23.66
14.55
22.36
29.11
/
23,96
15,84
22.61
29.04
w/ BBQ (AIC) [34]
22.05
11.41
20.72
28.21
1- 1-
w/ Bcta calibration [24]
22.55
12.29
21.27
28.49
1
p(clw)//e
/
x
24.18
24.85
19.43
18.88
22.95
23.67
27.87
28.54
w/ Isutonic reg. [64]
22.43
12.19
21.12
28.41
(ENS [6])
w/ Platt scaling [38]
22.55
12.29
21:27
28.49
exploc(<))\/ac
17.49
14.16
17.20
19.27
/
24.85
19.43
23.67
28.54
w/ NORCAL
25.22
19.33
24.16
28.98
exp(do(=)/aa))
x
17.52
14.04
17.38
19.20
Detection on v0.5
APá
AP
APP
AP}
/
24.77
17.99
23.81
28.83
Faster R-CNN [43]f
20.98
4.13
19.70
29.30
(CDT [61])
No
p(c|æ)/oo
/
x
24.50
25.22
18.34
19.33
23.42
24.16
28.41
28.98
r-normalized [27]]
21.61
6.18
20.99
28.54
exp()c(x))\/e
x 17.52
13.93
17.24
19.41
NORCAL
23.87
6.98
24.17
30.24
25.22
19.33
24.16
28.98
Table 2 shows the results. NORCAL significantly outperforms other techniques on both tasks and
can improve AP for all the classes. We attribute the improvement over methods studied in [7] to two reasons: first, NORCAL has only one hyper-parameter. while calibration methods [7] have hyper-parameters for every category and thus are sensitive to tune; second, NORCAL performs score normalization, while [7] does not. Compared to [22, 27], the use of per-class data count in NORCAL has been shown to outperform classifier norms for calibrating classifiers [23, 61].
4.3 Ablation Studies and Analysis
We mainly conduct the ablation studies on the Mask R-CNN model [18] (with ResNet-50 back- bone [17] and feature pyramid networks [29]), trained with repeated factor sampling (RFS) [12].
Effect of calibration mechanisms. In addition reducing the logits, i.e,, scaling down their exponentials (i.e, (xp()((x))/a, in Eq. 4), we investigate another two ways of score calibration. Specifically, scale down the output logits from the network (i.e., (x))/a)) or the probabilities from the classifier (i.e., p(cl=)/a.)
Again, we keep the background class intact and apply score normalization. In Table 3, we see that scaling down the exponentials and probabilities perform the same and outperform scaling down logits. We note that, logits can be negative; thus, scaling them down might instead increases the scores. In contrast, exponentials and probabilities are non- non-negative, scaling them down thus are guaranteed to reduce the scores of frequent classes more than rare classes.
Effeet of calibration factors aa. Beyond the class-dependent temperature (CDT) [61] presented in Eq. 5, we study an alternative factor, inspired by the effective number of samples (ENS) [6]. Specifically, study a (1 / - 7) with y [0, 1). Same CDT, ENS has single hyper- parra--ar y that controls the degree of dependency between de and No. If = O, we recover the original confidence scores. We report the comparison of these two calibration factors in Table 3. With appropriate post-calibration mechanisms, both provide consistent gains over the baseline model.
Importance of score normalization. Again in Table 3, we compare NORCAL with or without score normalization across classes. That is, whether we include the denominator Eq. 4 or not. By applying normalization, we see that NORCAL can improve all categories, including frequent objects. Moreover, it applicable to different types of calibration mechanisms as well as calibration factors. In contrast, the results without normalization degrade at frequent classes and sometimes even at common and rare classes. We attribute this to two reasons: first, score normalization enables the detected tuples of each class to be re-ranked (cf. Figure 3); second, with the background logits in the denominator, the calibrated and normalized scores can effectively prevent background patches from being classified
Wiih class normalization, they mathematically the same.
8

---

*+ overall
rare
common
frequent
40.0
28
40
37.5
26
35.0
24
22-
32.5
30
30.0
20
27.5
18
0
1
2
2
3
0.2
0.4
Background factor
Figure 4: Results of precision and recall by adjusting back- ground class scores. Results are vl instance segmentation.
Figure 5: Calibration factor y can be robustly tuned using training data.
Table 4: NORCAL improves average precision and recall. Results are on LVIS vl instance segmentation. (a) Average Procision (AP)
(b) Average Recall (AR)
AP
AP,
APc
AP,
AR
AR,
ARa
AR
RFS [12] 22.58
12.30
21.28
28.55
RFS [12]
30.61
13.73
28.64
40.24
NORCAL
25.22
19.33
24.16
28.98
w/ NORCAL
36.10
28.75
35.79
39.68
Table 5:
NORCAL can improve the baseline model in A and A with Boundary IoU. The baseline model uses ResNet-50 as the backbone with RFS [12]. Results are reported on LVIS vI instance segmentation.
(a) AP Fixed
(h) AP Fixed with Boundary IoU
AP
AP,
APc
AP,
AP
AP,
APc
APy
RFS [12] 25.68
20.07
24.82
29.11
RFS [12]
19.88
14.76
19.32
22.76
w/ NORCAL
26.26
20.56
25.39
29.73
w/ NORCAL
20.25
14.99
19.77
23.10
into foreground objects. Please be referred to the supplementary material for additional results and ablation studies on sigmoid-basel detectors (i.e., BALMS [42] and RetinaNet [30]).
How to handle the background class? NORCAL does not calibrate the background class logit. We ablate this design by multiplying the exponential of the background logit with a background calibration factor 3, i.e., exp( @c++ (z)) x B. If B = 1, there is no calibration on background class. Figure 4 shows the average precision and recall of the model with NORCAL w.r.t different B. We see consistent performance for 1. For 1, the average precision drops along with reduced B, especially for the rare classes whose average recall also drops. We note that, in the extreme case with 0, the background class will not contribute to the final calibrated score. Thus, many background patches may be classified as foregrounds and ranked higher than rare proposals. These results and explanation justifies one key ingredient of NORCAL- keeping the background logit intact.
Sensitivity to the calibration factor: NORCAL. has one hyper-parameter y in the calibration factor Gc, which controls the strength of calibration. We find that this be tuned robustly on the training data, even on 5K subset of training images: as shown in Figure 5, the AP trends on the training and validation sets different 7 are close to each other. In our experiments, we find that this observation applies to different models and backbone architectures.
NOR reduces false positives and re-ranks predictions within each class. In Table 4, we show that NORCAL can improve the AR for all classes but frequent objects (with a slight drop). The gains on AP for frequent classes thus suggest that NORCAL re-rank the detected tuples within each class, pushing many false positives to have scores lower than true positives.
NORCAL is effective in Fixed [7] and Boundary IoU [5]. Table reports the results in ApFixed and ApFixed with Boundary IoU. We see that NORCAL is metric-agnostic and consistently improve the baseline model in all groups of categories. It suggests that the improvements are due to both across-class and within-class re-ranking.
Limiting detections per image. Finally, we evaluate NORCAL by changing the cap on the number of detections per image. Specifically, we investigate reducing the default number of 300. The rationale that an image seldom contains over 300 objects. Indeed, each LVIS [12] image annotated with around 12 object instances average. We note that, to perform well in smaller cap requires

---

baseline NorCal
1.5
common
frequent
15 18 b
AP rare
10
200
Number of detections image
Figure 6: Limits on the number of detections per image. To perform well in small cap, a model must rank true positives higher such that they can be included in the cap. NORCAL performs much better than the baseline.
Ground truth
Baseline
NORCAL
Figure 7: Qualitative results. We superimpose red arrows to show the improvement, and Yellow and red boxes indicate the ground truth labels of frequent and classes, In the first example, NORCAL successfully detects the rare object sugar bowl without sacrificing other predictions. In the second example, even surprisingly, can detect a missed frequent object frishee by the baseline.
model rank most true positives in the front such that they be included in the cap. In Figure NORCAL shows superior performance against the baseline model under all settings. It is worth noting that NORCAL achieves better performance even using striet 100 detections per image than the baseline model with 300.
Qualitative results. We show qualitative bounding box results on LVIS vI in Figure 7. We compare the ground truths, the results the baseline, and the results of NORCAL. NORCAL can not only detect more objects from the rare categories that may be overlooked by the baseline detector, but also improve the detection results on frequent objects. For instance, in the upper example of Figure 7, NORCAL discovers rare object "sugar bowl" without sacrificing any other frequent objects. Moreover, NORCAL can improve the frequent classes, as shown in the bottom example of Figure Please the supplementary material for more qualitative results.
5 Conclusion
We present post-processing calibration method called NORCAL for addressing long-tailed object detection and instance segmentation. Our method is simple yet effective, requires no re-training of the already trained models, and can be compatible with many existing models to further boost the state of the art. We conduct extensive experiments demonstrate the effectiveness of our method in diverse settings, as well as to validate our design choices and analyze our method's mechanisms. We hope that our results and insights can encourage more future works on exploring the power of prot-processing calibration in long-tailed object detection and instance segmentation.
10

---

Acknowledgments and Funding Transparency Statement
This research is partially supported by NSF LIS-2107077 and the OSU GI Development funds. We are thankful for the generous support of the computational resources by the Ohio Supercomputer Center. We thank Zhiyun Lu (Google) for feedback on an early draft of this paper and Han-Jia Ye (Nanjing University) for the help on image classification experiments.
References
[1] Alexey Bochkovskiy, Chien- Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed accuracy of ohject detection. arXiv preprint arXiv:2004. 10934, 2020.
[2] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-250 2018. 1,4
[3] Nadine Chang. Zhiding Yu, Yu-Xiong Wang, Anima Anandkumar, Sanja Fidler, and Jose M Alvarez. Image-level object-level? tale of two resampling strategies for long-tailed detection. ICML, 2021.
[4] Liang-Chied Chen, Alexander Hermans, George Papandreou, Florian Schroff. Peng Wang, and Hartwig Adam. Masklab: Instance segmentation by refining object detection with semantic and direction features. In CVPR, 2018.
[5] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg. and Alexander Kirillov. Boundary loU: Improving object-centric image segmentation evaluation. In CVPR, 2021. 9
[6] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019. 2, 7,
Achal Dave, Piotr Dollár, Deva Ramanan, Alexander Kirillov, and Ross Girshick. Evaluating large- vocabulary object detectors: The devil the details. arXiv preprint arXiv:2102.01066, 2021. 2, 5. 6, 7, 8,
[8] Pedro F Felzenszwalb, Ross В Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE Trunsactions on Pattern Analysis and Machine Intelligence (TPAMI), 1277)11627 2009.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin Cubuk, Quac V Le, and Barret Zoph. Simple copy copy-paste a strong data augmentation method instance segmentation. In CVPR, 2021.
[10] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
[11] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
[12] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019. 2, 3. 4, 6, 8,
[13] Saurabh Gupta, Ross Girshick, Pablo Arbeláez, and Jitendra Malik. Learning rich features from rgb-d images for object detection and segmentation. In ECCV, 2014.
[14] Ahdul Muced Hañiz and Ghulam Mohiuddin Bhat. A survey on instance segmentation: state of the art. International Journal of Multimedia Information Retrieval, pages 1-19, 2020. 1
[15] Junwei Han, Dingwen Zhang, Gong Cheng, Nian Liu, and Dong Xu. Advanced deep-learning techniques for salient and category-specciic object delection: survey. IEEE Signal Processing Magazine, 35(1): 84-100, 2018.
[16] Haiho He and Edwardo Garcia. Learning from imbalanced data, IEFE Transactions on Knowledge and Data Engineering (TKDE), 2169)1126-1128, 2009. 1,
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 7,8
[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN, In 2017. 2017. 2, 7,

---

[19] Ting-I Hsieh, Esther Robb, Hwann-Tzoon Chen, and Jia-Bin Huang. Droploss for long-tail instance segmentation. In AAAI, 2021. 3,
[20] Xinting Hu, Yi Jiang. Kaihua Tang, Jingyuan Chen, Chunyan Miao, and Hanwang Zhang, Learning to segment the tail. In CVPR, 2020. 1,3
[21] Licheng Jiao, Fan Zhang, Fang Liu, Shuyuan Yang, Lingling Li, Zhixi Feng, and Rong Qu. A survey of deep learning-based object detection. IEEE Access, 7:1 7:128837-128868 2019. 1
[22] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2020. 1, 2, 3, 4.
6,
[23] Byungju Kim and Junmo Kim. Adjusting decision boundary for class imbalanced learning. IEEE Access. 8:81674-8166 2020. 5,
[24] Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well well-founded and casily implemented improvement on logistic calibration for binary classifiers. In AISTATS, 2017. 3, 7,
[25] Meelis Kull, Miquel Perello-Nicto, Markus Kängsepp, Hao Song, Peter Flach, et al. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration. In NewrIPS, 2019. 3
[26] Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, and Alan L Yuille. The secrets of salient object segmentation. In CVPR, 2014. 1
[27] Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In CVPR, 2020. 1, 2, 3, 6, 7, 8
[28] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 4,
[29] Tsung-Ye Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 1,7.8
[30] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017. 1,9
[31] Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep lcarning for generic object detection: A survey. International Journal of Computer Vision (IJCV), 128(2): 261-318, 2020. 1
[32] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yann Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV, 2016.
[33] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tai tail learning via logit adjustment. In ICLR, 2021. 1, 2,
[34] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilitics using bayesian binning. In AAAI, 2015. 7,8
[35] Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance prohlems in object detection; review. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
[36] Constantine P Papageorgion, Michael Oren, and Tomaso Poggio. general framework for object detection. In ICCV, 1998.
[37] Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, and Xiaowei Zhou. Deep snake for real-time instance segmentation. CVPR, 2020.
[38] John Platt al. Probabilistic outputs for support vector machines and comparisons to regularized likelihond methods. Advances Large Margin Classifiers, 10(3):61-74, 1999. 3, 7,
[39] Vignesh Ramanathan, Rui Wang, and Dhruv Mahajan. DLWL: Improving detection for lowshot classes with weakly labelled data. CVPR, 2020.
[40] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, 2017. 1
[41] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-timo object detection. In CVPR, 2016. 1
12

---

[42] Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In NewrlPS, 2020. 1, 6,9
[43] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI),
39(6):113-1119, 2016. 1, 4,
[44] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019. 6
[45] Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In ECCV, 2016. 3
[46] Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-taibe tailed object recognition. In CVPR, 2020. 6.7.
[47] Jingru Tan, Gang Zhang, Hanming Deng. Changbao Wang. Lewei Lu, Quanquan Li, and Jifeng Dai. 1st place solution of Ivis challenge 2020: A good box is not guarantee of good mask. arXiv preprint MmXiv:2009001555 2020.
[48] Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li. Equalization loss v2: A new gradient balance approach for long-tailed object detection. In CVPR, 2021. 1, 6,
[49] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In NeurlPS, 2020. 3
[50] Junjiao Tian, Yen-Cheng Liu, Nathan Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re-calibration for imbalanced datasets. In NewrlPS, 2020. 3
[51] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018. I
[52] Paul Viola, Michael Jones, et al. Robust real-time object detection. International Journal of Computer Vision (IJCV), 4(34-47):4, 2001. 1
[53] Jiaqi Wang. Wenwei Zhang. Yuhang Zang. Yubang Cao. Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long- loilltailed instance segmentation. In CVPR, 2021. 1, 3.7
[54] Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Jun Hao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng- Classification calibration for long-tail instance segmentation. arXiv preprint arXiv: 2019. 2,
[55] Tao Wang. Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng. The devil is in classification: simple framework for long-tail instance segmentation. In ECCV, 2020. 1, 3,
[56] Tong Wang, Yousong Zhu, Chaoyang Zhao, Wei Zeng. Jinqiao Wang, and Ming Tang. Adaptive class suppression loss for long-tail object detection. In CVPR, 2021.
[57| Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. In ICML, 2020. 3,
[58] Jialian Wu, Liangchen Song. Tiancai Wang. Qian Zhang, and Junsong Yuan. Forest R-CNN: Large- vocabulary long-tailed object detection and instance segmentation. In ACM MM, 2020.
[59] Tong Wu, Ziwei Liu, Qingqiu Huang, Yu Wang, and Dahua Lin. Adversarial robustness under long-tailed distribution. In CVPR, 2021. 2,
[60] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In CVPR, 2017. 7
[61] Han-Jia Ye, Hong- feature deviation in imbalanced deep learning, arXiv preprint a3Xiv:2001.01385, 2020. 2, 3, 4, 5, 6, 7, 8
You You Chen, De-Chuan Zhan, and Wei-Lam Chao. Identifying and compensating for
[62] Han-Jia Ye, De-Chuan Zhan, and Wei-Lun Chao, Procrustean training for imbalanced deep learning. In ICCV, 2021. 5
[63] Bianca Zadrozny and Charles Elkan, Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In ICML, 2001. 7,
13

---

[64] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In SIGKDD, 2002. 8.8
[65] Yuhang Zang, Chen Huang, and Chen Change Loy. FASA: Feature augmentation and sampling adaptation for long-tailed instance segmentation. In ICCV, 2021. 3
[66] Cheng Zhang, Tai-Yo Yu Pan, Yandong Li, Hexiang Hu, Dong Xuan, Soravit Changpinyo, Boqing Gong, and Wei-Lun Chao. MosaicOS: simple and effective of object-centric images for long-tailed object detection. ICCV, 2021. 3,7
[67] Songyang Zhang. Zeming Li. Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: unified framework for long-tail visual recognition. CVPR, 2021.
[68] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl, Joint coco and lvis workshop at ECCV 2020: LVIS challenge track technical report: CenterNet2.
[69] Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. Capturing long-tail distributions of object subcategories. CVPR, 2014.
[70] Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey, arXiv preprint 1rXiv:1005.00055, 2019.
14


Reviews:
[{"review_id": "LMZv7z2ale5", "reviewer": "Reviewer_bKAV", "summary": "This paper presents a normalized calibration method for long-tailed object detection and instance segmentation task. Experimental resutls show that separately treating the background class and normalizing the scores over classes for each proposal are keys to achieving superior performance. Experiments are conducted on the LVIS dataset with diverse evaluation sets. Results shows that consistent improvement can be achieved by the proposed scheme, and  extensive analysis and ablation studies are provided.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper apply simple post-calibration techniques from standard multi-way to object detection and instance segmentation. The main idea is to scale down the logit of each class with respect to its train size. \nOverall, the paper is well organized, and the proposed scheme is clearly presented. The idea is straightforward and easy to implement, as only Hyper-parameter is introduced for tune in the scheme. The extensive experiments offer insights into various modeling choices and mechanisms of the scheme.\nIt would be better to conduct experiments on a more general dataset, (e.g. Open Images) to demonstrate its effectiveness.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors doesn't address the limitation and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "KXAwftgtVQ3", "reviewer": "Reviewer_aBnL", "summary": "This paper presents a simple yet effective approach “Normalized Calibration” to post-process the class-specific logits from modern object detector using soft-max prediction for all object and one background classes. Experiments on LVIS has been conducted, and comparison with quite a few baseline methods are presented to show its superior performance for frequent, common and rare classes.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea seems to be original, which is to scale down the exponentialized logits by the class size in the training set and L1-normalize scores. Without L1-normalization, the calibration has a similar form to tar-normalization in [20] (e.g, scale down the logits before taking exponentiation) (also see table 3). It seems L1-normalization plays a more important role than logit calibration.\n\nThe paper writing is clean and easy to follow. External comparisons and internal ablative studies are convincing. There are enough clarity in the technical exposition.\n\nOne limitation is NorCal seems to work less well for detectors predicting sigmoid scores that are not necessarily L1-normalized (e.g. RetinaNet with focal loss). I would like to see more results using detectors using sigmoid rather than softmax to compute class scores.\n\nCurrently, only one task and one dataset LVIS is used for evaluation. The proposed NorCal seems to be generic enough to be evaluated on long-tailed classification such as ImageNet-LT and iNaturalist benchmark. Results will be more convincing if results on a 2nd dataset or task are provided.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not have concerns on the negative societal impact of this work.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not have Ethical Concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hrs"}, "extra_text": {}}, {"review_id": "1YhnXn1407m", "reviewer": "Reviewer_PQBv", "summary": "In this paper, the authors study the problem of \"calibration\" of long-tail object detection models. To be specific, they take an existing pre-trained object detector and modify its logits according to the number of samples in each class to obtain better logits/scores for each class. They demonstrate on the long-tailed LVIS dataset with different models that this significantly and consistently improves the performance of methods (in terms of both precision and recall). ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- A simple yet robust and generic method.\n- Extensive ablation analysis. \n- Strong results.\n- Clear and easy to follow text.\n\nWeaknesses:\n- The paper provides a very simple application of the normalization methods in other studies for object detection. In this sense, technical/methodological contribution is limited. Though, I favor simple solutions and I am not going to reflect this in my recommendation.\n\n- This is a paper for addressing an imbalance problem for object detection and therefore, the authors should cite the following review paper on imbalance problems in object detection:\n\nOksuz et al., \"Imbalance Problems in Object Detection: A Review\", IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2021.\n\n- Figure 2: It is not clear how these scores are obtained. Which split of the dataset is used? Which detector is used? How did you perform normalization for the plot on the left? Are the scores on the right not normalized? Why not? What do you mean by \"another rare\" classes? .. \n\nMinor comments:\n- Figure 2: It would be nicer if the font sizes & styles of the text in the figure match the font size and the style of the main text. Moreover, it is very clear that the spacing between the figure and the text has been played with. This is against the formatting instructions.\n- Footnote 2 on page 4: The period (.) is missing.\n- Line 133: N_c has not been introduced yet.\n- Line 136: \"confidential scores\" => \"confidence scores\".\n- Line 148: \"posts\" => \"posits\".\n- Line 149: \"background patches\" => \"background regions\".\n- Line 231: \"methods that trained\" => \"methods that are trained\".\n\n**AFTER REBUTTAL**\n\nI've looked at the comments of the other reviewers and noticed that we are generally very positive about the paper. Some of us requested more experiments and the authors provided the results for them in their rebuttal, which supported the claims of the authors. Therefore, I will keep my original score of acceptance for the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "1SB3EOYI78", "reviewer": "Reviewer_3gEu", "summary": "A simple and effective post-processing calibration method is proposed for long-tailed object detectors. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- The paper presents an interesting idea. It is quite effective and requires no re-training of the object detectors, making it a useful and low-effort tool to use. Experiments also show superior performance than other methods which requires a change of training.\n- The experiments revealed that the key to performance improvement is normalization especially for the frequent classes, which is an important result and helps us better understand how the method works. \n- The paper is well written and clear. The illustrations to explain why the proposed method works are very useful.\n\nWeakness\n- The proposed method is purely about post-processing of classification scores in object detectors. Thus the localization part is not important. Therefore it makes sense to start by studying the long-tailed image classification problem and then apply it to object detection/instance segmentation problem.  It'll be better if the authors can show the efficacy of this method on classification problems as well.\n- It makes sense that the proposed method works well with multi-class classifiers. However, for multiple binary classifiers, as described in Section 3.4 and Eq. 7, there's no normalization applied. In this case, the ranking of the boxes per class won't be changed because Eq. 7 is a monotonic function. So how is the performance improvement achieved?\n- Although the proposed method works well in the experiments, the overall novelty is not high. \n\nMinor comments\n- In Eq. 7, should it be dividing by the factor instead of multiplying it?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were generally positive about this paper, and most of the raised concerns were cleared after the responce. Thus I recommend accepting the paper.
Decision: Accept (Poster)