Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environments
Authors: Yang Yang
Wenhai Wang
Zhe Chen
Jifeng Dai
Liang Zheng
Venue: ICLR 2024 spotlight (2024)
Keywords: Object Detection
Model Generalization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Predict the accuracy of detectors on various real-world test sets without accessing test ground truths.

Abstract: Bounding boxes uniquely characterize object detection, where a good detector gives accurate bounding boxes of categories of interest. However, in the real-world where test ground truths are not provided, it is non-trivial to find out whether bounding boxes are accurate, thus preventing us from assessing the detector generalization ability. In this work, we find under feature map dropout, good detectors tend to output bounding boxes whose locations do not change much, while bounding boxes of poor detectors will undergo noticeable position changes. We compute the box stability score (BS score) to reflect this stability. Specifically, given an image, we compute a normal set of bounding boxes and a second set after feature map dropout. To obtain BS score, we use bipartite matching to find the corresponding boxes between the two sets and compute the average Intersection over Union (IoU) across the entire test set. We contribute to finding that BS score has a strong, positive correlation with detection accuracy measured by mean average precision (mAP) under various test environments. This relationship allows us to predict the accuracy of detectors on various real-world test sets without accessing test ground truths, verified on canonical detection tasks such as vehicle detection and pedestrian detection.

Published as a conference paper at ICLR 2024
BOUNDING Box STABILITY AGAINST FEATURE DROPOUT REFLECTS DETECTOR GENERALIZATION ACROSS ENVIRONMENTS
Yang Yang-.2r, Wenhai Wang Zhe Chen 4.21, Jifeng Dais.2, Liang Zheng'
I "The Australian National University 'OpenGVLab, Shanghai AI Laboratory 'hhe Chinese University of Hong Kong "Nanjing University 'Tsinghua University yangna au, whwange ie. cuhk. edu.hk, #zc294c2 @eg.almmaa...o.oom.ooo... [dai (daijifengets inghua edu. en, llang......... zheng@anu. ad
ABSTRACT
Bounding boxes uniquely characterize object detection, where a good detector gives accurate bounding boxes of categories of interest. However, in the real- world where test ground truths are not provided, it non-trivial to find out whether bounding boxes are accurate, thus preventing us from assessing the detector gen- eralization ability. In this work, we find under feature map dropout, good detec- tors tend to output bounding boxes whose locations do not change much, while bounding boxes of poor detectors will undergo noticeable position changes. We compute the box stability score (BoS score) reflect this stability. Specifically, given an image, we compute a normal set of bounding boxes and a second set after feature map dropout. To obtain Bo$ score, we use bipartite matching to find the corresponding boxes between the two sets and compute the average In- tersection over Union (IoU) across the entire test set. We contribute finding that BoS score has a strong, positive correlation with detection accuracy mea- sured by mean average precision (mAP) under various test environments. This relationship allows us to predict the accuracy of detectors on various real-world test sets without accessing test ground truths, verified canonical detection tasks such as vehicle detection and pedestrian detection. Code and data are available at https:/ / g thub. com/ Yang Langggannnrr/BBS.
INTRODUCTION
Object detection systems use bounding boxes to locate objects within an image (Ren et al., 2015; Lin et al., 2017; He et al., 2017), and the quality of bounding boxes offers an intuitive understand- ing of how well objects of a class of interest, e.g., vehicles, are detected in an image. To measure detection performance, object detection tasks typically use the mean Average Precision (mAP) (Ev- eringham et al., 2009) computed between predicted and ground-truth bounding boxes. However, is challenging to determine the accuracy of bounding boxes in the absence of test ground truths in real-world scenarios. which makes it difficult to evaluate the detector generalization ability.
In this work, start with specific aspect of bounding box quality, i.c., bounding box stability under feature perturbation. Specifically, given an image and a trained detector, we apply Monte Carlo (MC) dropout (Gal & Ghahramani, 2016) in the testing processing, which randomly set some elements in certain backbone feature maps to zero. Afterward, we use bipartite matching to find the mapping of bounding boxes with and without dropout perturbations, and then compute their intersection over union (IoU), called box stability score (BoS score). As shown in Fig. 1(a) and (b), under feature perturbation, bounding boxes remaining stable (with high BoS score) would indicate relatively good prediction, while bounding boxes changing significantly (with low BoS score) may suggest poor predictions.
Based on this idea, we contribute to discovering strong correlation between bounding box stability and detection accuracy under various real-world test environments. Specifically, given a trained de-
This work done when they are interns at Shanghai AT Laboratory.

---

Published as a conference paper at ICLR 2024
tector and set of synthesized sample sets, compute BoS score and actual mAP over each sample set. As the strong correlation (i.e., coefficients of determination R² > 0.94) shown in Fig. ((c), a detector overall would yield highly stable bounding boxes and thus have good accuracy (mAP) in relative environments, and in difficult environments where the detector exhibits low bounding box stability, object detection performance can be poor.
This finding endows us with the capacity to estimate detection accuracy in various test en- vironments without the need to access ground truth. This is because the BoS score (1) does not rely on labels in computation, which en- codes the difference between original predic- tions and predictions after feature dropout, and (2) strongly correlated with detection accu- racy. In fact, it is well known that detection ac- curacy drops when the test set has different dis- tribution from the training set. That said, with- out ground truths, it is challenging how to esti- mate test accuracy under distribution shifts. As a use case, being able to obtain model accuracy without ground truths allows us efficiently assess systems deployed in new environments.
Original Delection
Perturbed Detection
(a) Unstable Predictions
(b) Stable Predictions
35
mAP (%)
30
=²=0.943
p-0.937
sample set
Several methods (Deng Zheng, 2021; Guil- lory et al., 2021; Deng et al., 2021; Sun et al., 2021: Li al., 2021: Garg et al., 2022), col- lectively referred to as AutoEval, have emerged to predict the classifier performance in new unlabeled environments. But these methods are poor at predicting detector performance be- cause they do not take into account the regres- sion characterization of the detection task. With the aforementioned box stability score mea- surement, we can effectively estimate the de- tection accuracy of detector without ground truth. It is also the first method for AutoEval in object detection.
0.50
0.62
(c) Box Stability Score
Figure 1: Visual examples illustrating the corre- lation between bounding box stability and detec- tion accuracy. (a) Bounding boxes change sig- nificantly upon MC dropout, where we observe complex scenarios and incorrect detection results. (b) Bounding boxes remain at similar locations af- ter dropout, and relatively correct detection results are observed. (c) We observed that the box stabil- ity score is positively correlated to mAP (coeffi- cients of determination I²2 > 0.94, Spearman's 's Rank Correlation p > 0.93).
In sum, our main contributions are as follows:
(1) We report a strong positive correlation between the detection mAP and BoS score. Our proposed BoS score calculates the stability degree for the bounding box with and without feature perturbation by bipartite matching, which is an unsupervised and concise measurement.
(2) We demonstrate an interesting use case of this strong correlation: estimating detection accuracy in terms of mAP without test ground truths. To our best knowledge, we are the first to propose the problem of unsupervised evaluation of object detection. For the first time, we explore the feasibility of applying existing AutoEval algorithms for classification to object detection.
(3) To show the effectiveness of the BoS score in estimating detection mAP, this paper conducts experiments on two detection tasks: vehicle detection and pedestrian detection. For both tasks, we collate existing datasets that have the category of interest and use leave-one-out cross- validation for mAP estimator building and test mAP estimation. On these setups, we show that our method yields very competitive mAP estimates of four different detectors various unseen test sets. BoS score achieves only 2.25% RMSE on mAP estimation of the vehicle detection, outperforming confidence- based measurements by at least 2.21 points.
RELATED WORK
Mainstream object detectors are often categorized into two types, two-stage detectors (Ren et al., 2015; Cai & Vasconcelos, 2019) predict boxes w.r.t. proposals, and single-stage methods make pre-
2

---

Published as a conference paper at ICLR 2024
dictions w.r.t anchors (Lin al., 2017) grid of possible object centers (Tian al., 2019; Duan et al., 2019). Recently, transformer- based methods have shown an incredible capacity for object detection, including DETR (Carion et al., 2020), Deformable DETR (Zhu et al., 2020), and DINO (Zhang et al., 2022), etc. In the community, numerous popular datasets are available for benchmark- ing detector performance, such COCO (Lin al., 2014) and ADE20K (Zhou al., 2019), and so on, Nonetheless, due to the distribution shifts, the results on test sets in public benchmarks may not accurately reflect the performance of the model in unlabeled real-world scenarios. To address this issue, we focus on studying detector generalization ability on test sets from various distributions.
Label-free model evaluation predicts performance of certain model on different test sets (Madani et al., 2004; Donmez et al., 2010; Platanios et al.. 2016; 2017: Jiang et al., 2019). Some works (Guil- lory et al., 2021; Saito et al., 2019; Hendrycks Gimpel, 2016) indicate classification accuracy by designing confidence-haal hased measurements. In the absence of image labels, recent works (Deng & Zheng, 2021; Deng et al., 2021: Sun et al., 2021) estimated model performance for AutoEval with regression. Deng & Zheng (2021); Sun et al. (2021); Peng et al. (2024) used feature statistics to represent the distribution of a sample dataset to predict model performance. Deng et al. (2021) es- timated classifier performance from the accuracy of rotation prediction. ATC (Garg al., 2022) predicted accuracy as the fraction of unlabeled examples for which the model confidence exceeds the learned threshold. Different from the above works designed for the classification task, we aim to study a brand-new problem of estimating the detector performance in an unlabeled test domain.
Model generalization aims estimate the generalization error of model unseen images. Sev- eral works (Neyshabur et al., 2017; Jiang et al., 2018; Dziugaite et al., 2020) proposed various complexity measures on training sets and model parameters, to predict the performance gap of model between a given pair of training and testing sets. Neyshabur et al. (2017) derived bounds on the generalization gap based on the norms of the weights across layers. Jiang al. (2018) de- signed a predictor for the generalization gap based on layer-wise margin distributions. Corneanu et al. (2020) derived a set of topological summaries to measure persistent topological maps of the behavior of deep neural networks, and then computed the testing error without any testing dataset. Notably, these methods assume that the training and testing distributions are identical. In contrast, our work focuses on predicting the model performance test sets from unseen distributions.
3 STRONG CORRELATION BETWEEN BOUNDING Box STABILITY AND DETECTION ACCURACY
3.1 MEAN AVERAGE PRECISION: A BRIEF REVISIT
Mean average precision (mAP) (Everingham et al., 2009) is widely used in object detection evalua- tion. We first define a labeled test set D' {I., } in which x; is an image, ys is its ground truth label, and M is the number of images. Given trained detector fo v; - Dt. which is pa- rameterized by 0 and maps an image x; to its predicted objects ÿt. Since the D' is available, we can obtain the precision and recall curves of the detector by comparing the predicted objects y; with the ground truths yi- The Average Precision (AP) summarises the area under the precision/recall curve. mAPso and mAP75 correspond to the average AP over all K categories under 0.50, 0.75 IoU threshold, respectively. mAP corresponds to the average AP for preconfigured IoU thresholds.
In unsupervised detector evaluation, given detector fe and an unlabeled dataset 2 {r want to find a mAP predictor A : (fe. D") + mAP, which can output an estimated mAP E of detector fo this test set:
AM
[0, 1]
mAP mAPunsup unsup A (fo.
3.2 MEASUREMENT OF BOUNDING BOX STABILITY
This paper uses an existing technique (MC dropout (Gal & Ghahramani, 2016)) for measurement. Let us define Nori the number of objects detected in an image by the original detector, Nper as the number of objects detected in the same image by the perturbed detector with MC dropout. Let N and Nmax represent the smaller or larger value between Nori and N Yper, respectively. On this basis, denote the set with the fewest objects in the predicted set as y = (u5 ) and the most as
3

---

Published as a conference paper at ICLR 2024
40.2
727 pm00758
BSd DETRAC
p=0.937
IAP (%)
25.5
KIII Hizestow Traffi"
16.0
30.0
25.0
25.5
87-00750
20.0
2-00330
cm
0.24
0.85
0.00
(a) Prediction Score
(b) Avcrage Thresholded Confidence
(c) Box Stability Score (ours)
Figure 2: Correlation between different measurements and mAP "A" of different colors represents nine real-world datasets, such as BDD (Yu al., 2020) and Cityscapes (Cordts al., 2016), which are used as seeds to generate sample sets by various transformations. Each point " of different colors represents sample generated from different seed sets. mAP is obtained by RetinaNet (Lin et al., 2017) trained on the COCO training set. Three measurements including two existing methods (Hendrycks Gimpel, 2016; Garg al., 2022) and our box stability score are used in (a), (b), and (c). The trend lines are computed by linear regression, from which we can observe strong linear relationship (coefficients of determination R² 0.94, Spearman's Rank Correlation p 0.93) between box stability score and mAP.
ý ( kmr. To find a bipartite matching between these two sets, we search for a permutation of N elements a E Gw with the lowest cost:
= arg min OEEN
Σ Smatch (Uz, (o(s))
(2)
where Lmatch (Nis Be(i) is pair-wise matching cost between the original prediction and the per- turbed prediction with index j and index o(j.. respectively. This optimal assignment computed efficiently with the Hungarian algorithm following previous works (Kuhn, 1955; Carion et al., 2020).
The matching cost considers the similarity between the predicted boxes of the original detector and the perturbed detector. However, using the widely-used l loss will lead to different scales for small and large boxes, even if their relative errors are similar. Therefore, we adopt the GloU loss (Rezatofighi et al., 2019) and define the matching loss L(v, 9) as:
Σ/ Lgiou(by bo).)) N
Cly, 0)
where by and Bargs [0, 1]3 denote pair of matched boxes with index j and index o()) respectively. After that, we average the matching loss over all M test images D and obtain the box stability score of the unseen test set D
BS(D") = Gi Li
(4)
M
USE CASE: LABEL-FREE DETECTOR EVALUATION
Following the practice in (Deng Zheng, 2021), we formulate label-free evaluation as a regression which
problem. First, we create training meta-dataset composed of fully labeled sample transformed versions seed sets. Second, for each sample set, we compute the mAP and BoS score and train a regression model using BoS score as input and output mAP. Third, we use real-world test sets which are unseen during training, for mAP estimation.
4.1 TRAINING -ATA--
Take mAP estimation on vehicle detection as an example. We generate 500 sample sets from seed sets mentioned in Section 5.4. We employ the leave-one-out evaluation procedure, where each time we use 450 labeled sample sets (D',Di... generated from 9 out of 10 sources as the training meta-set. All images from the remaining source are 10 different permutations to construct the training meta-set and test set. We set the maximum number of testing images 5,000, the number of images in the test set differs in each permutation.
real- real-world test set D" Therefore, there
4

---

Published as a conference paper at ICLR 2024
4.2 REGRESSION MODEL TRAINING AND REAL-WORLD TESTING
From the strong correlation of box stability score and mAP shown in Fig. 2(c), we train an mAP estimator (i.e., linear regression model) on the previously constructed training meta-set to fit this re- lationship. Given detector fo, the linear regression model Alinear uses its BoS score input and the corresponding mAP as target. The i-th training sample of Atinear can be denoted as {BS,, mAP;). }. The linear regression model Atinear is written as,
mAP; = Alinaar (So, for D.) wi WSB + wo,
(5)
where f/ denotes the perturbed detector from fa, and wo; Wi are parameters of the linear regression model. We use standard least square loss to optimize this regression model. During real-world testing, given an unlabelled test set D and detector fe. compute the BoS score and then predict detector accuracy mAP using the trained mAP estimator.
5 EXPERIMENT
5.1 EXPERIMENTAL DETAILS
Datasets. We study label-free detector evaluation on two canonical object detection tasks the main experiment: vehicle detection and pedestrian detection, verify method. For vehicle detection, we use 10 datasets, including COCO (Lin et al., 2014), BDD (Yu et al., 2020), Cityscapes (Cordts al., 2015), DETRAC (Wen al., 2020), Exdark (Loh & Chan, 2019) Kitti (Geiger et al., 2013), Self-driving (Kaggle, 2020a), Roboflow (Kaggle, 2022), Udacity (Kaggle, 2021), 1), Traffic (Kaggle, 2020b). For pedestrian detection, we use 9 datasets, including COCO (Lin et al., 2014), Caltech (Griffin et al., 2007), Crowdhuamn (Shao et al., 2018), Cityscapes (Cordts et al., 2015), Self-driving (Kaggle, 2020a), Exdark (Loh & Chan, 2019), EuroCity (Braun al., 2019), Kitti (Geiger et al., 2013), CityPersons (Zhang et al., 2017). Note that the annotation standard of the datasets on the category of interest should be uniform.
Metrics. Following (Deng & Zheng, 2021), we use root mean squared error (RMSE) as the metric to compute the performance of label-free detector evaluation. RMSE measures the average squared difference between the estimated mAP and ground truth mAP. A small RMSE corresponds to good performance. Our main experiments are repeated 10 times due to the stochastic nature of dropout, after which mean and standard deviation reported.
Detectors. In the main experiment, we employ pre-trained ResNet-50 (He et al., 2016) as the back- bone and one-stage RetinaNet (Lin et al., 2017) the detection head. Some other detector archi- tectures, including Faster R-CNN (Ren et al., 2015) with R50, Faster R-CNN with Swin (Liu et al., 2021), RetinaNet with Swin, are also compared in Table 4 (left).
Compared methods. Existing methods in label-free model evaluation, including prediction score (PS) (Hendrycks & Gimpel, 2016), entropy score (ES) (Saito et al., 2019), average confidence (AC) (Guillory et al., 2021), ATC (Garg et al., 2022), and Fréchet distance (FD) (Deng & Zheng, 2021), are typically experimented on image classification. For comparison, we extend them to object de- tection. Because PS, ES, AC, and ATC are based on the softmax output, we use the softmax output of the detected bounding boxes compute these scores. To compute FD, we use the feature from the last stage of the backbone to calculate the train-test domain gap. Hyperparameters of all the compared methods are selected on the training set, such that the training loss is minimal.
Variant study: comparing bounding box stability with confidence stability. Using the same MC dropout and bipartite matching strategy, we can also measure the changes in bounding box softmax output instead of bounding box positions. In Fig. 3(c), we compare this variant with BoS score of their capability in estimating detection accuracy. Results show that the bounding box stability score is superior to confidence stability score. A probable reason is that confidence stability score does not change much due to over-confidence even the bounding box changes significantly. So it is less indicative of detection performance.
5

---

Published as a conference paper at ICLR 2024
Self-driving
Ave. RMSE
Citscapes
DETRACC
Roballow
COCO
BDD
31.5
40.2
41.1
Exdark
28.0
Kiiti
41.2
Self
30.5
32.8
Uducity
Tralii
25.2
Method
PS (llendrycks & Gimpel, 2016)
8.14
3.22
5.50
15.83
0.81
11.80
1.26
2.48
1.54
1.99
5.26
ES (Saito et sl., 2019)
7.06
1.37
8.40
16.20
0.14
13.08
0,93
3.14
1.23
3.60
5.52
AC (Guillory et al., 2021)
6.67
3,32
9.60
30.95
1.26
13.93
2.31
1.77
2.92
3.30
7.60
ATC (Garg al., 2022)
5.83
8.21
1.44
6.34
4.46
FD (Deng Zheng, 2021)
9.17
2.78
13.03
12.29
5,94
14.80
2.22
4.48
1.32
4.14
7,02
BoS (ours)
1.26
1.90
0.89
1.84
1,47
4.3811o
1,92
1.61 4.79
2,43.1.
2.25
BoS + PS
2. Loa 2.03gmme
1.13
0.04
1,22
5,56
1,71
4.09
1.42
2.24
2.24
Table 1: Method comparison of mAP estimation for vehicle detection. Specifically, we collect 10 source datasets, including COCO (Lin et al., 2014), BDD (Yu et al., 2020), and so on. For each column, we keep one original dataset as the unseen test set, and synthesize the meta-set using remained 9 sources. Given a detector trained on a COCO training set, we report its ground truth mAP (%) on different unseen test sets in the header and employ RMSE to measure the accuracy of the mAP estimation. For "Prediction Score" and "Entropy Score" methods, boxes with high prediction scores or low entropy scores are regarded as being detected correctly, respectively.
CrowdHuman
City
scapes
Self-driving
EuroCky
CityPersom
Avg. RMSE
CocO
Caltoch
Exdark
36.7
16.2
33.5
19.0
16.4
21.6
16.8
Kitti
13.3
11.8
Method
PS (Henddyckks Gimpel, 2016)
8.49
3.68
6.55
0,76
3.20
5.58
0.86
0.99
6.28
4.04
ES (Saito al., 2019)
5.91
1.02
9.44
1.06
1.63
3.64
0.13
2.23
5.71
3.42
AC or ATC (Guillory et 2021)
2021)
5.25
1.38
17.25
0.79
1.33
3.18
0.39
1.69
5.99
4.13
FD (Deng & Zheng, 2021)
3.88
3.29
3.93
2,98
(ours)
2.26+
2.53
6.39++x
4.42+taa
0.74++um
1.72+aa
5.16+1a
3.29
BoS PS
2.04
2.18
4.63
0.43
a.29
0.58
2.33
3,99
1.48
Table 2: Method comparison of mAP estimation for pedestrian detection. The settings same as those in Table I, except that datasets are collected from COCO (Lin et al., 2014), Caltech (Griffin al., 2007), Crowdhuman (Shao al., 2018), CityPersons (Zhang et al., 2017) et al.9 datasets and the detector is trained on a CrowdHuman training set.
5.2 MAIN EVALUATION
Comparison with state-of-the-art methods. We compare our method with existing accuracy eval- uators that are adapted to the detection problem. After searching optimal thresholds on the training meta-set, we set 71 = 0.4, 0.95, and 0.3 for ATC (Garg et al., 2022), PS (Hendrycks & Gimpel, 2016), and ES (Saito et al., 2019), respectively. As mAP estimation for vehicle detection is shown in Table 1, through the leave-one-out test procedure,
method achieves the lowest average RMSE 2.25% over ten runs. The second- best mAP estimator, ATC, yields an average RMSE = 4.46%. Across the ten test sets, method gives consistently good mAP estimations, evidenced by the rel- atively low standard deviation of 13%. Specifically, on COCO, BDD, and Cityscapes, our method yields the lowest RMSE of 1. 1.26%, 1.90%, 0.89%, respectively.
In Table 2, our method is compared with existing ones on label-free evaluation of pedestrian detec- tion. We have similar observations, where BoS score makes consistent predictions across the six test scenarios, leading to the lowest average RMSE. These results demonstrate the usefulness of our key contribution: the strong correlation between BoS score and mAP.
Using BoS score for multi-class detectors. BoS score can be easily extended to multi-class de- tectors theoretically because of its class-wise nature. To verify this, we should collect many object
COCO
Ciryscapes
Exdark
Kitti
Self-driving
RMSE
Method
person
car
person
car
person
car
person
car
person
car
43,08
33.98
23.88
39.66
23,94
28.05
11.85
43.66
14.68
31,66
FS
3.33
2.55
1.49
1.07
6.49
3.92
2.26
17.47
5.94
0.37
4.69
15.15
2.21
1.21
2.47
4.71
4.67
oC ATC
11.58
3,74
1.86
1.68
3,70
3.46
3.20
16.72
3.97
1.38
5,13
FD
10.50
3.79
1.93
0.90
5.71
2.84
3.35
13.55
6.79
2.91
5.23
BoS tours)
0.5
1.75
1.451
7.72
1,83
1,654
4.55
12.53
2.68
3,41
4.29-mmm
BoS + PS
11.16-034
2.57 use 2.04+
071
5.46 0.3
7.09
1.44
1.13
068
2,88 0.20
10.53+
0.3
2.26
047
3.91
057
4.90 0.66
Table 3: Method comparison of mAP estimation for multi-class detection. The settings same as those in Table I, except we implemented BoS score on the 5 datasets that contain pedestrians and vehicles simultaneously, including COCO, Cityscapes, Exdark and so on.

---

Published as a conference paper at ICLR 2024
Method
RetinaNet
Faster R-CNN
Method
RetimaNet RecNet-50
Swin 1
ResNet-50 50
Swin-T
mAP
mAPs0
mAPys
PS
5.71
5.26
5.34
5.21
FS
5.26
10.13
5.36
ES
5.26
6.26
6.63
5.84
ES
6.25
10.33
6.01
AC
6.27
7.60
5.61
5.02
AC
7.60
12.86
7.89
ATC
3,44
4.46
5.36
5.85
ATC
4.46
6.36
6.92
FD
8,70
7.02
8.54
7,44
FD
7.02
10.87
8.84
BoS (ours)
4.69:
2.25
3.79 4.83
Bos (ours)
2.25
4.84
3,47 +0.15
Table 4: Left: mAP prediction comparison under four detector structures. All the detectors are trained on COCO. Row 1: detector heads, i.e. RetinaNet and Faster R-CNN. Row 2: backbones, i.e. Swin-T and ResNet-50. Right: Comparing methods of their capabilities in predicting mAP, mAPso and mAP7s- We use the retinanet+RS0 R50 detector trained on COCO. RMSE (%) is reported.
Trining Mota-sel I
Training Metarsct 2
Training Mottesst 3
Bod-ES
Ood Of (W)
(P2)
Bus+FD
Bcb ATC
Pusips
Coet
0.86
Beb
8
821
(82al
0.22
0.05
0.1
0.15
02
0.5 03
6345
Ang RMSE
(a) Dropout Position
(b) Dropout Rate
(c) Fused Dataset-level Statistics
Figure 3: Comparing different (a) dropout positions and (b) dropout rate. We plot the coefficients of determination R obtained during training under different dropout configurations and matching times, measured on three different training meta-sets. We use greedy search for their optimal values during training, so that R is maximized. "[O]" means adding dropout layer after stage 0 of the backbone, "[O, 1J" means adding a dropout after stage 0 and stage I of the backbone respectively, and so (c) AutoEval performance (RMSE, %) of confidence stability score (CS score), box stability score (BoS score), and fused dataset-level statistics. No obvious improvement is observed after fusion. "n.s." represents that the difference between results is not statistically significant (i.e., p value > 0.05). corresponds to statistically significant (i.e., 0.01 < p value < 0.05). ** means the difference between results is statistically very significant (i.e., 0.001 < p-value < 0.01).
detection datasets that have diverse appearances and the same label space. We tried and found datasets that allow us to do pedestrian and vehicle detection. We implement BoS and report results in Table 3, where BoS is shown to be competitive compared with existing methods. Besides, we observe that all methods might not work as well in multi-category as it does in single-class in some datasets. This because when confronted with multi-class mAP predictions, the features between different categories may interfere with each other.
Effectiveness of our method for different detector structures. Apart from the detector structure composed of RetinaNet (Lin et al., 2017) head with ResNet-50 (He et al., 2016) 6) backhone evaluated in Table 1, we further apply this method for other structures such as Faster R-CNN (Ren et al., 2015) head with Swin (Liu et al., 2021) backbone on vehicle detection, and results summarized in Ta- ble 4 (left). Compared with existing methods, our method yields very competitive mAP predictions. For example, using Re backbone and RetinaNet as head, the mean RMSE of method is 2.25% while the second best method ATC gives 4.46%. Our method is less superior under the Swin-T backbone. It is perhaps because the transformer backbone is different from the convolutional neural networks w.z.f attention block, which may not be susceptible to dropout perturbation.
Effectiveness of combining BoS score with existing dataset-level statisties. To study whether existing label-free evaluation methods are complementary, we combine each of Entropy Score, Pre- diction Score, ATC, and FD with the proposed BoS score. As shown in Fig. 3(c), combining exist- ing statistics with BoS score does not result in noticeable improvement. For example, the average RMSE of the "box stability score + prediction score" on the same level as using BoS score alone. It suggests that confidence in this form does not add further value to bounding box stability, which somehow consistent with the findings in the previous paragraph. Performance even deteriorates when we use other measurements such as Entropy, FD, and ATC. While current measures based on confidence are not complementary, we believe it is an intriguing direction for further study.
Estimating mAP and mAP75 without ground truths. This paper mainly predicts the mAP score, which averages over 10 IoU thresholds, i.e., 0.50, 0.55, .... 0.95. Here, we further predict mAPs0 and mAPrs. which use certain IoU threshold of 0.50 and 0.75, respectively. In Table (right), we observe that BoS score is also very useful in predicting mAP50 and mAP7s. yielding the mean RMSE of 4.8 4.84% and 3.47%, respectively, which are consistently lower than the compared methods.

---

Published as a conference paper at ICLR 2024
5.3 FURTHER ANALYSIS
Impact of the stochastic nature of dropout. Because dropout disables some elements in the feature maps randomly, in main experiments such as Table 1, Table and Table 4, we use multiple runs of dropout, train multiple regressors and report the mean and standard deviation. We observe that the standard deviations are relatively small, indicating that the randomness caused by dropout does not have a noticeable impact the system.
Impact of test set size. We assume access to an unlabeled test set, so that a dataset-level BoS score can be computed. In the real world, might access limited test samples, we evaluate our system under varying test set sizes. Results are provided in Fig. 4(c). Our system gives relatively low RMSE (~2.54%% when the number of test images is more than 50.
Hyperparameters selection: dropout position p {[0], [1], [ ..., [0, 1, 313 and dropout rate c E [0, 1]. For the former, there are 15 pos- sible permutations for inserting dropout layer in 4 backbone stages. "[O, IJ" means adding dropout after stage 0 and stage of the back- bone respectively, and so on. We search the optimal configuration on the training meta set in greedy manner, where the interval for is 0.05 and permutations for dropout position are evaluated one by one. We use the configuration which results in the highest determination coef- ficient Re between mAP and BoS score during training. Noting that hyperparameters are se- lected training (see Section 4.1 for training meta-set details). Only for the purpose visualization convenience, we show how dropout rate and position p affect the training coefficient of determination in Fig. 3(a) and (b), respectively. When visualizing the impact of one hyperparameter, we fix the other at optimal value found in training. For the RetinaNet+R50 vehicle detector, the best dropout rate e is 0.15, and the best dropout position setting is adding a dropout layer after Stage 1 and Stage 2 (p [1, 2)) of the backbone, respectively.
1.28
2.2
50 100
200 250
50 100 150 200 250 (e) Test Set Size
(4) Meta Set Size
(b) Sample Set Size
Figure 4: Impact of (a) meta-set size, (b) sample set size and (c) test size the performance of mAP estimators. RMSE (%) is reported. Gener- ally speaking larger meta-set and larger sample sets are beneficial for regression model training. Our system gives relatively low RMSE when the number of test images is more than 50.
Impact of meta-set size and sample set size. The seta-set size defined by the number of sample sets it contains, and the size of sample set is defined by the number of images it has. We test various meta-set and sample set sizes and report the results in Fig. 4(a) and (b), respectively. We observe that larger meta-set size sample set size leads to improved system performance measured by RMSE Our results are consistent with (Deng & Zheng, 2021).
5.4 STRONG CORRELATION
One of our key contributions is the discovery of this strong relationship between mAP and BoS score. To study the relationship between mAP and BoS score on various test sets, we first train detector, where we adopt RetinaNet+R50 (Lin et al., 2017) trained on the COCO (Lin et al., 2014).
Second, we construct a meta set. The meta-set is a collection of sample sets. To construct the meta-set, we collect various real-world datasets that all contain the vehicle categories as seed sets, including BDD (Yu et al., 2020), Cityscapes (Cordts et al., 2016), Kitti (Geiger et al., 2013) and so on (all datasets are listed in Fig. 2). To increase the number of datasets, we follow prior work (Deng & Zheng, 2021) to synthesize sample sets using various geometric and photometric transforms. Specif- ically, we sample 250 images randomly for each seed set, and randomly select three transformations from (Sharpness, Equalize, SolorTemperature Solarize, Autocontrast, Brightness, Rotate). Later, apply these three transformations with random magnitudes the selected images. This practice generates 50 sample sets from each source, and the sample sets inherit labels from the seeds.
Then, we simply compute the mAp scores and BoS scores (Eq. 4) of the detector on these test sets. Finally, we plot these mAP scores against the BoS scores in Fig. 2. We clearly observe from Fig. 2(c) that BoS score has strong correlation with mAP, evidenced by the high R² score and Spearman's 's Rank Correlation P. It means that, for a given detector, its detection accuracy would be high in environments where the predicted bounding boxes are resistant feature dropout; Similarly,
8

---

Published as a conference paper at ICLR 2024
the mAP score would be low in test sets where its bounding box positions sensitive to model perturbation. If we think about this from the perspective of test difficulty for a given detector, results suggest that the test environment is more difficult if detector outputs are less robust model perturbation, and vice versa.
We then compare the correlation produced by BoS score with prediction score (Hendrycks & Gim- pel, 2016) and average thresholded confidence (Garg et al., 2022). We observe that the correlation strength of our method is much stronger than the competing ones. Apart from the study in Fig. 2, more correlation results, including other types of detectors, other detection metrics like mAP and other canonical detection tasks like pedestrian detection, are provided in the supplementary materi- als, where our method gives consistently strong correlations.
5.5 DISCUSSIONS AND LIMITATIONS
Is BoS score a kind of confidence/uncertaintyy measurement in object detection? It charac- terizes confidence the dataset level, but might not operate the level of individual bounding boxes/images. In our preliminary experiment, we tried to correlate BoS score with accuracy on in- dividual images and even bounding boxes, but the correlation was little weak. We find BoS score is correlated with accuracy when the number of images considered is statistically sufficient.
measure the change of bounding box confidence? Yes. By bipartite matching, technically possible to compute the confidence difference between corresponding bounding boxes. But find that this confidence stability score (CS score) does not correlate well with accuracy. Please refer to the bottom two pillars in Fig. 3(c) for this comparison.
Using MC dropout,
Can BoS score be loss function for object detection during training? No. There are two reasons. (1) the BoS-mAP correlation might not hold when a model just starts training, whose performance on test set extremely poor. (2) BoS score is not differentiable, since its calculation involves the perturbed detection results which are constantly changing during training. However, we might improve detector performance from the insights of our paper. For example, we can apply a resistance module like MC-DropBlock (Deepshikha et al., 2021) to improve detector generalization.
Can other self-supervised signals exhibit good correlation with mAP? This work uses the pretext metric, box stability score, for label-free model evaluation, because it well correlates with mAP, There exist other existing pretext tasks for unsupervised object detection pre-training (Xiao et al., 2021; Dai et al., 2021; Bar et al., 2022). It is interesting to study in the future whether these methods also relate to mAP in target environments.
Limitations on the working scope. Strong correlation is observed when the detector has reasonably good performance (e.g., mAp > 15%) on the sample sets. a test set is extremely difficult on which mAP could be as low as 1-2%, the correlation might not hold. The reason is when mAp is extremely low, there are too many missed detections. Without bounding boxes, BoS score cannot be computed. In fact, similar observation is made in image classification: existing confidence- based indicators (Saito et al., 2019; Deng & Zheng, 2021) are less correlated with accuracy where classification accuracy is very low. Moreover, we might encounter images from unseen classes in the open world (Ji et al., 2023). In addition, the effectiveness of the BoS score in predicting the mAP for multi-class detectors requires further investigation. This requires a large scale collection of object detection datasets across diverse appearances and the same label space. However, real- world datasets meeting these criteria are scarce, which limits us from exploring more findings. Part of our future work is to collect such datasets like Sun et al. (2023) to share with the community.
CONCLUSION
In this paper, we report very interesting finding: given a trained detector, its bounding box stability when feature maps undergo dropout positively correlates with its accuracy, measured on various test sets. The stability is measured by BoS score, computed as the intersection over union between corresponding bounding boxes found by bipartite matching. Because computing the BoS score does not require test labels, this finding allows to predict detection mAP on unlabeled test sets. We perform extensive experiments to verify this capability and show that the mAP evaluator based on BoS score yields very competitive results.

---

Published as a conference paper at ICLR 2024
REFERENCES
Amir Bar, Xin Wang, Vadim Kantorov, Colorado Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Detreg: Unsupervised pretraining with region priors for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14605-14615, 2022. 9
Markus Braun, Sebastian Krebs, Fabian Flohr, and Dariu M Gavrila. Eurocity persons: A novel benchmark for person detection in traffic scenes. IEEE transactions on pattern analysis and machine intelligence,
2019. 5, 14
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmn: high quality object detection and instance segmentation. IEEE transactions pattern analysis and machine intelligence, 43(5):1483-1498. 2019. 2
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020. 3,
Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint 1rXiv:1906.07/11 2019. 13
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Ro- drigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset. In CVPR Workshop the Future of Datasets in Vision, volume 2. sn, 2015. 5, 13, 14, 17
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016. 4, 8, 15, 16
Ciprian A Corneanu, Sergio Escalera, and Alcix M Martinez. Computing the testing error with- out testing set. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2677-2685, 2020. 3
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 1601-1610, 2021.
Kumari Deepshikha, Sai Harsha Yelleni, PK Srijith, and C Krishna Mohan. Monte carlo dropblock for modelling uncertainty in object detection. arXiv preprint arXiv:2108.0366 03614, 2021.
Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15069-15078, June 2021. 2, 3, 4, 5, 6, 8,9
Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell about classifier accuracy under varying testing environments? In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2579-2589, PMLR, 18 Jul 2021. URL https: https://prrceedingg. ....lrr..... mll..press/vil 39/ /e999//eg21111 html. 2,
Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian. Unsupervised supervised learn- ingi: i: Estimating classification and regression errors without labels. Journal of Machine Learning Research, (44), 2010. 3
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centemet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp. 6569-6578. 2019.
10

---

Published as a conference paper at ICLR 2024
Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, loannis Mitliagkas, and Daniel M Roy. In search of robust measures of generaliza- tion. Advances in Neural Information Processing Systems,
11733, 2020. 3
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-308, 2009. 3,
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016. 1,3
Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribation performance. arXiv preprint 0nt::2201.04234 2022. 2, 3, 6, 9, 14, 15, 16, 17
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 32(11):1231-1233, 2013. 5, 8, 13, 14
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007. 5, 6. 14
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence unseen distributions. Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 1134-1 144, 2021. 2, 3, 5,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 5, 13
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2961-2969, 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprini 1610v.0610.0200 2016. 3, 4, 5, 6, 9, 15, 16, 17
Ge-Peng Ji, Deng-Ping Fan, Peng Xu, Ming-Ming Cheng, Bowen Zhou, and Luc Van Gool. Sam struggles in concealed scenes-empirical study on" segment anything". Science China Information Sciences, 66:226101, 2023. 9
Yiding Jiang. Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. arXiv preprint arXlv.1810.00113. 2018.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint 1rXiv:1912.022773 2019.
Kaggle. Self driving cars.
solf-driving-cars 2020a. 5, 13, 14, 17
Kaggle. Traffic vehicles object detection. https: https::/www.kagglee..
wwwpectddeeeeeeee on, 2020b. 5, 14
Kaggle. Udacity self driving car dataset. https:/ www.kkaggle.coo sasglle.com//atastts//slllee.
udacity- seocity-see sel www.lfd-iiiigg----dddee carg--asrddtteet.... 2021. 5, 14
Kaggle. Car-person eustom object detection v2 roboflow. https::/ https://www.kggl...
datasets/ http:///wonannaaaaaaaaaaaeeeeeee
2022. 5, 14
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics
quarterly, 2(1-2):83-97, 1955. 4
Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2663-2673, 2021. 2

---

Published as a conference paper at ICLR 2024
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer EscionEECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014, 3, 5, 6, 8, 13, 14
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017. 3, 4, 5, 7, 8, 13, 15, 17
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021. 5, 7,
Yuen Peng Loh and Chee Seng Chan. Getting to know low-light images with the exclusively dark
dataset. Computer Vision and Image Understanding, 178:30 42, 2019. 5, 13, 14
Omid Madani, David Pennock, and Gary Flake. Co-validation: Using model disagreement un- labeled data to validate classification algorithms. Advances in neural information processing systems, 17, 2004. 3
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization deep learning. Advances neural information processing systems. 30, 2017.
Ru Peng, Heming Zou, Haobo Wang, Yawen Zeng, Zenan Huang, and Junbo Zhao. Energy-based automated model evaluation. arXiv preprint arXiv:2401. 2024.
Emmanouil Platanios, Hoifung Poon, Tom M Mitchell, and Eric Horvitz. Estimating accuracy from unlabeled data: A probabilistic logic approach. Advances in neural information processing
systems, 30, 2017. 3
Emmanouil Antonios Platanios, Avinava Dubey, and Tom Mitchell. Estimating accuracy from unla- beled data: A bayesian approach. In International Conference on Machine Learning, pp. 1416- 1425. PMLR, 2016. 3
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015. 2, 5,
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 658-666, 2019. 4
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised do- main adaptation via minimax entropy. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8050-8058, 2019. 3, 5, 6, 9, 15, 16, 17
Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun. Crowdhu- man: A benchmark for detecting human in a crowd. arXiv preprint DoXXxv/885.00/222 2018. 5, 6, 14, 17
Xiaoxiao Sun, Yunzhong Hou, Hongdong Li, and Liang Zheng. Label-free model evaluation with semi-structured dataset representations, arXiv preprint 1rXiv:2112.00694 2021. 2, 3
Xiaoxiao Sun, Xingjian Leng, Zijian Wang, Yang Yang. Zi Huang, and Liang Zheng. Cifar-10- warehouse: Broad and more realistic testbeds in model generalization analysis. arXiv preprint arXiv:2310.04414 2023. 9
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVÉ international conference on computer vision, pp. 9627-9636, 3
12

---

Published as a conference paper at ICLR 2024
Longyin Wen, Dawei Du, Zhaowei Cai, Zhen Lei, Ming-Ching Chang, Honggang Qi, Jongwoo Lim, Ming-Hsuan Yang, and Siwei Lyu. Ua-detrac: A new benchmark and protocol for multi-object detection and tracking. Computer Vision and Image Understanding, 193:102907, 2020. 5, 13
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10539-10548, 2021.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. 0dd100k: diverse driving dataset for heterogeneous multitask learn- ing. In Proceedings of the /EEE/CVF conference computer vision and pattern recognition, pp- 2636-2645, 2020. 4, 5, 6, 8, 13, 15, 16
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Harry Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In International Conference on Learning Representations, 2022. 3
Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele. Citypersons: diverse dataset for pedes- trian detection. In Proceedings of the IEEE conference computer vision and pattern recogni- tion, pp. 3213-3221, 2017. 5, 6, 14
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-323 221,, 2019. 3
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint urX1v:2010.04159, 2020.
A EXPERIMENTAL SETUP
A.1 MODELS BE EVALUATED
We consider both one-stage and two-stage detection models with different backbones, including RetinaNet+R50 (Lin et al., 2017; He et al., 2016), RetinaNet+Swin (Liu et al., 2021), Faster- RCNN+R50 (Ren et al., 2015) and FasterRCNN+Swin We follow common practices (Liu et al., 2021) to initialize the backbone with praitrei classification weights, and train models using a 3x (36 epochs) schedule by default. The models are typically trained with stochastic gradient descent (SGD) optimizer with a learning rate of 10-3 Weight decay of 10-4 and momentum of 0.9 are used. We synchronized SGD over 4 GPUs with total of 8 images per minibatch (2 images per GPU). During the training process, we resize the input images such that the shortest side is at most 800 pixels while the longest is at most 1333 (Chen al., 2019). We use horizontal image flipping as the only form of data augmentation. During the testing process, we resize the input images to fixed size of 1333.
A.2 DATASETS
The datasets we use are publicly available and have double-checked their license. Their open-source is listed as follows.
For vehicle detection, we use 10 datasets, including
COCO (Lin et al.. 2014): ht kttpss:
BDD (Yu al., 2020): httts::///dddddee http://dddddttaa.brrellllee...dd Cityscapes (Cordts et al.,
DETRAC (Wen et al., 2020): htt.ttpsss/....dllyysseeaaaaeae
Exdark
(Loh & Chan, 2019):
https://github..om. com chan/
Exclusively-Daae
Image-Dataset;
Kitti (Geiger et al., 2013):
Self-driving (Kaggle, 2020a):
owa i
13

---

Published as a conference paper at ICLR 2024
Perturbed Detection
Original Detection
Ca) Stable Prediction
(b) Urotable Predictions
Figure 5: Visual examples illustrating the correlation between bounding box stability and detec- tion accuracy. (a) Bounding boxes change significantly upon MC dropout, where we observe com- plex scenarios and incorrect detection results. (b) Bounding boxes remain at similar locations after dropout, and relatively correct detection results are observed.
Udacity (Kaggle, 2021):
self-drine salf-driving-carse
Roboflow (Kaggle, 2022):
Traffic (Kaggle, 2020b):
craffic-vehicles srafflc-eeeiccl----jee----eeeeee wew.llessojectt-eettttttee
For pedestrian detection, we use 9 datasets, including
COCO (Lin al., 2014):
Hhome
Caltech (Griffin et
Crowdhuamn (Shao et al., htt.:///.tts/////...oooaaaaarara
Cityscapes (Cordts et al., 2015): https:://ww..cd http:ppp//ww..tttosaeeeeaaaaaaaa
Self-driving (Kaggle, 2020a):
self driving cars; Exdark (Loh & Chan, 2019)
https://github.ccom/ssddhh
Exclusively-Darke Iaaek-IImmg---asset...
EuroCity (Braun et al., 2019): attps://aroottyy.tttee..........
Kitti (Geiger al., 2013): https: htw.:oolbb...ett/aaaattttt//tttt
CityPersons (Zhang et al., 2017):
com/
downloads/.
ATC + PS
Coefficients of Deteminnaioo (RP)
A.3 COMPUTATIONAL. RESOURCES
We conduct detector training experiments on four A100, and detector testing experiments
0.4
one A100, with PyTorch 1.9.0 and CUDA 11.1.
0.2
The CPU is Intel(R) Xeon(R) Gold 6248R 29. Core Processor.
Thresholds
A.4 HYPER-PARE PARAMETER EXPLORATION FOR BASELINE METHODS.
Figure Effect of thresholds for all baseline methods. We search the optimal threshold on training sample sets using RetinaNet+R50. We report the correlation results (coefficients of de- termination R²²) using various thresholds. We show that threshold 0.4, 0.95, and 0.3 helps for ATC, PS, and ES, respectively.
We empirically find that different measurements have their own optimal thresholds. According to the effect of thresholds in terms of correla- tion strength (R²) in Fig. 6, we use thresholds of 0.4, 0.95, and 0.3 for ATC (Garg et al., 2022),
14

---

Published as a conference paper at ICLR 2024
104
mAp
144 100
173
573
300
333
3.3
6.70
140.211
1660y (96)
Average
Suciaiity
Figure 7: Correlation between different measurements and detection metrics for vehicle detection. "A" of different colors represents nine real-world datasets, such as BDD (Yu et al., 2020) and Cityscapes (Cordts et al., 2016), which are used seeds to generate sample sets by various trans- formations. Each point " of different colors represents a sample set generated from different seed sets. Three detection metrics including mAP, mAP mAP7s. which obtained by RetinaNet (Lin et al., 2017) trained on the COCO training set. Four measurements including three existing methods (Saito al., 2019; Hendrycks & Gimpel, 2016; Garg al., 2022) and our box stability score used in (a), (b), (c), and (d). The trend lines are computed by linear regression, from which we can observe a relatively strong linear relationship between box stability score and mAP.
R²=0.492
R²=0.720
p=0.566
p=0.723
0.8
bbox
0.8
bbox
GT
0.6
GoU
0.4
0.4
0.2
0.0
0.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.7
0.9 1.0
(a) Confidence Score
(b) Box Stability Score (ours)
Figure 8: (a) The correlation between the loU of bounding boxes with the matched ground truth and the confidence score. (b) The correlation between the IoU of bounding boxes with the matched ground truth and box stability score. Considering detected bounding boxes having a confidence/box stability (> 0.5), the coefficients of determination Re point represents predicted box generated by RetinaNet+R50 run on COCO validation without the detector feature perturbed.
0.492, and (b) 0.720. Each
PS (Hendrycks & Gimpel, 2016), and ES (Saito et al., 2019), respectively. We have two observa- tions. First, ATC and ES tend to choose centering thresholds their optimal thresholds, while ES particularly sensitive to the setting of thresholds. Second, when using extremely high threshold temperature values, PS achieves its own strongest correlation with mAP.
15

---

Published as a conference paper at ICLR 2024
Rebinannt Seroont-SSiinn Swiin mal
200
164
p-d E5
p+d 536
FaxterRcNNN
mo4P (8)
800
Fasterli MORRCNNN-AAA Swii
R1et 003
nel (60
150
670
270
130
2.8
6.5
340
3.9
120
(4) Enitopy Score
011 Prediction Score
50 Average Threshalded Confidence
Rox Stabilty Score [ours]
Figure 9: Correlation between different measurements and mAP on different structures for vehicle detection. "A" of different colors represents nine real-world datasets, such as BDD (Yu et al., 2020) and Cityscapes (Cordts et al., 2016), which are used as seeds to generate sample sets by various transformations. Each point " of different colors represents sample generated from different seed sets. The mAP of each raw obtained by three detector structures trained on the COCO training set, including RetinaNet+Swin, FasterRCNN+R50, and FasterRCNN++Sinn respectively. Four measurements including three existing methods (Saito et al., 2019; Hendrycks & Gimpel, 2016; Garg et al., 2022) and our box stability score used in (a), (b), (c), and (d). The trend lines computed by linear regression, from which we can observe a relatively strong linear relationship between box stability score and mAP.
B DISCUSSION
Is BoS score a kind of confidenceuuncrtaiitty measurement in object detection? It charac- terizes confidence on the dataset level, but might not operate on the level of individual bounding boxes/images. As shown in Fig. 8(a), we tried to correlate box stability score with ground truth IoU (GT IoU) on individual bounding boxes. Compared to confidence score (coefficients of determina- tion R² 0.49), BS achieves stronger correlation (coefficients of determination R² > 0.72) with GT IoU, but the correlation was still weak. We find BoS score is correlated with accuracy when the number of images considered is statistically sufficient (coefficients of determination R² > 0.93). In Fig. (()) we give some visual examples of where BoS score can reflect whether the dataset is hard for the detector. Under feature perturbation, bounding boxes remaining stable (with high BoS score) would indicate a relatively good prediction, while bounding boxes changing significantly (with low BoS score) may suggest poor predictions.
C MORE CORRELATION RESULTS
C.1 CORRELATION RESULTS FOR OTHER METRICS
In Fig. additionally show the correlation between different measurements and different de- tection metrics. We observe that BoS score also has a strong correlation with mAP30 and mAP75. yielding the other three measurements.
16

---

Published as a conference paper at ICLR 2024
rV4P (60
WWM
20
160
60 E.
Q4l Entropy Score
36 Prediction Scor
13 Arenag Threshodded Confidence
699 Boa Stubiiit Seare tound)
Figure 10: Correlation between different measurements and mAP for pedestrian detection. "A" of different colors represents nine real-world datasets, such Caltech, Crowdhuman (Shao al., 2018), Cityscapes (Cordts et al., 2015), and Self-driving (Kaggle, 2020a), which are used as seeds generate sample by various transformations. Each point "*" of different colors represents a sample set generated from different seed sets. mAP obtained by RetinaNet (Lin et al., 2017) trained on the COCO training set. Four measurements including three existing methods (Saito al., 2019; Hendrycks & Gimpel, 2016; Garg et al., 2022) and our box stability score are used in (a), (b), (c), and (d). The trend lines are computed by linear regression, from which we can observe relatively strong linear relationship between box stability score and mAP.
C.2 CORRELATION RESULTS ON DIFFERENT DETECTORS
In this section, we additionally report the correlation results using other detectors: RetinaNet+Swin, FasterRCNN+R50, and FasterRCNN+Swin. As shown in Fig. there very strong correlation between BoS score and mAP Faster (R² = 0.830 and 0.783). But on Reti- naNet+Swin and CaNtrr, the correlation somehow weak, and we will try improve on this in future work.
C.3 CORRELATION RESULTS ON PEDESTRIAN DETECTION
In this section, we additionally report the correlation results on Pedestrian Detection. As shown in Fig. 10, the correlation between BoS score and mAP is significantly strong than the other methods when the mAP is greater than 15%. When mAP is less than 15%, the ATC method is slightly better among these four measures, but the sample are clustered together and hard to distinguish from each other.
17


Reviews:
[{"review_id": "QgYIPKR1PJ", "reviewer": "Reviewer_19ZD", "summary": "The authors discover and examine the phenomenon that - in an object detection setting - the stability of bounding box detection (in the presence of MC dropout variability) is strongly correlated to the ability of the detector to generalize to unseen datasets. Aside from a comprehensive set of experiments comparing the approach to various alternatives, the authors also spend time discussing what their insights are into the metric, as well as what it *cannot* do (a valuable insight into the metric),", "questions": "- I'm struggling to come up with any question - the authors have done a very good job covering all the bases I could ask for", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- An important discovery since dropout is performed on the neural network and so is dataset-independent. The fact that it works makes perfect sense as the approach is effectively testing the detector for robustness to unexpected/untrained differences in internal weights, which  can occur with distinctively-different datasets (especially real data), and is something that dropout is essentially simulating.\n\n- Results are comprehensive, covering multiple detection problems and comparing against several alternative algorithms.\n\n- The authors also spend a couple of sections on summarising what their scoring metric really is, and what it could be used for (and importantly what it could NOT be used for ie: as a loss function). This kind of clear elucidation is invaluable.", "weaknesses": "- I really don't have any weaknesses to mention - I can fully see this becoming part of the standard toolkit for evaluating detectors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sGcL9TqfPV", "reviewer": "Reviewer_h1Z9", "summary": "This paper provides a novel way to evaluate object detection systems without ground truth data, based on the stability of bounding boxes, and shows promising results in estimating detection accuracy. The authors propose a method for evaluating the quality of bounding boxes generated by object detection models and their correlation with detection accuracy. The paper introduces a metric named\"box stability score\" (BoS) and demonstrates a strong positive correlation between BoS scores and detection accuracy (mAP). The method is the first for AutoEval in object detection. Experiments on vehicle and pedestrian detection tasks demonstrate the effectiveness of the BoS score in estimating mAP and outperforming confidence-based measurements.", "questions": "nil", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a metric, the box stability score (BoS), for object detection. The metric is validated that it has a strong positive correlation with the detection accuracy (mAP). The correlation provides an agent for evaluating detectors on real-world datasets lacking ground truth bounding boxes. It can better test the generalization performance of various detectors. It first applies the AutoEval algorithms for object detection.\n\n2. The paper is well organized. The proposed method BoS is simple but effective for label-free detection evaluation. The discussions are sufficient to explain the insights of the paper. \n\n3. The paper is well written. It's easy to follow the main contributions of the paper.\n\n4. The experiments compare BoS with extended classification AutoEval methods. The comparison demonstrates that the BoS can better measure the detection tasks under label-free scenarios.", "weaknesses": "1. The BoS is correlated with the accuracy on the dataset level but failed on the individual level. It does not make sense. This part needs more explanation and provides more evidence.\n\n2. A limitation section is needed for a comprehensive analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HkcveUUshq", "reviewer": "Reviewer_mbpe", "summary": "This paper find that under feature map dropout, good detectors tend to output bounding boxes whose locations do not change much, while bounding boxes of poor detectors will undergo noticeable position changes. The proposed box stability score has a strong, positive correlation with detection accuracy.", "questions": "- Why there is a big gap between dropout rate=0.1 and 0.15 in Figure 3 (b)? \n- Why there is a peak at around 5 in Figure 4 (a)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ This paper is clear and easy to read.", "weaknesses": "- I find it hard to understand the experiments. In Table 1, the authors say \"we report its ground truth mAP (%) on different unseen test sets in the header, the estimated mAP (%) in the main table\". However, the numbers for estimated mAP reported in the main table are very low. So what do the numbers in Table 1 represent for? Same for Table 2, Table 3 and Table 4. What are the best numbers in each column?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IVtm9dVhbs", "reviewer": "Reviewer_xRLp", "summary": "This paper propose a box stability score (BoS score) for unsupervised detector evaluation on the real-world test sets. It has been well proved that BoS has a strong, positive correlation with detection accuracy measured by mAP.  BoS is the first metric proposed for unsupervised evaluation of object detection. Experiments demonstrate that it performs well with different network backbones and outperforms confidences based measurements.", "questions": "1. In Table 3, under the multi-class detection setting on Cityscapes and Self-driving datasets, the BoS performs much worse than other methods on car. While under single class setting, the RMSE of BoS of car on Cityscapes is 0.89, which is much better than others.There is no explaination on this point. Does the proposed method unstable in multi-class setting?\n\n2. In each table, does the number besides the dataset or class name reports the real mAP? For example, COCO (34.2) in Table 1. If so, in Table 2, on Kitti and CityPersons datasets, the mAP is 13.3 and 11.8 respectively, which are lower than the requirement (mAP > 15%).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. This paper is well written and organized. The idea is clear and easy to follow.\n2. Novelty: BoS is the first metric proposed for unsupervised evaluation of object detection.\n3. Correlation between BoS and mAP is well demonstrated.\n4. Good and consistent performance of BoS compared with exsiting confidence based metrics across meta-datasets, backbones, and iou thresholds.", "weaknesses": "1. My main concern on the proposed method is how it can be applied in the real world object detection and what it benefits. The proposed method has requirements on the detector's performance (mAP>15%) on target data. However, in real-world scenarios, we cannot be certain if the detector performs well enough to meet this requirement. We don't know how reliable is the BoS score. Can BoS be used to  improve the detection results?\n\n2. Another issue is that the evaluation of BoS is primarily focused on single-class scenarios, and it does not perform as well in multi-class settings. In real-world applications, we often encounter multiple bounding boxes detected across different classes. Consequently, the accuracy of matching between the original prediction and the prediction after feature perturbation may be compromised. It remains uncertain whether the performance of BoS will degrade when dealing with multi-class data.\n\n3. Additionally, the length of this paper exceeds the 9-page limit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)