Breaking Beyond COCO Object Detection
Authors: ali borji
Venue: Submitted to ICLR 2023 (2023)
Keywords: object detection
deep learning
performance analysis

Abstract: COCO dataset has become the de facto standard for training and evaluating object
detectors. According to the recent benchmarks, however, performance on this
dataset is still far from perfect, which raises the following questions, a) how far can
we improve the accuracy on this dataset using deep learning, b) what is holding
us back in making progress in object detection, and c) what are the limitations
of the COCO dataset and how can they be mitigated. To answer these questions,
first, we propose a systematic approach to determine the empirical upper bound
in AP over COCOval2017, and show that this upper bound is significantly higher
than the state-of-the-art mAP (78.2% vs. 58.8%). Second, we introduce two
complementary datasets to COCO: i) COCO_OI, composed of images from COCO
and OpenImages (from 80 classes in common) with 1,418,978 training bounding
boxes over 380,111 images, and 41,893 validation bounding boxes over 18,299
images, and ii) ObjectNet_D containing objects in daily life situations (originally
created for object recognition known as ObjectNet; 29 categories in common with
COCO). We evaluate models on these datasets and pinpoint the annotation errors
on the COCO validation set. Third, we characterize the sources of errors in modern
object detectors using a recently proposed error analysis tool (TIDE) and find that
models behave differently on these datasets compared to COCO. For instance,
missing objects are more frequent in the new datasets. We also find that models
lack out of distribution generalization. Code and data will be shared.

Under review as a conference paper at ICLR 2023
BREAKING BEYOND COCO OBJECT DETECTION
Anonymous authors
Paper under double-blind review
ABSTRACT
COCO dataset has become the de facto standard for training and evaluating object detectors. According to the recent benchmarks, however, performance on this dataset still far from perfect, which raises the following questions, a) how far can we improve the accuracy this dataset using deep learning, b) what is holding us back in making progress in object detection, and c) what are the limitations of the COCO dataset and how they be mitigated. To answer these questions, first, we propose systematic approach to determine the empirical upper bound in AP over COCOva/2017, and show that this upper bound is significantly higher than the state-of-the-art mAP (78.2% vs. 58.8%). Second, we introduce two complementary datasets COCO: i) COCO_OK composed of images from COCO and OpenImages (from 80 classes in common) with 1,418,978 training bounding boxes over 380,111 images, and 41.893 validation bounding boxes over 18,299 images, and ii) ObjectNet_]] _e containing objects in daily life situations (originally created for object recognition known as ObjectNet; 29 categories in common with COCO). We evaluate models on these datasets and pinpoint the annotation errors on the COCO validation set. Third, we characterize the sources of errors in modern object detectors using recently proposed error analysis tool (TIDE) and find that models behave differently on these datasets compared to COCO. For instance, missing objects more frequent in the new datasets. We also find that models lack out of distribution generalization. Code and data will be shared.
INTRODUCTION
Object recognition is believed, although debatable, to be solved in computer vision witnessed by the "superhuman" performance of the ofatc-of-the -hrtte--rttttttt (SOTA) models (~3% vs. ~5% top-5 error rate human vs. machine on ImageNet (He al. 2016 Dosovitskiy et al. 2020 Russakovsky et al.) 2015)). Unlike object recognition, however, object detection remains largely unsolved and models perform far below the theoretcial upper bound (mAP=1). The best performance on COCOva/2017 and CoCOtest-dev2017 are 58.8% and 11,, respectively based on the COCO detection leaderboard & codalab results According to the results from the 2019 OpenImaaes detection challenge, the best mAP this dataset is 65.9% Inspired by the recent stream of work such as Recht et al. (2019): Shankar et al.] (2020): : Beyer et al. (2020) examining the accuracy and out of distribution generalization of recognition models, strive to understand why detection performance is poor and study the limitations of object detectors and the ways models and datasets can be improved.
Several years of extensive research in object detection has resulted in accumulation of an overwhelming amount of knowledge regarding model backbones, tips and tricks for model training, optimization, data collection, augmentation, annotation, model evaluation, and comparison, that separating the wheat from the chaff is very difficult (Zou et al. 2019; Zhang et al. (019). For example, getting all details right in implementing average precision (AP) frustratingly difficult (see supplement). A quick Google search returns several blogs and codes with discrepant explanations of average precision (AP). In addition, is not clear whether AP has started saturate, whether small improvement in AP (e.g., 56.1 vs. 56 mAP) is meaningful, and more importantly how much we can improve following the current trend, making one wonder maybe we have reached the peak of detection performance using deep learning") IT] A critical concern here is that maybe detection datasets are not big enough to capture variations in object size? viewpoint, occlusion, and spatial relationships
point
This also known as ceiling analysis.
Thh median scale of the object relative to the image in ImageNet vs. COCO is 554 and 106, respectively. Therefore, most object instances in COCO smaller than 1% of the image area (Singh & Davis 2018.

---

Under review as a conference paper at ICLR 2023
91.6
14.1
78.2
78.2
B4.6
81.8
80
72.7
70.7
72.0
$2.6
61.0
56.0
64,9
63.5
60
47.9
40.7
NeahCVtab
40
FOOS
CenterNet FCDS
Casc, Mask RDM
20
MASK RCNN
11.1
srndet
DeepAft
Upper-bound AP
PASCAL VOC
Upper bp
Delectoits
COCO
AP50
APTS
AP5O
APTS
Figure 1: Upper bound AP (UAP) shown in red and scores of the best models using COCO evaluation tool (shown in black; numbers come from different models). Results over COCOva12017 are compiled from the top entries of the latest challenge on COCO available at COCO detectoin leaderboard. See also paperswithcode.com coco_ 1o__miiival Notice that the gap at AP is almost closed in COCO. There is, however, still a large gap between best models and UAP. The gap is wider over higher IOUs and small objects. Note: Higher mAPs have been reported over the VOC benchmark paperswitheode.cmm com YOClusing the VOC evaluation tool (89.1 AP50 Ghiasi al. (2020) which is close to 89.5 UAP computed by us using the VOC tool: please see also Appx A
among objects. In other words, scaling object detection seems to be much more challenging compared to scaling object recognition. Due to these, object detection can be considered as key task to assess the promises and limits of deep learning in computer vision.
Contributions. To shed light on blind spots that could be holding back progress, we carefully and systematically approximate the empirical upper bound in AP (UAP). We argue that UAP is the score of the object detector with access to ground-truth bounding boxes. An object recognition model is trained on the training target bounding boxes and then used to label the test target boxes (i.... localization is assumed to be solved). In nutshell, we find that there is a large gap between the best model mAP and the empirical upper bound as shown in Fig. The gap is wider at higher IOUs and over small objects. Using the latest results on COCO dataset (Fig. [1, the gap is narrow at IOU=0.5 (~~2 points). The computed UAP entails that there is hope to reach this peak with the current tools if we can find smarter ways to adopt the object recognition models and backbones for object detection, but going beyond it may require major breakthroughs. The current trend in deep learning, in particular in recognition and detection, has been scale the datasets. Less effort, however, has been spent to systematically inspect errors in datasets. Here. we perform such inspection, and investigate out of distribution generalization of object detectors by introducing new validation sets. We also introduce an extension of COCO by integrating it with OpenImages. Finally, we identify the bottlenecks in detectors and characterize the type of errors they make on these new sets.
2 RELATED RESEARCH
Four lines of work relate to our study. The first one includes works that strive to understand object detectors, identify their shortcomings, and pinpoint where more research is needed. Parikh & Zitnick T) aimed to find the weakest links in person detectors by replacing different components
with human annotations. Mottaghi ct al. (2015) proposed human- human-maching machine CRFs for identifying bottlenecks in scene understanding models. Hoiem et al. (2012) inspected detection models in terms of their localization errors and confusion with other c classes and the background over the PASCAL VOC dataset. They also conducted meta-analysii to measure the impact of object properties such as color, texture, and real-world size on detection performance. To overcome the shortcomings of the Hoiem et al. 's approach and COCO analysis tools, recently [Bolya et al. (2020) proposed to analyse the models by emphasizing the order in which the errors are considered. Russakovsky et al. (2013) analyzed the ImageNet localization task and emphasized on fine- fine-graine grained recognition. Zhang et al.] (2016) measured how far we are from solving pedestrian detection. Vondrick et al. (2013) proposed a method for visualizing object detection features to gain insights into their functioning, Some other related works in this regard include Li et al. (2019), Zhu al. (2012), Zhang et al. (2014), Goldman et al. (2019), and Petsiuk ctt al. (0200).
in pipeline (e.g.. part detection, non-maxima-supprese
The second line of work concerns research in comparing object detection models. Some works have analyzed and reported statistics and performances over benchmark datasets such as PASCAL VOC (Everingham et al.] 2010 2015), COCO (Lin et al. 2014), CityScapes (Cordts et al. 2016). and OpenImages (Kuznetsova et al. 2018). Recently, Huang et al. (2017) performed a speed/accuracy trade-off analysis of modern object detectors. Dollar et al. (201)) and Borji et al. (2055) compared
2

---

Under review as a conference paper at ICLR 2023
person detection and salient object detection models, respectively. Michaelis et al. (2019) assessed detection models on degraded images and observed about 60% performance drop, which could be mitigated by data augmentation. To resolve the issues with the AP score, some works have attempted to introduce alternative (e.g.. Hall et al. (2018) or complementary evaluation measures (e.g.. Oksuz et al. (2018); Rezatofighi al. (0199).
Works in the third line study the role of context in visual recognition Bar (2004); Wolf & Bileschi
(2006); Zhu al. (2016); Marat & Itti 2012); Heitz & Koller (2008); Torralba & Sinha (2001), );
Rabinovich et al. 2007): Rosenfeld et al. 2018); Galleguillos & Belongie (2010). Heitz & Koller
(2008) proposed probabilistic framework to capture contextual information between "stuff" and
"things" to improve detection. Barnea & Ben-Shahat (2019) utilized co-occurrence relations among
objects to improve the detection scores. Diivala et al. (2009) explored the role of different types of context in recognition. |Rosenfeld et al. (2018) showed that object detectors often miss objects that
are transplanted out of their usual context (e.g.. elephant in the living room). See also Heitz Koller
(2008), Chen et al. (2018), Song et al. 201 Hu et al. (2018). Marat & Itti (2012). and Alamri & (2019). (201).
The fourth line of work deals with generalization of object recognition models against synthetic and natural distribution shifts. As an example, |Recht et al. (2019) closely followed the original ImageNet creation process to build a new test set called ImageNetV2. They reported performance gap of about 11% in top-1 I accuracy between the performance of the best deep models on this dataset and the original test set. Similar observations have been made in (Shankar al. (2020); Beyer et al.] (2020); Barbu et al.] (2019). It is thus critical to ensure that models are learning meaningful generalizations, rather than just overfitting the idiosyncrasies in datasets. Towards this goal, a large number of works have assessed object recognition models and their robustness against image transformations and degradations such as spatial transformations. noise corruptions, simulated weather artifacts, and temporal changes. See for example example|Hendryck & Dietterich| (2019); (Azulay & Weiss 2018); Recht et al. (2019); | Mishkin et al. (2017); Gu al. (2019).
3 ESTIMATING THE EMPIRICAL UPPER BOUND IN AP
We define the empirical upper bound in AP as the score of the object detector that a) has access to the true location of the objects, and b) ground truth bounding boxes are labeled by the best object classifier. This way we essentially assume that the localization problem is solved and what remains is only object recognition. Beware that we do not mean undermine the importance of the localization component. What we intend to convey is that assuming no further progress in object recognition and investing all efforts in solving the localization problem can lead to this upper bound and not heyond that. Knowing the upper bound can help us to hetter coordinate our efforts. The above object detector, however, may not give us the true upper bound AP due to the subtleties involved in AP calculation (Appx B). Specifically, it might be possible to improve upon this detector in at least two ways: a) by exploiting the local scene context around an object to improve the classification accuracy and thus better UAP, and/or b) by searching among the bounding boxes around the target object (those with a certain overlap with it) and see whether any of them can be classified better, compared to the target box itself. This does not matter for determining the UAP at the perfect IOU (=1) but may affect UAP at IOUs lower than one. We carefully investigate these details in the following.
3.1 EXPERIMENTAL SETUP
Benchmarks and models. We establish our analysis based on two recent large-scale object detection benchmarks: MMDetection (Chen et al. (019b) and Detectron2 Wu et al.] (2019). The former evaluates more than 50 models. The latter includes several variants of FastRCNN (Girshick 2015). In both benchmarks, all COCO models have been trained on train2017 and evaluated on va/20/7. We consider major object detection models including several variants of the RCNN such as FasterRCNN (Ren et al. 20055), MaskRCNN (He et al. 2017), RetinaNet (Lin et al. 2017). GridRCNN (Lu et al.) 2019), LibraRCNN (Pang et al. 2019), CAN (Cai & Vasconcelos 2018), MaskScoringRCMM RCNN (Huang et al. 2019). GAFasterRCNN (Zhu et al. 2019), Hybrid Task Cascade (Chen et al.) 2019a). and other models such as SSD (Liu et al. 2016), FCOS (Tian et al.) 2019), and CenterNet (Zhou et al.] 2019). Different backbones for each model are also taken into
account. In addition, we also include the most recent models in the analysis.
3

---

Under review as a conference paper at ICLR 2023
Figure 2: Top: Visual context around an object, Bottom: Object recognition accuracy. Top rows: testing on the canonical object size (=1 in the table, used in the rest of the paper, Appx [D] ). Bottom rows: training and lesting are the same (e.g.. a classifier is trained on the object-only case 0.6 and is then tested on the object-only case 0.6).
object only
object + context
context only
Datawwr
oltject oully
obgjeel
cuntext
cantes ooly
tert
0.2 0:4 36 0.8
1.2
1.4
10
18
2
12.
all iug
YOU
293 68.9 8266 02.5
94.6
ME.
91.6
90.6
88.6
87.0
636
64.5
85.8
Full object (=1)
Openlmaages
COCO
67.1
7988
69.0
96.7
05.1
8299
60.7
78.3
12.5
07.4
63.0
63.
48.9
11.0
The best accuracy each row highlighted in bold.
YOC
61.1 79 87.2
92.4
94.8
94.4
54.0
90.7
92.4
91.3
61.8
70U6
73.6
same training
COCO
74
814
56.7
86.8
57.3
57.6
57.7
87.3
37.6
69.7
63.4
Datasets. Three datasets including PASCAL VOC (Everingham et al. 2015), COCO Lin al. 2014), and Openlmages (Kuznetsova et al., 2018) are employed. Over VOC, we use trainval0712 for training (16,551 images, 47,223 boxes) and tes12007 (4,952 images, 14,976 boxes) for testing. This dataset has 20 object categories. Over COCO, we use train2017 for training (118,287 images, 860,001 boxes) and val2017 (5,000 images, 36,781 boxes) for testing. COCO has carried the torch for benchmarking advances in object detection for the past 8 years. It has 80 object categories. Finally, we use Openlmages V4 dataset, used in the Kaggle competition It has 500 object categories and contains 1,743,042 images (12,195, 144 boxes) for training and 41,620 images (226,811 boxes) for validation. Its validation set is used here for testing.
3.2 UTILITY OF THE SURROUNDING CONTEXT
We trained 55 (He et al. 2016) on target bounding boxes in 3 settings as shown in Fig. 2 1) object-only, 2) object+context, and 3) context-only. Standard data augmentation techniques including color jittering. random horizontal flip, and random rotation (10 degrees) were applied. Boxes were resized to 224x224 pixels and models were trained for 15 epochs. Trained models were then tested on the target object boxes. Classification results (top-1 accuracy) are shown in the bottom panel of Fig. 2 We find that the canonical object size (full object) results in the best classification accuracy over all three datasets. Enlarging or shrinking the object bounding box lowers the performance. The context-only scenario results in a high classification score but still performs below other cases. Stretching the context to the whole scene drops the performance significantly.
3.3 SEARCHING FOR THE BEST LABEL & CONFIDENCE SCORE OF A TARGET BOUNDING BOX
Essentially the problem statement here is how best we can classify target box by utilizing all available information in the scene. This is different from recognition models where objects are treated in isolation (see discussion section). Notice that recognition accuracy is not the same as AP since detection scores also matter AP calculation (i.e., detections are ranked based on classification confidence). Having the best classifier in hand, are ready to approximate the UAP.
Strategies for labeling the boxes. We explore two strategies in pursuit of UAP. In the first one, we apply the best classifier from the previous section to the canonical target bounding boxes. The detector built this fashion gives the same AP regardless of the IOU threshold since our detections target boxes. As argued above, it is not possible to improve this detector at IOU=1. However, we are interested in upper bound at a lower IOU threshold (7), then might be possible to do better by searching among the candidate boxes near target box and choose the that can be classified better than the target box, or by aggregating the information from all its nearby boxes. To account for this, in our second strategy, we sample boxes around an object, similar to Oksuz al. (2020), and either use the original classifier (trained on the canonical object size) or train and test new classifiers on surrounding boxes. In any case, we always keep the target box but change its label and/or its classification confidence. First, let's 's take a look at our box sampling strategy illustrated in Fig. 3
Sampling boxes at IOU threshold 2 7. We are interested in finding the coordinates of the top-left corner of all rectangles with IOU 2 (7 < 1) with the ground-trutt bounding box. We use the coordinate system centered at the left corner of the target box (the PQRS rectangle; shown
3 Here, we assume that all boxes have the same width and height as the target box. The solution can be easily extended to the case where rectangles are non-homogencous.
4

---

Under review as a conference paper at ICLR 2023
A
GT box
Candidate boxes
Figure 3: A: Illustration of our setup for finding boxes with IOU 2 7 with the target box (corresponding to o 27/(1 + 7); aB 2/3 for IOU 0.5), B: The solutions are curves represented by Eqs. 4 to 7. Four sample rectangles are shown with dashed lines.
2g
-V=UV
R1
=1=UV UV
U
==VV
U-pa
=V
2V
-3
7-y
25 1 110
in black) which can be easily converted to the image level coordinate frame. Let's first find the relationship between the coordinates of the point marked with (< u, v >) and overlap threshold y.
According to the illustration in Fig. A we have:
RI
un,
R2 = UV. IOU 7 IOU
2R2 RI
From these equations and assuming u = at and
3V, it is easy to derive the following equations:
RI 2y R2
R1 = aUSV,
1+7
and from there we obtain:
2y (a. for 7 = 0.5) 1+7'
aB
3
The same equation governs the coordinates of the bottom-left, top-left, and top-right corners of the rectangles intersecting with the target box at points Q. R. and S, respectively (in the coordinate frames centered at each of these points, in order). Calculating the top-left corner of these rectangles in their corresponding coordinate frames and representing them in the coordinate frame of the image, we arrive at the following four equations. Notice that these are curves and not lines.
Zp : ( (a- 1)U zz. (B 1)V + yp
(4)
Zq: ( (1-a)) + zp, (B 2 1)V + Up
(5)
ZR: ((1- + (1 - B)V +yp
(6)
Zs: ( ((a-1)4 + zr, (1 B)V + UPp
2y
a. SL 1, s.t. aB 1+7
(8)
Using the above equations, we sample m (here m=4) rectangles
Most Confident Box
Most Frequent Label
Dataset
Acc.
with IOU 2 (Fig. B)
AP
AP:
APm
AP,
AP
AP,
APm
AP,
and label them with the label
VOC
93.7
88.7
91.7
81.4
63.8
89.1
92.0
82.9
of the target box. We then train a new classifier (same ResNet152 as above) on these boxes. This is effectively a new data augmentation technique. Notice that UAP is the direct consequence of the classification accuracy, meaning if we can classify objects better, we can reach a higher UAP. To estimate UAP, we sample m rectangles around a target box (with IOU 2 7), and then label the target box with a) the label and confidence of the bounding box with the highest classification score (i.c., the most confident box), or b) the most frequent label among the nearby boxes and the maximum confidence score among them.
COCO
84.8
76.9
81.8
80.6
62.8
76.4
82.0 80.4
60.7
Table 1: Upper bound AP using our second strategy (i.e., searching for the best bounding box object label near target box; among boxes with IOU > 0.5) using classifier trained and tested on surrounding boxes (see Appx C for results using the classifier trained on canonical objects). Notice that UAP, UAPso. and UAPrs are all the same.
3.4 UPPER BOUND AP RESULTS
Here, we report classification scores, upper bound APs, the score the models (mean AP over all IOUs; unless specified otherwise), and the breakdown AP over categories. See also Appxs E and
Comparison of strategies. Summary results of the first strategy are shown in Fig. There is a large gap between the UAP and the theoretical upper bound, and between UAP and model mAP. Results of
5

---

Under review as a conference paper at ICLR 2023
the second strategy are presented in Table 3 using classifier trained on surrounding boxes. Please see the Appx C for results using classifier trained on canonical objects and applied to surrounding boxes. Contrary to our expectation, the second strategy did not lead to higher UAPs perhaps because the surrounding boxes contain additional visual content which may introduce noise in the process. In other words, using surrounding boxes leads to less homogeneous samples (compared to using canonical objects) which hurts training. This leads to lower classification accuracy and hence a lower UAP. We found that sampling boxes at higher IOU thresholds (e.g., y = 0.9) did not improve the results. Also, setting the confidence of the detections to 1 lowered the UAP. In what follows we only discuss the results using the first strategy.
PASCAL VOC. Fig. 22 shows results on VOC using the COCO evaluation tool. We trained and tested models on this dataset including FasterRCNN, FCOS, SSD512, and two variants of CenterNet. The classification accuracy over VOC is very high about 94.8%. Consequently, UAP is high about 91.6%. FCOS does the best here with the mAP of 47.9% (dashed line). As can be seen, there is a large gap between the mAP of the best model and the UAP (~44 points). UAP is above model mAP in all categories. Models behave similarly across categories (i.e., low variance in each category). Results using the VOC evaluation code is given in Appx E VOC code is based on IOU=0.5 and calculates the area under the PR curve in a slightly different way than the COCO code (See also Fig. I
COCO. Borrowing the MMDetection benchmark and adding the results from CenterNet to it, we end up comparing 15 models (71 in total; the combination of models and backbones). Among the models we tested, the best ones are Hybrid Task Cascade model (Chen et al. (019a) and Cascade MaskRCNN (Cai Vasconcelos] 2018)) with 46.9% and 45.7% mAPs, respectively. See also Appx End Appx The UAP on COCO is 78.2% which about points above the mAP of the best model tested here, and 20 above the most recent model (see Fig. []. Recall that UAP does not depend the IOU threshold since detected boxes are ground-truth targets. The gap is much narrower at APsd 4 The gap between the UAP and model mAP over small objects is about 23 points which is almost 2x wider than the corresponding gap over large objects. The UAP over small objects is much lower than the UAP over large objects. This also holds for model mAPs.
14
AAD
0
0
0
0
0
2
Contorist dia
FCDS
RasteRONN
UppoBoued
VOC
pattedptant
chil
Usd
moop
Non 12n 9g NN
por
Figure 4: UAP and Model mAPs over the VOC dataset. Categories are sorted according the average model AP. Top bar chart shows classification accuracy. Dashed line represents FCOS mAP. See also Appx E
Breakdown APs over object categories are shown in Fig. 5 Here, we use the Detectron2 benchmark which reports per- category APs mainly over RCNN model family. Among the 80 categories, only 3 (snowboard, toothbrush, toaster) have UAPs below the best model APs. We noticed that aggregate scores on MMDeiection and Detectron2 are quite consistent. Among 18 variants of FasterRCNN and MaskRCNN, the best model has the AP of 44.3 (dashed line in Fig. [ which is lower than the best available model on COCO (58.8%; solid black line in Fig. T and the UAP (78.2%).
Openlmages. This dataset (Kuznetsova et al., 2018) is the latest endeavor in object detection and is much more challenging than its predecessors. Our classifier achieves 69.0% top-1 accuracy on the validation set of OpenImages V4 which is lower than the other three datasets.
DOTA and Fashion datasets. DOTA (here v1.0) is dataset of aerial images and includes 15 categories Xia et al. (2018). The fashion dataset includes 40 classes of fashion items such as jeans, T-shirt, and backpack. Sample images from these datasets and additional results are shown in Appx. The UAPs over the datasets in order are 88% and 71.2% 71.2%. The best model APs in order are 83.1 % and 59.7%. We obtain the UAP of 58.9%, using the TensorFlow evaluation code for computing the AP score on this dataset, which is slightly different than the COCO AP calculation tool (here we discarded grouping and super-category). We are not aware of any model scores on this particular set of OpenImages V4.
"The hest available scores COCOva12017 are shown in Fig. Interestingly, the gap at APso is almost closed (~x1. There however, still large gap at AP over all IOUs, s. and also over medium and small objects.

---

Under review as a conference paper at ICLR 2023
0.0
78.2
0
Rew cufo
58.8*
O
46.3
COCO
Figure 5: Detection APs over COCO dataset borrowed from the Detectron2 benchmark. The black dash line corresponds to the best model among the models we analyzed (the score shown with "s"). The black solid line shows the most recent results (the score shown with "+"). See Appx Eoro for results using MMDetection.
4 COMPLEMENTARY DATASETS TO COCO
The bulk of research on object detection has been primarily focused on developing network architectures or designing new loss functions; less attention has been devoted data. A notable exception is Openlmages dataset. However, despite being orders of magnitude larger than COCO, has been less adopted. To mitigate this, a) building on top of the COCO ecosystem, we extend the COCO dataset. We are motivated by studies (c.g.. Dosovitskiy et al. (2020); Kolesnikov et al. (2019) Radford et al. (22TT) that have used very large datasets (e.g., JFT-300M Sun et al. (2017)) to achieve SOTA accuracy in object recognition (about 90% top-1 acc; 10% improvement
models trained on ImageNet). Our approach is orthogonal to works that have employed data augmentation to increase data (e.g.. Zhang et al. (2019); Ghiasi al. (2020)), and b) we inspect the quality of data in existing datasets. As datasets grow larger in size it becomes increasingly harder to maintain high quality. High quality data is crucial for proper assessment of model performance. It is also important to have multiple evaluation sets with different characteristics to gauge generalization and robustness of detectors. We construct new validation sets evaluate detectors and diagnose model errors.
4.1 NEW DATASETS
COCO OL We selected images from all 80 categories of OpenImages that are in common with COCO, except per person, car, chair classes which are 3 most frequent classes in COCO (Fig. [4 Appx H.. This is to a) avoid making the dataset highly imbalanced and b) keep the number of samples and computation for model training under control. Some classes from Openimages fall under the same COCO category (e.g., bear, brown bear, polar bear all are mapped to the bear class in COCO). The class mappings are given in Appx G COCO image size, images from OpenImages are resized such that the larger dimension is 640px while preserving the aspect ratio. The train set of COCO OI has 1,418,978 annotations in total across 380, 1 images which larger than 1OCOtrain2017 with 860K boxes and 1 1182887 images. From these, 1824 images are selected from the train and test sets of OpenImages containing 558,977 boxes. As part of this dataset, we also create a validation set with 41,893 bounding boxes across 18,299 images chosen from validation set of OpenImages. See Fig. 6
To keep the image size in the range of
ObjectNet D. We annotated the object bounding boxes in the ObjectNet dataset originally introduced by Barbu et al. (2019). This dataset contains objects in daily life situations (https: //bp::///b// jectnee..eeeeeeee.e..e by Mechanical Turk workers using mobile app in a variety of backgrounds. rotations, and imaging viewpoints. Barbu et al. reported a dramatic performance drop of the SOTA models on ObjectNet compared to their accuracy on ImageNet (~ 40-45% drop). Therefore, it would be interesting to see how well object detectors perform on this dataset. In total, 5875
and its images (50,000 across 313 categories) are pictured
'When discarding some object classes from OpenImages, any image with any object instance from these classes, even though had some objects from other classes of interest, was discarded.
7

---

Under review as a conference paper at ICLR 2023
Dataset
Model
AP
APsa
APrs
APs
APM
AP
COCO
Efficientdet
51.2
70.4
55.1
35.7
55.8
64.8
DelectoRS
49.1
67.7
53.4
29.9
53
63.2
COCO_OT
Efficientdet
44
52.7
45.7
9
23.6
51.5
DetectoRS
51.5
51
45.1
11.4
25.5
50.9
ObjectNet_D
Efficientdet
23,9
41.6
24.5
5.7
24.7
DetectoRS
20.3
37.8
19.6
4.7
21.8
Table 2: Model performance over the validation sets of datasets.
bounding boxes across 5875 images (one annotated object per image) from 29 categories in common with COCO are annotated. Please see Appx for more details on this dataset.
4.2 MODEL PERFORMANCE AND ERROR ANALYSIS
ObjectNet
We tested two SOTA models. Efficientdet Tan al. 2019 and DetectoRS (Cai Vasconcelos 2018) on the new validation sets (trained COCO). As you can observe in Table 2 both models perform lower on ObjectNet_DD D and COCO_0 OI datasets compared to coco. In particular, over ObjectNet_D performance has been severely degraded partially because objects in this dataset are pictured in various, and sometimes odd, viewpoints and backgrounds (Fig. (). Also, APs and M on this dataset are both very low since almost all objects in ObjectNet_D are large, per the object size definitions in COCO.
To examine whether models behave similarly, we compare their errors over COCO and COCO_ON OI validation sets. To this end, we employ the recently proposed error diagnosis and analysis tool known as TIDE by Bolya et al. (2020). According to Fig. 7 both models behave the different. On COCO, errors are almost equally distributed over 4 major error types (Loc, C1 s, Bkg, and Mi ss), whereas over COCO OI, Bkg (i.e., classifying background as an object) significantly outweighs other error types. This result signifies the impact of distribution shift on models.
Figure Samples from validation sets of ObjectNet_D and COCO_OI.
each dataset. Their behavior across datasets, however, is drastically
Efficientdet
DetectoRS
Efficientdet
DetectoRS
Cis
Miss
Cls
Miss
Cts
COCO
COCO
Loc
Loc
Both
Loc
Bkg
Bkg
Both
Bkg
Bkg
Both
Css
26
25
15
Loc
Cla
25
1oC
Ok
25
40
LOC
Cbill
Dupe
10
Bkg
Sho
Big
Big
PM
FN
1822
FP PN
1012
Figure 7: Error analysis over validation sets of COCO and COCO_01 datasets.
4.3 LIMITATIONS OF THE COCO DATASET
To assess the quality of COCO annotations, we manually inspected val2017 images, and observed that annotations have similar error types in model predictions as illustrated in Fig. g We cluster the errors into four categories:
Misses, where some objects are not annotated (e.g.. the cup in Fig. A). Missed objects are often peripheral ones, and often small, occluded or cropped. Miss the most frequent error type, Non-existing objects or incorrect labels are false positives due to either confusing background as an object (Bkg error type; e.g., the sheep in Fig. B), or using a wrong label to annotate an object (Cls error type; i.c.. confusion with other classes, e.g., goat labeled cow in Fig B),
"efficientdet-d7,pp pth from
Anothor ant Pattttaa
8

---

Under review as a conference paper at ICLR 2023
Localization error, where bounding boxes around objects not accurately drawn (Loc error type; C.
Duplicates, where duplicate overlapping bounding boxes are drawn around the same object (Dupe; Fig.8 D). Also in some images large bounding box covers multiple objects close to each other. These annotations are tagged with iscrowd in COCO (group in OpenImages) and are treated differently during evaluation. Moving forward it would be better to make these annotations more precise since they often contain a large portion of the image background. OpenImages dataset suffers from the same errors as in COCO. Some shown in Fig. 6 ObjectNet D does not have these mistakes by construction, but is limited to only one annotated object per image (Fig.
In addition to above, in Fig. E we highlight some cases in which annotations are not consistent.
Missing annotation
Non-existing incorract labe!
For example, in some images, pictures of people posters are annotated, wheres in some other images they are not. The same holds about reflections in the glass or in the mirror, as well statues. In some images, an object is considered as part of another object and thus has been discarded (e.g.. keyboard images
Inaccurate boudning (localization error)
Duplicate
Inconsistent annotation
BROWN
laptop: last in Fig. E). Finally, some objects are very hard detect even for humans making the annotations debatable (i.e., low annotation consistency among people; Fig. F). For instance, it is not clear whether the right- most boxes in the first image contain cars, the animal on the right-side of the third image is actually zebra. Images and annotations of this sort can be either discarded or treated differently when evaluating models. In total, we found that COCOval2017, 52 objects are missed, 67 objects have incorrect labels, 89 objects have imprecise bounding boxes, and 19 objects have duplicate bounding boxes. We will share the corrected annotations.
BROW
Challenging' debatabie/ examples
Figure 8: Some annotation errors on COCOva12017. See also Appx
DISCUSSIONS, LIMITATIONS, AND FUTURE WORK
We found that the best detectors still underperform what is empirically possible, and the performance gap is wider over small objects. In measuring UAP, we label target boxes using trained classifier on objects. This essentially means that the deficit in mAP is mainly due to the classification error since other error types are fixed. Our approach is complementary to error analysis tools and provides additional insights into the bottlenecks in object detection from model-independdet perspective.
We foresee the following directions for future research: a) using better classifiers (e.g., transformers Dosovitskiy al. (2020) or ensembles of models to estimate the UAP, b) using the proposed techniques to generate even larger training datasets which can open up more research opportunities for robustness, transfer learning, and semi-supervised learning object detection, c) employing COCO_ OI for training models and comparing the results with models trained on COCO, d) fixing errors in detection datasets and re-evaluating models on them, e) analysing whether and how models utilize visual context in object detection, whether they do it similar to humans (Rosenfeld et al. 2018: Barnea & Ben-Shan ShahSha 2019 Singh et al. 2020), and whether results are consistent with studies of context prior to deep learning (e.g.. Rabinovich et al. (2007)), and f) invariance and robustness analysis of object detectors with respect to natural (c.g., domain shift) and synthetic distribution shifts ge.g., geometric transformations, blur).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Faisal Alamri and Nicolas Pugeault. Contextual relabelling of detected objects. arXiv preprint arXiv: 2019.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint
2018.
Moshe Bar. Visual objects in context. Nature Reviews Neuroscience, 5(8)6177 2004.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances neural information processing systems, 32:9453-9463, 2019.
Ehud Barnea and Ohad Ben-Shahar. Exploring the bounds of the utility of context for object detection. In Proceedings of the IEEE Conference on Computer Vixion and Puttern Recognition, pp. 7412-7420, 2019.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159 2020.
Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman, Tide: general toolbox for identifying object detection errors. 2020.
Ali Borji. Ming-Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A benchmark. IEEE transactions on image processing. 24(12):5706-5777 2015.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6154-6162, 2018
Kai Chen, Jiangmiao Pang. Jiaqi Wang. Xiong, Xiaoxiao Li, Shuyang Sun. Wansen Feng. Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 4974-4983, 2019a.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint 2019b.
Zhe Chen, Shaoli Huang, and Dacheng Tao. Context refinement for object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 71-86, 2018.
Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Santosh K Divvala, Derek Hoiem, James H Hays, Alexei A Efros, and Martial Hebert. An empirical study of context in object detection. In 2009 IEEE Conference computer vision and Pattern Recognition, pp. 1271-1278. IEFF, 2009.
Piotr Dollar, Christian Wojek, Bernt Schiele, and Pictro Perona. Pedestrian detection: An evaluation of the state of the art. IEEE transactions on pattern analysis and machine intelligence, 34(4):743-761, 2011.
Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 201: 1929,, 2020.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: retrospective. International journal of computer vision, 111(1): 03-136,77 2015.
Carolina Galleguillos and Serge Belongie. Context based object categorization: critical survey. Computer vision and image understanding, 114(6):112-77) 712-722, 2010.
Golnaz Ghiasi, Yin Cui. Aravind Srinivas, Rui Qian, Tsung Yi Lin, Ekin D Cubuk. Quoe V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. arXiv preprint 1727 2020.
10

---

Under review as a conference paper at ICLR 2023
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference computer vision, pp- 1440-1448, 2015.
Eran Goldman, Roei Herzig, Aviv Eisenschtat, Jacob Goldberger, and Tal Hassner. Precise detection in densely packed scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2227-5236, 2019.
Keren Gu, Brandon Yang, Jiquan Ngiam, Quoc Le, and Jonathon Shlens. Using videos to evaluate image model robustness. arXiv preprint arXiv:1104. 10076, 2019.
David Hall, Feras Dayoub, John Skinner, Peter Corke, Gustavo Carneiro, and Niko Sünderhauf. Probability detection quality (pdq): A prohabilistic approach detection evaluation. arXiv preprint arXiv: 1.11.11111100...11111 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings the IEEE international conference on computer vision, pp. 2961 2969, 2017.
Geremy Heitz, and Daphne Koller. Learning spatial context: Using stuff to find things. In European conference on computer vision, pp. 30-43. Springer, 2008.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 1at::::03332..66 2019.
Derek Hoiem, Yodsawalai Chodpathumwan, and Qieyun Dai. Diagnosing error in object detectors. In European conference on computer vision, pp. 340-353. Springer, 2012.
Jie Hu, Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 9401-9411, 2018.
Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, lan Fischer, Zbigniew Wojna, Yang Song. Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp.
7310-7311, 2017.
Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cnn. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6409-6418 18, 2019.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning, arXiv preprint arXiv: 11370, 6(2)8, 2019.
Alina Kuznetsova, Hassan Rom, Neil Alidrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci. Tom Duerig, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint urXiv:/811.00900 2018.
Hengduo Li, Bharat Singh, Mahyar Najibi, Zuxuan Wu, and Larry S Davis. An analysis of pre-training object 2019.
detection. arXiv preprint arXiv:
Tsung-Yi Yi Lin, Michacl Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp. 2980 2000-29888, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer. 2016.
Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid r-cnn. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7363-7372, 2019.
Sophie Marat and Laurent Itti. Influence of the amount of context learned for improving object classification when simultancously learning object and contextual cues. Visual Cognition, 20(4-5)5500-602, 2012.
11

---

Under review as a conference paper at ICLR 2023
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming, arXiv preprint arXiv: 1974474977.744, 2019.
Dmytro Mishkin, Nikolay Sergievskiy, and Jiri Matas. Systematic evaluation of convolution neural network advances on the imagenet. Computer Vision and Image Understanding, 161:11-19, 2017.
Roozbeh Mottaghi, Sanja Fidler, Alan Yuille, Raquel Urtasun, and Devi Parikh. Human- maaan- crts for identifying bottlenecks scene understanding. IEEE transactions on pattern analysis and machine intelligence, 38( 1):74877, 2015.
Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan. Localization recall precision (Irp): A new performance metric for object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 504-519, 2018.
Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan. Generating positive bounding boxes for balanced training of object detectors. In IEEE Winter Conference Applications of Computer Vision (WACV), 2020.
Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin. Libra r-cnm: Towards balanced learning for object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 921-830, 2019.
Devi Parikh and C Lawrence Zitnick. Finding the weakest link in person detectors. In CVPR 2011, pp. 1425-1432 Citeseer, 2011.
Vitali Petsiuk, Rajiv Jain, Varun Manjunatha, Vlad I Morariu, Ashutosh Mehra, Vicente Ordonez, and Kate Saenko. Black-box explanation of object detectors via saliency maps. arXiv preprint arXiv:2006.0324 03204. 2020.
Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, and Serge J Belongie. Objects in context. In ICCV, volume pp. Citeseer, 2007.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint ar Xiv:2103300022 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv: 108/1, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and a loss for bounding box regression. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 658-666, 2019.
Amir Rosenfeld, Richard Zemel, and John Tsotsos. The elephant the room, arXiv preprint arxiv: 1808.03305, 2018.
Olga Russakovsky, Jia Deng, Zhiheng Huang, Alexander C Berg, and Li Fei-Fei. Detecting avocados to zucchinis: what have we done, and where are we going? In Proceedings of the IEEE International Conference on Computer Vision, pp.
2013.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Adilya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 252, 2015.
Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang. Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy imagenet. In International Conference on Machine Learning (ICML), 2020.
Bharat Singh and Larry S Davis. An analysis of scale invariance in object detection snip. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3578 3587, 2018
Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jac Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't 't judge object by its context: Leaming overcome contextual bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10070-1008,, 2020.
Zheng Song, Qiang Chen, Zhongyang Huang, Yang Hua, and Shuicheng Yan. Contextualizing object detection and classification. In CVPR 201 1, pp. 1588-11592. IEFF, 2011.
12

---

Under review as a conference paper at ICLR 2023
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 943-855,220 2017.
Mingxing Tan, Ruoming Pang. and Quoe Le. Efficientdet: Scalable and efficient object detection. arXiv preprint arXi:: 2019.
Zhi Tian, Chunhua Shen, Chen, and Tong He. Foos: Fully convolutional one-stage object detection, arXiv preprint 1XXv::994.00355 2019.
Antonio Torralba and Pawan Sinha. Statistical context priming for object detection. Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 1, pp. 763-770, IEEE, 2001.
Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, and Antonio Torralba. Hoggles: Visualizing object detection features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1-8, 2013.
Lior Wolf and Stanley Bileschi. A critical view context. International Journal of Computer Vixion, 69(2): 151-261, 2006.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Lo. and Ross Girshick. Detectron2. 2019.
at tps
//github. htw.ttuu.coooo/ce http://cccooooeeeeccceeeeeeeeeee
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Dateu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In Proceedings of the IEEE conference on computer vision and pattemn recognition, 3074-3974999 9983,,983, 2018.
Peng Zhang, Jiuling Wang, Ali Farhadi, Martial Hebert, and Devi Parikh. Predicting failures of vision systems. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3566-386 3073,, 2014.
Shanshan Zhang. Rodrigo Benenson, Mohamed Omran, Jan Hosang, and Bernt Schiele. How far are we from solving pedestrian detection? In Proceedings of the iEEE conference on computer vision and pattern recognition, pp. 1259-1267, 2016.
Zhi Zhang, Tong He, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of freebies for training object detection neural networks. arXiv preprint 1rXiv:1902.041000 2019.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint arXiv:1904.07850 2019.
Xiangxin Zhu, Carl Vondrick, Deva Ramanan, and Charless C Fowlkes. Do we need more training data or better models for object detection". In BMVC, volume 3, pp. 5. Citeseer, 2012.
Xizhou Zhu, Dazhi Cheng, Zheng Zhang, Stephen Lin, and Jifeng Dai. An empirical study of spatial attention mechanisms in deep networks. arXiv preprint 1744v:190,.05873, 2019.
Zhuotun Zhu, Lingxi Xie, and Alan L. Yuille. Object recognition with and without objects, 2016.
Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection 20 years: A survey, arXiv preprint 105.v: 2019.
13

---

Under review as a conference paper at ICLR 2023
A STATE OF THE ART IN OBJECT DETECTION PERFORMANCE
Fig. showss the state of the art in object detection performance over VOC and COCO datasets taken from ht tps: / http://pppwwwtthoooooosoooooooee wwwperrewitccdee..... com/ It also shows our computed upper bound mAP (thick black lines). While there has been a steady progress over time, there is still a significant gap between models and what is empirically possible.
PASCAL VOC
Erpiinaal Uporr Bound
91.8
90
SNIPER
Nouple
MAP
0.
70
60
2015
2016
2017
2018
2019
2020
2021
Other models
Models highest
78.2
Empricai Upper
MS 0000
70
60
Cáscade FFT-87 NAS-FPN self-training Copy Paste, singlay
RetinaNet (SpineNet- 190, 1986x1536)
BDX
Mask RoCAN (tesNeXt- 150 (2P-ii-esssadd
Mask R-CNN (R55Next-152-1111
30
2017
2018
2019
2020
2021
Other models
Models highest box
80
MS COCO
70.2
Empiriall Upper Bound
EfficentDet-D DDx (sigle-sccal))
70
Mask R-CMI (kesNeXi- 152-1 TN
APSO
FPN
60
50
2017
2018
2019
2020
2021
Other models
Models highest
Figure State the art object detection performance taken from https::/papperwwi www.:/wwwwwwhwccccc.....coceeeee com/ well as our computed upper bound mAP.
14

---

Under review as a conference paper at ICLR 2023
B AP CALCULATION
AP calculation compiled from here
1. For the entire dataset, sort all the detections labeled with the same class with respect to their confidence scores.
2. Now, go these sorted detection list and check whether each detection be assigned to a ground truth. The assignment (or labeling as a True Positive) is based on Intersection Over Union (IoU) of the detection with a ground truth. There is true positive validation threshold in terms of loU and it is generally 0.5. In this step, note that a ground truth can be matched with only one detection (and this detection is the one with the highest confidence score since we go over sorted list).
3. At the end of the previous step, we have identified the True Positive (TP) detection boxes, False Positive (FP) detection boxes and False Negative (FN) ground truth boxes. So, by going over the sorted detection list again, we can find precision for each recall to draw the Recall-Precision (RP) curve. The blue curve in the figure the RP curve.
4. At the end of the previous step, have identified the True Positive (TP) detection boxes, False Positive (FP) detection boxes and False Negative (FN) ground truth boxes. So, by going over the sorted detection list again, we can find precision for each recall to draw the Recall-Precision (RP) curve. The blue curve in the figure is the RP curve.
5. To discard the wiggles of the RP curve, at some recall point interpolate the blue curve to the highest precision possible at the positive side of this recall point. Thus, the red curve, called the interpolated RP curve, obtained.
15

---

Under review as a conference paper at ICLR 2023
C RESULTS OF OUR SECOND STRATEGY FOR COMPUTING UAP
Here, we present the AP results of our second strategy using the original classifier trained on the canonical object size and applied the surrounding boxes (instead training a new classifier on the surrounding boxes). In the main text we reported results using our first strategy which is using the same canonical object size during training and testing (i.e., no search for the best surrounding box).
Most Confident Box
Most Frequent Label
Dataset
Acc.
AP
API
APm.
AP.
AP
APr
AP.
AP,
VOC
91.6
88.7
91.7
81.4
63.8
89.1
92.0
82.9
60
COCO
82.6
76.9
81.8
80.6
62.8
76.4
82.0
80.4
60.7
Table 3: Results our second strategy for estimating the upper-bound AP (i.e., searching for the best bounding box or object label near a target box; among boxes with IOU 2 0.5). Here, we have used the original classifier trained on the canonical object size to classify the surrounding boxes. instead of training new ones on the surrounding boxes.
D CONFUSION MATRICES
These the confusion matrices associated with the classifier trained the canonical objects results of which is presented in Fig. in the main text (i.e., our first strategy).
VOC
Confusion normalization
sersplane
ticycle
BB
2
bottle
bus
car
97
95
True labe
dningtable
dag
90
motorbike
pottesplan ant
sheep
94
sofo
bimoniter
wmopler
bicytie
brd
boile
bo
or
thar
dningable
dg
motorbiko
person
noltedpoen ent
sheep
Immoritor
Figure 10: Confusion matrix of the trained classifier the original object size over the VOC dataset (numbers
in %)..
16

---

Under review as a conference paper at ICLR 2023
CoCo
Confusion matrix, with normalization
boydle
car
motorcycle
arplane
buck
bost
traffic light
stop sign
parking
0.8
bench
brdd
cat
dbg
horse
sheap
dephant
bear
zebra
graffe
tackpack
uatcase
Ricboo
dus
0.6
mowbbard
sports
baseball glove
skateboard
True label
surfboard
tennis racket
bottle
Bark
inife
spoon
apple-
0.4
broccol
carrot
hot dog
2
auch
potted plant
bed
dnnnng
lppop
mouse
romoto
leyboard
0.2
cell phone
oven
baster
11
ank
wfrigerator
book
5.2
2
teddy bear
hair drier
288
botheruch
Biyder
metorcy
orglame
bast
traftic tight
Herse
Sheep
albra eephant
graffe
umbieela haniitine
goo
snoetii oports qall
Mlo
Sooetholi bat-
weeeee gloes
temms ricest
bettle
wime glass
kntio
spoon
baxe
banana
appie
cange
camm
hot dog
gizz
dinut
glkee
oouth
cant
boo
table
laghop
temooo
call Paaenii
100000
hald Aambuuu
tac
nottel
driing
Figure 11: Confusion matrix of the trained classifier the original object size over the COCO dataset (numbers
are in %).
17

---

Under review as a conference paper at ICLR 2023
E MODEL AP AND UAP OVER PASCAL VOC AND COCO DATASETS
UAP over VOC using the COCO and VOC evaluation codes shown in Fig. 12 using the MMDetection benchmark for object detection over COCO using the COCO evaluation code.
In 13 we show results
47.9
cat
snq
5op
taln
horse
Using COCO evaluation code
VOC
aerollane
oar
bicycle
moloriike
M00
lwonnloor
BjoS
CenterNet dia 512
parson
sheap
FasterRONN
UpperBBuud
bird
dninggable
FCOS
bottle
boat
chalr
poltddllant
80
100
80
40
CenterNet, dia 512
CenterNet dla 384
cat
FasterRCNN
UpperBound
horsa
Вор
SSD12
hoycle
FCOS
train train
Using VOC evaluation code
bb ane
mmm
sheep
brd
Bjfa
VOC
boat
botll
chalr
89.5
80.0
pottadiian
100
50
100
4P
60
Figure 12: Model scores and upper bound over PASCAL VOC dataset using VOC (left) and COCO AP evaluation codes (right). Categorics are sorted based on the average model AP. Bar charts show classification scores. Solid red and dashed black lines represent upper bound AP, and the best model AP, respectively.
18

---

Under review as a conference paper at ICLR 2023
Faster,
Cascade_RCNN
FCO8_ matrain
Lbbra Faster, RONN
70
80
AP_50
50
30
AP_S
101 61:10
X-101 -41-0411111111 Tx
R-50 20s
20t
SS03300 VODTE, 120
R30 1x
X-10 -1--8 11
R300 2X
X-101 6A 24
2-101-32x44..0 1x
R:101 (e) Tx
(0)
R:101 2x
(6) _2x
X-101 -4-44 2x
X-101 2ww-222ddd.. fx
R:101
101--4499
R401
R301
RA01
3.50
R:01
X-101
X-101
GA Faster RCNN
GA Retnallet
Mask RCNN
Mask_RCMN
Mask Scoring, RCNN Casc. Mask Mask_RCNN RCWN Hyrbrid Task Canc
Canterlist
50
30
10
80
30
10
X-101-4466 Tx
X-101-2x64_11 1x
X-101 1101-44-4 Tx
8-101c) Fx
18
Is
24
4-01-64w60_11 Tx
2.101-84-4,.2 2x
X-101 R-00 ffx
1-101-6466_11 TN
R:50 20
X-101 R-101 20s
88500 20
10101 20e
x-101 10.14444ddd.... 20
10
R-101 (9)
x-101-32x64.1
X-101-37X 101--2-111
2-101-41644
8-101
Figure 13: APs COCO dataset borrowed from the MMDetection benchmark. We add CenterNet results
MMDetection.
19

---

Under review as a conference paper at ICLR 2023
F RAW MODEL AP AND UAP PERFORMANCES OVER VOC AND COCO DATASETS
Here, show the detailed performance of UAP and AP of object detectors over COCO and VOC datasets results of which presented throughout the paper. In addition to precision, recall is also presented.
Upper-bound AP results
COCO dataset
~--- Summary metrics
Average Precision (AP) [[ IoU=0.50:0.99
area= all
maxDets=100
0.782
Average Precision (AP) [ IoU=0.50
area= all
maxDets=100 0.782
Average Precision (AP) [[ IoU=0.75
all maxDets=100 0.782
Average Precision (AP) [[ TcU==0.50:0.93
area= sma
maxDets=100 0.635
Average Precision
(AP) [ IcU-0.50:0.95
area-medium
maxDets-10c 0.816
Average Precision
(AP) @ IcU-0.50:0.35
area- large
maxDets-10C 0.846
Average Recall
(AR) [[ IcU=0.50:0.95
area= all
maxDets=
0.483
Average Recall
(AR) e[ IcU=0.50:0.95
area=
all
maxDets= 10
0.797
Average Recall
(AR) [[ IcU=0.50:0.95
area=
all
maxDets=100 0.812
Average Recall
(AR) el I00=0.50:0.95
area= small
maxDets=100 0.663
Average Recall
(AR) @l Io0=0.50:0.95
area=medi
maxDets=100
0.843
Average Recall
(AR) e 100
area= large
maxDets=100
0.893
PASCAL VOC dataset
---- Summary metrics
Average Precision (AP) 0[ IcU=0.50:0.39
area= all
maxDets=100 0.916
Average Precision (AP) IcU=0.50
area= all
maxDets=100 0.916
Average Precision
(AP) [[ IcU=0.75
area= all maxDets=10c 0.916
Average Precision
(AP) [[ IoU=0.50:0.95
area= small
maxDets=100 0.707
Average Precision
(AP) @ ToU=0.50:00.5
area=medi
maxDets=100
0.861
Average Precision
(AP) [ IcU=0.50:0.95
area= large
maxDet.s=100 0.941
Average Recall
(AR) el I00-0.50:0.95
area- all
maxDets=
0.579
Average Recall
(AR) 01 IcU=0.50:0.95
area= all
maxDets= 10 0.908
Average Recall
(AR) el I0U-0.50:0.95
area= all
maxDets=100 0.930
Average Recall
(AR) el IcU=0.50:0.95
area= small
maxDets=100 0.736
Average Recall
(AR) el IoU=0.50:0.95
area=medi
maxDets=100
0.877
Average Recall
(AR) @ IcU=0.50:0.95
area= large
maxDets=100
0.954
Model performance
COCO dataset
8 FasterRCNN
Summary metrics
Average Precision (AP) 0[ IcU=0.50:0.39
area= all
maxDets=100
0.364
Average Precision (AP) [[ IoU=0.50
area=
maxDets=100 0.584
Average Precision (AP) 0 IoU=0.75
area= all
maxDets=10c 0.391
Average Precision (AP) e[ IcU=0.50:0.99
area= small
maxDets=100 0.215
Average Precision
(AP) @ IoU=0.50:0.95
area=medine
maxDets=100
0.400
Average Precision
(AP) 01 IcU=0.50:0.95
area= large
maxDets=100 0.466
Average Recall
(AR) el Io0=0.50:0.95
area= all
maxDets=
0.304
Average Recall
(AR) e1 IcU=0.50:0.95
area= all
maxDets= 10 0.489
Average Recall
(AR) e1 IoU=0.50:0.95
area=
all
maxDets=100 0.514
Average Recall
(AR) el IcU=0.50:0.95
area= small
maxDets=100
0.324
Average Recall
(AR) 0 ToU=0.50:0.95
area=medium
maxDets=100 100
0.554
20

---

Under review as a conference paper at ICLR 2023
Average Recall
(AR) e I0U=0.50:0.95
area= large
maxDets=100 ] 0.645
$ FCOS
Summary metrics
Average Precision (AP) [[ 000=0.50:0..9
area= all
maxDets=100
0.428
Average Precision (AP) [[ IcU=0.50
area= all maxDets=100 0.626
Average Precision (AP) @ IoU=0.75
area= all maxDets=100 0.457
Average Precision (AP) [ :o0=0.50:0.95
area= small
maxDets=100 0.265
Average Precision (AP) @ To0=0.50:00.55
area=medi
maxDets=100
0.469
Average Precision
(AP) @1 IoU=0.50:0.95
area- large
maxDets=10c 0.545
Average Recall
(AR) e I0U-0.50:0.95
area- all
maxDets-
0.345
Average Recall
(AR) e IcU=0.50:0.95
area= all
maxDets= 10
0.552
Average Recall
(AR) e[ I0U=0.50:0.95
area= all
maxDets=100 0.582
Average Recall
(AR) 0 IcU=0.50:0.95
area= small
maxDets=100 0.388
Average Recall
(AR) el IcU=0.50:0.95
area=medium
maxDets=100
0.628
Average Recall
(AR) e I00=0.50:0.95
area= large
maxDets=100
0.735
RetinaNet
~--* Summary metrics
Average Precision (AP) [[ IcU=0.50:0..9
area= all
maxDets=100
0.400
Average Precision (AP) [[ IoU=0.50
area=
all maxDets=100 0.609
Average Precision (AP) 0 IcU=0.75
area= all
maxDets=100 0.430
Average Precision (AP)
TcU=0.50:0.95
11
maxDets=100 0.235
Average Precision
(AP) [[ IcU=0.500:..95
area=medi
maxDets=100 0.444
Average Precision
(AP) @ IoU-0.50:0.95
area- large
maxDets-100 0.526
Average Recall
(AR) @ IcU-0.50:0.95
area- all
maxDets=
0.328
Average Recall
(AR) e[ IcU=0.50:0.95
area=
all
maxDets= 10
0.522
Average Recall
(AR) e IcU=0.50:0.95
area= all maxDets=100
0.555
Average Recall
(AR) 0 IcU=0.50:0.95
area= small
maxDets=100
0.361
Average Recall
(AR) el Io0=0.50:0.95
area=medi imm
maxDets=100
0.599
Average Recall
(AR) 0 500=0.50:0.95
area= large
maxDets=100
0.704
8 SSD
Summary metrics
Average Precision (AP) 0 IcU=0.50:0.99
area= all
maxDets=100
0.293
Average Precision (AP) @[ IoU=0.50
area= all maxDets=100 0.492
Average Precision (AP) @[ ToU=0.75
area= all
maxDets=100 0.309
Average Precision
(AP) e[ IcU=0.50:0.95
area= small
maxDet.s=100 0.118
Average Precision
(AP) @l IoU-0.50:0.95
area-medi
maxDets-10c
0.341
Average Precision
(AP) 01 IcU=0.50:0.95
area= large
maxDets=100
0.447
Average Recall
(AR) e IoU=0.50:0.95
area= all
maxDets=
0.264
Average Recall
(AR) e[ IcU=0.50:0.95
area= all
maxDets= 10 0.400
Average Recall
(AR) 01 ToU=0.50:0.95
all
maxDets=100 0.425
Average Recall
(AR) 81 ToU=0.50:0.95
area= sma
maxDets=100
0.173
Average Recall
(AR) el ToU=0.50:0.95
area=medium
maxDets=100 =10
0.488
Average Recall
(AR) @ 1cU-0.50:0.95
area- large
maxDets= 100
0.607
21

---

Under review as a conference paper at ICLR 2023
VOC results using COCO evaluation code
$ CenterNet
Average Precision (AP) [[ ToU=50:
area= all
maxDets==00 = 47.8
Average Precision
(AP) [[ IcU=50
area= all maxDets=100 72.7
Average Precision
(AP) 0 IoU-75
area- all
maxDets-100 51.3
Average Precision
(AP)
IoU=50:95
area- small
maxDets-100
07.4
Average Precision
(AP) e[ IcU=50:95
area=medium
maxDets=100 = =26.2
Average Precision
(AP) e IcU=50:95
area= large
maxDets=100
61.0
Average Recall
(AR) @ IoU=50:95
area= all
maxDets= 1 = 40.2
Average Recall
(AR) @ ToU=50:95
area= all
maxDets= 10
57.2
Average Recall
(AR) el ToU=50:95
area= all
maxDets=100 = 58.4
Average Recall
(AR) el IoU=50:95
area" small
maxDets-100
18.6
Average Recall
(AR) el IoU-50:95
area-medium
maxDets-100
40.9
Average Recall
(AR) e IcU=50:95
area= large
maxDet.==10 = 70.5
FCOS
Average Precision (AP) 0 IoU=50:95
area= all
maxDets=100 = 47.9
Average Precision (AP) @ IoU=50
area= all maxDets=100 71.0
Average Precision (AP) 0 IoU=75
area= all
maxDets=100 51.4
Average Precision (AP) [ IcU=50:95
area= small
maxDets=100
-11.1
Average Precision
(AP) @ IcU=50:95
area=medi um
maxDets=100 = 33.11
Average Precision
(AP) @ IcU=50:95
area= large
maxDets=100
58.4
Average Recall
(AR) el IoU=50:95
area= all
maxDets= 1 41.2
Average Recall
(AR) [ IcU=50:95
area= all
maxDets= 10 58.5
Average Recall
(AR) e IoU=50:95
area= all
maxDets=100
59.5
Average Recall
(AR) el IcU=50:95
area= small
maxDets=100 19.5
Average Recall
(AR) el TcU=50:95
area=medive
maxDet.=100
45.2
Average Recall
(AR) e TcU=50:95
area= large
maxDetss=00 = 70.2
MASK RCNN
FasterRCNN
Average Precision (AP) [ IcU=50:95
area= all maxDets=100 = 47.3
Average Precision (AP) [[ IcU=50
area= all
maxDets=100 71.3
Average Precision (AP) @[ IoU=75
area= all
maxDets=100 52.6
Average Precision (AP) 0[ ToU=50:95
area= small
maxDets=100 08.6
Average Precision
(AP) @l IoU=50:95
area-medium
maxDets=100
30.7
Average Precision
(AP) 01 IoU=50:95
area- large
maxDets-100 58.1
Average Recall
(AR) [[ IcU=50:95
area= all
maxDets= 1 40.3
Average Recall
(AR) [[ IcU=50:95
area= all
maxDets= 10 = 53.8
Average Recall
(AR) IcU=50:95
area= all
maxDets=100
54.1
Average Recall
(AR) 0l IoU=50:95
area= small
maxDets=100 = 11.2
Average Recall
(AR) el IoU=50:95
area=medium
maxDets=100
36.9
Average Recall
(AR) e IoU=50:95
area= large
maxDets=100 100 = 65.7
VOC results categories using VOC evaluation code
Upper bound
AP for aeroplane 90901
AP for bicycle = 0. 9033
AP for bird 0.9065
AP for boat 0.8951
AP for bottle 0.9056
AP for bus 0.9039
AP for car 0.5052
22

---

Under review as a conference paper at ICLR 2023
AP for cat 0.9062
AP for chair 0.8722
AP for cow 0.8933
AP for diningtable = 0.8968
AP for dog 0.8950
AP for horse 0.8969
AP for motorbike 9.9054
AP for person 0.9066
AP for pottedplant 0. 9085
AP for sheep 0.9035
AP for sofa 0.7672
AP for train 0.9087
AP for tvmonitor 0.9012
Mean AP 0.8945
FCOS
AP for aeroplane 0.8701
AP for bicycle = 0484
AP for bird 0.7722
AP for boat 0.6895
AP for bottle 0.6709
AP for bus 0.8371
AP for car 0.8716
AP for eat 0.8704
AP for chair 0.6213
AP for cow 0.8362
AP for diningtable = 6900
AP for dog 0.8572
AP for horse 0.8414
AP for motorbike 0.7966
AP for person 0.8430
AP for pottedplant 0.5464
AP for sheep 0.8107
AP for sofa 0.7679
AP for train 0.8454
AP for tvmonitor 0.7735
Mean AP 782829
FasterRenn
AP for aeroplane 0.8147
AP for bicycle 0.8656
AP for bird 0.7555
AP for boat 0.6552
AP for bottle 0.6848
AP for bus 0.8660
AP for car 0.8821
AP for cat 0.8811
AP for chair 0.6376
AP for
0.8527
AP for diningtable = 0.7977
AP for dog 0.8691
AP for horse 0.8816
AP for motorbike 0.8602
23

---

Under review as a conference paper at ICLR 2023
AP for person 0.8008
AP for pottedplant = 0.5136
AP for sheep 0.7870
AP for sofa 0.8221
AP for train 0.8543
AP for tvmonitor - 0.7778
Mean AP 0. 7945
SSD12
AP for aeroplane 0.8388
AP for bicycle 0.8619
AP for bird = 0.8119
AP for boat 0.7161
AP for bottle 0.6090
AP for bus 0.8730
AP for car 0.8910
AP for cat 0.8951
AP for chair 0.6361
AP for cow 0. 0626
AP for diningtable = 7550
AP for dog 0.8681
AP for horse 0.8772
AP for motorbike = 0.8373
AP for person 0.8215
AP for pottedplant = 0. 066666
AP for sheep 0.8127
AP for sofa 0.7970
AP for train 0.8723
AP for tvmonitor 0.7855
Mean AP 7996
G LIST OF OBJECT CATEGORIES IN NEW DATASETS
The list of object categories present in the COCO along with their corresponding classes in the OpenImages dataset is given below. The mapping between COCO to ObjectNet D datasets is also presented. Note that some classes in COCO correspond more than one class in the destination dataset (i.e., one many relationship).
COCO to OpenImages category mapping:
COCO to tt_ooto______ dictict ('Baseball glove' Basseball glove', PParinng meter' : 'Pariing meter', 'kkateoarr'' 'Skateboard',
Car'' 'Car',
Cup'' l'coffee cup', 'Messuring cup'), ) 'Toilet' 'Toilet', 'bicycle' : 'bicycle', 'sink' 'sink', 'cow': ['Cattle', Bulll", 'cat': 'cat', 'Motorcycle's : 'Motorcycle', 'Bed' 'Bed',
'boat' 'boat',
'Cake' "Cake',
'refrigerator' : 'refrigerator', 'Ai 'Alrnnane' 'Airplane',
24

---

Under review as a conference paper at ICLR 2023
Carrot' 'Carrot', Stop sign' : Stop sign', 'Zebra' Zebra'', 'Tie' 'Tie', 'Pizza' 'Pizza',
"Banana' 'Banana',
'Tafffic light' 'Traffic light', Tennis racket' "Tennis racket', Microoaae'' : 'Microwave oven', :Apple': 'Apple',
'onutt: DDuughntt',
Bear' ['Bear', 'Brown bear', 'Polar bear'),
"Teddy bear' :
'Teddy bear',
:clock' ['Alarm clock', 'Clock', 'Digital clock' ,, ['Ball', 'Football', 'Volleyball' Tennis Ball', 'Rugby ball', 'golf ball', 'Cricket ball'],
sports ball'- :
'horse': 'horse', 'Hair drier's : 'Hair dryer',
'Fire hydrant': 'Fire hydrant',
"Bus': 'Bus',
'Dining Table' I'Table', 'Kitchen & dining room table' ) :Cocch': ['Couch', 'studio couch'),
moouse' 'computer mouse',
Remote' 'Remote control',
Bench': 'Bench',
Scissors'; 'Scissors', 'Tuckk': ' 'Truck',
Wine glass': Wine glass', Oven': 'Oven',
'ackpack'; 'Backpack', 'Bird' 'Bird', Handdbag' 'Handbag',
Sheep' 'Sheep',
Spoon': 'Spoon',
Tv' : Television',
Frisbee' "Flying disc',
'eyboard' : 'computr Keyboard',
Sandwich' ('Sandwich', Submarine sandwich' )
'Cell phone' : 'Mobile phone', 'Giraffe' 'Giraffe',
Chair'' 'Chair'
'Bottle' 'Bottle',
Potted plant' 'Houseplant', 'roccoli'' : 'Broccoli', Umbrella' 'Umbrella',
Orange' 'Orange', Hot dog' 'Hot dog', Knife' kitchen knife', Vasee 'Vase', Surfboard' : 'Surfboard', Toaster' 'Toaster',
Bowl' ['bowl', 'Mixing bowl'], 1,
'uutccase' : 'Suitcase', 'orr'' 'Frrk'
'kkis' "ski'
"Book' 'Book',
"Kite' 'Kite',
"Person' : 'Person',
"Dog" 'Dog',
25

---

Under review as a conference paper at ICLR 2023
'Lpptop' 'Laptop',
Elephant' : 'Elephant',
Toothbrush' 'Toothbrush',
Baseball bat' : Baseball bat', Snowboard' 'Snowboard', Train' 'Train')
COCO ObjectNet D category mapping:
coco to joco_to_ObjNeet jb - (' Baseballl glove' 'baeebal__glove',
'Cup' [' d'drinkingccup', 'measuring_cup' )
'bicycle' 'bicycle',
'Tie' 'tie',
'Bnanna' 'banana', Tennis racket' tenniss beannisrracke'', MMicrowae'' : 'microwave',
'clock': alarm_clock!
'Hair drier': 'hair_dryer',
Diiing Table' 'coffee_table',
mouse' 'computer_mouse'
Remote' 'remote_control',
Bench': 'bench', Backpack'; 'backpack', :'v'''Tv,
'Kyyborrd' : 'keyboard',
Cell phone' : 'cellphone"
Chair' 'chair'
Bottle'' l'water_bottle', 'cooking_oil bbeerboottle, 'wine_bottle" )
Umbrella' : 'Umbrella',
Orange' 'Orange',
"nifee I'butchers_knife'' 'bread_knifee)
VVase' 'aaee'
'oaster'' : 'Toaster',
'Bowl' ['mixing
soup_bowl'
Suitcase' : 'briefcase',
Bookk 'book_closed',
'aptop'' 'iaptop_open's
Baseball bat' 'Baseball bat'''
H DATASET STATISTICS
Some statistics of the proposed datasets shown here. In building COCO OI dataset, Fig. 14 have avoided to include samples from Person. Car, and Chair categories from the OpenImages dataset. Including these categories will make the COCO_ON __ dataset highly imbalanced. Our code for creating this dataset (link, however, is very general and allows discarding arbitrary object categories make more balanced dataset.
H.1 DETAILS ON OBJECTNET DATASET
ObjectNet dataset Barbu et al. (211)), is build with the purpose of having less bias than other recognition datasets. It consists indoor objects that are available to many people, are mobile, are not too large, too small, fragile dangerous. This dataset is supposed to be used solely as a test set and comes with a licence that disallows the researchers to finetune models on it.
ObjectNet images pictured by Mechanical Turk workers using mobile app in variety of backgrounds, rotations, and imaging viewpoints. ObjectNet contains 50,000 images across 313 categories, out of which 113 in common with ImageNet categories. Astonishingly, Barbu et al. found that the state of the art object recognition models perform drastically lower on ObjectNet compared to their performance ImageNet (about 40-45% drop).
26

---

Under review as a conference paper at ICLR 2023
Figure 14: Distribution of the number of objects per category in training sets of COCO and COCO_0 _I datascts.
27

---

Under review as a conference paper at ICLR 2023
ObjectNet D
MS COCO
COCO
1301 Dupued
Abrica reppiio
proreg
C000 1005
DIOM CMAS
209te
100
stoso
100000
1044
sgaqun
500
8
IN001 01RA1
1990 0p
Figure 15: Distribution of number of objects across validation sets the used datasets.
28

---

Under review as a conference paper at ICLR 2023
10
10
10
ObjectNet. D
MS COCO
COCO 01
Distribution of normalized
0.8
0.8
0.8
06
object area
0.6
object area
06
object area
object size
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
30000
25,000
20000
15000
10000
5000
20000
10000
5000
1750
1500
1250
1000
750
500
250
*
MS COCO
COCO_OI
ObjectNet.
Distribution of number of objects per image
016oj)
(caled 016o1)
(caled 016ot) #
Figure 16: Distribution of the number of objects and normalized object size across the used datasets.
29

---

Under review as a conference paper at ICLR 2023
The 113 object categories in the ObjectNet dataset, overlapped with the ImageNet, contain 18,574 images in total. On this subset, the average number of images per category is 164.4 (min=55, max=284). We drew bounding box around the object corresponding to the category label of each image. there were multiple nearby objects from the same category (e.g., chairs around a table), tried to include all of them in the bounding box. Some example scenes and their corresponding bounding boxes are given in Fig.
ImageNet
ObjectNet
Chairs by rotation
Chairs by background
Chairs by viewpoint
Chairs
Teapots
T-shirts
Figure 17: Sample images from the ObjectNet dataset along with manually annotated object bounding boxes from Chairs. Teapots s and T-shirts categories. The leftmost column shows three chair examples from the ImageNet dataset. ImageNet scenes often have a single isolated object them whereas images in the ObjectNet datasct contain multiple objects. Further, ObjectNet objects cover wider range of variation in contrast, rotation, scale, and occlusion compared to ImageNet objects (See arguments in Barbu al. (2019). In total, we annotated 18,574 images across 113 categories in common between the two datasets. This figure modified from Figure Barbu et al. (2019).
Efficientdet
DetectoRS
Cls
Miss
Miss
Loc
Loc
Both
Bkg
Both
Bkg
20
15
FPFN
Figure 18: Error analysis over the ObjectNet D dataset.
30

---

Under review as a conference paper at ICLR 2023
I ANNOTATION ERRORS IN COCO DATASET
Annotation errors over COC1va12017 images, categorized in four categories, shown in Fig. 19 (A, B, CC& & D). In addition, inconsistent annotations as well some challenging images are also shown.
A) Missing annotation (miss)
Non-exlsting object incorrect label
Inaccurate boudning box (localization error)
Duplicate
Inconsistent annotation
BROWN
F) Challenging/ debatable/ hard examples
Figure 19: Additional annoratio emors in COCOval2017.

---

Under review as a conference paper at ICLR 2023
J RESULTS OVER FASHION AND DOTA DATASETS
J.1 FASHION DATASET
This dataset covers 40 categories of clothing items (39 + humanss.. Trainval, and test for this dataset contain 206,530 images (776, 172 boxes) and 51, 51,650 images (193,689 boxes), respectively. Fig. 20 displays samples from this dataset (and also additional statistics). This is challenging dataset since clothing items are non-rigid opposed to COCO or VOC objects.
Results are shown in Fig- 22 The best classification accuracy on this dataset is 88.8%. The UAP 71.2 and the AP of the best model is 59.7 (FCOS). Interestingly, FCOS performs quite close to the upper bound at IOU=0.5. Models perform better here than over VOC. The FASHION UAP is lower than VOC UAP perhaps because classification is more challenging on the former dataset. The gap between UAP and model AP here, however, much smaller than VOC. This could be partly due to the fact that FASHION scenes have less clutter and larger objects than the VOC scenes. While per-class UAP above the AP of the best model over all VOC classes, UAPs of five FASHION categories fall below the best model AP (messenger bags, lunics, long sleeve shirts, blouses, and rompers). Looking at the classification scores, we find that they have low accuracy.
32

---

A
B
Train
Test
% of boxes
22
c20
saBelll Of %
020
% of
000
Spereeaan (m))
all objects distinct objects
50000
40000
20000
20000
6 otjects
3
Train
all objects
distinct objects
10000
8000
0000
2000
objects
Test
Figure 20: Statistics of the FASHION dataset. A) Sample images along with FCOS predictions, B) Percentage of annotated bounding boxes and images in train and test sets, and C) Aspect ratio, object size, and number of objects per image.

---

Under review as a conference paper at ICLR 2023
Fashion
Contusion matr with normalization
messangerbagg
pars
camss
tunes
roate
surhets, conboytatt hats
scarves serpo
sweaters
11
drcsocs
pants
80
duiches
shotts
legginge
Boots
65
1700 53001
jimpsutts
61
sandak
71
76
bear ielnscaps
15
Mouses-
slitt
38
bngeleeveshiit
bascballcaps
polcs
cardigane
bumans
unts
condggt
Figure 21: Confusion matrix of the trained classifier on the original object size over the FASHION dataset (numbers are in %).
1450
100
FASHION
71.2
69.7
AP
CenterNet
FCOS
MASK RCNN
Upper Bound
henleys
cardigans
messengerbage
berets
vests
scarves wraps
lunics
slippers
leggings
overalls
longsleeveshirss
hoodies
dutches
baseballcaps
watches
blouses
sweaters
handbags
sandals
rompers
tanks camis
shorts
buttondowns
pants
lees
beanieknitsaps
bools
coats jackets
jumpsuits
glasses
jeans
shoes
backpacks
polos
blazers
cowboyhals
skirts
dresses
suifcoats
hats,
Figure 22: Upper bound and model APs over the FASHION dataset.
34

---

Under review as a conference paper at ICLR 2023
J.2 DOTA DATASET
We are using the v1.0 set of this dataset (tttps://captain htt.tahhbbiiiuuuutt.....b.b..... index. html) which includes the following 15 categories: plane, ship, storage tank, baseball diamond, tennis court, basketball court, ground track field, harbor, bridge, large vehicle, small vehicle, helicopter, roundabout, soccer ball field and swimming pool. This dataset contains 2,806 images and 188,282 instances. The proportions of the training set, validation set, and testing set in DOTA-v1. 1.0 are 1/2, 1/6, and 1/3, respectively. We are computing the UAP over the validation set this dataset.
We are computing the UAP for the horizontal bounding box detection task on this dataset. Results are shown in Fig. 23 The trained classifier mentioned in the main text) scores about 89% on average over the categories. The UAP is about 88% which is higher than the AP of the best model on this dataset (about 83%). Please see https /tt:://capaaaaaaa whu, github. i Btt///OOA///euuuue html
mean: 0,9
classification accuracy
04
0.2
Plane
BD
Bridge
LV
Ship
SBF
RA
Harbor
HC
categories
AP, mean: 0.83
UAP, mean: 0.88
AP
0.4
Plane
BD
Bridge
GTF
SV
LV
Ship
TC
BC
ST
SBF
RA
Harbor
SP
HC
categories
Figure Upper bound model APs over DOTA dataset.
35

---

Under review as a conference paper at ICLR 2023
K COMPUTE REQUIREMENTS
All experiments were performed on machine with 4 NVIDIA V100 GPUs and 16GB RAM.
L DATASET LICENSES
Below we list each dataset's license, as provided either in the paper proposing the dataset or on the dataset website. For datasets where unable to find license, list "No License."
1. MSCOCO: Creative Commons Attribution 4.0 License, httppss//cocoodaastttt.tt org #termsofuse
2. PASCAL vOC: No license specified
3. OpenImages: CC BY 4.0 license, the images are listed as having a CC BY 2.0 license
html
4. ObjectNet:
Creative Commons Attribution 4.0 with only two additional clauses: a) ObjectNet may never be used to tune the parameters of any model, and b) Any individual images from ObjectNet may only be posted the web including their pixel red border. html
See http::://toecttteeeeee
36


Reviews:
[{"review_id": "za48i2ScHqK", "reviewer": "Reviewer_a5Z9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary:\n\nThis paper studies the limitations of the COCO benchmark, raises the questions about the future directions for improving object detection systems. Particularly, this paper studies the approximated empirical upper bound performance of average precision ()UAP), and conclude that there is still a big gap between it and the best model's AP. Furthermore, this paper also introduce new validation sets to study out-of-distribution generalization performance of current object detectors. Studies were conducted to analyze the bottleneck of existing object detectors and characterize the type of errors on the new validation sets. \n", "summary_of_the_review": "As pointed out in the weakness, I think this paper is a good attempt to revisit the topic of analyzing object detectors in the modern context. However, there are many issues in the execution and writing that made me to have difficulty in understanding the true insights behind. I thus could not support this paper for acceptance in its current presentation. ", "strength_and_weaknesses": "Strength:\n\n+ Some of the studies are providing interesting aspects in understanding object detectors. The way of finding UAP is quite solid and experiments are comprehensively conducted on multiple datasets. \n\n+ The authors have also spent effort in collecting new validation datasets to study the out of domain generalization performance of object detectors. \n\nWeakness:\n\n- The writing and presentation of this paper needs improvement. After reading the introduction, I could barely see any intuitive statement that briefly describes the results from the studies. Instead, it is full of statements that claims the results are important and interesting but without convincing me why and how. Meanwhile, I found it very hard to read Figures as they are placed in position that is far from the text. For instance, the actual text that explains Figure 1 is at Section 3.4 (page 5) but Figure 1 is at page 1. \n\n- While UAP is claimed to study the best model's upper bound performance (assuming near-perfect localization), utilizing an outdated classifiers such as ResNet152 does not seem ideal. Wouldn't it be more interesting to fine-tune a pre-trained classifiers such as CLIP model make UAP more accurately approximating the upper bound?\n\n- What other insights did UAP tell other than that there is a gap between the classifiers for detectors and object recognition models? I did not learn anything that is very unexpected from those studies. In fact, can such trained \"upper-bound\" classifiers be plugged into existing detectors to significantly improve their performance? If the answer is no, does this mean that the key challenge is to improve the localization performance? \n\n- What is the connection between the proposed new datasets and the UAP studies in section 3? In general I see a gap between section 3 and section 4, why do we need those two complementary datasets? \n\n- What is the role of these new validation sets comparing to the LVIS validation sets? In principle you can also use the LVIS dataset to study the out-of-domain generalization? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oNY_cpXvOOb", "reviewer": "Reviewer_4FQY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper evaluated the imperical upper bound of average precision of the COCO object detection dataset which is significantly higher than the current state of the art methods.  Two new datasets are introduced as complementary of COCO dataset. Source of error analysis showed that have state-of-the-art detectors have different behaviours on the proposed dataset. ", "summary_of_the_review": "The paper first attempts to analyze the empirical challenge in the current detection dataset, then propose two new datasets and point out the limitation of COCO. Despite the richness of the analysis, the paper is disconnected in overall logic and is not very clear in stating motivation. Overall the contribution of this paper is not very clear to me and I think a major revision/explanation is needed to justify the contribution.", "strength_and_weaknesses": "**Strength:** the paper provides a comprehensive analysis of the experiments\n\n**Weakness:**\n* The paper is not well organized, in a way that the motivation of the proposed method & experiments is not coherent throughout the paper. (e.g. what's the connection of upper bound AP to the new proposed dataset? what's the relationship between sections 4.1 4.2 and 4.3, is the limitation of the COCO dataset (4.3) inspire the composition of the new datasets (4.1)?) The paper is disconnected in logic.\n* The definition of the empirical upper bound is conditioned on knowing the exact location of the bounding box. So the gap between UAP and state-of-the-art AP largely indicates the difficulty in localization. But knowing this gap does not give us a hint about how to tackle the challenge that the current detector is facing, which makes this less exciting.  And it's debatable whether localization and recognition can be separated, this assumption needs more discussion in the paper\n* What's the motivation for the new proposed data? Simply saying that the OpenImages dataset is not largely adopted is not a very convincing reason for proposing an extension for the COCO dataset. how is it solving the limitation of the COCO dataset(4.3)? It's not clear to me since there is only limited analysis on the new dataset, and there is no claim of benefit besides its OOD nature.\n* The paper uses Efficientdet and DetectoRS to evaluate the difference in dataset, it would be greater to incorporate more detectors (e.g. deformable detr[1]) since different methods can have different properties when analyzing errors.\n\n*[1] Zhu, Xizhou, et al. \"Deformable DETR: Deformable Transformers for End-to-End Object Detection.\" International Conference on Learning Representations. 2020.*", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n-gUdtgghQS", "reviewer": "Reviewer_A2JS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a method to empirically estimate the upper-bound AP (UAP) for object detection performance using a robust classifier to classify cropped bounding boxes. Specifically, the technique disentangled the detection problem into localization and classification. Assuming the object localization is correct and classification is the bottleneck, they treated the resulting AP by a trained robust classification model (ResNet152) as the empirical upper bound on detector performance. Furthermore, the paper also analyzes the few variations of their settings to check the UAP change, for instance, by adding context (enlarging/shrinking the boxes) or by adding boxes to the evaluation set that don't perfectly overlap with the ground truth. The authors conclude that the deficit in AP lies in the classification error and want to provide these insights to the community for refocusing object detection research direction in the future.\n\n\n", "summary_of_the_review": "The topic of knowing the detector's gap to the upper bound performance is of interest to a wide scope of readers, however, the paper is at times hard to follow, and I do not see a concrete solution proposed based on the derived findings. Therefore, it is too soon to tell if the claimed findings are helpful to fellow researchers.", "strength_and_weaknesses": "Strengths\n1. The addressing topic is of interest to a wide-scope audience.\n2. The additional studies on the influence of context on detections and the experiments are interesting and comprehensive.\n\nWeaknesses\n1. The paper is hard to follow at times since all the text information and figures are not well organized and structured, and there are 35+ pages from the main paper plus the appendix, so readers tend to miss the emphasized points.\n2. The paper points out that a better classification area is what researchers should focus on, but it seems to be a bit handwaving for me. It would be great if the authors could prove the conclusion, presenting us with a better object detection solution based on the findings.\n3. The gap between the current detector's performance to the empirical upper bound might not hold true since the paper lacks a discussion on the most recent transformer-based solutions on the shelf now.\n4. Not sure if the proposed UAP is too loose since we over-simplify the problem by decoupling the detection into localization and classification and assuming the error types from the localization are fixed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fdjoll1OHd0", "reviewer": "Reviewer_GAsE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper compreehensively analyzed the upper bound performance of object detectors by partially uncovering the ground-truth labels to the models. The experiments demonstrate that the performance of current object detectors are far from perfect.\n\nThe paper also introduced a new dataset based on existing datasets (OpenImages, DOTA). The authors annotated the images from these existed datasets to create a new dataset for object detection.", "summary_of_the_review": "Given the limited technical novelty but a new dataset, I hold a neutral attitude toward this paper.", "strength_and_weaknesses": "+ paper is well written and easy to follow\n+ the new dataset is a good addon to limited existing detection datasets.\n\n- I think the technical contribution and novelty of this paper is limited. There are couple of papers analyzing the upper bound performance of object detectors using similar techniques.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MdqkPRwyIh", "reviewer": "Reviewer_5rhm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper attempts to present an empirical upper bound for COCO and show that the existing approaches to object detection are far from this upper bound. The authors further propose two datasets complementary to COCO. The authors show that object detection models behave differently on the proposed datasets compared to COCO.", "summary_of_the_review": "The idea of analyzing the performance of object detection models using an empirical upper bound is novel and interesting. However, the paper needs to be written better - in particular, making it self-contained by selecting the most important points, and clarifying some of the details of training and using the classifier. Therefore, I am recommending rejecting the paper. I would recommend that the paper include the most important details and experiments from the appendix into the main paper and submit to another venue - maybe a journal.", "strength_and_weaknesses": "The biggest strengths of the paper are the following:\n\n  1. The idea of coming up with an empirical upper bound for object detection on COCO. This provides a sort of target for object detectors and also shows the limitations of existing classifiers.\n\n  2. The analysis of the proposed UAP along with different ways of obtaining a UAP.\n\n  3. The motivation presented in the first paragraph of section 3 is very good and the explanation is commendable. \n\n\nHowever, the paper suffers from significant weaknesses which need to be addressed in a re-submission. In particular, I would mention the following:\n\n  1. The paper writing needs to improve. Currently, most of the paper is in the appendices. The authors need to select the most important things and bring them in the main paper. Appendices should be only for additional information.  The paper should be self-contained and there should not be so many things in the appendices. This is unfair to other papers which had to include everything within the paper. I would recommend that the authors write the paper properly and maybe submit it as a journal paper instead of a conference with page limits.\n\n  2. The authors use an object recognition model to classify objects. It's not clear how is this selected. The authors should provide motivation for why did they select one classifier over another. How do you determine which object detector's classification head works best? Doesn't the UAP depend on the selected classifier? And if yes, how much?\n\n  3. The proposed UAP assumes that the ground-truth bounding boxes are given and a classifier needs to be trained. Does this assume that the bigger problem is localization? Can there be another empirical UAT where the classes can be taken from GT but the model needs to generate boxes. \n\n  4. Though the authors mention that they include the most recent models in their analysis, I would encourage them to include transformer-based models like DETR/MDETR as well.\n\n  5. Section 3.2 needs more detail. It's not clear to me how exactly is the classifier trained and how the GT boxes are used.\n\n  6. In Table 1, please mention what is \"Acc.\" in the caption.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject