BBRefinement: an universal scheme to improve precision of box object detectors
Authors: Petr Hurtik
Marek Vajgl
Venue:  (2021)
Keywords: object detection
deep neural networks
refinement

Abstract: We present a conceptually simple yet powerful and flexible scheme for refining predictions of bounding boxes. Our approach is trained standalone on GT boxes and can then be combined with an object detector to improve its predictions. The method, called BBRefinement, uses mixture data of image information and the object's class and center. Due to the transformation of the problem into a domain where BBRefinement does not care about multiscale detection, recognition of the object's class, computing confidence, or multiple detections, the training is much more effective. It results in the ability to refine even COCO's ground truth labels into a more precise form. BBRefinement improves the performance of SOTA architectures up to 2mAP points on the COCO dataset in the benchmark. The refinement process is fast; it adds 50-80ms overhead to a standard detector using RTX2080, so it can run in real-time on standard hardware. The code is available at https://gitlab.com/irafm-ai/bb-refinement.

Under review as a conference paper at ICLR 2021
BBREFINEMENT: AN UNIVERSAL SCHEME TO IMPROVE PRECISION OF BOX OBJECT DETECTORS
Anonymous authors Paper under double-bling review
ABSTRACT
We present a conceptually simple yet powerful and flexible scheme for refining predictions of bounding boxes. Our approach is trained standalone GT boxes and can then be combined with an object detector to improve its predictions. The method, called BBRefinement, uses mixture data of image information and the object's 's class and center. Due to the transformation of the problem into domain where BBRefinement does not care about multiscale detection, recognition of the object's class, computing confidence, more effective. results in the ability to refine even COCO's ground truth labels into a more precise form. BBRefinement improves the performance of SOTA architectures up to 2mAP points on COCO dataset in the benchmark. The refinement process is fast; it adds 50-80ms overhead to standard detector using RTX2080, so it can run in real-time on standard hardware. The code is available at http::/:tabb..oommmaaaaaaabbbtte
multiple detections, the training is much
Input: an image
BEReftnemet
Generic oblect detector NHi
Whocle image
Rescaled
Lozstion of boves
crops
Centers
Predictad boxes
Retind boxes
Predicted
scorts
Prodicted classes
Prodicied clastes
The figure illustrates the proposed pipeline of prediction. A generic object detector pro- cesses an image, and then the detected boxes are taken from the original image, updated by BBRe- finement, and taken as the output predictions.
Figure 1:
PROBLEM STATEMENT
Object detection plays an essential role in computer vision, which attracts a strong emphasis on this field among the researchers. That leads situation when new, more accurate, faster object detectors replace the older ones with high frequency. A typical object detector takes an image and produces set of rectangles, so-called bounding boxes, which define borders of objects in the overlap between the detected box and ground truth
image. The detection quality is measured (GT), and it is essential for two reasons. Firstly, the criterion used in benchmarks mean Average Precision (mAP) is based particular thresholds for various values of Intersect over Union (IoU) between the prediction and the GT. Such thresholds are typically applied to distinguish between accepted and rejected boxes in detection. Therefore, precision here is crucial to filter valid boxes

---

Under review as a conference paper at ICLR 2021
from discarded. Secondly, the more precise the detected box is, the more accurate the classification should be. Although NN-based classifiers can deal with some tolerance in shifted or cropped data, the higher accuracy in the object detection may lead to the increased accuracy in the classification process.
Existing solutions for object detection yield accuracy around 0.3 0.3-O.5mAP the COCO dataset (Lin et al. 2014). Such a score allows the usage in many real applications. On the other hand, there is space for improvement. A combination of the following may reach such growth: more precisely distinguish between classes; increase the rate of true-positive detections; decrease false- positive detections; or increase the IoU of the detections. There are four points on why object detection may be difficult in general, which blocks further mAP growth. I) A neural network has to find all objects in an image. The number may vary from zero to hundreds of objects. 2) A neural network has to be sensitive to all possible sizes of an object. The same object class may be tiny or occupy the whole image. 3) network usually has no priori information, which should make the detection easier, like the context of the scene the number of objects. 4) There is lack of satisfactory big datasets. Therefore, the distribution of data is sampled roughly only. In this paper, we propose BBRefinement, which can suppress the effect of all the four mentioned difficulties. The proposed inference scheme 'Detection + Refinement' is achieved by a combination during prediction phase with a generic detector, and it increases the IoU of the detected boxes with its ground truth labels, resulting in higher mAP.
Related work. The problem of refinement can be tracked to the origin of two-stage detectors, where R-CNN (Girshick et al. 2014)) uses a region proposal algorithm that is used to generate a fixed num- ber of regions. The regions are classified and by bounding box regressor refined. Faster R-CNN (Ren) et al. 2013) replaces the region proposal generation algorithm with region proposal network. The same bounding box regressor can be used iteratively to obtain more precise detections (Gidaris & Komodakis 2015 Li et al. 2017). The effect of iterative refinement may be increased by involving
LSTM module (Gong et al. 2019). The aim of refinement can also be anchors; RefineDet (Zhang et al. 2018 2020) refines them to obtain customized anchors for each cell. Cascade R-CNN (Cai & Vasconceios 2018) uses a sequence of bounding box regressors to create n--saged object detector. In Cascade R- R-CNN, network head he takes proposals from the region proposal network and feeds the regressed bounding boxes to network head h1 and so on. All the heads work over the same features extracted from backbone network. The cascade scheme shows that h is dependant on the quality of ho head. If ho includes some bias, hy balances it. Therefore, all the heads have to be trained together (part by part), and if ho is retrained, hi should be retrained as well. In contrast, BBRefinement is a trained standalone, and it is not dependent on the quality of the object detec- tor with whom coupled during inference. That makes BBRefinement universal and able to be applied on various image detectors without retraining a detector or BBRefinement.
EXPLAINING BBREFINEMENT
The main feature of BBRefinement is a transformation of the problem into simpler scheme, where an NN can be trained easily. Compared with a standard object detector, BBRefinement is a spe- cialized, one-purpose neural network working a single object detector. It does not search for zero-to-hundreds objects, but it always detects only a single object and does not produce its confi- dence. It is also missing the part responsible for classification, it does not assess the object's s class. The only purpose is to take an image with a single object within a normalized scale and generate super-preeise bounding box. The training is realized on boxes extracted from dataset according to ground truth labels. When BBRefinement is trained, the fixed model can be coupled with an ar- bitrary detector to realize the inference. Here, the feed for BBRefinement is bounding boxes in the form of image data produced by the detector.
2.1 PROBLEM WITH A NAIVE SINGLE OBJECT DETECTOR
Let bounding box be given by its top-left and bottom-right coordinates b = (x1.3/- 22, V2). Further, let us suppose color image f D c N² + L R3. Then neural network detecting single object is generally noted as g : + b. To train such a network, we generally minimize term bb s(S) or its alternatives.
2

---

input vector in the case of Under review as a conference paper at ICLR 2021
The issue comes when includes two objects at once, and the network is extended to produce two bounding boxes. The network should return b for the first object and b2 for the second one. How- ever, generic solution will predict a box 0.5(b) b2) for the both cases, generally 1/n Σi=1 b, for n. if we consider that the boxes have the same frequency occurrence. modify the network to detect sparse of objects assumes boxes in a fixed order. Such detection scheme is not possible in general without deeper modification of an architecture leading the presence of grid, etc. With no guarantee of sin- gle object's presence only, the naive solution cannot be used. This problem was solved later in chronological order by a sliding deformable models/ / window technique (Felzenszwalb al. 2009). two-stage techniques such (Fast/Faster) R-CNN (Ren et al.] (015)), single-stage techniques as SSD (Liu et al., 2016) or YOLO (Redmon & Farhadi 2017) and finally by anchors-free techniques that are mainly keypoint-based (Law & Deng (018)). Every such approach affects the architecture
naive solution is to B. where B (b); b2.-... b,.
of the neural network and related to a specific model.
BBRefinement, as single object detector, would suffer the shortcomings mentioned above. The reason is that even we have an extracted object defined by its bounding box, such bounding box for non-rectangular object will also involve some background, which may contain other objects. The presence of other objects leads to the problem described in the previous paragraph, and in conclusion, it may confuse the neural network, thus cause improper refinement. The examples from the COCO dataset are shown in Figure - for the easy case and in Figure 3 for the problematic case. For illustration, the COCO dataset includes 1.7M boxes from which 47% of all boxes have an intersection with box with the same class, and 84% of boxes have an intersection with an arbitrary- class box. To solve the problem, BBRefinement uses information about the object's class and center as well.
STOP
Figure 2: The figure shows crops which can be refined even with the naive way because crop includes only one, nice visible object
The figure shows crops that cannot be refined by the naive way because a crop includes multiple objects, usually of the same class. Note, precise labeling of such images is hard task even for humans.
Figure 3:
2.2 THE PRINCIPLE OF BBREFINEMENT
Firstly, need to define neural network in detail, we suppose convolutional neural network F to be set of & layers, F = f2.. fx). Here, all the layers are meant as fs D c R. L C In: which are for the sake simplicity defined as convolutions layers without poolings/residual connections/batch norms etc, i.c.. fM, = a(W, & M,). where a is activation function, W, weights and M, output of the previous layer, or, an input image in the case of i = 1. Such neural network is generally called as backbone, with the aim to map an input image iteratively into feature space. Here, we suppose ex : C R - L C R to be an embedding of the k-th layer created as ex(,x (M.)) = (M.)). operation.
(M,.)
where is global average pooling or flattening
Furthermore, suppose a fully connected network G to be set of layers, G {01, 92...................... }. All layers are meant as 9i : D C g + L C IR as g, (M.) = (WW,)), where a is activation function, W, weights and M, is an output of the previous layer, i ==1.
mixture data input to the suggested scheme of refinement. Convolution neural network processes the input image (crop with a fixed resolu- tion) containing object, fully connected neural network G processes fixed-size vector that
According the motivation, we propose
3

---

Under review as a conference paper at ICLR 2021
holds information about class and an expected center of the object. The both networks are de- signed in order to lex (fa (Ma)) = [g, (M,) be valid. Then, both information is mixed together as ex (fff((Me)) 93 (M3), where is dot product. Finally, we connect æ with the output layer o consisting of four neurons and utilizing the sigmoid activation function. Such a neural network is trained in a full end-to-end supervised scheme. From practical point of view,
can use an
arbitrary SOTA backbone such as ResNeSt, ResNeXt, or EfficientNet. to mention few. For BBRe- finement, we use EfficientNet (Tan & Le 2019) due its easy scalability. In the benchmark section, we are presenting results for versions BO B4. According to the version, an input image's 's resolution 224², 240 260 300 and 380 The version affects |es (f& (M:)] as well, is 1280 (BO and B1), 1408, 1536, and 1792.
The pipeline for the prediction with BBRefinement is illustrated in Figure T and is as follows. The boxes detected by generic object detector are taken (with small padding) from the input image and rescaled into the BBRefinement input resolution. That has several beneficial consequences. Firstly, bigger objects are downscaled, and smaller are upscaled to fit the resolution, so all the objects have the same scale, which is much more effective than train a network for multiscale detection. Second, one image from the dataset yields multiple boxes. In the case of COCO, a standard detector uses 0.2M images (one image as an input), while BBRefinement uses 1.7M images (one box as an input). Also, a standard detector rescales the input images into specific resolution to fit GPU memory, so many pixels are thrown out. BBBrfinement does not use non-object parts from the image, but it allows us to use more pixels from the object-are due weaker downsample. Third, thanks to mixture data usage, BBRefinement obtains information about the object's ss detected class and center. Although such data may be imprecise, it is piece of priory information, making the task more accessible. Finally, there is a guarantee that each crop includes just one main object. That explains why BBRefinement can produce more precise coordinates of bounding boxes than a general object detector. Note, it is necessary to take into account BBRefinement is placed top of an object detector, which may be imprecise. As a result, the data fed into BBRefinement may be ambiguous. Therefore, the crops should not be extracted precisely but surrounded by padding; during the training, the padding is random. For the same reason, it is beneficial to distort the center by random shifts for training. Such augmentation is visualized in Figure 4
Figure 4: Augmentations of a box. The green box represents the GT label. The original crop is randomly padded. The center's position is slightly distorted (and visualized as a green dot) as suppose BBRefinement will be applied to a generic detector's predictions, which can produce such distortion.
To realize the refinement, there are several options on how to define loss function & used for training BBRetinement. The first option is to compare each coordinate of the box with the GT label using, e.g., binary cross-entropy (BCE). The second option is to use BCE for comparing top-left points and then euclidean distance for evaluating the width and height of box. Such an approach is used, e.g., in YOLO. The third way is and compute IOU. Let us imagine a situation where a box is shifted by a pixel according to its label in vertical and horizontal direction. If we shift the box in the directions separately, we will observe that the sum of such partial losses will be equal to the loss produced by shifting in both directions at once. That is not valid behavior; the second loss should be bigger. On the other hand, the triangle inequality is fulfilled when IoU is used. The next problematic situation is when the euclidean distance is used: bigger boxes tend to produce bigger differences than small boxes. This means they produce bigger losses, and neural network tends to focus on them more than on the small boxes. IoU is computed as a relative value; therefore, the same difference in width or height creates bigger loss for small boxes, which is the desired behavior. Based on these reasons, we use loU-based loss for training BBRefinement. We have two available options on how to define the IoU loss function. Namely (b. b') -log(i(b, b') (/)b/uubb b')) for the logarithmized form and (b,,, b') = 1.0 - i(b, b')/u(b. b') for the linear form, where i represents intersection two boxes, their union, b is GT box, and b' predicted box.
coordinates of all points to determine the boxes' areas
4

---

Under review as a conference paper at ICLR 2021
Logarithmized form. Let consider task where the evaluation criterion involves IoU with some threshold, such as 0.5, which can be found in many real competition websites such as Kaggle or Signate. Here, it is much more to satisfy the threshold rather than to reach the best possible IoU. Here, the goal of the BBRefinement is to take unprecise boxes (with loU<0..5 and push them over the threshold. The logarithm-shape of IoU loss generates the biggest loss for unprecise boxes, while the loss is vanishing for high-precision boxes, similar to Focal loss (Lin et al. 2017). That is also beneficial if we take into account that no dataset perfect, and labels created by humans are not accurate. Here, the property of smaller loss for near-perfect detections would be beneficial.
Linear form. The COCO dataset's 's official benchmark computes mAP for several IoU thresholds, such as 0.5, ..... 0.95. Here, the situation is the opposite: by refinement, pushing box from, e.g., 0.4 into 0.95, is more beneficial than refine three boxes from 0.4 to 0.54. The reason that a box with IoU 00.95 will be taken into account in all the IoU thresholds, while box with IoU 0.54 will be taken into account only for the threshold of 0.5; other thresholds will count it as a false positive. Therefore, we use the linear form in the following benchmarks.
Note, both forms of the loss function can be based on a more efficient version of IoU. The other choices may be Generalized IoU loss (Rezatofighi et al. 2019). Complete IoU, or Distance IoU (Zheng et al., 2020). Generally, these loUs converge faster and are able to compute loss effec- tively, even for boxes without overlap.
BENCHMARK
The setting of the training: BBRefinement was trained using two different computers with cards RTX 2060 or 2080. The resolution of models corresponds to the default setting of EfficientNet (Tan) & Le 2019) version, namely side size of 224, 240, 260, 300, and 380px for version BO-B4 with the batch size of 7-40 according to the version and memory of the used graphic card. For training, we used COCO dataset (Lin et al., 2014) as follows. We merged train 2014, train 2017, and a part of valid 2014. The unused part (5000 images) of valid 2014 has been used as the valid set. The testing set represented by valid 2017. The loss function linear IoU described above, optimizer AdaDelta [Ziler] 2012) with default learning rate, i.e., a = 1.0, and functionality of decrease learn- ing rate by factor 0.5 with patience equal to two. We also experimented with cyclic LR (Smith) 2017), which converged faster but generally produced significantly worse the best loss than the used scenario. During one epoch, all training images were processed, and a single random box has been taken from each one of them. Each such box was augmented by random padding (each side sep- arately), by linear/non- linear HSV distortion, CLAHE, and by flipping. The information about the box center has been augmented by distorting the coordinates. The illustration of the augmented box is shown in Figure 4 Models were trained until loss did not stop decrease, which took approx 70-90 epochs. For illustration, the heaviest used backbone, EfficientNet B4, was trained for nine days on a computer with an RTX2080Ti. For the comparison, we selected SOTA networks, namely Faster R-CNN (Ren et al. 2015). RetinaNet (Lin et al., 2017) (both for two various backbones), and Cas- cade R-CNN Cai & Vasconcelos (2018). All of them are available through Detectron2 framework Next, we used DETR (Carion et al. 2020), which is available through official implementation de- rived from MMDetection framework. We used the reference models trained on the COCO dataset and realized the inference only.
The detailed results are presented in T We want to emphasize that BBRefinement improves the mAP of all but Cascade R-CNN models, considering the standard [IoU=0.50:0.0. 95] setting, while holds that the heavier backbone of BBRefinement usually brings stronger boost. Also, it holds that the worse the baseline model, the bigger increase of mAP. Considering the objects' size according the situation is not so straightforward. In the case of small
to the COCO tools (small-medium-biin objects, EfficientNet-Bi I can be marked as the best backbone with the claim that it may be beneficial to refine only the less precise models; otherwise, BBRefinement may even decrease the performance. For medium objects, BBRefinement EfficientNet B2 is the best one, and the usage of refinement leads to an increase of accuracy in all but Cascade R-CNN models. A similar situation is for the large objects where BBRefinement EfficientNet-BB -B4 is the best one in all the cases. There is a hypothesis that strong upscaling of small objects leads to distortion and, therefore, to decreased performance.
wwtphhaaaaacceeeeeeeeeeceeeeeeen dempo ipmo.
5

---

Under review as a conference paper at ICLR 2021
Table 1: mAP [loU=0.50:0.95] performance of original and refined predictions the COCO dataset. The table shows accuracy in the form of IOU of a generic detector when its official, pre- trained model used marked as a baseline. The right part shows IOU accuracy when the same pre-trained model is coupled with BBRefinement All BBRefinement versions are trained only once, and the same trained version is used for all multiple detectors. For the training of BBRefinement, use the same split it is common and has also been used by the authors of the generic detectors.
BeRefinement, EfficientNet
Model
Baseline
B0
B1
B2
B3
B4
Boost
All objects
Faster R-CNN, ResNet-50 C4 1x
35.7
37.4
37.7
37.9
38.0
37.9
+2.3
Faster R-CNN, ResNeXt-101 FPN 3x
43.0
43.1
43.3
43.6
43.6
43.6
+0.6
RetinaNet, ResNet-50 FPN Ix
37.4
38.3
38.6
38.8
38.8
38.8
+1.4
RetinaNet, ResNet-101 FPN 3x
40.4
40.6
40.8
41.1
41.1
41.1
+0.7
DETR, ResNet-50 Cascade R-CNN, ResNet-50 FPN 3x
34.3
35.6
35.8
36.0
36.0
36.0
+1.7
44.3
43.0
43.2
43.4
43.4
43.5
-0.8
Small objects
Faster R-CNN, ResNet-50 C4 1x Faster R-CNN, ResNeXt-101 FPN 3x
19.2
19.3
19.5
19.1
19.1
19.2
+0.3
27.2
25.8
25.9
25.9
25.7
25.9
-1.3
RetinaNet, ResNet-50 FPN Ix
23.1
22.1
22.1
22.1
22.0
22.0
11.
RetinaNet, ResNet-101 FPN 3x
24.0
23.5
23.4
23.4
23.4
23.4
005
DETR, ResNet-50 Cascade R-CNN, ResNet-50 FPN 3x
14.3
15.9
16.0
15.9
15.9
15.7
+1.7
26.6
24.6
24.5
24.3
24.4
24.4
2.0
Medium objects
Faster R-CNN, ResNet-50 C4 1x Faster R-CNN, ResNeXt-101 FPN 3x
40.9
42.4
42.6
43.0
43.0
43.1
+2.2
46.1
46.6
46.8
47.2
47.0
47.2
+1.1
RetinaNet, ResNet-50 FPN Ix
41.6
42.9
43.1
43.6
43.4
43.4
+2.0
RetinaNet, ResNet-101 FPN 3x
44.3
44.8
45.1
45.5
45.4
45.4
+1,2
DETR, ResNet-50 Cascade R-CNN, ResNet-50 FPN 3x
36.6
38.4
38.7
38.9
38.8
38.9
+2.3
47.7
46.7
47.0
47.3
47.2
47.4
-.3
Large objects
Faster R-CNN, ResNet-50 C4 1x Faster R-CNN, ResNeXt-101 FPN 3x
48.7
52.5
53.2
53.5
53.5
53.6
+4.9
54.9
56.6
57.1
57.3
57.5
57.7
+2.8
RetinaNet, ResNet-50 FPN Ix
48.3
50.7
51.0
51.3
51.3
51.4
+3.0
RetinaNet, ResNet-101 FPN 3x
52.2
53.8
54.2
54.4
54.4
54.5
+2.3
DETR, ResNet-50 Cascade R-CNN, ResNet-50 FPN 3x
51.5
52.0
52.3
52.7
52.8
52.8
+1.3
57.7
57.3
57.8
58.0
58.1
58.5
+0.8
DISCUSSION
Bugs in dataset: Deep learning, data-driven approach, is directly dependent the quality of data. On the other hand, is impossible to create a flawless dataset. The object detection task's general issues are incorrect classes, imprecise box boundaries, and missing boxes. BBRefinement is (as standard object detectors) vulnerable to the first two issues, but (opposite to standard object detectors) resistant the third issue. we consider missing labels illustrated in Figure we will penalize a detector during training if the detector will produce predictions for such missing labels. That will lead decreased performance. In the case of BBRefinement, the training data are created from the labels. If some label is missing, cropped image will not be produced. So, the missing labels only decrease the training set's size but not affect BBRefinement's performance.
Refinement of dataset: We also tested the most precise object detector, DetectoRS (Qiao et al. 2020), which can reach the mAP above 0.5. In that case, we observed a decrease of mAP by 1.3 after the refinement. We analyzed visual outputs and recognized interesting behavior: DetectoRS's predictions are closer to GT, but the refined predictions look visually better, even better than the GT. Therefore, realized second experiment. We took GT test labels, refined them, and visualized both of them into an image. Surprisingly, we can claim that BBRefinement can produce more precise labels than COCO dataset. On the other hand, because the boxes' positions not identical, refined

---

Under review as a conference paper at ICLR 2021
GOODNAYES BOYS' SCHOOL. APRIL, 1969.
Figure 5: The figure illustrates two images taken from the COCO dataset, where the boxes are inpainted ground truth labels. It is evident that some labels are imprecise, and a lot of labels are missing. Such behavior can be seen mainly in images that include groups, and it is known issue of the COCO dataset.
ROOK
Figure 6: The image illustrated crops with green ground truth and yellow refined inpainted labels. Here, BBRefinement creates labels with significantly higher precision than is the ground truth. Best see zoomed-in.
boxes do not yield to IoU of 1.0, and therefore, the mAP can be decreased when BBRefinement is applicd on a high-precise object detector. The reason why an object detector can produce predictions on test set closer to GT than the refined version is unelear for Figure shows crops from the test set with inpainted boxes: green color marks GT boxes given by the COCO dataset, and yellow the labels produced by BBRefinement. We selected the images in Figure such cases, where it is obvious that BBRefinement yields more precise boxes. Note, IoU between predictions and GT varies here around 0.8. Because the dataset is big, and eight selected crops were chosen selectively, also selected eight additional crops as follows. The first one has index 100 in the ordered list of images, the second one 200, the third one 300, etc., the selection is not affected by our preference. They are illustrated in Figure 7 We can proudly claim that BBRefinement, although not so significant as for the previous cases, still produces more precise boxes than GT (see best zoomed-in), Also, we applied BBRefinement trained on COCO to the Cityscapes dataset. Again, BBRefinement makes visually more precise labels than the Cityscapes ground truth is. Such a finding leads us to three conclusions. First, it is ambiguous to compare high-mAP object detectors because the high mAp does not necessarily mark a better detector the meaning of real-world truth as the labels are affected by human subjection and error. Next, thanks to the high number of boxes, BBRefinement can be trained in such a generalized manner that the labeling error can vanish, so it can used for re- re-labeging dataset. Finally, there is hypothesis that IoU between BBRefinement trained

---

Under review as a conference paper at ICLR 2021
The image illustrated crops with green ground truth and yellow refined inpainted labels. The crops were selected uniformly according to their index to show general cases. Here, BBRefine- ment proposes slightly higher precision than the ground truth has, Best see zoomed-in.
Figure 7:
specific dataset, and its GT labels be used to express the quality of labels. The verification of this hypothesis a theme for future work.
5 ABLATION STUDY
Comparing with naive refinement without mixture data We have trained BBRefinement with EfficientNetBI with the same setting as is used in Section 3 but without mixture data. It means that only visual information represented by image crops is available during training and inference. During the inference, we coupled it with 'Faster R-CNN, ResNet-50 C4 1x'. It achieved mAP performance on all/smal//medium//ig areas of 34.0/15. 1tt.1//..9// 9944.5. That better than the baseline performance, 33. 1/15.0038.0/46. but worse than the full couple with the mixture data, which yields 34.9/15.5/39.5550. This finding confirms the meaningfulness of the proposed scheme.
Influence of the accuracy of center and class: The performance of BBRefinement is affected by the accu- racy of the used object detector. Therefore, we realized an experiment where GT data were distorted, fed into
10U bxtwenn seliaal and GT
Distorted position of center by of widlh/eight
% of dutnect objects have distorted class
0.9
BBRefinement, and IOU between the refined and GT was measured. In the ideal case, IOU would be 1.0.
0.8
As show in Figure R distorted the position of the center and the correct class separately. The distor- tion for center e realized as (cz de. cy + dy), where de, dy ~ U(-d, d) and by d we mean the maxi- mum distortion. For the class distortion, of correct classes with random incorrect classes. The figure shows that BBRefinement is robust on incorrect class, but it is sensitive to center position distortion. Such strong robustness in the dis- tortion of class may, on the other hand, mean that information about the class is not important for BBRefinement and therefore, BBRefinement can trained even without
0.7
10
15
Distorted [X]
Figure 8: Influence of distortions
replace n%
Speed of the inference: In some applications, the ability to run in real-time may have the same importance as high precision. Therefore, BBRefinement should not increase the inference time significantly. Via selection of BBRefinement's 's backbone, the tradeoff between speed and precision can be controlled. The BBRefinement' 's inference time is consisting of two parts, the preparation of the crops and the inference itself. While the first part depends the CPU, the second part relies on GPÜ power only. Speaking in numbers, we measured the time of BBRefinement with backbone EffnetB1 and BffnetB3. For both cases, the non-optimized preparation of crops CPU costs 32ms
8

---

Under review as a conference paper at ICLR 2021
per image, where the image can include multiple crops. The time GPU is 23ms for B1 and 44ms for B3. We could predict all crops from a single image in one batch, which helped keep the time
small. The time means that BBRefinement runs 18FPS for the B1 backbone and 13FPS for the
B3 one. The speed can be further increased by parallelizing crops' preparation and optimizing the model's speed by automatic tools.
Influence of crops size We have selected BBRefinement with EfficientNeiB2 backbone and trained it for various crops size to reveal the impact. To converge faster, we have weakened the setting compared to the 'full exper- iment', namely 3000 steps per epoch, batch size 8, and
Small
mAP (o0-0-0000000
0.4
0.3
0.1
patience I in reducing the learning rate. During the in-
224
240
260
280
460
000
ference, it was coupled with Faster R-C R-CNN, ResNet-50 C4 1x', According to the graph in Figure we can con- clude the experiment the model is stable, and the crop size has a minor impact for sizes but 600, where a decrease in performance. For the extreme case of crops size 600, almost all objects, including the big ones, are upsampled, which distorts them.
Side size ips]
Figure 9: Influence of crops size
CONCLUDING REMARKS
We discussed the difficulties of the object detection problem. We shown the difficulties can be suppressed by the refinement stage coupled with an object detector, if the refinement is given as single-object detector. To solve the problem when one bounding box includes more objects, we propose to use mixture data where the image information is complemented with information about the object's class and center, which helps the network to refine the desired object. We showed the simple scheme could increase the mAP of the SOTA models. Finally, we presented that our scheme, BBRefinement, is able to produce predictions that are more precise than ground truth labels.
As the refinement process is partially independent of the detector, this approach opens new re- search direction. The original research, which is focused on increasing accuracy by proposing new architectures, etc.,
be complemented with independent research of refinement networks. The final system, which can be deployed to real productions on various competitions (such as Kag- gle Signate), may consist of combination of the best algorithms from both types of research.
ACKNOWLEDGMENTS
The work is supported by ERDF/ESF "Centre for the development of Artificial Intelligence Methods for the Automotive Industry of the region" (No. 100.0.111100.0000077000000000000
REFERENCES
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6154-6162, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. arXiv preprint 1722 2020.
Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained bast- models. IEEE transactions pattern analysis and ma- chine intelligence, 1624-15677 2009.
Spyros Gidaris and Nikos Komodakis. Object detection via a multi-region and semantic segmentation-aware cnn model. In Proceedings of the IEEE international conference on com- puter vision, pp. 1134-113 1142, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for curate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.

---

Under review as a conference paper at ICLR 2021
Jicheng Gong, Zhao Zhao, and Nic Li. Improving multi-stage object detection via iterative proposal refinement. In BMVC, pp. 223, 2019.
Hei Law and Jia Deng. Cornernt: Detecting objects as paired keypoints. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 734-750, 2018
Jianan Li, Xiaodan Liang, Jianshu Li, Yunchao Wei, Tingfa Xu, Jiashi Feng, and Shuicheng Yan. Multistage object detection with group recursive learning. IEEE Transactions on Multimedia, 20 10:1645-1655 2017.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He. and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. European conference on computer vision, pp. 21-37. Springer, 2016.
Siyuan Qiao, Liang-Chieh
Chich Chen, and Alan Yuille. Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. arXiv preprint urXiv:2006.02334 2020.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 658-666, 2019.
Leslie N Smith. Cyclical learning rates for training neural networks. 2017 IEEE Winter Confer- ence on Applications of Computer Vision (WACV), pp. 464-472. IEEE, 2017.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1194, 2019.
Matthew D Zeiler. 2012.
Adadelta: an adaptive learning rate method. arXiv preprint arXiv: arXiv: 1212.5701,
Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Li. Single-shot refinement neural network for object detection. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 4203-4212, 2018.
Shifeng Zhang, Longyin Wen, Zhen Lei, and Stan Z Li. Refinedet++: Single-shot refinement neural network for object detection. IEEE Transactions Circuits and Systems for Video Technology, 2020.
Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren. Distance-iou loss: Faster and better learning for bounding box regression. In AAAI, pp. 12993-13000. 2020.
10


Reviews:
[{"review_id": "cNLrqvODeX8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach, BBRefinement, to refine the bounding box predictions of an object detector. After the object detector predicts boxes, a patch is cropped around each box and passed into a separate network for refinement. The authors show that this improves the performance of several object detectors on the COCO dataset.\n\nI think the novelty of this paper is limited as a similar idea has already been published in Cascade R-CNN [a]. The authors do not compare their work to Cascade R-CNN. To me, the major difference between this and Cascade R-CNN seems to be that Cascade R-CNN uses the RoI-pooled features for refinement, but this uses the image crops, which does not seem to be significant. Other differences such as BBRefinement use the centers and the classes of the boxes as additional information also seem to be minor. Furthermore, Cascade R-CNN seems to be able to provide better improvement on the APs. Since the authors do not provide any comparison with Cascade R-CNN, it is unclear whether BBRefinement is better at refining boxes than Cascade R-CNN or significantly different.\n\nThe authors claim an advantage of BBRefinement is that it can be trained with more images as it takes crops around each box as input while conventional detector uses the full images as input. I find this argument a little bit weird. Considering the number of actual pixels, BBRefinement actually has much fewer training data in terms of the number of pixels as large number of pixels are thrown away during cropping.\n\nThe authors also claim that BBRefinement is more robust to missing labels. The conventional detector may get penalized incorrectly for producing boxes for missing labels, while BBRefinement does not suffer from such issue as it only takes cropped images around each label as input. I don’t think this claim is fair. The authors should not compare BBRefinement to a detector because they are different. BBRefinement only refines the predictions from a detector but a detector classifies and localizes objects from an image. And actually, the regressors in convetional detectors also only process cropped images/feature maps. I am not seeing why this is an advantage to BBRefinement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RgmaLuaZ1w3", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents a simple yet powerful and flexible framework to refine the predictions of a two-stage detector. The approach can produce more precise predictions by using mixture data of image information and the objects' class and center. They showed a simple scheme can increase the mAP of the SOTA models and it is able to produce predictions that are more precise than ground truth.\n\nWeakness:\n+ The idea of this paper is to use a refinement module to boost the performance of the two-stage detectors. I find this work to contain very limited novelty that other researchers can use/build on. The proposed method simply uses a naive refine module to extract the Region feature from the crop. In my opinion, this simple module is similar to Cascade R-CNN. The only difference is that it extracts features from the crop of the images. It does not advance the understanding of this field although is reasonable to me.\n\n+ The ablation experiments are weak and inadequate. Only one experiment is provided to compare the performance of the refine module. The author should do more ablation studies to support his contribution. e.g. (1) the comparison with the Cascade R-CNN which extracts the feature from the region feature maps rather than the images. (2) the architecture or the refinement module number. \n\n+ The claim of run in real-time on standard hardware, without any time cost or FPS results in this paper.  However, the speed of the refinement module may be slow owing to the extracting feature from the crops. For the two-stage detector, e.g. FPN, the proposals of the detector are 512 under the common setting. \n\n+ The results would have been more complete if results were shown in a setting where the region feature is used without the use of the original crops. In other words, an ablation study on the effect of the feature extraction strategies. \n\n+ How important is the crop size to the proposed method? Considering the paper states that this is required to get a good crop, some ablation studies on showing the crop strategies would be useful for understanding.\n\n+ In Abstract, the author of this paper provides his code which is non-anonymous. It shows that the repository of BBREFINEMENT is the \"IRAFM AI\" and the author's name is easy to be found.  This behavior violates the rules of the anonymous code mentioned in the Author Guide. \n\nFinally, I suggest rejecting the paper. BTW, The author should pay attention to the rules the next time.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BES1WrhrSLp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-The idea of this paper is just to crop the detections and then forwarded to a second stage for more accurate predictions. This idea can be traced back to the original R-CNN paper, which is not even referred and discussed. There are also many papers having a second stage to refine the detection predictions, e.g. Cascade R-CNN, RefineDet, Revisiting RCNN, but none of them are discussed in this paper.\n\n-The writing is terrible.\n\n-Tons of literature is missing.\n\n-To be honest, this paper should be desk rejected.\n\n=====updates======== \n\nAfter reviewing the other reviews and rebuttal, I will remain my original recommendation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6ztLO2NuAx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary: \n\nThis paper proposes BBRefinement, which is a post-processing for object detection to refine the predicted bounding boxes. BBRefinement takes cropped images from predicted bounding boxes as input and refine the bounding box with a separate network that is only targeted in predicting box offsets.\n\n##########################################################################\n\nPros:\n\nThe proposed method is simple and experiment shows the effectiveness of proposed method.\n\n##########################################################################\n\nCons:\n\n1. Missing literature review. This paper is not the first one to study how to refine bounding boxes. There are a lot of works on refining bounding boxes [A, B, and many more], but this paper fails to discuss related works and explain the connection as well as the differences with them.\n\n2. Missing ablation studies. Section 2.2 discusses principals of BBRefinement, including the importance of using mixture data. However, there is no experiment supporting this claim. Also the expanding ratio of bounding boxes is an important parameter, but there is also no experiment on this parameter.\n\n3. This paper only applies the proposed method to very simple baselines like Faster R-CNN and RetinaNet. These methods (Faster R-CNN, RetinaNet) are known to predict not tight bounding boxes. I wonder if BBRefinement is still necessary when a method already predicts tight bounding boxes like Cascade R-CNN [C].\n\n4. It is not clear to me how the model is trained, especially how boxes are sampled during training. Is it an image-centric sampling or an instance-centric sampling? Does sampling strategy matter? There could also be false positives in the prediction, does these boxes harm the training (e.g. existence of background box)?\n\n5. Table 1 needs more explanation, do you need to train separate BBRefinement for each model? If not, what boxes do you use to train the BBRefinement.\n\n6. The timing of BBRefinement, is \"23ms for B1 and 44ms for B3\" a single box? What is the average end-to-end runtime on the whole dataset? The timing is also much faster than EfficientNet speed in [D], [D] reports 52 ms for B1 and 114 ms for B3. Can you explain why your timing is 2x faster?\n\n7. The dataset split on COCO also has problem, COCO train 2014 and part of val 2014 is exactly the same as train 2017. And the 5000 minival 2014 is exactly the same as val 2017.\n\n##########################################################################\n\nReasons for score:\n\nAlthough this paper presents a simple and effective solution, the overall quality of the paper is poor. First, this paper does not have discussion on related works **AT ALL**. Second, it misses important implementation details and important ablation studies. Third, this paper only applies the method to weak detectors and fails to apply it to methods that give tight bounding boxes like Cascade R-CNN. Finally, the authors put a [link to the code](https://gitlab.com/irafm-ai/bb-refinement) which leaks the authorship (one author's name and institute); this is a violation of the double-blind review policy. Considering all this facts, this paper is a clear reject to me.\n\n##########################################################################\n\nReferences:\n\n[A] Object detection via a multi-region & semantic segmentation-aware CNN model, ICCV 2015  \n[B] A MultiPath Network for Object Detection, 2016  \n[C] Cascade R-CNN: Delving into High Quality Object Detection, CVPR 2018  \n[D] Designing Network Design Spaces, CVPR 2020  \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the simplicity of the approach, but found the exposition lacking. There were also concerns about strong similarities to CascadeRCNN, which were not resolved in the rebuttal.
In the end all reviewers recommend rejection. The AC sees no reason to overturn this recommendation.
Decision: Reject