BBRefinement: an universal scheme to improve precision of box object detectors

1 PROBLEM STATEMENT . Object detection plays an essential role in computer vision , which attracts a strong emphasis on this field among the researchers . That leads to a situation when new , more accurate , or faster object detectors replace the older ones with high frequency . A typical object detector takes an image and produces a set of rectangles , so-called bounding boxes , which define borders of objects in the image . The detection quality is measured as an overlap between the detected box and ground truth ( GT ) , and it is essential for two reasons . Firstly , the criterion used in benchmarks – mean Average Precision ( mAP ) – is based on particular thresholds for various values of Intersect over Union ( IoU ) between the prediction and the GT . Such thresholds are typically applied to distinguish between accepted and rejected boxes in detection . Therefore , precision here is crucial to filter valid boxes from discarded . Secondly , the more precise the detected box is , the more accurate the classification should be . Although NN-based classifiers can deal with some tolerance in shifted or cropped data , the higher accuracy in the object detection may lead to the increased accuracy in the classification process . Existing solutions for object detection yield accuracy around 0.3–0.5mAP on the COCO dataset ( Lin et al. , 2014 ) . Such a score allows the usage in many real applications . On the other hand , there is space for improvement . A combination of the following may reach such growth : more precisely distinguish between classes ; increase the rate of true-positive detections ; decrease falsepositive detections ; or increase the IoU of the detections . There are four points on why object detection may be difficult in general , which blocks further mAP growth . 1 ) A neural network has to find all objects in an image . The number may vary from zero to hundreds of objects . 2 ) A neural network has to be sensitive to all possible sizes of an object . The same object class may be tiny or occupy the whole image . 3 ) A network usually has no a priori information , which should make the detection easier , like the context of the scene or the number of objects . 4 ) There is a lack of satisfactory big datasets . Therefore , the distribution of data is sampled roughly only . In this paper , we propose BBRefinement , which can suppress the effect of all the four mentioned difficulties . The proposed inference scheme ’ Detection → Refinement ’ is achieved by a combination during prediction phase with a generic detector , and it increases the IoU of the detected boxes with its ground truth labels , resulting in higher mAP . Related work . The problem of refinement can be tracked to the origin of two-stage detectors , where R-CNN ( Girshick et al. , 2014 ) uses a region proposal algorithm that is used to generate a fixed number of regions . The regions are classified and by bounding box regressor refined . Faster R-CNN ( Ren et al. , 2015 ) replaces the region proposal generation algorithm with a region proposal network . The same bounding box regressor can be used iteratively to obtain more precise detections ( Gidaris & Komodakis , 2015 ; Li et al. , 2017 ) . The effect of iterative refinement may be increased by involving LSTM module ( Gong et al. , 2019 ) . The aim of refinement can also be anchors ; RefineDet ( Zhang et al. , 2018 ; 2020 ) refines them to obtain customized anchors for each cell . Cascade R-CNN ( Cai & Vasconcelos , 2018 ) uses a sequence of bounding box regressors to create n-staged object detector . In Cascade R-CNN , network head h0 takes proposals from the region proposal network and feeds the regressed bounding boxes to network head h1 and so on . All the heads work over the same features extracted from a backbone network . The cascade scheme shows that h1 is dependant on the quality of h0 head . If h0 includes some bias , h1 balances it . Therefore , all the heads have to be trained together ( part by part ) , and if h0 is retrained , h1 should be retrained as well . In contrast , BBRefinement is a trained standalone , and it is not dependent on the quality of the object detector with whom it is coupled during inference . That makes BBRefinement universal and able to be applied on various image detectors without retraining a detector or BBRefinement . 2 EXPLAINING BBREFINEMENT . The main feature of BBRefinement is a transformation of the problem into a simpler scheme , where an NN can be trained easily . Compared with a standard object detector , BBRefinement is a specialized , one-purpose neural network working as a single object detector . It does not search for zero-to-hundreds objects , but it always detects only a single object and does not produce its confidence . It is also missing the part responsible for classification , so it does not assess the object ’ s class . The only purpose is to take an image with a single object within a normalized scale and generate a super-precise bounding box . The training is realized on boxes extracted from a dataset according to ground truth labels . When BBRefinement is trained , the fixed model can be coupled with an arbitrary detector to realize the inference . Here , the feed for BBRefinement is bounding boxes in the form of image data produced by the detector . 2.1 PROBLEM WITH A NAIVE SINGLE OBJECT DETECTOR . Let bounding box b be given by its top-left and bottom-right coordinates b = ( x1 , y1 , x2 , y2 ) . Further , let us suppose a color image f : D ⊂ N2 → L ⊂ R3 . Then a neural network detecting single object is generally noted as g : f → b . To train such a network , we generally minimize term |b− g ( f ) | or its alternatives . The issue comes when f includes two objects at once , and the network is extended to produce two bounding boxes . The network should return b1 for the first object and b2 for the second one . However , a generic solution will predict a box 0.5 ( b1 + b2 ) for the both cases , or generally 1/n ∑n i=1 bi for n if we consider that the boxes have the same frequency of occurrence . A naive solution is to modify the network to detect a sparse set of objects g : f → B , where B = { b1 , b2 , . . . , bn } assumes boxes in a fixed order . Such a detection scheme is not possible in general without a deeper modification of an architecture leading to the presence of a grid , etc . With no guarantee of a single object ’ s presence only , the naive solution can not be used . This problem was solved later in chronological order by a sliding deformable models / window technique ( Felzenszwalb et al. , 2009 ) , two-stage techniques such as ( Fast/Faster ) R-CNN ( Ren et al. , 2015 ) , single-stage techniques as SSD ( Liu et al. , 2016 ) or YOLO ( Redmon & Farhadi , 2017 ) and finally by anchors-free techniques that are mainly keypoint-based ( Law & Deng , 2018 ) . Every such approach affects the architecture of the neural network and is related to a specific model . BBRefinement , as a single object detector , would suffer the shortcomings mentioned above . The reason is that even if we have an extracted object defined by its bounding box , such bounding box for a non-rectangular object will also involve some background , which may contain other objects . The presence of other objects leads to the problem described in the previous paragraph , and in conclusion , it may confuse the neural network , thus cause improper refinement . The examples from the COCO dataset are shown in Figure 2 for the easy case and in Figure 3 for the problematic case . For illustration , the COCO dataset includes 1.7M boxes from which 47 % of all boxes have an intersection with a box with the same class , and 84 % of boxes have an intersection with an arbitraryclass box . To solve the problem , BBRefinement uses information about the object ’ s class and center as well . 2.2 THE PRINCIPLE OF BBREFINEMENT . Firstly , we need to define a neural network in more detail , so we suppose a convolutional neural network F to be set of k layers , F = { f1 , f2 , . . . , fk } . Here , all the layers are meant as fi : D ⊂ Rmi → L ⊂ Rni which are for the sake of simplicity defined as convolutions layers without poolings/residual connections/batch norms etc , i.e. , fi ( Mi ) = a ( Wi ⊗ Mi ) , where a is activation function , Wi weights and Mi is output of the previous layer , or , an input image in the case of i = 1 . Such a neural network is generally called as a backbone , with the aim to map an input image iteratively into feature space . Here , we suppose ek : D ⊂ Rnk → L ⊂ R to be an embedding of the k-th layer created as ek ( fk ( Mk ) ) = p ( fk ( Mk ) ) , where p is global average pooling or flattening operation . Furthermore , we suppose a fully connected network G to be set of j layers , G = { g1 , g2 , . . . , gj } . All layers are meant as gi : D ⊂ Rqi → L ⊂ Rei as gi ( Mi ) = a ( WiMi ) , where a is activation function , Wi weights and Mi is an output of the previous layer , or an input vector in the case of i = 1 . According to the motivation , we propose to use mixture data as an input to the suggested scheme of refinement . Convolution neural network F processes the input image ( crop with a fixed resolution ) containing an object , a fully connected neural network G processes a fixed-size vector that holds information about a class and an expected center of the object . The both networks are designed in order to |ek ( fk ( Mk ) ) | = |gj ( Mj ) | be valid . Then , both information is mixed together as x = ek ( fk ( Mk ) ) · gj ( Mj ) , where · is a dot product . Finally , we connect x with the output layer o consisting of four neurons and utilizing the sigmoid activation function . Such a neural network is trained in a full end-to-end supervised scheme . From a practical point of view , we can use an arbitrary SOTA backbone such as ResNeSt , ResNeXt , or EfficientNet , to mention a few . For BBRefinement , we use EfficientNet ( Tan & Le , 2019 ) due to its easy scalability . In the benchmark section , we are presenting results for versions B0–B4 . According to the version , an input image ’ s resolution is 2242 , 2402 , 2602 , 3002 , and 3802 . The version affects |ek ( fk ( Mk ) | as well , it is 1280 ( B0 and B1 ) , 1408 , 1536 , and 1792 . The pipeline for the prediction with BBRefinement is illustrated in Figure 1 and is as follows . The boxes detected by a generic object detector are taken ( with small padding ) from the input image and rescaled into the BBRefinement input resolution . That has several beneficial consequences . Firstly , bigger objects are downscaled , and smaller are upscaled to fit the resolution , so all the objects have the same scale , which is much more effective than train a network for multiscale detection . Second , one image from the dataset yields multiple boxes . In the case of COCO , a standard detector uses 0.2M images ( one image as an input ) , while BBRefinement uses 1.7M images ( one box as an input ) . Also , a standard detector rescales the input images into a specific resolution to fit GPU memory , so many pixels are thrown out . BBrefinement does not use non-object parts from the image , but it allows us to use more pixels from the object-are due to a weaker downsample . Third , thanks to mixture data usage , BBRefinement obtains information about the object ’ s detected class and center . Although such data may be imprecise , it is a piece of priory information , making the task more accessible . Finally , there is a guarantee that each crop includes just one main object . That explains why BBRefinement can produce more precise coordinates of bounding boxes than a general object detector . Note , it is necessary to take into account BBRefinement is placed on top of an object detector , which may be imprecise . As a result , the data fed into BBRefinement may be ambiguous . Therefore , the crops should not be extracted precisely but surrounded by padding ; during the training , the padding is random . For the same reason , it is beneficial to distort the center by random shifts for training . Such augmentation is visualized in Figure 4 . To realize the refinement , there are several options on how to define loss function ` used for training BBRefinement . The first option is to compare each coordinate of the box with the GT label using , e.g. , binary cross-entropy ( BCE ) . The second option is to use BCE for comparing top-left points and then euclidean distance for evaluating the width and height of a box . Such an approach is used , e.g. , in YOLO . The third way is to use coordinates of all points to determine the boxes ’ areas and compute IoU . Let us imagine a situation where a box is shifted by a pixel according to its label in a vertical and horizontal direction . If we shift the box in the directions separately , we will observe that the sum of such partial losses will be equal to the loss produced by shifting in both directions at once . That is not valid behavior ; the second loss should be bigger . On the other hand , the triangle inequality is fulfilled when IoU is used . The next problematic situation is when the euclidean distance is used : bigger boxes tend to produce bigger differences than small boxes . This means they produce bigger losses , and a neural network tends to focus on them more than on the small boxes . IoU is computed as a relative value ; therefore , the same difference in width or height creates a bigger loss for small boxes , which is the desired behavior . Based on these reasons , we use IoU-based loss for training BBRefinement . We have two available options on how to define the IoU loss function . Namely ` 1 ( b , b′ ) = −log ( i ( b , b′ ) /u ( b , b′ ) ) for the logarithmized form and ` 2 ( b , b ′ ) = 1.0− i ( b , b′ ) /u ( b , b′ ) for the linear form , where i represents intersection of two boxes , u their union , b is GT box , and b′ predicted box . Logarithmized form . Let us consider a task where the evaluation criterion involves IoU with some threshold , such as 0.5 , which can be found in many real competition websites such as Kaggle or Signate . Here , it is much more to satisfy the threshold rather than to reach the best possible IoU . Here , the goal of the BBRefinement is to take unprecise boxes ( with IoU < 0.5 ) and push them over the threshold . The logarithm-shape of IoU loss generates the biggest loss for unprecise boxes , while the loss is vanishing for high-precision boxes , similar to Focal loss ( Lin et al. , 2017 ) . That is also beneficial if we take into account that no dataset is perfect , and labels created by humans are not accurate . Here , the property of smaller loss for near-perfect detections would be beneficial . Linear form . The COCO dataset ’ s official benchmark computes mAP for several IoU thresholds , such as 0.5 , . . . , 0.95 . Here , the situation is the opposite : by refinement , pushing a box from , e.g. , 0.4 into 0.95 , is more beneficial than refine three boxes from 0.4 to 0.54 . The reason is that a box with IoU > 0.95 will be taken into account in all the IoU thresholds , while a box with IoU 0.54 will be taken into account only for the threshold of 0.5 ; other thresholds will count it as a false positive . Therefore , we use the linear form in the following benchmarks . Note , both forms of the loss function can be based on a more efficient version of IoU . The other choices may be Generalized IoU loss ( Rezatofighi et al. , 2019 ) , Complete IoU , or Distance IoU ( Zheng et al. , 2020 ) . Generally , these IoUs converge faster and are able to compute loss effectively , even for boxes without overlap .

Summary: -The idea of this paper is just to crop the detections and then forwarded to a second stage for more accurate predictions. This idea can be traced back to the original R-CNN paper, which is not even referred and discussed. There are also many papers having a second stage to refine the detection predictions, e.g. Cascade R-CNN, RefineDet, Revisiting RCNN, but none of them are discussed in this paper.