Bootstrap Your Object Detector via Mixed Training

1 Introduction . Object detection is a fundamental task of computer vision . Its goal is to locate the bounding boxes of objects in an image as well as to classify them . Due to the complexity and diversity of the real world , this problem remains challenging despite the considerable attention it attracts . Most previous works focus on developing better detection frameworks [ 24 , 34 , 5 , 2 , 21 , 23 , 16 ] or stronger network architectures [ 18 , 1 , 14 , 28 , 12 , 14 ] . In addition , there are some that study data augmentation strategies [ 10 , 6 , 4 ] , label assignment [ 39 , 17 ] , or training losses [ 19 , 25 ] . In general , the existing works follow a standard training paradigm where the network takes training images that are augmented by a single data augmentation strategy and the human-annotated bounding boxes are simply used as the training targets . We refer to this approach as SiTraining . Few works explore alternative training methods , which have been based on distillation [ 3 , 32 , 40 ] or dynamic adjustment of label assignment criteria and regression loss [ 35 ] . In this work , we expand the power of an augmentation strategy and the utility of human-annotated boxes through a new training paradigm for object detection . We observe that suitable magnitudes of an augmentation can vary from image to image , where a strong augmentation of certain images will enrich the training data , but may degrade the data when applied to other images , such as by altering the object appearance to become less compatible with the object class . Based on this , we present a mixed augmentation strategy that utilizes both strong and normal augmentations in a manner that takes advantage of strong augmentations only on images where they are expected to be helpful . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We additionally note that human annotation of bounding boxes is often noisy or incomplete , which can be harmful to training . To alleviate this issue , we introduce the use of mixed training targets , composed of both human-annotated ground-truths and pseudo boxes that are intended to compensate for annotation errors . The pseudo boxes are determined by bootstrapping on the detector . Because of the robustness of neural networks to label noise , the online detector can produce pseudo boxes that capture object locations missed or inaccurately localized by human labelers . We therefore utilize these pseudo boxes in conjunction with the human-annotated boxes to improve detection . Furthermore , the online detector can predict whether a strong augmentation of a training image would help training . This is accomplished by using the online detector to compute the foreground score of a training target . A high score indicates that the target can be easily trained on , while those with a low score are discarded from training . Our training paradigm , called MixTraining , integrates the mixed augmentation and mixed training targets as illustrated in Figure 1 . In the bottom two branches , normally augmented and strongly augmented images are passed to the detector for training . In the top branch , an exponential moving average ( EMA ) model of the online detector is used to generate pseudo boxes and to predict the foreground scores of the training targets . Only for training targets with a high score will their strong augmenta- tions be included for training . As the detector progresses through training , the mixed augmentation and mixed training targets become increasingly better via bootstrapping . MixTraining is a general training framework for object detection that can enhance existing object detectors without introducing extra computation or model parameters at inference time . Our experiments show that MixTraining can appreciably improve the performance of leading object detectors such as Faster R-CNN [ 24 ] with a ResNet-50 [ 13 ] backbone ( from 41.7 mAP to 44.0 mAP ) and Cascade R-CNN [ 1 ] with the recently proposed Swin-Transformer [ 22 ] backbone ( from 50.9 mAP to 52.8 mAP ) . 2 Related works . Framework and Network Design in Object Detection Designing more effective frameworks has been a research focus in object detection . Most object detectors can be separated into two framework categories : single-stage object detectors [ 21 , 23 , 19 , 29 ] and two-stage object detectors [ 24 , 5 , 16 ] . Although we mainly conduct experiments of our method on two-stage detectors to verify the effectiveness this work , as a general training paradigm , MixTraining is suitable for both single-stage and two-stage detectors . There are also many works exploring stronger network architectures . For example , feature pyramid network [ 12 ] utilizes a pyramid network to tackle the challenge of scale diversity , and Cascade R-CNN [ 1 ] introduces a multi-stage head for improving localization accuracy . These methods can significantly elevate detection performance , and our work is compatible with these methods . Data Augmentation in Object Detection Recently , the importance of data augmentation has become evident in the research community . Existing works can be classified into two categories . Methods in the first category focus on developing new transformation components . For example , Mixup [ 36 ] proposes to use mixed images and labels as training samples . CutOut [ 8 ] masks a part of a region in the original inputs . CopyPaste [ 11 ] and InstaBoost [ 10 ] further consider data augmentation at the instance level , but both of them require instance mask annotations . Another type of method studies how to combine existing transformations more effectively . Representative works include AutoAug [ 41 ] and RandAug [ 6 ] . Different from those methods that focus on how to improve a single augmentation strategy , our method considers two augmentations of different magnitudes . Current augmentation methods are , in fact , compatible with our mixed augmentation strategy , which can be applied with arbitrary forms of augmentation . Label Assignment in Object Detection Existing works on label assignment mainly focus on improving assignment accuracy . GuidedAnchoring [ 30 ] dynamically changes the shape of the anchor and leverages semantic features to fit the objects better . ATSS [ 37 ] presents an adaptive label assignment mechanism by dynamically adjusting the IoU threshold . AutoAssign [ 39 ] assigns positive or negative weights for each sample based on the training loss . Different from those methods , MixTraining does not change the mechanism of label assignment itself , but changes the availability of each training target based on its predicted foreground score and the magnitude of the data augmentation . Alternative Training Methods for Object Detection Only a few works explore alternative training methods for object detection , and they have primarily been based on distillation . For example , in [ 32 , 40 ] , feature distillation is presented to obtain fine-grained features or reduce invalid contextual information . Another work [ 38 ] focuses on the localization ambiguity issue , which is also explored in this work but we arrive at a different conclusion . In [ 3 ] , the distillation of features and detection results are considered at the same time . However , they rely on a well-trained teacher detector and mainly focus on distilling knowledge from a stronger teacher model to a weaker student detector . Compared with these distillation-based methods , MixTraining also consists of two branches but mainly focuses on annotation noise and measuring the training difficulty of training targets . Dynamic R-CNN [ 35 ] examines the existing training paradigm from the training dynamics perspective . During training , it dynamically adjusts the label assignment criteria and the parameters of the regression loss for fuller training of the network . In comparison , MixTraining enhances training by gradually introducing new well-trained samples into the set of strongly augmented images . 3 MixTraining . In this section , we introduce our new training scheme , MixTraining , which integrates the two major components : mixed training targets and mixed data augmentation . The whole training paradigm is illustrated in Figure 1 . In the following , we will introduce each component in detail . Mixed Training Targets Conventional SiTraining uses only human-annotated ground-truths to train the detector . However , human annotation is imperfect and often includes annotation noise such as inaccurate localization or missing labels which can harm training . Examples are shown in Figure 2 . Because of the robustness of deep neural networks to label noise [ 26 ] , they can predict pseudo boxes , also known as pseudo labels , that are relatively free of localization noise and resilient to missing labels . This property has been exploited in semi-supervised object detection [ 27 ] , weakly-supervised object detection [ 9 ] , and sparsely annotated object detection [ 31 ] . However , its value in supervised object detection has rarely been explored . In this work , we introduce the use of mixed training targets that include both human-annotated ground-truth boxes and pseudo boxes to alleviate the issue of noisy annotation . To generate high-quality pseudo boxes , we follow the common practices [ 27 , 33 ] of applying an exponential moving average ( EMA ) model on the input images without any data augmentation except for scale jitter . Since the predicted boxes of an object detector are often noisy and redundant , non-maximum suppression ( NMS ) is employed to eliminate redundancy , where only pseudo boxes with a foreground score greater than 0.9 are retained . We next examine how to properly combine the pseudo boxes predicted by the object detector with human-annotated ground truth . For this , we design three strategies to deal with the missing label and noisy box localization issues : • Missing labels . In this strategy , we retain pseudo boxes whose IoU ( intersection-over-union ) with the ground truth is less than 0.5 . They can be considered as missing annotations and are added to the training targets . An example is shown in Figure 3 ( c ) . • Box localization noise . Due to occlusion , small object size , and human subjectivity , the annotation quality for box localization is often inconsistent . We alleviate this issue by replacing the ground-truth boxes by pseudo boxes whose IoU with ground-truth boxes is greater than 0.5 . An example is shown in Figure 3 ( d ) . • Hybrid Approach . The above two strategies are compatible with each other , and can be used at the same time , we named this strategy as hybrid approach , which has the advantages of the above two strategies . An example is shown in Figure 3 ( e ) . We test the three strategies on the COCO dataset [ 20 ] in Sec . 4.3 . The results show that this treatment of missing labels notably improves detection performance , and there is no significant affects that applying the strategy for box localization noise alone . However , the hybrid strategy , which takes the advantages of other two strategies , show better performance than each of the above two strategies . By thus , we adopt the hybrid strategy in producing mixed training targets . 0.85 0.93 In MixTraining , we expand single data augmentation to a mixed strategy : during training , we randomly sample and apply an augmentation from among strong and normal augmentations for each image . Training over various levels of augmentation can be advantageous because of the greater diversity of data . However , strong augmentations can potentially hurt training if the resulting appearance becomes incongruous with object class . We address this issue by using a training loss in which strongly augmented images are taken only for targets that can be easily trained on . A direct approach to implement this idea is to simply remove non-easy targets from all the training samples . However , a consequence of this is that many foreground proposals would be incorrectly assigned as background in the label assignment , as shown in Figure 4 . Therefore , we instead assign a weight w for each training target g : w ( g ) = { 1 , g is an easy target or a normal augmentation 0 , otherwise . ( 1 ) For all images that are normally augmented , w ( g ) is set to 1 for all of their training targets . For the image that are strongly augmented , w ( g ) is set to 1 only if g is an easy training target . Then , we use this weight w as a loss weight for each proposal in the training stage : Ldet = N∑ i=0 w ( gi ) Ldet ( pi , gi ) ( 2 ) where pi is the i-th proposal , and gi is the training target assigned to pi . To determine whether g is an easy target , we employ a simple approach where the foreground score of g is predicted by the EMA model , and g is considered an easy target if its score is greater than a threshold ( 0.9 by default ) . Since this method uses the same model and input images as pseudo box generation , the extra computational overhead can be largely reduced by treating the targets { gi } as proposals in the pseudo box generation process .

Summary: Object detection is a fundamental task in computer vision. In this paper a new paradigm of training, MixTraining, is proposed which utilizes augmentation of different strengths. Either standard augmentation and standard processing is used for an image or strong augmentation is applied (mixed data augmentation) and changed ground truth is used (loss is computed only on simple boxes and mixed training targets is used). As soon as human-annotated bounding boxes often are noisy (missing or inaccurate localizations) the other key ingredient when strong augmentation is applied to an image is to use only confident bounding boxes predicted by the model itself (fixes errors in human annotations) in addition to the confident bounding boxes from the ground truth (making simpler task as soon as strong augmentation is used). This key ingredient is obtained via pseudo-labeling by EMA detector (pseudo boxes). The idea is that this pseudo boxes and bootstrapping from the model itself help to detect images where strong augmentation can be expected to be helpful. MixTraining is shown to consistently improve various detectors on the COCO dataset.