Bootstrap Your Object Detector via Mixed Training
Authors: Mengde Xu
Zheng Zhang
Fangyun Wei
Yutong Lin
Yue Cao
Stephen Lin
Han Hu
Xiang Bai
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: object detection

Abstract: We introduce MixTraining, a new training paradigm for object detection that can improve the performance of existing detectors for free. MixTraining enhances data augmentation by utilizing augmentations of different strengths while excluding the strong augmentations of certain training samples that may be detrimental to training. In addition, it addresses localization noise and missing labels in human annotations by incorporating pseudo boxes that can compensate for these errors. Both of these MixTraining capabilities are made possible through bootstrapping on the detector, which can be used to predict the difficulty of training on a strong augmentation, as well as to generate reliable pseudo boxes thanks to the robustness of neural networks to labeling error. MixTraining is found to bring consistent improvements across various detectors on the COCO dataset. In particular, the performance of Faster R-CNN~\cite{ren2015faster} with a ResNet-50~\cite{he2016deep} backbone is improved from 41.7 mAP to 44.0 mAP, and the accuracy of Cascade-RCNN~\cite{cai2018cascade} with a Swin-Small~\cite{liu2021swin} backbone is raised from 50.9 mAP to 52.8 mAP.


Bootstrap Your Object Detector via Mixed Training
Mengde Xu Xu1.3+ 1.3+
Zheng Zhang 133+
Fangyun Wei
Yutong 2.3
Yue Cao
Stephen Lin
Han Hu
Xiang Bai'
Huazhong University of Science and Technology
2 an Jiaotong University Microsoft Research Asia
Abstract
We introduce MixTraining, a new training paradigm for object detection that can improve the performance of existing detectors for free. MixTraining enhances data augmentation by utilizing augmentations of different strengths while excluding the strong augmentations of certain training samples that may be detrimental to training. In addition, it addresses localization noise and missing labels in human annotations by incorporating pseudo boxes that can compensate for these errors. Both of these MixTraining capabilities are made possible through bootstrapping on the detector, which can be used to predict the difficulty of training on strong augmentation. as well as to generate reliable pseudo boxes thanks to the robustness of neural networks to labeling error. MixTraining is found to bring consistent improvements across various detectors on the COCO dataset. In particular, the performance of Faster R-CNN (24] with ResNet-50 [13] backbone is improved with a backbone is raised from 50.9 mAP to 52.8 mAP.
from 41.7 mAP to 44.0 mAP, and the accuracy of Rascade-RCNi Swin-Small 22
Introduction
Object detection is fundamental task of computer vision. Its goal is to locate the bounding boxes of objects in an image as well as to classify them. Due to the complexity and diversity of the real world, this problem remains challenging despite the considerable attention it attracts. Most previous works focus on developing better detection frameworks 22 34 2 23 [6 or stronger
In addition, there are some that study data augmentation or training losses [19 25 In general, the existing works follow standard training paradigm where the network takes training images that are augmented by a single data augmentation strategy and the human-annotated bounding boxes are simply used as the training targets. We refer this approach as SiTraining. Few works explore alternative training methods, which have been based on distillation 3 32 40 or dynamic adjustment of label assignment
network architectures [18 114 28
12
14)
strategies [10 4 label assignment 39
criteria and regression loss (3]]
In this work, we expand the power of an augmentation strategy and the utility of -nnontttd boxes through new training paradigm for object detection. We observe that suitable magnitudes of an augmentation can vary from image to image, where a strong augmentation of certain images will enrich the training data, but may degrade the data when applied to other images, such by altering the object appearance to become less compatible with the object class. Based on this, we present mixed augmentation strategy that utilizes both strong and normal augmentations in manner that takes ad advantage of strong augmentations only on images where they are expected to be helpful.
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

EMA
Revision
Detector
Raw Image
Prediction
Missing Label
normal
0.91
listron
Image (Normal Aug)
GT (Prediction Score>0.9)
Prob Map (Normal)
Detector
Image (Strong Aug)
Prob Map (Strong)
Figure 1: The illustration of our training paradigm MixTraining. It integrates mixed augmentation
and mixed training targets. In the bottom two branches. normally augmented images and strongly augmented images are passed the detectors for training. In the top branch, an EMA detector used to generate pseudo boxes and predict the foreground scores of the training targets. Only targets with score higher than 0.9 will be used for training strongly augmented images.
We additionally note that human annotation of bounding boxes is often noisy or incomplete, which can be harmful to training. To alleviate this issue, we introduce the use of mixed training targets, composed of both human-ana aaman--nnotated ground-trun -ttuths and pseudo boxes that are intended compensate for annotation errors.
The pseudo boxes are determined by bootstrapping the detector. Because of the robustness of neural networks to label noise, the online detector can produce pseudo boxes that capture object locations missed or inaccurately localized by human labelers. We therefore utilize these pseudo boxes in conjunction with the human-annotatee annotated boxes to improve detection. Furthermore, the online detector can predict whether strong augmentation of training image would help training. This accomplished by using the online detector to compute the foreground score of training target. A high score indicates that the target can be easily trained on, while those with low score are discarded from training.
Our training paradigm, called Mix- Training, integrates the mixed aug- mentation and mixed training targets as illustrated in Figure [ In the bottom two branches, normally aug- mented and strongly augmented im- ages are passed the detector for training. In the top branch, an expo- nential moving average (EMA) model of the online detector is used to gen- erate pseudo boxes and predict the foreground scores of the training tar- gets. Only for training targets with a high score will their strong augmenta- tions be included for training. As the detector progresses through training, the mixed augmentation and mixed training targets become increasingly better via bootstrapping.
Figure 2: Illustration of annotation noise in COC02017.
(left) The red box is inaccurate annotation, and the correct localization is the green box. (right) Red boxes are original annotations, and the green box is the missing label.
MixTraining general training framework for object detection that can enhance existing object detectors without introducing extra computation or model parameters at inference time. Our experi- ments show that MixTraining can appreciably improve the performance leading object detectors such as Faster R-CNN 24 with a ResNer-50 [I3 backbone (from 41.7 mAP to 44,0 mAP) and Cascade R-CNN I with the recently proposed 52.8 mAP).
22 backbone (from 50.9 mAP to

---

2
Related works
Framework and Network Design in Object Detection Designing more effective frameworks has been a research focus in object detection. Most object detectors can be separated into two framework categories: single-stage object detectors 21
23 19 29] and two-stage object detectors [24] [ [6]. Although we mainly conduct experiments of our method two-stage detectors to verify the effectiveness this work, as general training paradigm, MixTraining suitable for both single-stage and two-stage detectors.
There are also many works exploring stronger network architectures. For example, feature pyramid network [[2] utilizes a pyramid network to tackle the challenge of scale diversity, and Cascade R-CNN [ introduces a multi-stage head for improving localization accuracy. These methods can significantly elevate detection performance, and our work is compatible with these methods.
Data Augmentation in Object Detection Recently, the importance of data augmentation has become evident in the research community. Existing works can be classified into two categories. Methods in the first category focus on developing new transformation components. For example, Mixup 36 proposes to use mixed images and labels training samples. CutOut [8] masks part of region in the original inputs. CopyPaste [1] and InstaBoost [10] further consider data augmentation at the instance level, but both of them require instance mask annotations. Another type of method studies how to combine existing transformations more effectively. Representative works include AutoAug [41] and RandAug (6) Different from those methods that focus how improve single augmentation strategy, our method considers two augmentations of different magnitudes. Current augmentation methods are, in fact, compatible with our mixed augmentation strategy, which can be applied with arbitrary forms of augmentation.
Label Assignment in Object Detection Existing works on label assignment mainly focus on improving assignment accuracy. GuidedAnchoring B0 dynamically changes the shape of the anchor and leverages semantic features to fit the objects better. ATSS (37] presents an adaptive label assignment mechanism by dynamically adjusting the IoU threshold. AutoAssign [39] assigns positive or negative weights for each sample based on the training loss. Different from those methods, MixTraining does not change the mechanism of label assignment itself, but changes the availability of each training target based on predicted foreground score and the magnitude of the data augmentation.
Alternative Training Methods for Object Detection Only few works explore alternative train- ing methods for object detection, and they have primarily been based on distillation. For example, in 33 40). feature distillation is presented to obtain fine- graigrd features or reduce invalid contextual information. Another work (38] focuses the localization ambiguity issue, which is also explored in this work but we arrive at different conclusion. In [3], the distillation of features and detection results are considered at the same time. However, they rely
well-trained teacher detector and mainly focus on distilling knowledge from stronger teacher model to a weaker student detector. Compared with these distillation-base methods, MixTraining also consists of two branches but mainly focuses on annotation noise and measuring the training difficulty of training targets.
Dynamic R-CNN 335 examines the existing training paradigm from the training dynamics perspective. During training, dynamically adjusts the label assignment criteria and the parameters of the regression loss for fuller training of the network. In comparison, MixTraining enhances training by gradually introducing new well-trained samples into the set of strongly augmented images.
3 MixTraining
In this section, we introduce our new training scheme, MixTraining, which integrates the two major components: mixed training targets and mixed data augmentation. The whole training paradigm is illustrated in Figure In the following, we will introduce each component in detail.
Mixed Training Targets Conventional SiTraining uses only human-annotated ground-truths to train the detector. However, human annotation imperfect and often includes annotation noise such as inaccurate localization missing labels which can harm training. Examples are shown in Figure 2
3

---

(a)
(b)
(c)
(d)
(e)
Figure 3: (a) Human-annotated ground-truth boxes (blue boxes); (b) Predicted pseudo boxes (green
boxes); (c) Mixed training targets that address "Missing Label" issue; (d) Mixed training targets that
address "Box Localization Noise" issue; (e) Mixed training targets that applying hybrid approach.
Because of the robustness of deep neural networks to label noise [26.. they can predict pseudo boxes, also known pseudo labels, that relatively free of localization noise and resilient to missing labels. This property has been exploited in semi-supervised object detection [27], weakly
object detection [9], and sparsely annotated object detection [31] However, its value in supervised object detection has rarely been explored. In this work, we introduce the use of mixed training targets
that include both human-annotate ground-truth boxes and pseudo boxes to alleviate the issue of noisy annotation.
To generate high-quality pseudo boxes, we follow the common practices (27 33 of applying an exponential moving average (EMA) model on the input images without any data augmentation except for scale jitter. Since the predicted boxes of an object detector are often noisy and redundant, non-maximum suppression (NMS) is employed to eliminate redundancy. where only pseudo boxes with foreground score greater than 0.9 are retained.
We next examine how to properly combine the pseudo boxes predicted by the object detector with human-annotate ground truth. For this, we design three strategies to deal with the missing label and noisy box localization issues:
Missing labels. In this strategy, we retain pseudo boxes whose IoU cittersection-over-mne with the ground truth less than 0.5. They can be considered as missing annotations and are added the training targets. An example is shown in Figure
Box localization noise. Due to occlusion, small object size, and human subjectivity, the annotation quality for box localization is often inconsistent. We alleviate this issue by replacing the ground-truth boxes by pseudo boxes whose IoU with ground-truth boxes is greater than 0.5. An example is shown in 3
Hybrid Approach. The above two strategies are compatible with each other, and can be used at the same time, we named this strategy as hybrid approach, which has the advantages of the above two strategies. An example is shown in Figure
We test the three strategies on the COCO dataset 20 in Sec. 4.3 of missing labels notably improves detection performance, and there is no significant affects that applying the strategy for box localization noise alone. However, the hybrid strategy, which takes the advantages of other two strategies, show better performance than each of the above two strategies. By thus, we adopt the hy strategy in producing mixed training targets.
The results show that this treatment

---

Mixed Data Augmentation Appro- priate data augmentation is essential in training an object detector. In SiTraining, a single data augmentation strategy is typically utilized, and the augmentation syper-parameters need to be carefully tuned to achieve good detection performance. However, set- ting suitable hyper-parameters is challenge. Weak data augmentation prone to over-fitting and the model capacity may not be fully utilized. On the other hand, strong data augmen- tation may be detrimental, as it can lead to difficulty in training for some images and training targets.
Figure 4: The boxes with solid line are training targets, and the boxes with dashed line are proposals. (Left) There are two training targets: a person and a surfboard, and their foreground score are 0.9 and 0.85, respectively. (Right) If we simply remove the surfboard (non-easy target) from the training target set, then the red proposal will assigned as background, because its IoU with the person is less than 0.5.
In MixTraining, expand single data augmentation to a mixed strategy: during training, we randomly sample and apply an augmenta- tion from among strong and normal augmentations for each image. Training over various levels of augmentation can be advantageous because of the greater diversity of data. However, strong augmentations can potentially hurt training if the resulting appearance becomes incongruous with object class. We address this issue by using a training loss in which strongly augmented images are taken only for targets that can be easily trained
direct approach to implement this idea is to simply remove non-easy targets from all the training samples. However, a consequence of this is that many foreground proposals would be incorrectly assigned as background the label assignment, as shown Figure Therefore, we instead assign a weight w for each training target g:
= ( 0, otherwise.
1, g is an easy target or normal augmentation
w(g)=
(1)
For all images that are normally augmented, w((g is set to for all of their training targets. For the image that are strongly augmented, w(g) is set to I only if g an easy training target. Then, we use this weight w as a loss weight for each proposal in the training stage:
N
Liet
Σ O
Laed 9i)
where p; is the i-th proposal, and gs is the training target assigned to Ps-
To determine whether g is an easy target, we employ a simple approach where the foreground score of g is predicted by the EMA model, and is considered an easy target if its score is greater than threshold (0.9 by default). Since this method uses the same model and input images as pseudo box generation, the extra computational overhead can be largely reduced by treating the targets {gi} as proposals in the pseudo box generation process.
Experiments
4.1 Experimental Settings and Implementation Details
Dataset and Evaluation Protocol We validate our method on the COC02017 dataset 20). which contains 80 object categories, 118k images for training (train2017), 5k images (minival) for validation and 20k images for testing (test-dev). The mean average precision (mAP) is adopted as the default metric for measuring performance. In our experiments, we mainly conduct experiments on the validation set for both system-level comparison and ablation study.
In our experiments, two different data augmentation strategies are adopted: normal augmentation and strong augmentation. We summarize the details Table Compared with normal augmentation, the strong augmentation includes greater spatial/geometric transformation.
Implementation Details
5

---

Normal Augmentation
Strong Augmentation short edge E (0.5, 1.5)
Scale jitter
short edge E (0.5. 1.5)
Solarize jitter
p=0.25, ratio E (0, 1)
p=0.25, ratio E (0, 1)
Brightness jitter
p=0.25, ratio E (0,
p=0.25, ratio E (0,
Constrast jitter
p=0.25, ratio E (0,1)
p=0.25, ratio E (0,1)
Sharpness jitter
p=0.25, ratio E (0,1)
p=0.25, ratio E (0,
Translation
p=0.3, translation ratio E (0, 0.1)
Rotate
p=0.3, angle E (0, 30%
Shift
p=0.3, angle E (0, 30
Cutout
E (1, 5), ratio (0.05, 0.2)
Table 1: Summary of the transformations we used in normal augmentation and strong augmentation. "_" indicates that the augmentation is not used.
For updating the EMA model that predicts pseudo boxes, the momentum coefficient is set to 0.999. To examine the effectiveness of MixTraining, we conduct experiments on various object detectors benefit from longer training
and backbone architectures. In practice, find that MixTraining schedule, while the performance of other models may degrade due to over-fitting (we discussed multiple training schedules for each model and report
in Sec. 4.2). For fair comparison, its best performance in our experiments. For models using ResNet-50 backbone. we adopt the SGD (stochastic gradient descent) as the default optimizer, and for models using Swin-Small as the backbone, we adopt the AdamW [1] as the optimizer. Besides, all the backbone weights are pre-trained on ImageNet- IK dataset 7 All the models run on 32 x Nvidia V100. For other training settings and saper-pprraeters for each detector, we following the default settings if not otherwise
specified.
models
backbone
method
mAP
mAP@0.5
mAP@0.75
SiTvaining
41.7
62.8
45.6
Faster R-CNN
ResNet-50
MixTraining
44.0
64.9
47.9
+2.3
+2.1
+2.3
Sifraining
48.7
70.5
53.6
Faster R-CNN
Swin-S
MixTraining
50.3
71.6
55.2
A
+1.6
+1.1
+1.6
SiTraining
50.9
70.3
55.6
Cascade R-CNN
Swin-S
MixTraining
52.8
72.1
57.9
A
+1.9
+1.8
+2.3
Table 2:
SiTraining MixTraining the COC02017 validation set. Across various object detectors and backbones, MixTraining consistently outperforms SiTraining by a large margin.
Method
180K
360K
540K
720K
SiTraining (Normal)
41.7
41.7
40.2
36.1
SiTraining (Strong)
40.1
40.7
40.5
38.9
MixTraining
42.4
44.0
Table 3: Performance of SiTraining and MixTraining different training iterations.
4.2 Comparison to SiTraining
MixTraining Benefits from Longer Training We found that MixTraining can benefit from a longer training schedule due to the use the mixed data augmentation. As shown in Figure 5 and Table 3 by extending the training iterations from 360k to 720k, MixTraining can improve Faster R-CNN with ResNet-50 backbone from 42.4 mAP to 44.0 mAP. On the contrary, extending the training schedule of SiTraining with normal data augmentation results in performance degradation because of over-fitting Although using strong augmentation in SiTraining can alleviate the over-fitiig issue, its best performance is worse than that for normal data augmentation because of difficulty in training.

---

40
40
35
MAP
MAP
30
25
SiTraining Normal
OoTraal
25
SiTraining- Normall
SiTraining-Stroon
SiTrrinning-Srrom Strrog
20
MixTraining
20
MixTraining
0
100
200
300
0 200
400
600
Iteration/K
Iteration/K
Figure 5: We illustrate the validation accuracy (mAP) of different models during training. All models shown in the left figure are trained over 360K total iterations (short training schedule), and models shown in the right figure are trained over 720K total iterations (long training schedule). The model trained by SiTraining (normal augmentation) is heavily over-fitted in the long training schedule. Although the use of strong augmentation can alleviate the over-fitting issue, its performance worse than the model trained by SiTraining (normal augmentation) in the short training schedule. In comparison, our MixTraining achieves better performance while avoiding over-fitting.
To fairly compare different models, all the following experiments, we report the best performance of different models by training the models under various training schedules.
Comparison Different Detectors We also compare MixTraining to SiTraining using different detectors. The results are shown in Table 2 When using Faster R-CNN to evaluate the two methods, MixTraining outperforms SiTraining by 2.3 points with the ResNet-50 backbone and by 1.6 points with the Swin-S backbone, which demonstrates that our method is compatible with different backbone architectures. We further conduct experiments on Cascade R-CNN to validate our method on stronger detector, and consistent improvements are achieved, with MixTraining outperforming SiTraining by 1.9 points with the Swin-S backbone.
Mixed Data Augmentation
Mixed Training Targets
mAP
mAP@0.5
mAP@0.75
41.7
62.8
45,6
42.5
63.0
46.6
44.0
64.9
47.9
Table 4: Ablation of different components. Compared with the model trained by SiTraining with normal augmentation, using mixed data augmentation improves performance by 0.8 points. Adding mixed training targets leads to further improvement of 1.5 points.
4.3
Ablation Study
In this section, we validate our design choices on Faster R-CNN with a ResNet-50 backbone.
Effects of Different Components We first study the effects of different components of MixTruining. The results are shown in Table 4 Compared to the model trained by SiTraining with normally augmented images (41.7 mAP), adopting mixed data augmentation improves mAP by 0.8 points. Further integrating the mixed training targets improves the model by 1.5 points.
Strategies for Mixing Targets We present three different strategies for combining the pseudo boxes and uuman-annotaal ground-truth boxes, and the results each are shown in Table Compared with the baseline model that uses only human-annotatee annotated ground-truite boxes during training, the strategy designed for dealing with the missing label issue brings notable improvements. In comparison, using the strategy that only deals with box localization noise only show a certain advantage. Furthermore, applying both strategies together (i.e. hybrid approach) leads to results that surpass either of the strategies alone.
7

---

strategy
MAP
m.P@0.5
mAP@0.75
baseline
42.5
63.0
46.6
box loc noise
42.9
64.3
47.0
missing label
43.7
64.7
48.0
hybrid 44.0
64.9
47.9
Table 5: Different strategies for combining pseudo boxes and human-annotated ground truth. "base- line" indicates the model trained with mixed data augmentation but without pseudo boxes. "box loc noisy" includes pseudo boxes that address localization noise. "missing label" includes pseudo boxes that deal with missing labels. "hybrid" includes both types of pseudo boxes.
Number of Total Pseudo Boxes/K
50
160
Number Missing Targets/K
45
140
40
120
35
100
80
25
200
400
600
400
600
Iteration/K
Iteration/K
Figure Pseudo box plots. (left) the number of generated pseudo boxes at different training iterations; (right) the number of pseudo boxes retained as missing labels different training iterations.
In addition, examine how the number of predicted pseudo boxes and the number of pseudo boxes retained as missing labels changes the training process. Figure shows the results. training progresses, both of these quantities increase, indicating that the quality of pseudo boxes improves during training. In Figure 8 some qualitative results of the generated pscudo boxes are displayed.
Data Aug
Weighted Loss
mAP
mAP@0.5
mAP@0.75
41.7
62.8
45.6
42.1
63.2
46.3
42.5
63.0
46.6
Table 6: Effects of the data augmentation itself and the weighted loss for strongly augmented images.
Mixed Data Augmentation The mixed data augmentation consists of two parts: the mixed data augmentation strategy, and the weighted training loss that includes strong augmentations only on easy training targets. We study the impact of these two parts, and the results are shown in Table Compared with the baseline model learned through SiTraining with normal data augmentation, simply applying the mixed data augmentation strategy improves the performance by 0.4 points, and further adopting the weighted loss to alleviate the training difficulty issue for strongly augmented images further improves the performance by 0.4 points.
Figure 7 (Ieft) illustrates how the proportion of easy training targets changes during training. In the early stage, only small number of training targets identified easy targets, strongly augmented images have little impact on training. As the training progresses, as increasing number of training targets are well fitted and identified easy targets. At the end of training, more than 50% the training targets are identified as easy samples and used for strong augmentation. Figure 7 (middle) and (right) illustrate the proportion of objects of different sizes in easy targets and non-easy targets when the model completed the training. In easy target, the large objects is much more than small objects, while the conclusion opposite in non-easy target.
Since our mixed data augmentation contains both normal augmentation and strong augmentation, we also compared to applying just type of augmentation, and the results shown in Table We

---

Easy Targets
Non-easy Targets
Small Medium
Ratio Toining Targeto
0.55
Large
0.50
0.45
17%
15%
0.40
48%
49%
0.30
34%
0.25
0.20
200
400
600
Iteration/K
Figure 7: (left) Plot of the proportion of easy targets different training iterations. (middle) The chart shows the proportion of object of different sizes in easy target. (right) The chart shows the proportion of objects of different sizes in non-easy target.
Se 92%
Cog 36
shar 98%
DSAL
table
(b)
(c)
(d)
Figure 8: Qualitative results of pseudo boxes. (a) and (b) show the detected pseudo boxes at an early training stage (3k iterations); (c) and (d) show the pseudo boxes generated at the end of training.
Data Aug
mAP
mAP@0.75 mAP@0.5
Normal
41.7
62.8
45.6
Strong
40.7
62.2
44.4
Mixed
42.5
63.0
46.6
Table 7: Mixed data augmentation strategy vs. single augmentation strategy.
found that simply adopting strong augmentation in SiTraining does not improve performance. On the contrary, it hurts the training process, resulting in worse performance than using normal augmentation. Our mixed data augmentation strategy can alleviate the training difficulty of strong augmentation and improve the performance by 0.8 points.
Conclusion
In this work, we introduce new training paradigm, MixTraining, for object detection that can improve the performance of existing detectors for free. Our method consists an enhanced data augmentation that combines different strengths of augmentation and excludes the strong augmentations certain training targets to reduce training difficulty. In addition, a pseudo box mechanism is introduced to address label noise human annotation. The experiments conducted on the COC02017 benchmark verify the effectiveness of Mix Training across various detectors and backbones.

---

References
[1] Cai, Z. and Vasconcelos, N. (2018). Cascade r-cnn: Delving into high quality object detection. In CVPR.
[2] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020). End-to-end object detection with transformers. In ECCV.
[3] Chen, G., Choi, W. Yu, X., Han, T., and Chandraker, M. (2017). Learning efficient object detection models with knowledge distillation. In NeurIPS.
[4] Chen, Y., Li, Y., Kong, T., Qi, L.,, Chu, R., Li, L.., and Jia, J. (2021). Scale-aware automatic augmentation for object detection. arXiv preprint arXiv:2 17220.
[5] Chen, Y., Zhang, Z., Cao, Y., Wang, L., Lin, S., and Hu, H. (2020). Reppoints v2: Verification meets regression for object detection. NeurIPS.
[6] Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. (2020). Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshop.
[7] Deng, J., Dong, W., Socher, R., Li, L.-J, Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In CVPR.
[8] DeVries, T. and Taylor, G. W. (2017). Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04: 1008:0488.
[9] Dong, X., Zheng. L., Ma, F., Yang. Y., and Meng. D. (2018). Few-example object detection with model communication. TPAMI.
[10] Fang, H.-S., Sun, J., Wang, R., Gou, M., Li, Y.-L.., and Lu, C. (2019). Instaboost: Boosting instance segmentation via probability map guided copy-pasting. In ICCV.
[11] Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.-Y, Cubuk, E. D., Le, Q. V., and Zoph. B. (2020). Simple copy copy-paste is a strong data augmentation method for instance segmentation. arXiv preprint arXiv:2012.07117
[12] Ghiasi, G., Lin, T.-Y., and Le, Q. V. (2019). Nas-fpn: Learning scalable feature pyramid architecture for object detection. In CVPR.
[13] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In CVPR.
[14] Hu, H., Gu, J., Zhang. Z., Dai, J., and Wei, Y. (2018). Relation networks for object detection. CVPR.
[15) Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv: 112660.4...6980.
[16] Law, H. and Deng, J. (2018). Cornernet: Detecting objects as paired keypoints. In ECCV.
[17] Li, Cheng, B., Feris, R., Xiong, J., Huang, S., Hwu, W.-M., and Shi, H. (2021). Pseudo-iou: Improving label assignment in anchor-free object detection. arXiv preprint arXiv:2104. 140.22
[18] Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017a). Feature pyramid networks for object detection. CVPR.
[19] Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017b). Focal loss for dense object detection. In ICCV.
[20] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In ECCV.
[21] Liu, W. Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y,, and Berg, A. C. (2016). Ssd: Single shot multibox detector. In ECCV.
10

---

[22] Liu, Z., Lin, Y., Cao. Y., Hu. H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). Swin trans- former: Hierarchical vision transformer using shifted windows. arXiv preprint arX1v:2103.14030.
[23] Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016). You only look once: Unified, real-time object detection. In CVPR.
[24] Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS.
[25] Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S. (2019). Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR.
[26] Rolnick, D., Veit, A., Belongie, S., and Shavit, N. (2017). Deep learning is robust to massive label noise. arXiv preprint arXiv:1705,
[27] Sohn, K., Zhang, Z., Li, C.-L., Zhang, H., Lee, C.-Y., and Pfister, T. (2021). A simple semi-supervised learning framework for object detection. ICLR.
[28] Tan, M., Pang, R., and Le, Q V. (2020). Efficientdet: Scalable and efficient object detection. In CVPR.
[29] Tian, 7., Shen, C., Chen, H., and He, T. (2019). Fcos: Fully convolutional one-stage object detection. In ICCV.
[30] Wang. J.. Chen, K., Yang, S., Loy, C. C., and Lin, D. (2019a). Region proposal by guided anchoring. In CVPR.
[31] Wang, T., Yang, T., Cao, J., and Zhang, X. (2020). Co-mining: Self-supervised learning for sparsely annotated object detection. arXiv preprint arXiv:2012.01950.
[32] Wang. T., Yuan, L., Zhang, X., and Feng, (2019b). Distilling object detectors with fine-grained feature imitation. In CVPR.
[33] Xie, Q., Luong, M.-T., Hovy, F.,, and Le, Q. V. (2020). Self-training with noisy student improves imagenet classification. In CVPR.
[34] Yang, Z., Liu, S., Hu, H., Wang, L., and Lin, (2019). Reppoints: Point set representation for object detection. In ICCV.
[35] Zhang, H., Chang, H., Ma. B., Wang, N., and Chen, X. (2020a). Dynamic r-cnn: Towards high quality object detection via dynamic training. In ECCV. Springer.
[36] Zhang. H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09411
[37] Zhang, S., Chi, C., Yao, Y., Lei, 7., and Li, S. 7. (2020b). Bridging the gap between anchor- based and anchor-free detection via adaptive training sample selection. In CVPR.
[38] Zheng. Z. Ye, R., Wang, P., Wang. J., Ren, D., and Zuo, W. (2021). Localization distillation for object detection. arXiv preprint 12X22:2102.12252.
[39] Zhu, B., Wang. J., Jiang, Z., Zong, F., Liu, S., Li, Z., and Sun, (2020). Autoassign: Differentiable label assignment for dense object detection. arXiv preprini arXiv:2007.03496.
[40] Zhu, X., Hu, H., Lin, S., and Dai, J. (2019). Deformable convnets v2: More deformable, better results. In CVPR.
[41] Zoph, B., Cubuk, E. D., Ghiasi, G., Lin, T.-Y,, Shlens, J., and Le, Q. V. (2020). Learning data augmentation strategies for object detection. In ECCV,


Reviews:
[{"review_id": "mb1pipc-yXD", "reviewer": "Reviewer_WPyC", "summary": "The study offers MixTraining, an object detection training technique that may be used to improve the performance of existing detectors. The suggested method improves data augmentation by leveraging augmentations of various strengths while eliminating strong augmentations of particular training samples, which could be harmful to training. It addresses localization noise and missing labels in human annotations as well. The efficacy of MixTraining is validated on the COCO dataset using different deep learning models and backbones.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This study utilizes augmentations of varying strengths while avoiding strong augmentations of certain training samples commendably. Psedo-labeling has been around for a long time. Overall, the authors enhanced a few existing techniques in an effective manner.\n\nThe authors demonstrate how to effectively use an augmentation technique and the value of human-annotated bounding boxes to bootstrap an object detector by extending its capabilities. They show that a model/backbone trained using the suggested mechanism outperforms a model/backbone trained using a typical setting.\n\nThe manuscript is well-organized, well-written, and easy to follow. The motivation of the study is clear and well-defined. The authors evidently explain how to apply an augmentation strategy to improve an object detector's performance.\n\nThe proposed mechanism is useful. The study does a great job of improving the performance of a detector by employing a previously less explored mechanism. However, in my opinion, the contribution of the paper is somewhat incremental which effectively uses the MixTraining technique to improve the capabilities of existing detectors.\n\nTo sum up, the authors demonstrate how an augmentation method and the efficacy of human-annotated bounding boxes can be extended beyond a typical training paradigm for object detection. I think this is an interesting approach that could prove beneficial to the research community. The proposed method can also be utilized effectively in real-world applications to enhance the performance of an object detector. Overall, I think this is a good study and am inclined to accept the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not think there is any potential negative societal impact of this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "lHN38s7MDyW", "reviewer": "Reviewer_eNWD", "summary": "Object detection is a fundamental task in computer vision. In this paper a new paradigm of training, MixTraining, is proposed which utilizes augmentation of different strengths. Either standard augmentation and standard processing is used for an image or strong augmentation is applied (mixed data augmentation) and changed ground truth is used (loss is computed only on simple boxes and mixed training targets is used). As soon as human-annotated bounding boxes often are noisy (missing or inaccurate localizations) the other key ingredient when strong augmentation is applied to an image is to use only confident bounding boxes predicted by the model itself (fixes errors in human annotations) in addition to the confident bounding boxes from the ground truth (making simpler task as soon as strong augmentation is used). This key ingredient is obtained via pseudo-labeling by EMA detector (pseudo boxes). The idea is that this pseudo boxes and bootstrapping from the model itself help to detect images where strong augmentation can be expected to be helpful. MixTraining is shown to consistently improve various detectors on the COCO dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nAuthors propose a novel paradigm of training for object detection when both normal and strong augmentations are involved and pseudo boxes are used to define how particular box is hard or not to use as a target in strongly augmented sample.  Augmentation, EMA and pseudo labeling are very well known, however, the latest  is more commonly used in semi/self-supervised setting and not for the supervised one to guide how hard or simple to detect particular bounding box. Known techniques are used from a new point of view.\n\n**Quality**\n\nThere is no any theoretical analysis and work is purely experimental. However, all experiments are very well defined and looks reasonable and technically strong:\n- different backbones and detectors are considered\n- careful training schedules are used as long as different models and paradigms have different convergence speed (of course due to stronger augmentation used in MixTraining for example).\n- extensive ablations are performed to understand the mixed targets and mixed augmentations dynamics\n- extra analysis on pseudo boxes is conducted\n\nHowever some extra information on choice of crucial MixTraining hyper-parameters is not provided/discussed:\n- When is strong augmentation branch involved into training: right at the beginning or after EMA model reaches some good performance? Did authors observe any problem on bootstrapping with pseudo boxes?\n- Are pseudo boxes regenerated every update? Does model generate pseudo boxes and strong targets once per N updates for all data?\n- What is the dynamics of the parameters from above two bullets?\n- Is probability to apply normal and strong  augmentations 0.5? Or do for the update step we apply both normal and strong augmentation for an image and compute Loss_normal + Loss_strong? Do we apply only strong/normal augmentation to the whole batch or mix of normal and strong ones?\n\nOther comments:\n- Is prediction score threshold tuned somehow, why is its value chosen to be 0.9?\n- Would be great to have experiment also with on-stage detector to have broader experimental proof that method works.\n- How were the augmentations and their parameters found for normal and strong augmentation strategies?\n- In fig. 5 seems the learning rate decay happens at some point when there is a gap in model performances. Is it true? If yes then the best learning strategy is perform decay when the validation mAP is at maximum. Did authors try this schedule to avoid overfitting in SiTraining earlier to decay learning rate?\n\n**Clarity**\n\nThe paper is very well written and organized (no any typos found during reading). Explanations with proper references on original works are well described too. Comparison with existed methods is fully covered in Section 2. All details on the reproduction steps are provided and the new training paradigm is clear described in the text and by Fig. 1.\n\n**Significance** \n\nMixTraining is a novel general framework, which demonstrates ability on bootstrapping and self-improving even with noisy labels like missed or inaccurate localizations. MixTraining does not introduce extra parameters/computations at inference time, while improve leading detectors significantly via the training strategy itself (Faster R-CNN 41.7 -> 44 mAP; Cascade R-CNN 50.9 -> 52.8 mAP). This also opens potential research for semi-supervised learning as soon as bootstrapping via pseudo-labeling by EMA detector is demonstrated to work.\n\n**Update on decision:**\nChanging final decision from 6 to 7 as authors addressed all concerns in the below comments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Didn't discussed in the paper. It is referenced to Appendix in the Checklist, however, Appendix was not attached to the submission. Either include in the Appendix and post it or add in the main text.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "K3qLsFsI6bj", "reviewer": "Reviewer_iVVT", "summary": "This paper proposes a method to improve the performance of object detectors with supervised training. The main ideas consist of two parts: (1) The human annotation of bounding boxes is often noisy or incomplete. Adopting a teacher model to generate pseudo boxes and combining them with original annotation can improve the performance of object detectors. (2) Strong augmentation may generate data which degrades the training performance. Adopting a teacher model to select easy targets in the augmented images for training can improve the performance.\nIn this paper, the author adopts an exponential moving average model as the teacher model. The proposed MixTraining strategy improves the performance (1.6~2.3 mAP) of several object detectors in COCO benchmark.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper is well written and easy to follow. The idea of adopting teacher model to handle the out of distribution data generated by strong augmentation is simple and effective.\nThe ablation shows the effectiveness of adopting pseudo boxes in the supervised object detection training. \n\nWeakness:\nThe proposed method could only address the missing label problem or inaccurate bounding box localization. As for false positive annotation, the proposed method can not handle.\nMissing references for data augmentation works which adopt different augmentation policies for different data samples during training.\nAs claimed in this paper, the author considers strong data augmentation may degrade the performance of object detectors and provide experiments in Table 6. However, as shown in [1] and [2], the searched data augmentation can improve the performance significantly. Adopting the proposed method on [1] or [2] will make the experiments stronger.\n\n[1] Learning Data Augmentation Strategies for Object Detection.\n[2] Scale-aware Automatic Augmentation for Object Detection.\n\nComments:\nCould the authors adopt the proposed method for one-stage object detectors? Why do the authors only provide results of two-stage detectors?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "C_JcIM2pnO1", "reviewer": "Reviewer_7Km7", "summary": "This paper introduces a new training paradigm named MixTraining for object detection that can improve the performance of existing detectors for free. The main idea is that MixTraining enhances data augmentation by utilizing augmentations of different strengths. MixTraining is found to bring consistent improvements across various detectors on the COCO dataset. It improves faster R-CNN from 41.7 to 44.0 and Swin-small from 50.9 to 52.8. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The motivation to perform instance-wise augmentation for different instance is clear and make sense;\n2) The method is straightforward and easy to follow;\n3) The effectiveness of the method is validated on a strong baseline, which is a 50 + mAp detector on COCO dataset.\n4) No further computation cost is found in the proposed method, MixTraining can bring improvement for free.\n5) Good experimental results and ablation studies.\n\nWeaknesses:\n1) The network has to recognize each instance with different augmentations, which may slow down the training time.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new training procedure for object detection that relies on a teacher model and data augmentation. The approach adds a loss that is computed on a strongly augmented image but only on boxes that are above a threshold for the teacher (exponential moving average) model on the raw image. It is quite simple and generic, and provides a significant boost in final performance. The experiments are conducted on COCO with strong standard models (Faster R-CNN and Cascade-R-CNN) models. The reviewers have made constructive observations and all agree that this is a good paper. Overall, the paper is a significant contribution and is a welcome addition to the program of NeurIPS.
Decision: Accept (Spotlight)