Unbiased Teacher for Semi-Supervised Object Detection
Authors: Yen-Cheng Liu
Chih-Yao Ma
Zijian He
Chia-Wen Kuo
Kan Chen
Peizhao Zhang
Bichen Wu
Zsolt Kira
Peter Vajda
Venue: ICLR 2021 Poster (2021)
Keywords: Object Detection

Abstract: Semi-supervised learning, i.e., training networks with both labeled and unlabeled data, has made significant progress recently. However, existing works have primarily focused on image classification tasks and neglected object detection which requires more annotation effort. In this work, we revisit the Semi-Supervised Object Detection (SS-OD) and identify the pseudo-labeling bias issue in SS-OD. To address this, we introduce Unbiased Teacher, a simple yet effective approach that jointly trains a student and a gradually progressing teacher in a mutually-beneficial manner. Together with a class-balance loss to downweight overly confident pseudo-labels, Unbiased Teacher consistently improved state-of-the-art methods by significant margins on COCO-standard, COCO-additional, and VOC datasets. Specifically, Unbiased Teacher achieves 6.8 absolute mAP improvements against state-of-the-art method when using 1% of labeled data on MS-COCO, achieves around 10 mAP improvements against the supervised baseline when using only 0.5, 1, 2% of labeled data on MS-COCO.

Published as a conference paper at ICLR 2021
UNBIASED TEACHER FOR SEMI-SUPERVISED OBJECT DETECTION
Yen-Cheng L Chih- Yao Ma², Zijian He?, Chia-Wen Kuo', Kan Chen', Peizhao Zhang?, Bichen Wu², Zsolt Kira', Peter Vajda"
"Georgia Tech, Facebook Inc.
{ycliu, cwkuo, zkira (airraa@gate @gatech. edu,, (cyma, zi zijian, kanchenl 8, st stzpz, wbc, vajjaa)@fftccco com
ABSTRACT
Semi-supervised learning, i..., training networks with both labeled and unlabeled data, has made significant progress recently. However, existing works have pri- marily focused image classification tasks and neglected object detection which requires more annotation effort. In this work, we revisit the Semi-Supervised Object Detection (SS-OD) and identify the pscudo-labeling bias issue in SS- OD. To address this, we introduce Unbiased Teacher', simple yet effective approach that jointly trains a student and a gradually progressing teacher in a mutually-beneficial manner. Together with a blass-balance loss to downweight overly confident pseudo-labels, Unbiased Teacher consistently improved state-of- the-art methods by significant margins on COCO-standard, COCO-additonal,, and VOC datasets. Specifically, Unbiased Teacher achieves 6.8 absolute mAp improvements against state-of-the of-the-art method when using 1% of labeled data MS COCO, achieves around 10 mAP improvements against the supervised base- line when using only 0.5, 1, 2% of labeled data on MS-COCO.
INTRODUCTION
The availability of large-scale datasets and computational resources has allowed deep neural net- works to achieve strong performance on wide variety of tasks. However, training these networks requires large number of labeled examples that are expensive to annotate and acquire. As an al- ternative, Semi-Supervised Learning (SSL) methods have received growing attention (Sohn et al., 2020a; Berthelot et al., 2020; 2019; Laine & Aila, 2017: Tarvainen Valpola, 2017; Sajjadi et al., 2016; Lee, 2013; Grandvalet & Bengio, 2005). Yet, these advances have primarily focused on image classification, rather than object detection where bounding box annotations require more effort.
In this work, we revisit object detection under the SSL setting (Figure I): an object detector is trained with a single dataset where only a small amount of labeled bounding boxes and a large amount of unlabeled data provided, or object detector is jointly trained with large labeled dataset as well as a large external unlabeled dataset. A straightforward way to address Semi-Supervised Object Detection (SS-OD)
adapt from existing advanced semi-supervised image classification meth- ods (Sohn et al., 2020a). Unfortunately, object detection has some unique characteristics that interact poorly with such methods. For example, the nature of class-imbalance in object detection tasks impedes the usage of pseudo-labeling. In object detection, there exists bregrounddbackkroown imbalance and foreground classes imbalance (see Section 3.3). These imbalances make models trained in SSL. settings prone to generate biased predictions. Pseudo-laheling methods, one of the most successful SSL methods in image classification (Lee, 2013; Sohn et al., 2020a), may thus be biased towards dominant and overly confident classes (background) while ignoring minor and less confident classes (foreground). As a result, adding biased pseudo-labels into the semi-supervised training aggravates the ure 2, taking a two-stage object detector as an example, there exists heavy overfitting on the fore-
issue and introduces severe overfitting. As shown in Fig-
Wook done partially while interning at Facebook. Code: https: / //ggtgttitt thub. ccm httm://fccoookkeeeerrcccooo
ased teacher.

---

Published as a conference paper at ICLR 2021
Semi-Supervised Object Detection
35-
Spaslard only
Labeled Train Set
Unlabeled Train Set
30-
C5D Unblaee Teacher
25-
20-
10-
Objject
10
0.5
Suparvision (S)
(a)
(b)
Figure 1: (a) Illustration of remi-supervised object detection, where the model observes set of labeled data and a set of unlabeled data in the training stage. (b) Our proposed model can efficiently leverage the unlabeled data and perform favorably against the existing semi-supervised object de- tection works, including CSD (Jeong et al., 2019) and STAC (Sohn et al., 2020b).
ground/background classification in the RPN and multi-class classification in the ROlhead (but on bounding box regression).
To overcome these issues, we propose general framework Unbiased Teacher: an approach that jointly trains Student and a slowly progressing Teacher in -bbeefiicl manner, in which the Teacher generates pseudo-labels to train the Student, and the Student gradually updates the Teacher via Exponential Moving Average (EMA)", while the Teacher and Student are given different augmented input images (see Figure 3). Inside this framework, (i) we utilize the pseudo-labels as explicit supervision for both RPN and ROlhead and thus alleviate the overfitting issues both RPN and ROIhead. (ii) We also prevent detrimental effects due to noisy pseudo-labels by exploiting the Teacher-Student dual models (see further discussion and analysis in Section 4.2). (ii) With the use of EMA training and the Focal loss (Lin et al., 2017b), 7b), we address the pseudo-labeling bias problem caused by class-imbalance and thus improve the quality of pseudo-labels. As the result, our object detector achieves significant performance improvements.
We benchmark Unbiased Teacher with SSL setting using the MS-COCO and PASCAL VOC datasets, namely COCO-standard, COCO-addiional, and VOC. When using only 1% labeled data from MS-COCO (COCO-standari), Unbiased Teacher achieves 6.8 absolute mAp improvement against the state-of-the-ar method, STAC (Sohn et al., 2020b). Unbiased Teacher consistently achieves around 10 absolute mAP improvements when using only 0.5, 2, 5% of labeled data com- pared supervised baseline.
We highlight the contributions of this paper as follows:
By analyzing object detectors trained with suprrvd-uupprrisionnn we identify that the nature of class- imsalance in object detection tasks impedes the effectiveness of pseudo-labeling method on SS-OD task.
We thus proposed simple yet effective method, Unbiased Teacher, to address the pseudo- labeling bias issue caused by class-imbalance existing in ground-truth labels and the over- fitting issue caused by the scareity of labeled data.
Our Unbiased Teacher achieves state-of-the-arr performance on SS-OD across COCO- standard, COCO addo-iddiion,,l, and VOC datasets. We also provide an ablation study to verify the effectiveness of each proposed component.
RELATED WORKS
Semi-Supervised Learning. The majority of the recent SSL methods typically consist of (1) input augmentations and perturbations, and (2) consistency regularization. They regularize the model to be invariant and robust to certain augmentations on the input, which requires the outputs given the original and augmented inputs to be consistent. For example, existing approaches apply convention data augmentations (Berthelot et al., 2019; Laine & Aila, 2017; Sajjadi al., 2016; Tarvainen &
2 Note that there have been many works that leverages EMA, egg., ADAM optimization (Kingma & Ba, 2015), Batch Normalization (loffe & Szegedy, 2015), self supf-supee learning (He et al., 2020; Grill et al., 2020), and SSL image classification (Tarvainen & Valpola, 2017). We, for the first time, show its effectiveness in combating class imbalance issues and detrimental effect of pseudo-lahels for the object detection task.
2

---

Published as a conference paper at ICLR 2021
RPN Classification Loss
RPN Regression Loss
ROlhead Val Classification
ROlhead Val Regression
1.0
0.11
0.8
0.35
1605
0.10
0.6
0.5
0.08
0.4
0.30
0.07
0.2
0.25
20K
BOX
Training Beritions.
Training iterations
Training Iterations
Training Iterations
1% (Supervised)
1% (Ours) --- 5% (Supervised)
5% (Ours)
100% (Supervised)
Validation Losses of our model and the model trained with labeled data only. When the labeled data is insufficient (1% and 5%), RPN and ROlhead classifiers suffer from overfitting, while RPN and ROlhead regression do not suffer from overfitting. Our model can significantly alleviates the overfitting issue in classifiers and also improves the validation box regression loss.
Figure 2:
Valpola, 2017) to generate different transformations of the semantically identical images, perturb the imput images along the adversarial direction (Miyato et al., 2018; 88; Yu et al., 2019), utilize mul- tiple networks to generate various views of the same input data (Qiao et al., 2018), mix input data to generate augmented training data and labels (Zhang et al., 2018; Yun al., 2019; Guo et al., 2019; Hendrycks et al., 2020), or learn augmented prototypes in feature space instead of the image space (Kuo et al., 2020). However, the complexities in architecture design of object detectors hinder the transfer of existing semi-supervised techniques from image classification to object detection.
Semi-Superviieed Object Detection. Object detection is one of the most important computer vision tasks and has gained enormous attention (Lin et al., 70; He et al., 2017; Redmon & Farhadi, 2017: Liu al., 2016). While existing works have made sigmificant progress over the years, they have primarily focused on training object detectors with fully-labeled datasets. On the other hand, there exist several semi-supervised object detection works that focus on training object detector with combination of labeled, weakly-labeled, the resurgence of deep learning (Rosenberg et al., 2005). Later, along with the success of deep learning, Hoffman et al. (2014) and Gao et al. (2019) trained object detectors data with bounding box labels for some classes and image-level class labels for other classes, enabling detection for categories that lack bounding box annotations. Tang et al. (2016) adapted the image-level classifier of a weakly labeled category (no bounding boxes) into a detector via similarity-based knowledge transfer. Misra et al. (2015) exploited few sparsely labeled objects and bounding boxes in some video frames and localized unknown objects in the following vidcos.
unlabeled data. This line of work began even before
Unlike their settings, we follow the standard SSL setting and adapt it the object detection task, in which the training contains a small set of labeled data and another set of completely unlabeled data (i.e., only images). In this setting, Jeong al. (2019) proposed a consistency- based method, which enforces the predictions of an input image and its flipped version to be consistent. Sohn et al. (2020b) pre-trained a detector using a small amount labeled data and generates pseudo-labels on unlabeled data to fine-tune the pre-trained detector. Their pseudo-labels are generated only once and are fixed through out the rest of training. While they can improve the performance against the model trained on labeled data, imbalance issue is not considered in existing SS-OD works. In contrast, method not only improve the pseudo-label generation model via teacher-student mutual learning regimen (Sec. 3.2) but address the crucial imbalance issue in generated pseudo-labels (Sec. 3.3).
3 UNBIASED TEACHER
Problem definition. Our goal is to address object detection in a semi- semi-supervised setting, where set N, are available for
of labeled images D. {mf yf 1= and a set of unlabeled images D,, = {a:;) )== training. N₂ and Nr, are the number of supervised and unsupervised data. For each labeled image x", the annotations y" contain locations, sizes, and object categories of all bounding boxes.
Overview. As shown in Figure 3, our Unbiased Teacher consists of two training stages, the Burn- In stage and the Stacher-Student Mutual Learning stage. In the Burn-In stage (Sec. 3.1), we simply train the object detector using the available supervised data to initialize the detector. At the beginning of the Peacher-Student Mutual Learning stage (Sec. 3.2), duplicate the initial- ized detector into two models (Teacher and Student models). Our Teacher-Student Mutual Learning stage aims evolving both Teacher and Student models via mutual learning mechanism, where
3

---

Published as a conference paper at ICLR 2021
Burn-In Stage
Teacher-Student Mutual Learning Stage
Student Learning
Teacher Refinement
Teacher
Teacher
Weak Aug.
N
RPN
ROIHead
Pseudo
ROlHead
Labeled
Unlabeled
labeling
RPN
EMA
ROIHead
Student
Student
Strong Aug.
RPN
RPN
+
ROIHead
ROlHead
Overview of Unbiased Teacher. Unbiased Teacher consists of two stages. Burn-In: we first train the object detector using available labeled data. Peacher-Student Mutual Learning con- sists of two steps. Student Learning: the fixed teacher generates pseudo-labels to train the Student, while Teacher and Student are given weakly and strongly augmented inputs, respectively. Teacher Refinement: the knowledge that the Student learned is then transferred to the slowly progressing Teacher via exponential moving average (EMA) network weights. When the detector is trained until converge in the Burn-In stage, we switch to the Teacher- Student Mutual Learning stage.
Figure 3:
the Teacher generates pscudo-labels train the Student, and the Student updates the knowledge learned back to the Teacher; hence, the pseudo-labels used to train the Student itself are improved. Lastly, there exists slass-imbalance and foreground-backgroundd imbalance problems in object de- tection, which impedes the effectiveness of semi-supervised techniques of image classification (e.g. pseudo-labeling) being used directly on SS-OD. Therefore, Sec. 3.3, we also discuss how Focal loss (Lin et al., 2017b) and EMA training alleviate the imbalanced pseudo-label issue.
3.1 BURN-IN
It important have good initialization for both Student and Teacher models, as will rely on the Teacher to generate pseudo-labels to train the Student in the later stage. To do so, we first use the available supervised data to optimize our model 9 with the supervised loss Csup- With the supervised data D, = {x; y 1 the supervised loss of object detection consists of four losses: the RPN classification loss Lats the RPN regression loss Crey. the ROI classification loss Drss. and the ROI regression loss Lreg (Ren et al., 2015),
Soup EE (a(, + Freg (2t y!) ( + Lrei (zef, y.)-
After Burn-In, we duplicate the trained weights for both the Teacher and the Student models (0, +- 0,0, <- 0). Starting from this trained detector, we further utilize the unsupervised data to improve the object detector via the following proposed training regimen.
3.2 TEACHER-STUDEN MUTUAL LEARNING
Overview. To leverage the unsupervised data, we introduce the Teacher-Student Mutual Learning regimen, where the Student is optimized by using the pseudo-labels generated from the Teacher, and the Teacher is updated by gradually transferring the weights of continually learned Student model. With the interaction between the Teacher and the Student, both models can evolve jointly and continuously to improve detection accuracy. With the improvement on detection accuracy, this also means that the Teacher generates more accurate and stable pseudo-labels, which we identify as one of the keys for large performance improvement compared to existing work (Sohn et al., 2020b), In another perspective, we can also regard the Teacher as the temporal ensemble of the Student models in different time steps. This aligns observation that the accuracy of the Teacher is consistently higher than the Student. As noted in prior works (Tarvainen Valpola, 2017; Xie et al., 2020), one crucial factor in improving the Teacher model is the diversity of Student models; we thus use the
4

---

Published as a conference paper at ICLR 2021
strongly augmented images as input of the Student, but we use the weakly augmented images as input of the Teacher to provide reliable pseudo-labels.
Student Learning with Pseudo-Labeling. To address the lack of ground-truth labels for unsuper- vised data, we adapt the pscudo-labeling method to generate labels for training the Student with unsupervised data. This follows the principle of existing successful examples in semi-supervised image classification task (Lee, 2013; Sohn et al., 2020a). Similar to classification- prevent the consecutively detrimental effect of noisy pseudo-labels (i.e., confirmation bias or error accumulation), we first set a confidence threshold ó of predicted bounding boxes to filter low- confidence predicted bounding boxes, which are more likely to be false positive samples.
methods,
While the confidence threshold method have achieved tremendous success in the image classifica- tion, it is however not sufficient for object detection. This is because there also exist duplicated box predictions and imbalanced prediction issues in the SS-OD (we leave the discussion of the imbal- anced prediction issue in Sec. 3.3). To address the duplicated boxes prediction issue, we remove the repetitive predictions by applying class- wise non-maximum suppression (NMS) before the use of confidence thresholding as performed in STAC (Sohn et al., 2020b).
In addition, noisy pseudo-labels can affect the pseudo-label generation model (Teacher). As a result, we detach the Student and the Teacher. To be more specific, after obtaining the pseudo-labels from the Teacher, only the learnable weights of the Student model is updated via back-propagation.
0, - 02 + (Lrup >unCunuup)
Lunsup" = =E (x) (.) (x.
09,
Note that we do not apply unsupervised losses for the bounding box regression since the naive con- fidence thresholding box regression (because the confidence of predicted bounding boxes only indicate the confidence of predicted object categories instead of the quality of bounding box locations (Jiang et al., 2018)).
able filter the pseudo-labels that are potentially incorrect for bounding
Teacher Refinement via Exponential Moving Average. To obtain more stable pscudo-labels, we apply EMA to gradually update the Teacher model. The slowly progressing Teacher model can be regarded as the ensemble of the Student models in different training iterations.
+ a0,- + (1 10--ap a))
This approach has been shown be effective in many existing works, e.g., ADAM optimiza- tion (Kingma & Ba, 2015), Batch Normalization (loffe & Szegedy, 2015), self-supervised learn- ing (He al., 2020; Grill al., 2020), and SSL image classification (Tarvainen & Valpola, 2017), while we, for the first time, demonstrate its effectiveness also in alleviating pseudo-labeling bias issue for SS-OD (see next section).
3.3 BIAS IN PSEUDO-LABEL
Ideally, the methods based pseudo-labels can address problems caused by the scarcity of labels, yet the inherent nature of imbalance in object detection tasks/datasets impedes the effectiveness of pseudo-laheling methods. As mentioned in (Oksuz al., 2020), in object detection, there imbalance (e.g., background instances accounts for 70% of all training instances) and foreground classes imbalance (e.g., human instances accounts for 30% of all fore- ground training instances in MS-COCO (Lin et al., 2014)). If standard cross-entropy is applied in the condition of insufficient training data, the model is likely prone to predict the dominant classes. This makes the prediction bias toward prevailing classes and leads to the imbalance issue in generated pseudo-labele Relying on the biased pseudo-labels during training makes the imbalanced prediction issue even more severe. To address the imbalance issue in object detection, existing works have proposed several methods (Shrivastava et al., 2016; Lin al., 2017b; Li al., 2020).
ists
In this work, we consider simple yet effective method; we replace the standard cross-entropy with the multi-class Focal loss (Lin et al., 2017b) for the multi-class classification of ROlhead classifier (i.e., Crz.). Focal loss is designed to put more loss weights on the samples with lower-confidence instances. As a result, it makes the model focus on hard samples, instead of the easier examples that likely from dominant classes. Although the Focal loss is not widely used for vanilla supervised object detection settings (the accuracy of YOLOv3 (Redmon & Farhadi, 2018) even drops if the focal loss is applied), argue that it is crucial for SS-OD due to the issue of biased pseudo-labels.
5

---

Published as a conference paper at ICLR 2021
Table 1: Experimental results on COCO-standan comparing with CSD (Jeong et al., 2019) and STAC (Sohn et al., 2020b). *; we implement the CSD method and adapt it on the MS-COCO dataset. The results of 0.5% with STAC is from their released code.
COCO-standrd
0.5%
1%
2%
5%
10%
Supervised
6.83 0.15
9.05 0.16
12.70 + 0.15
18.47
0.22
23.86 0.81
CSD*
7.41 0.21 (40
10.51 :: 0.06 (b)
13.93 (.12))
18.63
0.07 (+A
22.46 0.08 (-140)
STAC
9.78 0.53 (2299)
13.97 0.35
18.25 0.25 (6555)
24.38
0.12
28.64 1: 0.21 (-4.7)
Unbiased Teacher
16.94 =: 0.23 (11011))
20.75 = 0.12 (111.7))
24.30 + 0.07 (111.00)
28.27 0.11 (19,00)
31.50 +: 0.10 (-764)
On the other hand, we also observe that the EMA training can also alleviate the imbalanced pseudo- labeling biased issue due to the conservative property of the EMA training. To be more specific, with the EMA mechanism, the new Teacher model regularized by the previous Teacher model, and this prevents the decision boundary from drastically moving toward the minority classes In detail, the weights of the Teacher model can be represented as follows:
1-1
a(Lsup + Ay Cunnsp)))
0
(4)
where 0 is the model weight after the burn-in stage, 0 is the Teacher model weight in i-th iteration, 8h is the Student model weight in k-th iteration, is the learning rate, and a is the EMA coefficient.
The regularization of the previous Teacher model is equivalent putting additional small co- efficient on the gradients on Student models in previous steps. With the slowly altered decision boundary (i.e., higher stability), the pseudo-labels of these unlabeled instances are less likely to change dramatically, and this prevents the decision boundary from moving toward minority classes (i.e,, majority class bias). Thus, the EMA-trained Teacher model beneficial for producing more stable pseudo-labels and addressing the class-imbalance issue in SS-OD.
We note that the class- iaaas-imaalance issue is crucial when using peeudollablligg method to address semi-supervies or other low-label object detection tasks. There indeed exist other imbalance methods that can potentially improve the performance, but we leave this for future research.
EXPERIMENTS
Datasets. We benchmark our proposed method on experimental settings using MS-COCO (Lin et al., 2014) and PASCAL VOC (Everingham et al., 2010) following existing works (Jeong et al., 2019; Sohn al., 2020b). Specifically, there are three experimental settings: (1) COCO-standan we randomly sample 0.5, 1, 5, and 10% of labeled training data labeled set and use the rest of the data as the training unlabeled set. (2) COCO-addiional:: we use the standard labeled training set as the labeled set and the additional COCO2017-umlabe
data the unlabeled set. (3) VOC:
use the VOCO7 trainval set as the labeled training set and the VOC12 trainval set as the unlabeled training set. Model performance is evaluated the VOCO7 test set.
Implementation Details. For a fair comparison, we follow STAC (Sohn et al., 2020b) to use Faster- RCNN with FPN (Lin et al., 201 7a) and ResNet-50 backbone (He et al., 2016) our object detec- tior, where the feature weights are initialized by the works (Jeong et al., 2019; Sohn et al., 2020b). We use confidence threshold 8 0.7. For the data augmentation, we apply random horizontal flip for weak augmentation and randomly add color jit- tering, grayscale, Gaussian blur, and cutout patches for strong augmentations. Note that we do not apply any geometric augmentations, which are used in STAC. We use AP30:95 (denoted as mAP) as evaluation metric, and the performance is evaluated on the Teacher model. More training and implementation details can be found in the Appendix.
model, same as existing
4.1 RESULTS
COCO-standard. We first evaluate the efficacy of our Unbiased Teacher on COCO-standard (Ta- ble 1). When there are only 0.5% to 10% of data labeled, our model consistently performs favorably

---

Published as a conference paper at ICLR 2021
Table 2: Experimental results on COCO comparing with CSD (Jeong et al., 2019) and STAC (Sohn et al., 2020b). *; we implement the CSD method and adapt it on the MS-COCO dataset. Note that Ix represents 90K training iterations, and Nx represents N x90K training iterations.
COCO-additional
Supervised (Ix) Supervised (3x) CSD (3x)
STAC (6x)
Ours (3x)
p60-95
37.63
40.20
38,82
39.21
41.30
Box Accuracy
Box mloU
Number of Pseudo Boxes
Accuracy (%)
labels Accuracy
70
Number of Boxes Per Image
10.0
Pscudo boxes
Ummt
80
miou (9%)
7.5
60
5.0
70
50
20K
40K
20K
40K
20K
40K
Training Iteraiions
Training Iterations (b)
Training Iterations
(c)
Figure 4: Pseudo-label improvement on (a) accuracy, (b) mloU, and (c) number of bounding boxes in the case of ConO-standard 1% labeled data. We measure the (a) accuracy and (b) mloU by comparing the ground-truth boxes and pseudo boxes. The Burn-In limit curves indicate the pseudo- boxes obtained from the model right after the Burn-In stage without further refinement (i.e, the model trained on labeled data only). GT curve on the number of boxes figure indicates the averaged number bounding boxes in the GT labels, and we showed that there are around bounding boxes per image average in MS-COCO. This result indicates our model can generate more accurate pseudo-labels after the Burn- stage (i.e., 2k iterations).
against the state-of-the-art methods, CSD (Jeong et al., 2019) and STAC (Sohn et al., 2020b). It worth noting that our model trained on 1% labeled data achieves 20. 20.75% mAP, which is even higher than STAC trained on 2% labeled data (mAP 18.25 18.25%), CSD trained on 5% labeled data (mAP 18. (8.57%), and the supervised baseline trained on 5% labeled data (mAP 18.47%). We also observe that, as there are less labeled data, the improvements between our method and the existing approaches becomes larger. Unbiased Teacher consistently shows around 10 absolute mAP im- provements when using less than 5% of labeled data compared to supervised method. We attribute the improvements to several crucial factors:
1) More accurate pseudo-labels. When leveraging the pseudo-labeling and consistency regulariza- tion between two networks (Teacher and Student in our case). it is critical to make sure pseudo-labeit are accurate and reliable. Existing method attempts do this by training the pseudo-label gener- ation model using all the available labeled data and is completely frozen afterwards. In contrast, in our framework, our pseudo-label generation model (Teacher) continues to evolve gradually and smoothly via Stacher-Studen Mutual Learning. This enables the Teacher to generate more accurate pseudo-labels as presented in Figure 4, which are properly exploited in the training of the Student.
2) Class-imbalance on pseudo-labels. Our improvement also comes from both the use of the EMA and the Focal loss (Lin et al., 2017b), which addresses the baas--ibbllanced pseudo-labeling issue. As mentioned in Sec. 3.3, using more balanced pseudo-labels not only avoids the consecutive biased prediction problem but also benefits the predictions on the minority classes. Later in Sec. 4.2, we present the details of the ablation study on the EMA and the Focal loss.
COCO-additional and VOC. In the previous section, we presented Unbiased Teacher can suc- cessfully leverage very small amounts of labeled data. We now aim to verify whether the model trained on 100% supervised data can be further improved by using additional unlabeled data. We and VOC and present the results in Table and 3.
thus consider
In the case of COCO-additional (Table 2). compared with supervised only model, our model has 1.10 absolute AP improvement. We also found a similar trend in the VOC experiment (Table 3). With VOCO7 as labeled set and VOC12 as an additional unlabeled set, STAC shows 2.51 absolute MP improvement with respect the supervised model, whereas our model demonstrates 6.56 ab-
7

---

Published as a conference paper at ICLR 2021
Table 3: Results VOC comparing with CSD (Jeong et al., 2019) and STAC (Sohn et al., 2020b).
Backbone
Labeled
Unlabeled
Al's
AP30:95
Supervised (from Ours)
ResNett--PP
VOCO7
None
72.63
42.13
CSD
ReNNc101-R-FCN
74.70 (42.07)
STAC
ResNe1SOFPP
VOCU7
VOC12
77.45 (1487)
44.64 (42.5)
Unbiased Teacher
ResNet50.FPN
77.37 (+4.74)
48.69 (+6.56)
CSD
ResNet101-R-FCN
VOC12
75.10 (42.47)
STAC
ResNet50-FFN
VOCO7
79.08 (+6.45)
46.01 (+3.88)
Unbiased Teacher
RoNFe55-FFPN
COCO20els
78.82 (+6.19)
50.34 (+8.21)
KLedivergnce: 0.0851
Kl-divengunce: 0.2001
Focal Loss wio EMA
Foral Loss w/ EMA
10
Uis
Focal Loso 598
Cross Intropy wi EMA4
Cross Entrapy weo EHA
20
10
20
30
50
60
70
15
20
70
86
Clvv Vitex
Cluss Irdew
KL-divergence
KL-divergence:
30-
2.50
50K
100K
150c
10
20
(a)
(b)
Figure 5: Ablation study on the EMA and the Focal loss in the case of COCO 1% labeled data. (a) mAP of the models using the Focal loss or cross-entropy and applying the EMA or standard training. (b) Class empirical distribution (i.e., histogram) of pseudo-label, generated by each model and compute KC--divergence between the ground-truth labels distribution and the pseudo-label dis- tribution. Among these models, the model using the Focal loss and EMA training (i.e., green curve) achieves the best mAP with the most balanced pseudo-labels.
solute mAP improvement. To further examine whether increasing the size of unlabeled data can further improve the performance, we follow CSD and STAC to use COCO20els dataset as an ad- ditional unlabeled set. STAC shows 3.88 absolute mAp improvement, while our model achieves 8.21 absolute mAp improvement. These results demonstrate that our model can further improve the object detector trained on existing labeled datasets by using more unlabeled data, Note that, follow- ing STAC, APss since the metric of APso has been indicated as a saturated metric by the prior work (Cai & Vasconcelos, 2018; Sohn al., 2020b).
more challenging metric, A.s0:95, which averages the ten values over APsu to
4.2 ABLATION STUDY
Effect of the EMA training. We first examine the effect of EMA training and present a comparison between our model with EMA and without EMA. Our model without EMA is where the model weights of Teacher and Student are shared during the training stage, and it implies the Teacher model is also updated when the student model is optimized by using unlabeled data and pseudo- labels. Note that the state-of-the-an the semi-supervieed classification model, FixMatch (Sohn et al., 2020a) similarly shares the model weights of the Teacher and the Student models.
From Figure 5, we observe that our model with EMA is superior to without EMA, and this trend can be found both in the model using the Focal loss and cross-entropy. To further analyze the diverged results, visualize the class distribution of pseudo-labes K.C-divergence between the ground-truth labels distribution and the pseudo-labed distribution. With the use of cross-entropy and standard training (i.e., without EMA training), the model generates the imbalanced pseudo-labels. To be more specific, the instances of most object categories in pseudo- labels disappear, while only instances of specific object categories remain. We observe that using the EMA training can alleviate the imbalanced pseudo-labels issue and reduces the X.C-divergence
generated by each model and measure the
cOC022cls is generated by only leaving COCO images which have object categories that overlap with the object categories used in PASCAL VOCO7.
8

---

Published as a conference paper at ICLR 2021
from 1.7915 to 0.2482. On the other hand, we also observe that the model with EMA has smoother learning curve compared with the model without EMA. This is because the model weight of the pseudo-label generation model (Teacher) is detached from the optimized model (Student). The pseudo-label generation model can thus prevent the detrimental effect caused by the noisy pseudo- labels (e.g. false positive boxes)
describe in Sec. 3.2.
In sum, the EMA training has several advantages: 1) prevents the imbalanced pseudo-labels issue caused by the imbalanced nature in low-labeled object detection tasks, 2) prevents the detrimental effect caused by the noisy pseudo-labels, and 3) the Teacher model can be regarded as the temporal ensembles model of Student models in different time steps.
Effeet of the Focal loss. In addition to the EMA training, we also verify the effectiveness of the Focal loss. As presented in Figure 5, the model using Focal loss can perform favorably against the model using cross-entropy. The model trained with the Focal loss can generate the pseudo-label which distribution is more similar the distribution of ground-truth labels, and it can improve the KCC-divrgeecce from 1.7915 (Cross entropy w/o EMA) to 0.2001 (Focal loss w/o EMA) and mAP from 13.42 to 17.85. When EMA training applied, the KC-divergence of the model with the Focal loss can be further improved from 0.2482 (Cross entropy w/ EMA) to 0.0851 (Focal loss w/ EMA) and mAP improve from 16.91 to 21.19. This confirms the effectiveness of the Focal loss in handling the class imbalance issues existed in the semi-supervised object detection. The reduction of KC--divereence (i.e, better-fitting pseudo-label distributions to ground-truth label distributions) results in the mAP improvement.
Other ablation studies. We also ablate the effects of the Burn-In stage, pseudo-labeling threshold- ing, EMA rates, and unsupervised loss weights in the Appendix.
5 CONCLUSION
In this paper, we revisit the semi-supervised object detection task. By analyzing the object detectors in low-labeled scenarios, we identify and address two major issues: overfitting and class imbalance. We proposed Unbiased Teacher - a unified framework consisting of a Teacher and a Student that jointly learn improve each other. In the experiments, we show our model prevents pseudo-labeling bias issue caused by class imbalance and overfitting issue due to labeled data scarcity. Our Unbi- ased Teacher achieves satisfactory performance across multiple supi-super object detection datasets.
ACKNOWLEDGMENTS
Yen- Ba--Cheng Liu and Zsolt Kira were partly supported by DARPA's Learning with Less Labels (LwLL) program under agreement HR0011-188 1110011-1----044, as part of their affiliation with Georgia Tech.
REFERENCES
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), pp. 5049-5059, 2019.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Delving into high quality object detection. In
Terrance Driei and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (IJCV) 88(2):303-338, 2010.

---

Published as a conference paper at ICLR 2021
Jiyang Gao, Jiang Wang, Shengyang Dai, Li-Jia Li, and Ram Nevatia. Note-renn: Noise tolerant ensemble renn for semi-supert ceni-supereised object detection. In Proceedings of the IEEE international conference on computer vision (ICCV), pp. 9008-951,,2111
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances in neural information processing systems (NeurIPS), pp. 529-536, 2005.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprini arXiv:2006.07733, 2020.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup locally linear out-of-manifold regular- ization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 33, pp. 3714 1712, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision (ICCV), pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Judy Hoffman, Sergio Guadarrama, Eric S Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, and Kate Saenko. Lsda: Large scale detection through adaptation. In Advances in Neural Information Processing Systems ((NeurIPS)), pp. 3536-3544, 2014.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference Machine Learning (ICML), 2015.
Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak. Consistency-basee semi-supervised learning for object detection. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Borui Jiang. Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In Proceedings of the European Conference on Com- puter Vision (ECCV), 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Chia-Wen Kuo, Chih-Yar Ma, Jia-Bin Huang, and Zsolt Kira. Featmatch: Feature-hased augmen- tation for semi-supervised learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Hei Law and Jia Deng. Cornerne: Detecting objects as paired keypoints. Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning. ICML, volume 3, pp. 2, 2013.
10

---

Published as a conference paper at ICLR 2021
Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng. Overcom- ing classifier imbalance for long-tail object detection with balanced group softmax. Proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming Hc, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference Com- puter Vision and Pattern Recognition (CVPR), 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (CVPR), pp. 2980-2988, 2017b.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision (ECCV), pp. 21-37. Springer, 2016.
Ishan Misra, Abhinav Shrivastava, and Martial Hebert. Watch and learn: Semi-supervised learning for object detectors from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3593-3602, 2015.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervie learning. IEEE transactions pattern analysis and machine intelligence (PAMI), (8999 2018.
Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance problems in object detection: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep co-training for semi- supervised image recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 135-152, 2018.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 7263-7271, 2017.
Joseph Redmon and Ali Farhadi. Yolov3:
An incremental improvement.
arXiv preprint
McXiv:180.002766 2018
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems (NeurIPS), pp. 91-99, 2015.
Chuck Rosenberg. Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of ob- ject detection models. In 2005 Seventh IEEE Workshops on Applications of Computer Vision, volume 1, pp. 29-36, 2005.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1 1163-1171, 2016.
Abhinay Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (VVPR).016
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying superviee leam- ing with consistency and confidence. Advances in Neural Information Processing Systems (NeurIPS). 2020a.
11

---

Published as a conference paper at ICLR 2021
Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen- Yu Lee, and Tomas Pfister. A sim- ple semi-supervised learning framework for object detection. arXiv preprint arXiv:2005.04757, 2020b.
Peng Tang, Chetan Ramaiah, Ran Xu, and Caiming Xiong. Proposal learning for semi-supervised object detection. arXiv preprint urXiv:2001.0555 1.0066,6 2020.
Yuxing Tang, Josiah Wang, Boyang Gao, Emmanuel Dellandréa, Robert Gaizauskas, and Liming Chen. Large scale semi-supervised object detection using visual and semantic knowledge transfer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp-
2119-2128, 2016.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- Seight-average consis- tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems (NeurIPS), pp. 1195-1204, 2017.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. hww.ceeokkeeaaarceeeeeeeeeeeeeee 2019.
https: /ttp//ggtttt/bt//
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Bing Yu, Jingfeng Wu, Jinwen Ma, and Zhanxing Zhu. Tangent-normal adversarial regularization for semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10676-10684, 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. Proceed- ings of the IEEE International Conference Computer Vision (ICCV), pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Pat Paz. mixup: Beyond empirical risk minimization. Proc. International Conference on Learning Representations (ICLR), 2018.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. arXiv preprint arXiv:1708. 04896, 2017.
12

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 EMA ON IMBALANCED PSEUDO-LABELING ISSUE
To empirically examine the effectiveness of EMA on imbalance, we present the pseudo-label dis- tribution in different training iterations as presented in Figure At the beginning of training (i.e, 30k), both Teacher models with and without EMA could generate the balanced pseudo-labels (the KL divergence between ground-truth labels and pseudo-labels are both small). However, since the Student model is trained with the pscudo-labels generated by the Teacher models, the model without EMA starts biasing towards specific classes. In contrast, with the EMA training, the model gener- ates less imbalanced pseudo-labels. Note that, although the EMA is applied, the balance issue still exists. We thus apply Focal loss to enhance the ability to mitigate the imbalance issue further.
30K iterations
140K iterations
KL- Kl-divergence: 0.0920
KL-divergence: 1.7915
60
60
Without EMA
40
Ratio (%)
Ratio
10.
10
0
20
40
60
80
20
40
60
80
Class index
Class index
KL-divergence: 0.0730
KL-divergence: 0.2.482
60
With EMA
40
Ratio (%)
40
Ratio
10
5
20
60
index
Class
Figure 6: Ablation study on EMA at different training iterations. Both the models with EMA and without EMA have pseudo-label distributions, which are similar the ground-truth distributions in the early stage of training iterations. However, the model without EMA tends to generate more biased pscudo-label distribution later during training.
A.2 ADDITIONAL. ABLATION STUDY
In addition to the ablation studies provided in the main paper, we further ablate Unbiased Teacher in the following sections.
A.2.1 EFFECT OF BURN-IN STAGE
As mentioned Section 1. it crucial to have a good initialization for both Student and Teacher models. We thus present a comparison between the model with and without the Burn-In stage in Figure 7. We observe that, with the Burn-In stage, the model can derive more accurate pseudo- boxes in the early stage of the training. As result, the model can achieve higher accuracy in the early stage of the training, and it also achieves better results when the model is converged.
A.2.2 EFFECT OF PSEUDO-LABELING THRESHOLD
As mentioned in Section 3.3, we apply confidence thresholding to filter these low-confidence pre- dicted bounding boxes, which are more likely to be false- positive instances. To show the effective- ness of thresholding, we first provide the accuracy of predicted bounding boxes before and after the pseudo-labeling in Figure 8.
13

---

Published as a conference paper at ICLR 2021
100
25
Burn-In Without Burn-In
80
20
mAp (%6
Accuracy
60
15
10
20
With Burn-In
Without Burn-In
50K
100K
150K
50K
100K
150K
Training teerations
Training Iterations
(a) mAP
(b) Box Accuracy
Number of Boxes Per Image
mlou (%6
40
20
2
Burn-nn
With Burn-In
Without Burn-In
Without BurnIn
100K
100K
Training Iterations
Training Iterations
(c) mloU
(d) Number of Secudo-Boxes
Figure 7: the case of COCO -tandddd 1% labeled data, (a) Unbiased Teacher with Burn-In stage achieve higher mAP against Unbiased Teacher without Burn-In stage. Using Burn-In Stage results in the early improvement of (b) box accuracy and (c) mloU. (d) Unbiased Teacher with Burn-In stage can derive more pseudo-boxes than Unbiased Teacher without Burn-In stage.
Quality of Pseudo-labels
100
Before Thresholding
Thresholding
Accuracy
20K
40K
60K
100K 120K 140K 160K
Training Iterations
Figure 8: Pseudo-label accuracy improvement with the use of confidence thresholding. We measure the accuracy by comparing the ground-truth labels and predicted labels before and after confidence thresholding. This result indicates that confidence thresholding can significantly improve the quality of pseudo-labels.
When varying the threshold value 8 from 0 to 0.9, expected, the number of generated pseudo- boxes increases as the threshold reduces (Figure 9). The model using excessively high threshold (e.g., 8 0.9) cannot perform satisfactory results, as the number of generated pseudo-labels is very low. On the other hand, the model using a low threshold (e.g.. å = 0.6) also cannot achieve favorable results since the model generates too many bounding boxes. which are likely to be false- positive instances. We also observe that the model cannot even converge if the threshold below 0.5.
14

---

Published as a conference paper at ICLR 2021
Average Preceision
Pseudo-label Boxes
- 0.6
-- 0.8
Number of Boxes Per Image
5 0.6
0.8
Ground-Truth
6 0.7
0.9
20
0.7
0,9
15
(96)
5
0
20K
40K
60K
BOK
100K 120K
140K
20K
40K
BOK
BOK
100K
120K 140K
Training
Training Iterations
(a)
(b)
Figure 9: (a) Validation AP and (b) number of pseudo-label bounding boxes per image with various -aauudig thresholds With an excessively low threshold (e-g., 0.6), model has lower AP, as it predicts more pseudo-labeled bounding boxes compared to the number of bounding
boxes in ground-truth labels. On the other hand, the performance the model using an excessively high threshold (e.g. d = 0.9) drops as it cannot predict sufficient number of bounding boxes in its generated pseudo-labels.
A.2.3 EFFECT EMA RATES
We also evaluate the model using various EMA rate a from 0.5 0.9999 and present the mAP result of the Teacher model in Figure 10. We observe that, with smaller EMA rate (e.g., = 0.5), the model has lower mAP and higher variance, the Student contributes more the Teacher model for each iteration. This implies the Teacher model is likely to suffer from the detrimental effect caused by noisy pseudo-labels. This unstable learning curve can be stabilized and improved as the EMA rate increases. When the EMA rate a achieves 0.99, it performs the best mAP. However, the EMA rate a keeps increasing, the teacher model will grow overly slow as the Teacher model derive the next model weight mostly from the previous Teacher model weight.
EMA rate
EMA Training
22
40
EMA=0.5
20
EMA=0.9
30
EMA-0.99
18
Mmw
EMA=0
AP (%)
MAP
EMA=0
16
0.5
20
== 0.9
14
0.99
10
12
0.999
0.9999
10
20K
40K
60K
80K 100K
120K 140K 160K
AP50 APSS AP6O
APES AP7O AP75 AP8O AP8S AP90 AP95
Training Iterations
(a)
(b)
Figure 10: Validation AP on Teacher model with various MMA rates a. (a) With a small MMA rate (e.g., 0.5), the Teacher model has lower AP and larger variance. In contrast, the MMA rate grows to 0.99, the Teacher model can gradually improve along the training iterations. However, when the MMA grows 0.9999, the Teacher model grows overly slow but has lowest variance. (b) We breakdown the AP metric into APs from P3 to AP
A.2.4 EFFECT OF UNSUPERVISED LOss WEIGHTS
To examine the effect unsupervised loss weights, vary the unsupervised loss weight A,, from 1.0 to 8.0 in the case of COCO-standard 10% labeled data. As shown Table 4, with a lower unsupervised loss weight \, 1.0, the model performs 29.30%. On the other hand, observe that the model performs the best with unsupervised loss weight À = 5.0. However, when the weight increases 8.0, the training of the model cannot converge.
15

---

Published as a conference paper at ICLR 2021
Table Ablation study of varying unsupervised loss weight X, on the model trained using 10% labeled and 90% unlabeled data.
A
1.0
2.0
4.0
5.0
6.0
8.0
AP (%)
29.30
30.64
31.82
32.00
31.80
Cannot Converge
A.3 AP BREAKDOWN FOR COCO-STANDARD
We present an AP breakdown for COCO-standard 0.5% labeled data. As mentioned in Section 4, our proposed model can perform favorably against both STAC (Sohn et al., 2020b) and CSD (Jeong et al., 2019). This trend appears in all evaluation metrics from APso to AP95. as shown in Figure 11, and it confirms that our model is preferable for handling extremely low-label scenario compared to the state of the arts.
Figure 11: Evaluation metric breakdown of all methods on 0.5% labeled data.
A.4 IMPLEMENTATION AND TRAINING DETAILS
Network and framework. Our implementation builds upon the Detectron2 framework (Wu et al., 2019). For fair comparison, we follow the prior work (Sohn et al., 2020b) to use Faster-RCNN with FPN (Lin et al., 2017a) and ResNet-50 backbone (He et al., 2016)
object detection network.
Training. At the beginning of the Burn-In stage, the feature backbone network weights are ini- tialized by the pretraned model, which is same as existing works (Jeong et al., 2019; Tang et al., 2020; Sohn et al., 2020b). We use the SGD optimizer with a momentum rate 0.9 and learning rate 0.01, and we use constant learning rate scheduler. The batch size of supervised and unsupervised data are both 32 images. For the COCO-standarts we train 180k iterations, which in- cludes 1/2/6 12/ 2/20k iterations for 0.5%/ 1% 11%2%%/52 55% / 100% the Burn-In stage and the remaining iterations in the Teacher-Studeet Mutual Learning stage. For the COCO-additioal, we train 360k iterations, which includes 90k iterations in the Burn-Up stage and the remaining 270k iterations in the Teacher-Student Mutual Learning stage.
sypper-arrameerr. We use confidence threshold experiments, the unsupervised loss weight X,, 4 is applied for COCO-standard and VOC, and the unsupervised loss weight A 2 is applied for COCO-additional. We apply a = 0.9996 as the EMA rate for all our experiments. flyper-parameter used are summarized in Table 5.
0.7 to generate pseudo-labels for all our
Data augmentation. As shown in Table we apply randomly horizontal flip for weak augmentation and randomly add color jittering, grayscale, Gaussian blur, and cutout patches (DeVries & Taylor, 2017) for the strong augmentation. Note that we do not apply any image-level or box-level geometric augmentations, which are used in STAC (Sohn et al., 2020b). In addition, do not aggressively search the best hyper-parameter for data augmentations, and is possible to obtain better hyper- parameters.
16

---

Published as a conference paper at ICLR 2021
Table 5: Meanings and values of the hyper-parameters used in experiments.
Hyper-panee parameter
Description
COCO-standard and VOC
COCO-additional
Confidence threshold
0.7
0.7
Ag
Unsupervised loss weight
a
EMA rale
0.9996
0.9996
bt
Batch size for labeled
Batch size for unlabeled data Learning rate
7
0.01
0.01
Table Detail of data augmentations. Probability in the table indicates the probability of applying the corresponding image process.
Weak Augmentation
Process
Probubility
Parameters
Descriptions
Horizontal Flip
None
Strong Augmentation
Probubility
Descriptions
Brightness factor chosen unifarmly from [0.6, 1.4),
(brightmess, contrasi, saluration, lue) (0.4, 0.4, 0.1)
contrast factor chosen uniforuly from [0.6. 1.4).
Color Jittering
0.8
saturation factor chosen uniformly from [0.6, 1.4).
and hue value is chosen uniformly from [-D.1, 0.1].
Grayscale
Nane
None
GaussianBlur
0.5
(sigma sigma (0.1,
Gaussian filter with 0.1 and dy 2.0 applied.
Randomly selects rectangle region in an image
CutoutPatternl
0,7
scale=(0.05,, 0.2), ratio=(0.3, 3.3)
crases pixels. We refer detail Zhong et (2017).
Randomly selects rectangle region in image
CutoutPatterm2
scale=(0.02, 0.2), ratic=(0.1.
erases its pixels. refer detail Zhong et (2017).
Randomly selects rectangle region in image
3
scale=(0.02, 0.2), matio=(0.05,
and crases its pixcls. We refer the detail in Zhong et al. (2017).
Evaluation Metrics. APs0:95 is used to evaluate all methods following the prior works (Law & Deng, 2018; Sohn et al., 2020b).
17


Reviews:
[{"review_id": "s_VugpC9XfT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\nThis paper focuses on the pseudo-labeling bias issue in semi-supervised object detection (SS-OD), and proposes an Unbiased Teacher framework to address this issue. More specifically, the unbiased teacher framework combines the Mean Teacher model for semi-supervised image classification (Tarvainen and Valpola 2017) and Focal Loss (Lin et al. 2017) for fully supervised object detection to address the bias issue. Experiments on COCO and PASCAL VOC show that the proposed method obtains the state-of-the-art semi-supervised object detection results.\n\n\nPros:\n\n+ The paper is well written and easy to understand.\n\n+ The motivations of this paper are clear and interesting.\n\n+ A very simple but effective solution is proposed.\n\n+ Very solid experiments are conducted.\n\n\nCons:\n\n- The main weakness of this paper is that the proposed method is a simple combination of the previous methods including Mean Teacher and Focal Loss.\nIn addition, for the foreground/background imbalance issue, it is straightforward to generate pseudo gts to address this issue, which also has been studied by previous semi-supervised object detection work (Sohn et al. 2020). For the class imbalance issue in object detection, it is also straightforward to use Focal Loss to address this issue.\n\n- One minor thing: It would be better to show results of different detectors / CNN backbones.\n\n\nReview summary:\nIn summary, I think this is a good semi-supervised object detection paper because of its simple but effective solution and solid experiments. So I would like to give a weak accept to this paper.\n\n\nPost-rebuttal comments: \nThe authors have addressed my concerns in their rebuttals. All reviewers give positive comments to this paper. So I would like to give an accept to this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "l-b9NP1HUh6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addressed an essential task for large-scale application of object detection -- semi-supervised learning. It introduced a simple but effective Unbiased Teacher to solve the traditionally problematic data imbalance issue. \n\nPros\n\n(1) Provides strong evidences and analysis on the class-imbalance problem inherited in pseudo-labeling methods;\n(2) Proposed \n      (a) an interesting learning paradigm where the teacher is the temporal ensemble of student networks;\n      (b) focal loss in place of cross entropy;\n(3) The experiment and ablation suggested that the resulting teacher model is not prone to class-imbalance-induced overfitting and the improvement from SOTA is significant.\n\nCons\n\nCurious to know why there is a drop in AP_50 comparing to STAC. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z4h3F5P6bS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "+This paper presents a good work on semi-supervised object detection (SSOD), which is a very challenging task. Although there are great progresses on semi-supervised classification, the SSOD is lying behind. This paper shows very good results over the supervised baselines, even when all annotations are used in COCO. \n\n+The proposed method is very simple. It seems the paper is easy to be reproduced.\n\n+It is a good idea to use EMA of mean teacher for SSOD. In addition, the focal loss (FL) is shown to be very useful in SSOD. \n\nA few questions:\n\n-The default FL is mainly for +/- class imbalance. Do you have modified for imbalance among all positive classes?\n\n-It seems FL is more useful than EMA. However, FL is not well ablated. For example, it is the balance between +/- classes more important or among all positive classes? How about the other SSOD, e.g. STAC, using FL?\n\n-It is understandable that EMA model is more reliable. But I don't see it can be beneficial for class imbalance issue explicitly. \n\n-The comparisons with the baseline supervised models seem to be not very fair. For example, in the last column of Table 1, the supervised model uses 1x schedule (90k iterations), but the proposed method uses 4x schedule. From the Detectron2 github, the 3x model should have AP of 40.2. When compared with 4x supervised baseline, the gain of this paper should be much smaller. The same issue could also be in the other COCO-standard experiments. More fair comparisons should be presented.\n\n-Don't quite understand Fig. 4. Too few description. What are burn-in limit and GT. Why are they constant?\n\n-For the paragraph of \"Class-imbalance on pseudo-labels\", if you don't show results, why you write it in Sec. 4.1, instead of Sec. 4.2?\n\n-It is very weird to cite Law & Deng, 2018, when you refer to AP50 saturation. There are tons of paper out there showing that. \n\n-For the AP curves, do you evaluate on the full val2017 set? It seems the data points are very dense. Isn't it expensive?\n\n-I don't think the smoother curve of EMA teacher is the weights of teacher model is detached from the student model. \n\n=====updates========\n\nMost of my concerns have been addressed by the rebuttal and all the other reviews are positive. l will remain my original recommendation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QU0_W0jZcmn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work tackles the task of semi-supervised object detection via a teacher-student method. The authors introduced a training regime where a teacher and student network, who share the same initial weights pre-trained on labeled data, jointly learns on unsupervised data. They find that label imbalance in the object detection task can lead to inefficient pseudo-label training under the usual SSL training pipeline, and therefore proposes to train the teacher network via exponential moving average to avoid bias in pseudo-labels.\n\n\nPros:\n1. Revisiting of pseudo-labeling bias issue in semi-supervised object detection is good.\nThis paper analyze both classification and regression loss with different ratios of labelled data, which shows that classification branch can easily suffer from overfitting that limits current state-of-the-art semi-supervised object detectors. It shows that the misalignment between classification and regression branch not only exists in fully-supervised learning object detection, but also in self-supervised object detection. \n\n2. Experiments are solid and convincing.\nTable 1 & Table 2 show that unbiased teacher consistently improves state-of-the-art methods CSD and STAC by a large margin on both COCO and VOC dataset.\n\nCons:\n1. Process of VOC12 dataset.\nIn Table 2, since VOC12 has a large overlap with VOC07, is the VOC07 part removed from VOC12 as unlabeled data?\n\n2. Novelty\nNevertheless, some contributions seem a bit straight-forward and without significant novelty. The use of EMA is a direct adaptation from successful methods from classification, while Focal loss is already known to tackle class imbalance. The authors might need to provide more insights on the use of these methods (e.g. explaining in more detail how EMA alleviates such bias).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new semi-supervised object detection approach using Unbiased Teacher to jointly address the pseudo-labeling bias and overfitting issues. Significant improvements over SOTA were reported on COCO and VOC. Reviewers agree that the proposed method is simple and effective, and the experimental results are solid and convincing.  While the novelty of technical contributions for individual components may not be very significant, the idea is simple and well executed with strong results and good presentation. Overall, the paper is recommended for acceptance (poster). 
Decision: Accept (Poster)