Unbiased Teacher for Semi-Supervised Object Detection

1 INTRODUCTION . The availability of large-scale datasets and computational resources has allowed deep neural networks to achieve strong performance on a wide variety of tasks . However , training these networks requires a large number of labeled examples that are expensive to annotate and acquire . As an alternative , Semi-Supervised Learning ( SSL ) methods have received growing attention ( Sohn et al. , 2020a ; Berthelot et al. , 2020 ; 2019 ; Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ; Sajjadi et al. , 2016 ; Lee , 2013 ; Grandvalet & Bengio , 2005 ) . Yet , these advances have primarily focused on image classification , rather than object detection where bounding box annotations require more effort . In this work , we revisit object detection under the SSL setting ( Figure 1 ) : an object detector is trained with a single dataset where only a small amount of labeled bounding boxes and a large amount of unlabeled data are provided , or an object detector is jointly trained with a large labeled dataset as well as a large external unlabeled dataset . A straightforward way to address Semi-Supervised Object Detection ( SS-OD ) is to adapt from existing advanced semi-supervised image classification methods ( Sohn et al. , 2020a ) . Unfortunately , object detection has some unique characteristics that interact poorly with such methods . For example , the nature of class-imbalance in object detection tasks impedes the usage of pseudo-labeling . In object detection , there exists foreground-background imbalance and foreground classes imbalance ( see Section 3.3 ) . These imbalances make models trained in SSL settings prone to generate biased predictions . Pseudo-labeling methods , one of the most successful SSL methods in image classification ( Lee , 2013 ; Sohn et al. , 2020a ) , may thus be biased towards dominant and overly confident classes ( background ) while ignoring minor and less confident classes ( foreground ) . As a result , adding biased pseudo-labels into the semi-supervised training aggravates the class-imbalance issue and introduces severe overfitting . As shown in Figure 2 , taking a two-stage object detector as an example , there exists heavy overfitting on the fore- ∗Work done partially while interning at Facebook . 1Code : https : //github.com/facebookresearch/unbiased-teacher . ground/background classification in the RPN and multi-class classification in the ROIhead ( but not on bounding box regression ) . To overcome these issues , we propose a general framework – Unbiased Teacher : an approach that jointly trains a Student and a slowly progressing Teacher in a mutually-beneficial manner , in which the Teacher generates pseudo-labels to train the Student , and the Student gradually updates the Teacher via Exponential Moving Average ( EMA ) 2 , while the Teacher and Student are given different augmented input images ( see Figure 3 ) . Inside this framework , ( i ) we utilize the pseudo-labels as explicit supervision for both RPN and ROIhead and thus alleviate the overfitting issues in both RPN and ROIhead . ( ii ) We also prevent detrimental effects due to noisy pseudo-labels by exploiting the Teacher-Student dual models ( see further discussion and analysis in Section 4.2 ) . ( iii ) With the use of EMA training and the Focal loss ( Lin et al. , 2017b ) , we can address the pseudo-labeling bias problem caused by class-imbalance and thus improve the quality of pseudo-labels . As the result , our object detector achieves significant performance improvements . We benchmark Unbiased Teacher with SSL setting using the MS-COCO and PASCAL VOC datasets , namely COCO-standard , COCO-additional , and VOC . When using only 1 % labeled data from MS-COCO ( COCO-standard ) , Unbiased Teacher achieves 6.8 absolute mAP improvement against the state-of-the-art method , STAC ( Sohn et al. , 2020b ) . Unbiased Teacher consistently achieves around 10 absolute mAP improvements when using only 0.5 , 1 , 2 , 5 % of labeled data compared to supervised baseline . We highlight the contributions of this paper as follows : • By analyzing object detectors trained with limited-supervision , we identify that the nature of class-imbalance in object detection tasks impedes the effectiveness of pseudo-labeling method on SS-OD task . • We thus proposed a simple yet effective method , Unbiased Teacher , to address the pseudolabeling bias issue caused by class-imbalance existing in ground-truth labels and the overfitting issue caused by the scarcity of labeled data . • Our Unbiased Teacher achieves state-of-the-art performance on SS-OD across COCOstandard , COCO-additional , and VOC datasets . We also provide an ablation study to verify the effectiveness of each proposed component . 2 RELATED WORKS . Semi-Supervised Learning . The majority of the recent SSL methods typically consist of ( 1 ) input augmentations and perturbations , and ( 2 ) consistency regularization . They regularize the model to be invariant and robust to certain augmentations on the input , which requires the outputs given the original and augmented inputs to be consistent . For example , existing approaches apply convention data augmentations ( Berthelot et al. , 2019 ; Laine & Aila , 2017 ; Sajjadi et al. , 2016 ; Tarvainen & 2Note that there have been many works that leverages EMA , e.g. , ADAM optimization ( Kingma & Ba , 2015 ) , Batch Normalization ( Ioffe & Szegedy , 2015 ) , self-supervised learning ( He et al. , 2020 ; Grill et al. , 2020 ) , and SSL image classification ( Tarvainen & Valpola , 2017 ) . We , for the first time , show its effectiveness in combating class imbalance issues and detrimental effect of pseudo-labels for the object detection task . Valpola , 2017 ) to generate different transformations of the semantically identical images , perturb the input images along the adversarial direction ( Miyato et al. , 2018 ; Yu et al. , 2019 ) , utilize multiple networks to generate various views of the same input data ( Qiao et al. , 2018 ) , mix input data to generate augmented training data and labels ( Zhang et al. , 2018 ; Yun et al. , 2019 ; Guo et al. , 2019 ; Hendrycks et al. , 2020 ) , or learn augmented prototypes in feature space instead of the image space ( Kuo et al. , 2020 ) . However , the complexities in architecture design of object detectors hinder the transfer of existing semi-supervised techniques from image classification to object detection . Semi-Supervised Object Detection . Object detection is one of the most important computer vision tasks and has gained enormous attention ( Lin et al. , 2017a ; He et al. , 2017 ; Redmon & Farhadi , 2017 ; Liu et al. , 2016 ) . While existing works have made significant progress over the years , they have primarily focused on training object detectors with fully-labeled datasets . On the other hand , there exist several semi-supervised object detection works that focus on training object detector with a combination of labeled , weakly-labeled , or unlabeled data . This line of work began even before the resurgence of deep learning ( Rosenberg et al. , 2005 ) . Later , along with the success of deep learning , Hoffman et al . ( 2014 ) and Gao et al . ( 2019 ) trained object detectors on data with bounding box labels for some classes and image-level class labels for other classes , enabling detection for categories that lack bounding box annotations . Tang et al . ( 2016 ) adapted the image-level classifier of a weakly labeled category ( no bounding boxes ) into a detector via similarity-based knowledge transfer . Misra et al . ( 2015 ) exploited a few sparsely labeled objects and bounding boxes in some video frames and localized unknown objects in the following videos . Unlike their settings , we follow the standard SSL setting and adapt it to the object detection task , in which the training contains a small set of labeled data and another set of completely unlabeled data ( i.e. , only images ) . In this setting , Jeong et al . ( 2019 ) proposed a consistency-based method , which enforces the predictions of an input image and its flipped version to be consistent . Sohn et al . ( 2020b ) pre-trained a detector using a small amount labeled data and generates pseudo-labels on unlabeled data to fine-tune the pre-trained detector . Their pseudo-labels are generated only once and are fixed through out the rest of training . While they can improve the performance against the model trained on labeled data , imbalance issue is not considered in existing SS-OD works . In contrast , our method not only improve the pseudo-label generation model via teacher-student mutual learning regimen ( Sec . 3.2 ) but address the crucial imbalance issue in generated pseudo-labels ( Sec . 3.3 ) . 3 UNBIASED TEACHER . Problem definition . Our goal is to address object detection in a semi-supervised setting , where a set of labeled imagesDs = { xsi , ysi } Ns i=1 and a set of unlabeled imagesDu = { xui } Nu i=1 are available for training . Ns and Nu are the number of supervised and unsupervised data . For each labeled image xs , the annotations ys contain locations , sizes , and object categories of all bounding boxes . Overview . As shown in Figure 3 , our Unbiased Teacher consists of two training stages , the BurnIn stage and the Teacher-Student Mutual Learning stage . In the Burn-In stage ( Sec . 3.1 ) , we simply train the object detector using the available supervised data to initialize the detector . At the beginning of the Teacher-Student Mutual Learning stage ( Sec . 3.2 ) , we duplicate the initialized detector into two models ( Teacher and Student models ) . Our Teacher-Student Mutual Learning stage aims at evolving both Teacher and Student models via a mutual learning mechanism , where the Teacher generates pseudo-labels to train the Student , and the Student updates the knowledge it learned back to the Teacher ; hence , the pseudo-labels used to train the Student itself are improved . Lastly , there exists class-imbalance and foreground-background imbalance problems in object detection , which impedes the effectiveness of semi-supervised techniques of image classification ( e.g. , pseudo-labeling ) being used directly on SS-OD . Therefore , in Sec . 3.3 , we also discuss how Focal loss ( Lin et al. , 2017b ) and EMA training alleviate the imbalanced pseudo-label issue . 3.1 BURN-IN . It is important to have a good initialization for both Student and Teacher models , as we will rely on the Teacher to generate pseudo-labels to train the Student in the later stage . To do so , we first use the available supervised data to optimize our model θ with the supervised loss Lsup . With the supervised data Ds = { xsi , ysi } Ns i=1 , the supervised loss of object detection consists of four losses : the RPN classification loss Lrpncls , the RPN regression loss Lrpnreg , the ROI classification loss Lroicls , and the ROI regression loss Lroireg ( Ren et al. , 2015 ) , Lsup = ∑ i Lrpncls ( x s i , y s i ) + Lrpnreg ( xsi , ysi ) + Lroicls ( xsi , ysi ) + Lroireg ( xsi , ysi ) . ( 1 ) After Burn-In , we duplicate the trained weights θ for both the Teacher and the Student models ( θt ← θ , θs ← θ ) . Starting from this trained detector , we further utilize the unsupervised data to improve the object detector via the following proposed training regimen .

Summary: This work tackles the task of semi-supervised object detection via a teacher-student method. The authors introduced a training regime where a teacher and student network, who share the same initial weights pre-trained on labeled data, jointly learns on unsupervised data. They find that label imbalance in the object detection task can lead to inefficient pseudo-label training under the usual SSL training pipeline, and therefore proposes to train the teacher network via exponential moving average to avoid bias in pseudo-labels.