Unbiased Teacher for Semi-Supervised Object Detection

1 INTRODUCTION . The availability of large-scale datasets and computational resources has allowed deep neural networks to achieve strong performance on a wide variety of tasks . However , training these networks requires a large number of labeled examples that are expensive to annotate and acquire . As an alternative , Semi-Supervised Learning ( SSL ) methods have received growing attention ( Sohn et al. , 2020a ; Berthelot et al. , 2020 ; 2019 ; Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ; Sajjadi et al. , 2016 ; Lee , 2013 ; Grandvalet & Bengio , 2005 ) . Yet , these advances have primarily focused on image classification , rather than object detection where bounding box annotations require more effort . In this work , we revisit object detection under the SSL setting ( Figure 1 ) : an object detector is trained with a single dataset where only a small amount of labeled bounding boxes and a large amount of unlabeled data are provided , or an object detector is jointly trained with a large labeled dataset as well as a large external unlabeled dataset . A straightforward way to address Semi-Supervised Object Detection ( SS-OD ) is to adapt from existing advanced semi-supervised image classification methods ( Sohn et al. , 2020a ) . Unfortunately , object detection has some unique characteristics that interact poorly with such methods . For example , the nature of class-imbalance in object detection tasks impedes the usage of pseudo-labeling . In object detection , there exists foreground-background imbalance and foreground classes imbalance ( see Section 3.3 ) . These imbalances make models trained in SSL settings prone to generate biased predictions . Pseudo-labeling methods , one of the most successful SSL methods in image classification ( Lee , 2013 ; Sohn et al. , 2020a ) , may thus be biased towards dominant and overly confident classes ( background ) while ignoring minor and less confident classes ( foreground ) . As a result , adding biased pseudo-labels into the semi-supervised training aggravates the class-imbalance issue and introduces severe overfitting . As shown in Figure 2 , taking a two-stage object detector as an example , there exists heavy overfitting on the fore- ∗Work done partially while interning at Facebook . 1Code : https : //github.com/facebookresearch/unbiased-teacher . ground/background classification in the RPN and multi-class classification in the ROIhead ( but not on bounding box regression ) . To overcome these issues , we propose a general framework – Unbiased Teacher : an approach that jointly trains a Student and a slowly progressing Teacher in a mutually-beneficial manner , in which the Teacher generates pseudo-labels to train the Student , and the Student gradually updates the Teacher via Exponential Moving Average ( EMA ) 2 , while the Teacher and Student are given different augmented input images ( see Figure 3 ) . Inside this framework , ( i ) we utilize the pseudo-labels as explicit supervision for both RPN and ROIhead and thus alleviate the overfitting issues in both RPN and ROIhead . ( ii ) We also prevent detrimental effects due to noisy pseudo-labels by exploiting the Teacher-Student dual models ( see further discussion and analysis in Section 4.2 ) . ( iii ) With the use of EMA training and the Focal loss ( Lin et al. , 2017b ) , we can address the pseudo-labeling bias problem caused by class-imbalance and thus improve the quality of pseudo-labels . As the result , our object detector achieves significant performance improvements . We benchmark Unbiased Teacher with SSL setting using the MS-COCO and PASCAL VOC datasets , namely COCO-standard , COCO-additional , and VOC . When using only 1 % labeled data from MS-COCO ( COCO-standard ) , Unbiased Teacher achieves 6.8 absolute mAP improvement against the state-of-the-art method , STAC ( Sohn et al. , 2020b ) . Unbiased Teacher consistently achieves around 10 absolute mAP improvements when using only 0.5 , 1 , 2 , 5 % of labeled data compared to supervised baseline . We highlight the contributions of this paper as follows : • By analyzing object detectors trained with limited-supervision , we identify that the nature of class-imbalance in object detection tasks impedes the effectiveness of pseudo-labeling method on SS-OD task . • We thus proposed a simple yet effective method , Unbiased Teacher , to address the pseudolabeling bias issue caused by class-imbalance existing in ground-truth labels and the overfitting issue caused by the scarcity of labeled data . • Our Unbiased Teacher achieves state-of-the-art performance on SS-OD across COCOstandard , COCO-additional , and VOC datasets . We also provide an ablation study to verify the effectiveness of each proposed component . 2 RELATED WORKS . Semi-Supervised Learning . The majority of the recent SSL methods typically consist of ( 1 ) input augmentations and perturbations , and ( 2 ) consistency regularization . They regularize the model to be invariant and robust to certain augmentations on the input , which requires the outputs given the original and augmented inputs to be consistent . For example , existing approaches apply convention data augmentations ( Berthelot et al. , 2019 ; Laine & Aila , 2017 ; Sajjadi et al. , 2016 ; Tarvainen & 2Note that there have been many works that leverages EMA , e.g. , ADAM optimization ( Kingma & Ba , 2015 ) , Batch Normalization ( Ioffe & Szegedy , 2015 ) , self-supervised learning ( He et al. , 2020 ; Grill et al. , 2020 ) , and SSL image classification ( Tarvainen & Valpola , 2017 ) . We , for the first time , show its effectiveness in combating class imbalance issues and detrimental effect of pseudo-labels for the object detection task . Valpola , 2017 ) to generate different transformations of the semantically identical images , perturb the input images along the adversarial direction ( Miyato et al. , 2018 ; Yu et al. , 2019 ) , utilize multiple networks to generate various views of the same input data ( Qiao et al. , 2018 ) , mix input data to generate augmented training data and labels ( Zhang et al. , 2018 ; Yun et al. , 2019 ; Guo et al. , 2019 ; Hendrycks et al. , 2020 ) , or learn augmented prototypes in feature space instead of the image space ( Kuo et al. , 2020 ) . However , the complexities in architecture design of object detectors hinder the transfer of existing semi-supervised techniques from image classification to object detection . Semi-Supervised Object Detection . Object detection is one of the most important computer vision tasks and has gained enormous attention ( Lin et al. , 2017a ; He et al. , 2017 ; Redmon & Farhadi , 2017 ; Liu et al. , 2016 ) . While existing works have made significant progress over the years , they have primarily focused on training object detectors with fully-labeled datasets . On the other hand , there exist several semi-supervised object detection works that focus on training object detector with a combination of labeled , weakly-labeled , or unlabeled data . This line of work began even before the resurgence of deep learning ( Rosenberg et al. , 2005 ) . Later , along with the success of deep learning , Hoffman et al . ( 2014 ) and Gao et al . ( 2019 ) trained object detectors on data with bounding box labels for some classes and image-level class labels for other classes , enabling detection for categories that lack bounding box annotations . Tang et al . ( 2016 ) adapted the image-level classifier of a weakly labeled category ( no bounding boxes ) into a detector via similarity-based knowledge transfer . Misra et al . ( 2015 ) exploited a few sparsely labeled objects and bounding boxes in some video frames and localized unknown objects in the following videos . Unlike their settings , we follow the standard SSL setting and adapt it to the object detection task , in which the training contains a small set of labeled data and another set of completely unlabeled data ( i.e. , only images ) . In this setting , Jeong et al . ( 2019 ) proposed a consistency-based method , which enforces the predictions of an input image and its flipped version to be consistent . Sohn et al . ( 2020b ) pre-trained a detector using a small amount labeled data and generates pseudo-labels on unlabeled data to fine-tune the pre-trained detector . Their pseudo-labels are generated only once and are fixed through out the rest of training . While they can improve the performance against the model trained on labeled data , imbalance issue is not considered in existing SS-OD works . In contrast , our method not only improve the pseudo-label generation model via teacher-student mutual learning regimen ( Sec . 3.2 ) but address the crucial imbalance issue in generated pseudo-labels ( Sec . 3.3 ) . 3 UNBIASED TEACHER . Problem definition . Our goal is to address object detection in a semi-supervised setting , where a set of labeled imagesDs = { xsi , ysi } Ns i=1 and a set of unlabeled imagesDu = { xui } Nu i=1 are available for training . Ns and Nu are the number of supervised and unsupervised data . For each labeled image xs , the annotations ys contain locations , sizes , and object categories of all bounding boxes . Overview . As shown in Figure 3 , our Unbiased Teacher consists of two training stages , the BurnIn stage and the Teacher-Student Mutual Learning stage . In the Burn-In stage ( Sec . 3.1 ) , we simply train the object detector using the available supervised data to initialize the detector . At the beginning of the Teacher-Student Mutual Learning stage ( Sec . 3.2 ) , we duplicate the initialized detector into two models ( Teacher and Student models ) . Our Teacher-Student Mutual Learning stage aims at evolving both Teacher and Student models via a mutual learning mechanism , where the Teacher generates pseudo-labels to train the Student , and the Student updates the knowledge it learned back to the Teacher ; hence , the pseudo-labels used to train the Student itself are improved . Lastly , there exists class-imbalance and foreground-background imbalance problems in object detection , which impedes the effectiveness of semi-supervised techniques of image classification ( e.g. , pseudo-labeling ) being used directly on SS-OD . Therefore , in Sec . 3.3 , we also discuss how Focal loss ( Lin et al. , 2017b ) and EMA training alleviate the imbalanced pseudo-label issue . 3.1 BURN-IN . It is important to have a good initialization for both Student and Teacher models , as we will rely on the Teacher to generate pseudo-labels to train the Student in the later stage . To do so , we first use the available supervised data to optimize our model θ with the supervised loss Lsup . With the supervised data Ds = { xsi , ysi } Ns i=1 , the supervised loss of object detection consists of four losses : the RPN classification loss Lrpncls , the RPN regression loss Lrpnreg , the ROI classification loss Lroicls , and the ROI regression loss Lroireg ( Ren et al. , 2015 ) , Lsup = ∑ i Lrpncls ( x s i , y s i ) + Lrpnreg ( xsi , ysi ) + Lroicls ( xsi , ysi ) + Lroireg ( xsi , ysi ) . ( 1 ) After Burn-In , we duplicate the trained weights θ for both the Teacher and the Student models ( θt ← θ , θs ← θ ) . Starting from this trained detector , we further utilize the unsupervised data to improve the object detector via the following proposed training regimen .

Summary: This paper focuses on the pseudo-labeling bias issue in semi-supervised object detection (SS-OD), and proposes an Unbiased Teacher framework to address this issue. More specifically, the unbiased teacher framework combines the Mean Teacher model for semi-supervised image classification (Tarvainen and Valpola 2017) and Focal Loss (Lin et al. 2017) for fully supervised object detection to address the bias issue. Experiments on COCO and PASCAL VOC show that the proposed method obtains the state-of-the-art semi-supervised object detection results.