Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification

1 Introduction . Deep neural networks ( DNNs ) have developed significantly in the computer vision area [ 15 , 18 ] . Despite this , DNNs highly rely on the fully-supervised learning with a large amount of humanannotated data , which consumes a tremendous amount of time to annotate . In comparison , unlabeled images are much easier to access . Semi-supervised learning [ 4 ] is thus studied to address this problem . By involving large-scale unlabeled images in training , semi-supervised learning becomes more valued [ 16 , 26 , 40 , 28 ] on benchmark datasets . However , most of the existing semi-supervised learning methods focus on image classification . Object detection , where complete annotations include category-aware tags and location-aware bounding boxes , requires much more efforts to construct large-scale fully-annotated datasets . In this work , we aim at semi-supervised learning for object detection [ 35 ] : an object detector is trained on a dataset where only a small fraction of images are fully-labeled and the rest of them are unlabeled . In this setting , easily obtained unlabeled data are utilized to improve the performance of fully-supervised object detection . Most of the current semi-supervised learning methods in object detection are based on pseudo labeling [ 19 , 36 ] . A fully-supervised model is firstly pre-trained on completely labeled images then performs inference on unlabeled data to generate pseudo labels . These unlabeled images associated with pseudo labels are further combined with labeled data to learn the semi-supervised model . This pseudo labeling scheme has been widely adopted in semi-supervised object detection [ 31 ] . However , the performance is still limited since the noise inherently exists in pseudo labels , i.e. , pseudo labeling based semi-supervised learning methods severely suffer from the inherent noise . ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To promote semi-supervised learning , it is natural to ask : what kinds of adverse effects are caused by the noisy pseudo labels in detection ? A common way of object detection is to firstly generate several candidate region proposals then extract object-centric feature representations . Because of the noisy pseudo labels , region proposals are likely to be assigned with incorrect labels . Based on the error analysis , we discover that three kinds of errors ( i.e. , the missed GT error , the classification error , and the assignment error ) can be attributed to noisy pseudo labeling . Another question arises : how to combat noise inherent with pseudo labels and facilitate semi-supervised learning ? To answer this , we estimate the noise by quantifying the uncertainty of noise-polluted region proposals . In particular , we measure the region uncertainty from the perspective of incorrect or imprecise predictions , then involve this uncertainty for noise-resistant semi-supervised learning . In this paper , we propose a region uncertainty quantification based semi-supervised learning method for object detection . Specifically , we first present a detailed investigation into the effects of noisy pseudo labels . We observe that different types of region proposals behave with different sensitivity in the face of noisy labels . By associating the varied sensitivity with noisy pseudo labels , we present a quantitative metric to measure the uncertainty degrees of regions and construct an uncertainty-aware soft target as the learning objective . In addition , we remove the competitive effect among classes to allow multi-peak probabilistic confidence and avoid over-confident predictions for uncertain regions . By quantifying and embracing the region uncertainty , we obtain a novel semi-supervised learning approach for object detection with high noise resistance and superior performance . Our main contributions can be summarized as follows : • We investigate the negative effects of noisy pseudo labels . A novel region uncertainty quantification metric is proposed from the perspective of the sensitivity to noisy pseudo labeling . • We propose a noise-resistant semi-supervised learning approach by formulating an uncertaintyaware soft target as the learning objective , which prevents the performance from deterioration caused by noisy pseudo labeling . • By removing the competition among classes and allowing multi-peak probability distributions , we further alleviate the overfitting to noisy pseudo labels . Our method achieves the state-of-the-art results on the PASCAL VOC and MS COCO dataset , exceeding supervised baseline methods by 6.2 % and 4.2 % respectively . 2 Related Work . Semi-supervised learning . To reduce the difficulty of obtaining large-scale labeled data , semisupervised learning is presented by adopting unlabeled images into training . Until recently , current semi-supervised methods are mainly about consistency regularization [ 16 , 26 , 40 , 28 ] , which usually performs data augmentation or perturbation first then constrains their output to be consistent ; self-training [ 44 , 4 , 30 ] , which usually adopts the co-training mechanism of teacher and student models ; label propagation [ 48 , 2 ] that helps improve the quality of pseudo labels and so on . These methods usually target at classification or semantic segmentation . Object detection , however , is by nature different . Large numbers of region proposals within a single image and the existence of the background category make these methods hard to transfer well to object detection . Semi-supervised object detection . Object detection , one of the most important tasks in computer vision , has developed rapidly in recent years [ 9 , 8 , 33 , 32 , 23 ] . But the large annotation cost restricts the scale of current detection datasets , further limits the performance of detection models . Currently , a part of semi-supervised object detection methods are based on pseudo labels [ 31 , 24 , 43 , 39 , 47 ] . But they usually fail to consider noise within pseudo labels thus are easy to overfit noisy labels . Some methods also adopt the idea of data augmentation and consistency regularization [ 12 , 13 , 37 ] . But they usually need to adopt extensive forms of data augmentation for consistency regularization and increase the training budget . We mainly focus on the usage of pseudo labels . Compared with these methods , we target a noise-robust semi-supervised learning approach when using pseudo labels . Uncertainty-based pseudo-labeling . Considering that standard deep learning methods do not possess the ability to model its uncertainty , a series of methods [ 7 , 25 , 17 ] have been proposed . They usually aim at estimating how uncertain a model is about its predictions . Since uncertainty estimation is strongly related to the quality of pseudo labels , some methods [ 27 , 34 , 46 ] have adopted it for pseudo label based semi-supervised learning . Recent works [ 14 , 42 ] also try to improve the calibration of the network in object detection . However , these works usually focus on measuring the uncertainty of the pseudo label itself . Object detection , in comparison , is different , as pseudo labels do not supervise the learning process directly . Instead , assigned labels for different regions are the learning targets . Therefore , we focus on uncertainty quantification for regions in object detection . 3 Region Uncertainty Quantification . In this section , we present the quantification metric for estimating the uncertainty of region proposals . Specifically , we investigate different forms of noise in pseudo labels and observe that different types of proposals possess different noise resistance . Base on this fact , we quantify the uncertainty of regions based on their sensitivity to noisy pseudo labeling . 3.1 Noisy Labeling Effects in Semi-supervised Object Detection . Most existing methods for semi-supervised object detection are through pseudo labels . For an unlabeled image , its pseudo labels are usually extracted by a detection model pre-trained on fullylabeled images . To improve the quality of pseudo labels , we perform data distillation [ 31 ] on unlabeled images . Specifically , different data transformations are applied to the unlabeled images for test augmentation . Detection results from images under different transformations are then averaged to form the final pseudo labels G = { bbn , cn , sn } Nn=1 , where bbn denotes the bounding box , cn is its category label , and sn is the corresponding probability score from the classification branch . For an unlabeled image , after the region proposal network ( RPN ) [ 33 ] , a series of region proposals are generated . A certain number of region proposals are then randomly sampled , which forms a set of region proposals { ri } for the later classification and regression . For a region proposal ri , its label is assigned according to its overlaps with pseudo labels . We consider the maximum value among it : Ii = maxg∈G IoU ( ri , g ) . If Ii is larger than a pre-defined lower bound threshold ∆b , it is regarded as a positive proposal , and the corresponding label { bbi , ci , si } from the pseudo label set G will be assigned . Otherwise , it is regarded as a negative one . Its category ci will be assigned as background , while the bounding box bbi and the probability score si are absent . In the semi-supervised object detection setting , noise inherently exists in pseudo labels . The assigned category ci is thus likely to be incorrect . The origin of incorrect assigned labels is mainly three-fold : 1 . Missed GT error . The proposal ri contains an object , but the corresponding annotation is missed in pseudo labels G , like the red dashed toaster bounding box in Fig . 2a . ri fails to match with any annotations thus is assigned with the background category . 2 . Classification error . The categories of pseudo labels may be incorrect , like the blue ’ bottle ’ bounding box in Fig . 2a , which should be a cup . Consequently , although the proposal ri is able to be assigned with the correct object , its category ci is incorrect . 3 . Assignment error . Some pseudo labels are inaccurately localized , like the oversized oven in Fig . 2a , so Ii is erroneous . For the proposal ri , inaccurate bounding box labels in G make it easily be assigned to another object or the background category , resulting in the incorrectly assigned ci . These three types of errors have different effects on different types of region proposals . According to their definitions , the classification error only happens on positive proposals , while the missed GT error only occurs on negative ones . The missed GT error only matters when the specific negative region is close enough to a miss-annotated object . However , since the number of negative regions is quite large , the noisy negative bounding box is a little improbable to be randomly sampled . As a result , in the region proposal set { ri } , it is much less likely for a negative proposal to be affected by the missed GT error , compared to positive proposals under the classification error . We thus conclude that negative proposals suffer from less noise than positive ones . The assignment error affects region proposals whose I is close to ∆b most severely . For these proposals , even a slight localization error of pseudo labels may change the value of I , then disturb the comparison of I and ∆b . The assignment error thus occurs and leads to the wrong category label . Therefore , we hold that proposals whose I is close to ∆b are exposed to more noise . We conduct a baseline semi-supervised object detection experiment on the COCO dataset [ 22 ] , extract all region proposals on unlabeled images , and record whether their assigned labels are correct . The relation between the accuracy of their assigned labels and I is plotted in Fig . 2b . We observe that the accuracy of negative proposals is universally larger than that of positive ones , and proposals whose I are close to ∆b ( 0.5 ) are more noisy . This result confirms our analysis above .

Summary: This paper tackles semi-supervised learning for object detection, where only a small amount of the training data is labeled with bounding boxes, while the rest of the training data is unlabeled. To solve this problem, the authors present a region uncertainty quantification method that quantifies the uncertainty of regions in unlabeled images based on their sensitivity to noisy pseudo labels. In particular, a quantification metric for the uncertainty is proposed for positive region proposals according to the classification error and the assignment error. Experiments are performed on both PASCAL VOC and MS COCO, with a comparison to three competitors: DD, CSD, STAC.