Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification

1 Introduction . Deep neural networks ( DNNs ) have developed significantly in the computer vision area [ 15 , 18 ] . Despite this , DNNs highly rely on the fully-supervised learning with a large amount of humanannotated data , which consumes a tremendous amount of time to annotate . In comparison , unlabeled images are much easier to access . Semi-supervised learning [ 4 ] is thus studied to address this problem . By involving large-scale unlabeled images in training , semi-supervised learning becomes more valued [ 16 , 26 , 40 , 28 ] on benchmark datasets . However , most of the existing semi-supervised learning methods focus on image classification . Object detection , where complete annotations include category-aware tags and location-aware bounding boxes , requires much more efforts to construct large-scale fully-annotated datasets . In this work , we aim at semi-supervised learning for object detection [ 35 ] : an object detector is trained on a dataset where only a small fraction of images are fully-labeled and the rest of them are unlabeled . In this setting , easily obtained unlabeled data are utilized to improve the performance of fully-supervised object detection . Most of the current semi-supervised learning methods in object detection are based on pseudo labeling [ 19 , 36 ] . A fully-supervised model is firstly pre-trained on completely labeled images then performs inference on unlabeled data to generate pseudo labels . These unlabeled images associated with pseudo labels are further combined with labeled data to learn the semi-supervised model . This pseudo labeling scheme has been widely adopted in semi-supervised object detection [ 31 ] . However , the performance is still limited since the noise inherently exists in pseudo labels , i.e. , pseudo labeling based semi-supervised learning methods severely suffer from the inherent noise . ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To promote semi-supervised learning , it is natural to ask : what kinds of adverse effects are caused by the noisy pseudo labels in detection ? A common way of object detection is to firstly generate several candidate region proposals then extract object-centric feature representations . Because of the noisy pseudo labels , region proposals are likely to be assigned with incorrect labels . Based on the error analysis , we discover that three kinds of errors ( i.e. , the missed GT error , the classification error , and the assignment error ) can be attributed to noisy pseudo labeling . Another question arises : how to combat noise inherent with pseudo labels and facilitate semi-supervised learning ? To answer this , we estimate the noise by quantifying the uncertainty of noise-polluted region proposals . In particular , we measure the region uncertainty from the perspective of incorrect or imprecise predictions , then involve this uncertainty for noise-resistant semi-supervised learning . In this paper , we propose a region uncertainty quantification based semi-supervised learning method for object detection . Specifically , we first present a detailed investigation into the effects of noisy pseudo labels . We observe that different types of region proposals behave with different sensitivity in the face of noisy labels . By associating the varied sensitivity with noisy pseudo labels , we present a quantitative metric to measure the uncertainty degrees of regions and construct an uncertainty-aware soft target as the learning objective . In addition , we remove the competitive effect among classes to allow multi-peak probabilistic confidence and avoid over-confident predictions for uncertain regions . By quantifying and embracing the region uncertainty , we obtain a novel semi-supervised learning approach for object detection with high noise resistance and superior performance . Our main contributions can be summarized as follows : • We investigate the negative effects of noisy pseudo labels . A novel region uncertainty quantification metric is proposed from the perspective of the sensitivity to noisy pseudo labeling . • We propose a noise-resistant semi-supervised learning approach by formulating an uncertaintyaware soft target as the learning objective , which prevents the performance from deterioration caused by noisy pseudo labeling . • By removing the competition among classes and allowing multi-peak probability distributions , we further alleviate the overfitting to noisy pseudo labels . Our method achieves the state-of-the-art results on the PASCAL VOC and MS COCO dataset , exceeding supervised baseline methods by 6.2 % and 4.2 % respectively . 2 Related Work . Semi-supervised learning . To reduce the difficulty of obtaining large-scale labeled data , semisupervised learning is presented by adopting unlabeled images into training . Until recently , current semi-supervised methods are mainly about consistency regularization [ 16 , 26 , 40 , 28 ] , which usually performs data augmentation or perturbation first then constrains their output to be consistent ; self-training [ 44 , 4 , 30 ] , which usually adopts the co-training mechanism of teacher and student models ; label propagation [ 48 , 2 ] that helps improve the quality of pseudo labels and so on . These methods usually target at classification or semantic segmentation . Object detection , however , is by nature different . Large numbers of region proposals within a single image and the existence of the background category make these methods hard to transfer well to object detection . Semi-supervised object detection . Object detection , one of the most important tasks in computer vision , has developed rapidly in recent years [ 9 , 8 , 33 , 32 , 23 ] . But the large annotation cost restricts the scale of current detection datasets , further limits the performance of detection models . Currently , a part of semi-supervised object detection methods are based on pseudo labels [ 31 , 24 , 43 , 39 , 47 ] . But they usually fail to consider noise within pseudo labels thus are easy to overfit noisy labels . Some methods also adopt the idea of data augmentation and consistency regularization [ 12 , 13 , 37 ] . But they usually need to adopt extensive forms of data augmentation for consistency regularization and increase the training budget . We mainly focus on the usage of pseudo labels . Compared with these methods , we target a noise-robust semi-supervised learning approach when using pseudo labels . Uncertainty-based pseudo-labeling . Considering that standard deep learning methods do not possess the ability to model its uncertainty , a series of methods [ 7 , 25 , 17 ] have been proposed . They usually aim at estimating how uncertain a model is about its predictions . Since uncertainty estimation is strongly related to the quality of pseudo labels , some methods [ 27 , 34 , 46 ] have adopted it for pseudo label based semi-supervised learning . Recent works [ 14 , 42 ] also try to improve the calibration of the network in object detection . However , these works usually focus on measuring the uncertainty of the pseudo label itself . Object detection , in comparison , is different , as pseudo labels do not supervise the learning process directly . Instead , assigned labels for different regions are the learning targets . Therefore , we focus on uncertainty quantification for regions in object detection . 3 Region Uncertainty Quantification . In this section , we present the quantification metric for estimating the uncertainty of region proposals . Specifically , we investigate different forms of noise in pseudo labels and observe that different types of proposals possess different noise resistance . Base on this fact , we quantify the uncertainty of regions based on their sensitivity to noisy pseudo labeling . 3.1 Noisy Labeling Effects in Semi-supervised Object Detection . Most existing methods for semi-supervised object detection are through pseudo labels . For an unlabeled image , its pseudo labels are usually extracted by a detection model pre-trained on fullylabeled images . To improve the quality of pseudo labels , we perform data distillation [ 31 ] on unlabeled images . Specifically , different data transformations are applied to the unlabeled images for test augmentation . Detection results from images under different transformations are then averaged to form the final pseudo labels G = { bbn , cn , sn } Nn=1 , where bbn denotes the bounding box , cn is its category label , and sn is the corresponding probability score from the classification branch . For an unlabeled image , after the region proposal network ( RPN ) [ 33 ] , a series of region proposals are generated . A certain number of region proposals are then randomly sampled , which forms a set of region proposals { ri } for the later classification and regression . For a region proposal ri , its label is assigned according to its overlaps with pseudo labels . We consider the maximum value among it : Ii = maxg∈G IoU ( ri , g ) . If Ii is larger than a pre-defined lower bound threshold ∆b , it is regarded as a positive proposal , and the corresponding label { bbi , ci , si } from the pseudo label set G will be assigned . Otherwise , it is regarded as a negative one . Its category ci will be assigned as background , while the bounding box bbi and the probability score si are absent . In the semi-supervised object detection setting , noise inherently exists in pseudo labels . The assigned category ci is thus likely to be incorrect . The origin of incorrect assigned labels is mainly three-fold : 1 . Missed GT error . The proposal ri contains an object , but the corresponding annotation is missed in pseudo labels G , like the red dashed toaster bounding box in Fig . 2a . ri fails to match with any annotations thus is assigned with the background category . 2 . Classification error . The categories of pseudo labels may be incorrect , like the blue ’ bottle ’ bounding box in Fig . 2a , which should be a cup . Consequently , although the proposal ri is able to be assigned with the correct object , its category ci is incorrect . 3 . Assignment error . Some pseudo labels are inaccurately localized , like the oversized oven in Fig . 2a , so Ii is erroneous . For the proposal ri , inaccurate bounding box labels in G make it easily be assigned to another object or the background category , resulting in the incorrectly assigned ci . These three types of errors have different effects on different types of region proposals . According to their definitions , the classification error only happens on positive proposals , while the missed GT error only occurs on negative ones . The missed GT error only matters when the specific negative region is close enough to a miss-annotated object . However , since the number of negative regions is quite large , the noisy negative bounding box is a little improbable to be randomly sampled . As a result , in the region proposal set { ri } , it is much less likely for a negative proposal to be affected by the missed GT error , compared to positive proposals under the classification error . We thus conclude that negative proposals suffer from less noise than positive ones . The assignment error affects region proposals whose I is close to ∆b most severely . For these proposals , even a slight localization error of pseudo labels may change the value of I , then disturb the comparison of I and ∆b . The assignment error thus occurs and leads to the wrong category label . Therefore , we hold that proposals whose I is close to ∆b are exposed to more noise . We conduct a baseline semi-supervised object detection experiment on the COCO dataset [ 22 ] , extract all region proposals on unlabeled images , and record whether their assigned labels are correct . The relation between the accuracy of their assigned labels and I is plotted in Fig . 2b . We observe that the accuracy of negative proposals is universally larger than that of positive ones , and proposals whose I are close to ∆b ( 0.5 ) are more noisy . This result confirms our analysis above .

Summary: **Goal:** This paper aims to address semi-supervised object detection, where the object is trained with a set of labeled images (fully-annotated box and object category) and a set of unlabeled images.   **Method:** - The authors propose to use both classification score and IoU to pseudo-box to compute the uncertainty of each proposal boxes, and this is based on their analysis that assignment errors are large when the IoU is closer to 0.5.   - Rather than using the hard label for the classification branch, they construct the soft labels by using uncertainty-based to weigh the ground-truth label.   - The author also proposes to use focal loss and sigmoid to alleviate the overfitting issue of incorrect categories.   **Experiments:** - They present the results on VOC and COCO datasets and show the improvement against CSD and STAC.