Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
Authors: Zhenyu Wang
Ya-Li Li
Ye Guo
Shengjin Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: object detection
semi-supervised learning
semi-supervised object detection

Abstract: Semi-supervised learning aims to leverage a large amount of unlabeled data for performance boosting. Existing works primarily focus on image classification. In this paper, we delve into semi-supervised learning for object detection, where labeled data are more labor-intensive to collect. Current methods are easily distracted by noisy regions generated by pseudo labels. To combat the noisy labeling, we propose noise-resistant semi-supervised learning by quantifying the region uncertainty. We first investigate the adverse effects brought by different forms of noise associated with pseudo labels. Then we propose to quantify the uncertainty of regions by identifying the noise-resistant properties of regions over different strengths. By importing the region uncertainty quantification and promoting multi-peak probability distribution output, we introduce uncertainty into training and further achieve noise-resistant learning. Experiments on both PASCAL VOC and MS COCO demonstrate the extraordinary performance of our method.

Combating Noise: Semi-supervised Learning by
Region Uncertainty Quantification
Zhenyu Wang
Yali Li
Ye Guo
Shengjin Wang
Beijing National Research Center for Information Science and Technology (BNRist) Department of Electronic Engineering, Tsinghua University wangzyy0, guo-yl @emmmmmlla 1s. tsinghua www.gaaaeuu....eeeeeeeo cn (liyali13, wgsgi) @ts irssing inghua edu.
Abstract
Semi-supervised learning aims to leverage a large amount of unlabeled data for performance boosting. Existing works primarily focus image classification. In this paper, we delve into semi-supervised learning for object detection, where labeled data are more labor-intensive to collect. Current methods are easily dis- tracted by noisy regions generated by pseudo labels. To combat the noisy labeling, we propose noise-resistant cumi-supervised learning by quantifying the region uncertainty. We first investigate the adverse effects brought by different forms of noise associated with pseudo labels. Then we propose to quantify the uncertainty of regions by identifying the noise-resistant properties of regions over different strengths. By importing the region uncertainty quantification and promoting multi- peak probability distribution output, we introduce uncertainty into training and further achieve noise-resistant learning. Experiments on both PASCAL VOC and MS COCO demonstrate the extraordinary performance of our method.
Introduction
Deep neural networks (DNNs) have developed significantly in the computer vision area |5 18 Despite this, DNNs highly rely on the fully-supervised learning with a large amount human- annotated data, which consumes tremendous amount of time annotate. In comparison, unlabeled images are much easier to access. Semi-supervised learning 4 thus studied to address this problem. By involving large-scale unlabeled images training, semi-supervised learning becomes more valued on benchmark datasets.
16
26
40
28
However, most of the existing semi- semi-supervised learning methods focus on image classification. Object detection, where complete annotations include category-award tags and location-aware bounding boxes, requires much more efforrs to construct large-scale fully-annotated datasets. In this work, aim at semi-supervised learning for object detection [35]: an object detector is trained on a dataset where only small fraction of images are fully-labeled and the rest of them unlabeled. In this setting, easily obtained unlabeled data are utilized to improve the performance of sully-super object detection. Most of the current remi-supervised learning methods in object detection are based model is firstly pre-trained on completely labeled images then performs inference unlabeled data generate pseudo labels. These unlabeled images associated with pseudo labels are further combined with labeled data to learn the semi-supervised model. This pseudo labeling scheme has been widely adopted in semi-supervised object detection 31]. However, the performance is still limited since the noise inherently exists in pseudo labels, i.c., pscudo labeling based ceni-supervised learning methods severely suffer from the inherent noise.
on pseudo labeling 19 36).
*Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

region proposals
cake X
person X
background
labeled
unlabeled images
images
multi-peak probability
uncertainty
p
quantification
dls scorn $
model
bets
noise resistant
training
uncertainty
uncertainty-aware soft target y
Figure The working framework our proposed method. To combat noise associated with pseudo labels, we quantify uncertainty for different regions first. By further constructing uncertainty- aware soft target and promoting multi-peal, probability distribution, we introduce uncertainty into training and achieve noise-resistad learning.
To promote semi-super learning, it is natural to ask: what kinds of adverse effects are caused by the noisy pseudo labels in detection? common way object detection to firstly generate several candidate region proposals then extract object-centric feature representations, Because of the noisy pseudo labels, region proposals are likely to be assigned with incorrect labels. Based on the error analysis, discover that three kinds of errors (i.e., the missed GT error, the classification the assignment error) can be attributed to noisy pseudo labeling. Another question arises: how to combat noise inherent with pseudo labels and facilitate semi-supervvise learning? To answer this, we estimate the noise by quantifying the uncertainty of noise-polluted region proposals. In particular, measure the region uncertainty from the perspective of incorrect imprecise predictions, then involve this uncertainty for noise-resistant semi-supervised learning.
and
In this paper, propose region uncertainty quantification based semi-supervised learning method for object detection. Specifically, we first present a detailed investigation into the effects of noisy pseudo labels. We observe that different types of region proposals behave with different sensitivity in face of noisy labels. By associating varied sensitivity with noisy pseudo labels, we present quantitative metric measure the uncertainty degrees of regions and construct uncertainty-avee aware soft target as the learning objective. addition, we remove the competitive effect among classes to allow mulli-peak probabilistic confidence and avoid over-confident predictions for uncertain regions. By quantifying and embracing the region uncertainty, we obtain a novel semi-supervised learning approach for object detection with high noise resistance and superior performance.
Our main contributions can be summarized as follows:
We investigate the negative effects of noisy pseudo labels. A novel region uncertainty quantification metric is proposed from the perspective of the sensitivity to noisy pseudo labeling.
We propose a noise-resistant semi-supervised learning approach by formulating an uncertainty- aware soft target as the learning objective, which prevents the performance from deterioration caused by noisy pseudo labeling.
By removing the competition among classes and allowing multi-peak probability distributions, further alleviate the overfitting to noisy pseudo labels.
Our method achieves the sfate-of-the-ar results the PASCAL VOC and MS COCO dataset, exceeding supervised baseline methods by 6.2% and 4.2% respectively.
Related Work
Semi-supervised learning. To reduce the difficulty of obtaining large-scale labeled data, semi- supervised learning is presented by adopting unlabeled images into training. Until recently, current

---

accuracy of assigned labels
bottle
0.6
0.4
0.2
0.5
1.0 I
googmmmnn
closeificarinn
entor
(b)
Figure 2: Illustration of different types of noise within pseudo labels (a) and their effects on different types of region proposals (b). Noisy pseudo labels mainly cause the missed GT error, the classification eтror and the assignment error to hurt the model's performance. These effects behave differently on regions according to their different IoUs with pseudo labels.
semi-supervised methods are mainly about consistency regularization 16
which usu- ally performs data augmentation perturbation first then constrains their output be consistent; self-training 44 30), which usually adopts the co-training mechanism of teacher and student models; label propagation [88 2 that helps improve the quality of pseudo labels and so These methods usually target at classification or semantic segmentation. Object detection, however, is by nature different. Large numbers of region proposals within single image and the existence of the background category make these methods hard to transfer well to object detection.
26
40
28
Semi-supervised object detection. Object detection, one of the most important tasks in computer vision, has developed rapidly recent years 9 g 33 32 23). But the large annotation cost restricts the scale of current detection datasets, further limits the performance of detection models. Currently, a part of semi-supervised object detection methods are based on pseudo labels [1 But they usually fail to consider noise within pseudo labels thus easy overfit noisy labels. Some methods also adopt the idea of data augmentation and consistency regularization [12 B 37 But
[4]
43
39 47).
they usually need to adopt extensive forms of data augmentation for consistency regularization and increase the training budget. We mainly focus on the usage of pseudo labels. Compared with these methods, target noise-robust semi-supervised learning approach when using pseudo labels.
Uncertainty-based pseudo-labeling. Considering that standard deep learning methods do not possess the ability to model its uncertainty, series of methods 7 25 17 have been proposed. They usually aim at estimating how uncertain a model about its predictions. Since uncertainty estimation is strongly related to the quality of pseudo labels, some methods (27 34 46) have adopted for pseudo label based semi-supervised learning. Recent works [4 42 also try to improve the calibration of the network object detection. However, these works usually focus on measuring the uncertainty of the pseudo label itself. Object detection, in comparison, is different, as pseudo labels do not supervise the learning process directly. Instead, assigned labels for different regions the learning targets. Therefore, we focus on uncertainty quantification for regions in object detection.
3 Region Uncertainty Quantification
In this section, we present the quantification metric for estimating the uncertainty of region proposals. Specifically, we investigate different forms of noise in pseudo labels and observe that different types of proposals possess different noise resistance. Base on this fact, we quantify the uncertainty of regions based on their sensitivity to noisy pseudo labeling.
3.1 Noisy Labeling Effects in Semi-supervised Object Detection
Most existing methods for semi-supervised object detection are through pseudo labels. For an unlabeled image, its pseudo labels are usually extracted by a detection model pre-trained on fully- labeled images. To improve the quality of pscudo labels, perform data distillation [3] on
3

---

unlabeled images. Specifically, different data transformations are applied to the unlabeled images for test augmentation. Detection results from images under different transformations are then averaged to form the final pseudo labels G = {bb,,, Cz 8n n=1 where bb, denotes the bounding box, cn is its category label, and 8, is the corresponding probability score from the classification branch.
For an unlabeled image, after the region proposal network (RPN) are generated. A certain number of region proposals are then randomly sampled, which forms set of region proposals {re} for the later classification and regression. For a region proposal ris its label is assigned according to its overlaps with pseudo labels. We consider the maximum value among it: I = maxxeG (r,, g). If I, is larger than a pre-defined lower bound threshold A,, is regarded as positive proposal, and the corresponding label {6b, 81) from the pseudo label set G will be assigned. Otherwise, it is regarded as a negative one. Its category c; will be assigned as background. while the bounding box 66, and the probability score s; are absent.
series of region proposals
In the semi-superee supervised object detection setting, noise inherently exists in pseudo labels. The assigned category c; is thus likely to be incorrect. The origin of incorrect assigned labels is mainly three-fold.
1. Missed GT error. The proposal r; contains an object, but the corresponding annotation is missed in pseudo labels G, like the red dashed toaster bounding box in Fig. 2a Y; fails to match with any annotations thus is assigned with the background category.
2. Classification error. The categories of pseudo labels may be incorrect, like the blue 'bottle' bounding box in Fig. Za] which should be a cup. Consequently, although the proposal r, is able to be assigned with the correct object, its category C: is incorrect.
3. Assignment error. Some pseudo labels are inaccurately localized, like the oversized oven in Fig. 2a so I; is erroneous. For the proposal ri. inaccurate bounding box labels in G make it easily be assigned to another object or the background category, resulting in the incorrectly assigned o;.
These three types of errors have different effects on different types of region proposals. According to their definitions, the classification error only happens on positive proposals, while the missed GT error only occurs on negative ones. The missed GT error only matters when the specific negative region is close enough to miss-annotated object. However, since the number of negative regions is quite large, the noisy negative bounding box is a little improbable to be randomly sampled, As a result, in the region proposal set {r:}. it is much less likely for negative proposal to be affected by the missed GT error, compared to positive proposals under the classification error. We thus conclude that negative proposals suffer from less noise than positive ones.
The assignment error affects region proposals whose I close to A, most severely. For these proposals, even a slight localization eιror of pseudo labels may change the value of I, then disturb the comparison of I and A. The assignment error thus occurs and leads to the wrong category label. Therefore, we hold that proposals whose I is close to As are exposed to more noise.
We conduct baseline semi supervisd object detection experiment on the COCO dataset (22] extract all region proposals on unlabeled images, and record whether their assigned labels are correct. The relation between the accuracy of their assigned labels and I is plotted Fig. 26 We observe that the accuracy of negative proposals is universally larger than that of positive ones, and proposals whose I are close to Ab (0.5) are more noisy. This result confirms our analysis above.
3.2 Uncertainty Quantification
According to the above section, as the assigned category q; for the proposal r; may be incorrect, the region proposals are kind of noisy. However, since the image is unlabeled, we do not know whether the proposal is noisy not. This causes region proposals to be uncertain. Current models lack strategies to estimate the uncertainty degrees of regions, not to mention avoiding noisy ones. In this section, we present quantification metric for the uncertainty of region proposals.
The uncertainty of a region proposal r; closely related whether its assigned category e; is correct. If the category c; is likely to be incorrect, the proposal r; is also uncertain for the model. The definition of uncertainty quantification, therefore, should be consistent with the likelihood that the corresponding region proposal is noisy. According to section 3.1 negative proposals are less likely to be noisy than positive ones. Also, since background samples are significantly more, is universally acknowledged that foreground samples occupy a more important position in training

---

Therefore, we only quantify the uncertainty for positive proposals, and assume that negative proposals are all accurate and certain. Positive proposals are likely to be noisy because of the classification error and the assignment error. We thus need to take both emrors into consideration for the uncertainty quantification.
3]
For the proposal T; and its assigned pseudo label {bb,, C2 s.). the classification error derives from the incorrect c₁. Si, the classification probability score, can well reflect its self-confidence, thus can be used for measuring the classification error. Through test time augmentation, a simple but effective strategy for assessing the uncertainty of the classification results, $; contains more information thus is more accurate for evaluating the classification error.
The assignment error is closely related to the overlaps between the proposal and all pseudo labels the distance hetween I, and the threshold A,- So we adopt Z; to measure this. Since our uncertainty quantification only aims positive samples, only region proposals whose Iς is larger than A, is considered. Plus, the assignment error is prominent for proposals with a close-to-A, A, I, We thus set an upper bound threshold A y and only consider proposals with Z; in the range from As A In this range, we normalize the Z, to 0 ~ I with a sigmoid mapping function, with I." denoting the normalized I, We think Z,¹ is a reasonable metric for estimating the assignment error:
11/(1+ cxx(((cxCCI) if A <I<A
I T=11
otherwise
where Tς¹ is the linear normalization of I; from 1. This ensures the symmetry of I,. C' is pre-defined hyper-parameter. I,¹ for proposals whose I; is not in the range from As to A, is set to 1 manually, since we assume that the assignment error has little effect on them. From Fig. B
A, ~ A,
0.8
C=3
that only if C' is not too small, I. Is approximately continuous for positive proposals. For proposals whose Ię is quite close to As. its value is quite small, since these proposals are much more likely be affected by the assign- ment error. Then, I increases as Iç increases, until Z; is close to As. At this time, I," close to and the assignment error can be neglected. As we can see, Z," is in the range 0 and is negatively correlated to the assignment error, thus can be used for estimating the assignment error.
0.4
C=5
C=10
C=15
0.2
C-20
C=25
0.0
0.40
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
1,
Figure 3: ZT,, the normalized maximum overlap, w.r.t. Z, the maimum overlap, under different C. Ag is 0.5 and A, is 0.7.
Based on the above analysis, 8; and T. can be utilized for assessing the classification error and the assignment error separately. A region is certain only when both the classification error and assignment error are small. We thus multiply them to consider both errors. Then, we propose our uncertainty quantification u; for the proposal r;, follows:
8t I if I2 > A
otherwise
For a positive proposal, its uncertainty metric u; is relevant to its 8; and Z," If both of them are close to I, the possibility that is prone the classification error and the assignment error is small. This means that its assigned category e; is relatively accurate and it is somewhat certain. So its is close O, according Equ. 2 vice versa. For negative proposals, we presume that they are less likely to be noisy, they are certain to the model. So their uncertainty quantification is set to 0 manually.
Noise-Resistant Semi-supervised Learning
When region proposals are uncertain, current detection models lack means handling uncertainty. They cannot discriminate uncertain regions or alleviate uncertainty during training, which makes easy to overfit noise. In this section, we propose to introduce uncertainty into semi-supervised learning, so that it better adapts to uncertain regions. Finally we obtain noise-resistant learning.
5

---

4.1 Constructing Uncertainty-aware Soft Target
In object detection, for the proposal rt after assigned with the label {bb,, Ct, si). classification and regression are conducted on this single proposal. Generally, softmax function is used to convert the output logits into the probability distribution P: [pt)], where Pik denotes the probability prediction of the proposal r; for the k-th category. Cross entropy loss is then utilized for classification: Tidss = Cyy Pt). yi = [Mbs] is the one-hot category label for r,, where yie, = and others are O. When p. = yi, the cross entropy is minimized. At this time, the probability distribution p; is also one-hot and its certainty maximized.
In fully-supervised object detection, e; is always the actual ground truth category of ri. denoted with o, so this is of no problem. But in the semi-supervised setting, assigned labels may be incorrect and * ct may occur. When G * c,", the network still tries to maximize Pic, so that P: can approach y:- The only outcome is that the cross entropy between p: and the actual one-hot label of is enlarged, which increases the actual loss. As result, the optimized p; approximates y:, the distribution of pseudo labels. The model thus overfits to noise, which degrades its performance.
The above phenomenon occurs because when region proposals are uncertain, the model still tries to optimize itself towards a certain objective. This disparity causes the model hard to accommodate the uncertain regions. Since the assigned label {bb,, Ct2 S for Ti is uncertain, just adopting the hard label y: unreasonable. To address this issue, we utilize our proposed uncertainty metric u, and construct a soft target label for the proposal 7ú. denoted with y: = [sa]
(1 (3)))) if l * bg (background) Enyhn Dik otherwise
Uik
(3)
where
.B(B) [1 (s: ) 0
if I otherwise
B= (t
(4)
For foreground categories, the soft target is calculated by multiplying its certainty quantification 1 u, and hard label y/k. The soft target of the background category is then obtained by restricting the summation of ÿ. to 1. For a positive proposal r,, if it is relatively certain and its assigned label oa trustworthy, its u; close to O, and y; is close to yi. Since it is certain, the hard label y: is of no problem. If the label e; for r; is very likely to be incorrect, its uncertainty quantification u, is high, close to Then the soft label ÿJ approaches the hard label of negative proposals. In this way, the network regards r; as background category. Since negative samples are noise-resilient, they are less likely to be interfered with by these uncertain samples. With this action, positive proposals with higher quality are involved in classification. Noisy and uncertain regions do not participate in the training of positive samples, thus mitigate the noise overfitting problem.
( in Equ. and Equ. 4 is dynamic exponent, where t denotes the current number of iteration and T the total number of iterations. q is a pre-defined parameter to make 9 increase in a polynomial manner. The introduction of 3 is to make the soft target yi dynamically change with the semi- supervised training process. For deep learning networks under noisy labels, they will usually learn from clean and easy samples at the beginning, and overfit to noisy labels later [ 45
[0]. In the beginning, 8 close to 0 thus (3( is close to 0 too. At this time, Ú; is basically the same as y, and the training is just like previous methods, which guarantees enough amount of positive data for the initial training. Later, 33 starts to increase thus (33 also increases. ti begins to change and provides high-quality samples when the model is gradually easy to overfit to noise. With this schedule, the soft target y) can better adapt to the training.
Finally, the soft target label participates in the learning. For the classification branch, we utilize the KL divergence. For the regression branch, we choose the commonly used LI loss, and re-weight with the dynamic uncertainty quantification metric 1 - (B))
L (L(l)))
Lyeg (1 uz (3))
(5)
4.2 Promoting Cross-category Uncertainty
The above section aims to solve the problem by avoiding the hard label y:- However, besides the hard label y: the probability distribution p, is also problematic. P; is usually obtained with the

---

Table 1: Experimental Results on PASCAL VOC 2007 test for different methods, FS is the supervised model. We re-implement DD and CSD method based on the Faster RCNN.
Method
Labeled
Unlabeled
AP50.95
APso
AP7s
FS
VOCO7
43.1
76.9
43.1
DD
B1
VOCO7
VOC12
45.2
77.8
47.1
CSD
[2
VOC07
VOC12
45.0
77.5
46.7
STAC 33
VOCO7
VOC12
44.6
77.5
ours
VOCO7
VOC12
49.3
80.6
53.0
FS
VOC0712
52.9
83.9
57.6
DD
VOCO7
VOC12 + coco-20cls
46.6
79.0
49.0
CSD
VOC07
VOC12 + coco-20cls
45.5
77.4
47.8
STAC
VOCO7
VOC12 + coco-20cls
46.0
79.1
ours
VOC07
VOC12 + coco-20cls
50.2
81.4
54.2
softmax function, which restricts the summation of p; to This generates single-peak distribution Ps, where Pic. reaches its highest value. The entropy of such probability distribution is usually small, as the strong lateral inhibition effect softmax represses the cross-category uncertainty of pt. When region proposals uncertain, this kind of probability distribution also inappropriate. If e & of enlarging Pic: means a smaller Pict", finally resulting in overfitting to incorrect categories.
To avoid the lateral inhibition effect of softmax, we adopt sigmoid for outputting the probability distribution p;. With sigmoid function, the cross-category uncertainty is thus promoted and the multi-peak distribution p; is allowed. In this way, raising Pic, does not necessarily mean suppressing
Pic when C # ctt. The probability distribution p; turns more uncertain since its entropy is enlarged under sigmoid, which better adapts to the uncertain region proposals. We finally choose to use focal loss 21 to promote the cross- uncertainty, which adopts sigmoid for probability output.
Based on focal loss, we further introduce our constructed soft target y; into semi-supervises learning, Following [4]] we propose to use soft-target focal loss with y; as the soft target. In this way, uncertainty is introduced into both the target and the probability distribution, making the learning more noise-resistant.
Experiment
5.1
Experimental Setting
Dataset. We mainly conduct our proposed method on PASCAL VOC 6 and MS COCO 22 For simple notation, refer subset containing 35k images from COCO 2014 validation set as coco-35, the training set with 80k images as coco-80, and their union set as coco-115. The 120k images from COCO 2017 unlabeled set are denoted with coco-120. We also select images from COCO trainval consisting of only 20 categories as PASCAL VOC, and denote these 19,592 images coco-20cls. We mainly adopt four settings: 1) VOCO7 trainval (5,011 images) as labeled set and VOC12 trainval (1 1,540 images) as unlabeled set; 2) VOCO7 trainval as labeled set, VOC12 trainval and coco-20cls unlabeled set; 3) coco-35 35 as labeled set and coco-80 labeled set; 4) coco- 1eco-115 as labeled set and coco-120 as unlabeled set. Besides, we also follow previous methods for experiments with gradually increasing percentage of labeled examples.
Implementation Details. We conduct our experiment using Pytorch [29] and MMDetection [5]. Unless otherwise specified, we Faster RCNN [33) with ResNet50 [11] and FPN (20). The standard Ix schedule is adopted. The generation of pseudo labels and their filtration are basically the same as that in (II] For syper-parameters after the first decay of learning rate. C' is set to 15 and is set to 0.1.
set A to 0.5. A is set to 0.7 first, and changes to 0.8
5.2 Comparison with Existing Methods
PASCAL VOC. We perform the comparative study with Faster RCNN on the VOC dataset. The results are presented Tab. T The fully-supervised detection model trained on VOCO7 trainval obtains 43.1% 1% AP and 76.9% APso. When using the pseudo labels without consideration
7

---

Table 2: Experimental Results on COCO minival for different methods. FS is the supervised
model. We re- re-implement DD method based on the Faster RCNN. i denotes adopting strong data augmentation in training.
Method
Labeled
Unlabeled
AP50:95
AP
AP45
FS
coco-35
31.3
52.0
33.0
DD [31
coco-35
coco-80
33.1
53.3
35.4
ours
coco-35
coco-80
35,5
54.5
38.7
FS
coco- 115
37.4
58.1
40.4
DD
coco-1 15
coco-120
37.9
60.1
40.8
PL [38
coco-1 115
coco-120
38.4
59.7
41.7
CSD
coco-115
coco-120
38.9
STAC
37
coco-115
coco-120
39.2
Ubteacher
24
coco-115
coco-120
41.3
Humble teacher
coco-115
coco-120
42.4
Instant-teaching
coco-115
coco-120
40.2
ours (1x)
coco-115
coco-120
40.6
59.7
44.4
ours (3x)
coco-115
coco-120
41.7
61.0
45.9
ours (3x)
coco-115
coco-120
43.2
62.0
47.5
Table 3: Experimental Results on COCO minival with a gradually increasing percentage of
labeled examples. All methods except CSD adopt strong data augmentation in training.
Method
1% COCO
2% COCO
5% COCO
10% COCO
CSD
12
10.51 = 0.06
13.93 + 0.12
18.63 0.07
22.46 + 0.08
STAC
37
13.97 = 0.35
18.25 0.25
28.64 0.21
Ubteacher 24
17.84 = 0.12
21.98 +
26.30
29.64 + 0.10
Humble teacher 39
16.96 = 0.38
21.72
2.110
31.61 0.28
Instant-teaching
47
18.05 + 0.15
22.45
2.00
30.40 + 0.05
ours
18.41 + 0.10
24.00 0.15
28.96 + 0.29
32.43 0.20
of noise, the method (i.e. DD) obtains 45.2% AP and 77.8% APs0- Our method, in comparison, achieves a 49.3% AP and 80.6% AP30- The final AP outperforms the baseline method by more than 4% and also performs significantly better than other methods. This large margin strongly demonstrates the effectiveness of our method. Compared to other methods, performance metrics from A to APrs show consistent improvement, which proves that method better adapts to uncertain region proposals in semi-supervised learning and achieves a more noise-resistant training.
Further, augment the unlabeled dataset by adding images from coco-20cls. With more data available, the performance is further improved to 50.2% AP and 81.4% AP5o- This illustrates the usefulness of our proposed method on datasets of different distribution, which is usually inevitable problem when utilizing large-scale datasets. Also, this proves the potential of semi-supervised object detection to break through the upper bound of fully-supervised object detection.
MS COCO. We also evaluate our method on the COCO dataset, more challenging dataset, and the results are listed in Tab. 2 In the coco-35/80 setting, compared to the baseline method DD, our method achieves 2.4% AP improvement, which is prominent for the COCO dataset. In the coco- 115/10 setting, where a larger-scale dataset is adopted, we observe that previous methods obtain higher accuracy compared to that on PASCAL VOC, which indicates that more available data is beneficial semi-supervised learning. And we notice that approach also achieves better results. Even with a Ix schedule, our method still outperforms many current methods, which usually adopt a 3x schedule and more training time. We further adopt 3x schedule and the same data augmentation strategy as that [24] 39 47). Our method achieves a better result, a 43.2% AP, 0.8% more than the state-of-the-an which further indicates the superiority of our method.
To further demonstrate the effectiveness of our method, we follow the same setting as that previous methods 37
24
39 47 with different degrees of supervision using the COCO dataset. For fair
comparison, we adopt the same data augmentation strategy. The results are listed in Tab. 3 For unbiased teacher [4 which uses larger batch size and longer training schedules, retrain it under

---

Table 4: Ablation Study on VOCO7 test. CCU:
Table 5: Ablation Study on COCO. CCU:
cross-category uncertainty, UST: uncertainty- aware soft target.
cross- categryy uncertainty, UST: uncertainty- aware soft target.
Setting
CCU
UST
APs0:s
A
A/4s
Setting
CCU
UST
A
A
A/4s
43.1
76.9
43.1
31.3
52.0
33.0
fully
fully
/
42.5
77.2
41.6
31.2
52.5
32.9
45.2
77.8
47.1
33.1
53.3
35.4
46.2
79.4
47.9
34.3
55.3
36.4
scmi
semi
48.2
78.3
52.2
4
33.7
51.1
37.0
49.3
80.6
53.0
35.5
54.5
38.7
the common training schedules with the released official implementation. It noteworthy that our method outperforms CSD [2] and STAC [37 by a large margin. Compared to recent works such as unbiased teacher [24]. instant-teaching 47 and humble teacher [39]. our uncertainty-
noise- resistant learning can also consistently achieve better results under different degrees of supervised data. Particularly under the 2% supervision data setting, our method outperforms the instant settantteachhng by 1.6%. With 5% supervision data, our method outperforms humble teacher by 1.2%. This comparative study further validates the performance of our proposed method.
5.3 Ablation Study
We perform ablation study on VOC and COCO to analyze the impact of our designation. The results where we use VOC07/12 and COCO35/80 setting separately. We do
are listed in Tab. and Tab. 5 not adopt any data augmentation strategy in this section.
notice that the enhancement of promoting cross- category uncertainty is limited in the fully-sue setting. AP50:95 and AP7s even decrease a little. This is reasonable. In the fully-supervised setting, region proposals are certain, so suppressing the cross-category uncertainty is no problem. In addition. in two-stage detectors. the extreme class- imbalance problem mollified by RPN and random sampling, leaving little space for focal loss to improve by hard example mining, But in the semi-supervised setting, it is effective. By promoting cross-category uncertainty, the lateral inhibition effect is mitigated and the probability of the actual category is increased, which better accommodates the uncertain regions. It brings about 1.6% A.3o improvement. The more precise classification result even helps the localization, increasing A AP30:95 by 1.0% and APrs by 0.8%. From Tab. 5 a similar trend is also observed. This again demonstrates that promoting cross-category uncertainty is effective for semi-supervised detection.
Cross-category Uncertainty. From Tab.
Uncertainty-aware Soft target. From Tab. we observe that after adopting the uncertainty-aware soft target and optimizing with the KL divergence, the overall AP is improved by 3.0%. The localization ability is boosted even more - AAPrs increases by over 5%. This is because under the soft target, those certain regions contribute more the classification for positive proposals and bounding box regression. With the uncertainty quantification involved in the soft target, the network better avoids uncertain regions. After further combining with the focal loss, the performance further boosted, finally achieving 49.3% AP and 80.6% AP3o This illustrates that adopting uncertainty into both the probability distribution and classification target can further accommodate the uncertain region proposals. Experimental results from the COCO dataset in Tab. alsso verify its effectiveness.
Visualization of Noise Quantification. We perform illustrative analysis to explain the dynamic uncertainty quantification u(3). Specifically, we collect positive proposals that are assigned with the correct labels (clean ones) and the wrong labels (noisy ones). The probability distribution of their u(3) is plotted in Fig. [4] where Fig. 4a 4b ic the distribution in the early, middle, late stage separately. We observe that clean region proposals generally have lower u(3) than noisy ones, which demonstrates that our designed u(3) is reasonable for uncertainty quantification. From Fig. 4a we notice that most clean proposals and about half of noisy proposals possess small u(3). This ensures enough data for the model's 's initial training. As the training proceeds, the value of u(B) turns larger because of the dynamic 3, from the middle stage of Fig. 4bb to the late stage of Fig. 4c this time, almost all noisy samples possess close-to-1 u(3). As cost, u(3) for many clean proposals is also enlarged. But some of them are still preserved. This guarantees that the model learned with high-quality proposals when it is sensitive to noise.

---

0.7
0.7
0.7
0.8
clean
0.8
clean
0.8
clean
noisy ones
noisy ones
noisy ones
0.5
0.5-
0.5
probabiityy
probabiiity
0.4
04-
633
0.3-
0.3
0.2
0.1
0.1
0.0
0.6
0.0
0.0
((B)
u(B)
u(B)
(a)
(b)
(c)
Figure 4: Distribution of the uncertainty quantification u(3) on clean or noisy region proposals in the early stage of training (a), the middle stage (b) and the late stage (c). Clean regions' u(3) is usually smaller than noisy ones, and w(3) increases gradually as the training proceeds.
The choice of byper-parameters. We also evaluate method under different typer-parameter settings, including the value of C, 4y. The results are in the supplementary material. We find that our method is relatively robust the variance of ayper-parameters Therefore, even in new environment, it is not so hard to tune the syper-parameters.
6 Conclusion
In this paper, focus pseudo label based semi-supervised learning and target combating noise. By utilizing our proposed uncertainty quantification as the soft target and facilitating multi-peak probability distribution, we introduce uncertainty into semi-supervised learning. Experimental results on multiple semi-supervised object detection settings demonstrate that our proposed method can better adapt the uncertain regions, thus achieves the state-of-the-art results. the future, we will further explore more efficient approaches to utilize unlabeled data.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under Grant No. 61771288, Cross-Media Intelligent Technology Project of Beijing National Research Center for Information Science and Technology (BNRist) under Grant No. BNR2019TD01022 and the research fund under Grant No. 2019GQGO0R from the Institute for Guo Qiang, Tsinghua University.
References
[1] Devansh Arpit, Stanislaw K. Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron C Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In ICMI., 2017.
Yoshua Bengio. Olivier Delalleau, and Nicolas Le Roux. label propagation and quadratic criterion. 2006.
[3] Yuhang Cao, Kai Chen, Chen Change Loy, and Dahua Lin. Prime sample attention in object detection. In CVPR, 2020.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Sumi-supe learning. IEEE Transactions
on Neural Networks, 2009.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint
wrXiv:1906.07155 2019.
[6] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
[7] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty
in deep learning. In /CML, 2016.
[8] Ross Girshick. Fast r-cnn. In /CCV, 2015.
[9] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate
object detection and semantic segmentation. In CVPR, 2014,
[10] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama.
Co Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NewrIPS, 2018
10

---

[11] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR. 2016.
[12] Jisoo Jeong. Scungeui Lec. Jeesoo Kim, and Nojun Kwak. Consistency- baatee semi-supervised leamnng for object detection. In NeurlPS, 2019.
[13] Jisoo Jeong. Vikas Verma. Minsung Hyun, Juho Kannala, and Nojun Kwak. Interpolation- semi-
supervised learning for object detection. In CVPR, 2021.
[14] Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In ECCV, 2018.
[15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
[16] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICTR, 2017.
[17] Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NewrlPS, 2017.
[18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
[19] Dong-Hyun Lee. Pseudo-label: The simple and efficient suni-super learning method for deep neural networks. In JCMLW. 2013,
[20] Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, 2017.
[21] Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[22] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[23] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yan Fu, and Alexander Berg. Ssd: Single shot multibox detector. ECCV, 2016.
[24] Yen-Cheng Liu, Yhih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt
Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. In ICLR, 2021.
[25]
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In NeurIPS, 2018. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. TPAMI, 2018.
[26]
[27] Subhabrata Mukherjee and Ahmed Awadallah. Uncertainty-ave aware self self-training for few-shot text classifica-
tion. In NewrIPS, 2020.
[28] Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross- consistency training. In CVPR, 2020.
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NewrlPS, 2019.
[30] Jizong Peng. Guillermo Estrada, Marco Pedersoli, and Christian Desrosiers. Deep co-training for semi- supervised image segmentation. Pattern Recognition, 2020.
[31] lija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. In CVPR, 2018
[32] Joseph Redmon. Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
[33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[34] Mamshad Nayeem Rizve. Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense pseudo-
labeling: An uncertainty-an aware pseudo-label selection framework for semi-supervised learning. In ICLR, 2021.
[35] Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. In WACVMOTION' '05, 2005.
[36] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NewrIPS, 2020.
[37] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li. Han Zhang, Chen-Yu Yu Lee, and Tomas Pfister. A simple semi-supen suni-supereieed learning framework for object detection. arXiv preprint arXiv:2005 047557 2020.
[38] Peng Tang, Chetan Ramaiah, Ran Xu, and Caiming Xiong. Proposal learnigg for semi-supervised object detection. WACV, 2021.
[39] Yihe Tang. Weifeng Chen, Yijun Luo, and Yuting Zhang. Humble teachers teach better students for semi-supervise object detection. In CVPR, 2021.
[40] Antti Tarvainen and Harri Valpola. Mcan teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
[41] Dong Wang. Yuan Zhang, Kexin Zhang, and Liwei Wang. Focalmix: Semi-supervised learning for 3d medical image detection. In CVPR, 2020.
[42] He Wang, Yezhen Cong, Or Litanyy Yue Gao, and Leonidas J Guibas. 3dioumatch: Leveraging iou prediction for semi-supervised 3d object detection. In CVPR, 2021.
[43] Zhenyu Wang, Yali Li, Ye Guo, Lu Fang, and Shengjin Wang. Data-uncertainty guided multi-phase leamnig for semi-supervised object detection. In CVPR, 2021.
11

---

[44]
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, 1995.
[45]
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep
learning requires rethinking generalization. Communications the ACM, 2016.
[46] Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. IJCV, 2021.
[47] Qiang Zhou, Chaohui Yu, Zhibin Wang. Qian, and Hao Li. Instant-teaching: An end-to-end semi-
supervised object detection framework. In CVPR, 2021.
[48] Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation.
2002.
12


Reviews:
[{"review_id": "nqJtWurM3fc", "reviewer": "Reviewer_jnH3", "summary": "A method for semi-supervised 2D object detection is proposed. The main contribution is an uncertainty estimate of pseudo-labels that comes from a combination of classification score (s) \nand a localization score. The latter is estimated by the IoU score with the set of pseudo-labels and is normalized to values between 0 and 1 only in a certain range. The estimated uncertainty is then utilized in setting the soft target and penalizing the L1 regression loss. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* Experimental results are quite impressive\n* The analysis of different failure modes in section 3.1 is pretty interesting. \n\nWeaknesses - major:\n* A large effort is devoted to supervising uncertain pseudo-labels. I wonder how important that is. What if all uncertain labels were simply ignored (a-la fix-match)? \n* I'm missing a table with a gradually increasing percentage of labeled examples. This is quite common in semi-supervised detection. \n\nminor:\n* Please spell out the maximization of IoU with the set G. (max over all g in G, etc)\n* Writing could be improved. In particular, section 3 is very wordy and could be more concise. Same goes for e.g lines 247-253 where the idea is repeated using the same terminology instead of explaining from different views or simply explaining once. Also please rephrase sentences like 215 that are vague. \n* Confidence of localization was discussed before in [B] and then used in [C] for semi-supervised 3D detection. \n \nmissing references:\n[A] UNBIASED TEACHER FOR SEMI-SUPERVISED OBJECT DETECTION\n[B] Acquisition of Localization Confidence for Accurate Object Detection\n[C] 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "limitation could be further discussed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "n0qZzwONFd", "reviewer": "Reviewer_1uTd", "summary": "**Goal:**\nThis paper aims to address semi-supervised object detection, where the object is trained with a set of labeled images (fully-annotated box and object category) and a set of unlabeled images. \n\n**Method:**\n- The authors propose to use both classification score and IoU to pseudo-box to compute the uncertainty of each proposal boxes, and this is based on their analysis that assignment errors are large when the IoU is closer to 0.5. \n\n- Rather than using the hard label for the classification branch, they construct the soft labels by using uncertainty-based to weigh the ground-truth label. \n\n- The author also proposes to use focal loss and sigmoid to alleviate the overfitting issue of incorrect categories. \n\n**Experiments:**\n- They present the results on VOC and COCO datasets and show the improvement against CSD and STAC. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n- Analyses and categorization of errors in pseudo-labels are clear and reasonable, and, based on the observation, the authors further proposed to use IoU to construct the region uncertainty.\n\n**Cons:**\n- The authors did not mention and compare to the prior works, including Unbiased Teacher [1], Instant-Teaching [2], or Humble Teacher [3], as they show the significant improvement against CSD and STAC. \n\n- In the prior works of semi-supervised object detection, one important experimental benchmark is to examine the effectiveness of methods under different degrees of supervision using COCO-dataset (i.e., 1%-10% of COCO images as labeled and the remaining images as unlabeled set). This important experiment is missing in the paper, and this makes the proposed method hard to be compared with the existing works on semi-supervised object detection tasks. \n\n- As the proposed method has more hyper-parameters (such as C in Eq.1, beta and q in Eq. 4) and some hyper-parameter scheduling than the prior works, this makes the proposed method hard to be directly applied to new datasets or environments. These hyper-parameters might need to be tuned for the new datasets. \n\n\n[1] \"Unbiased Teacher for Semi-Supervised Object Detection\", ICLR 2021\n\n[2] \"Instant-Teaching: An End-to-End Semi-Supervised Object Detection Framework\", CVPR 2021\n\n[3] \"Humble Teachers Teach Better Students for Semi-Supervised Object Detection\", CVPR 2021", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not mention the potential negative societal impact in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "Yh8gE0znIGp", "reviewer": "Reviewer_123Z", "summary": "This submission addresses the semi-supervised object detection task. Compared to previous methods, the authors propose to quantify the uncertainty degree and apply it into the model training. Besides, the authors modify the classification error to allow multi-peak in probability distribution from soft label targets. The experiments on four settings of semi-supervised object detection show the model achieves good performances. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n* The motivation and technical novelties are clearly explained. The design of the uncertainty from noisy labels is reasonable and can be easily plugged into other two-stage semi-supervised object detectors' training\n\n* The authors provide full details of hyper-parameter ablation study in both submission and supplementary materials, which is much appreciated. \n\n### Weakness\n* The novelty is incremental in engineering. The design of noisy labels' uncertainty can be only applied into the scenario of two-stage object detectors' training. For other one-stage detectors or anchor-free detectors, there may be no $\\Delta_b$ calculated from IOU, thus limited the scope of the novelty. \n\n* The experiment results are not significant to show the effectiveness of the design. The compared the method is not the state-of-the-art performance. Before the NeuRIPS submission deadline, there is an ICLR 2021 paper [R1]. The reported performance on Table 1 VOC+coco-20cls is 50.34 $AP_{50:95}$, which is higher than the proposed model's performance. In Table 2, the reported performance on coco-115 + coco-120 is 41.3 $AP_{50:95}$, with only 0.4 decrease compared to that from the proposed model. \n\n[R1]: Unbiased Teacher For Semi-supervised Object Detection, ICLR 2021. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please check the \"weakness\" part mentioned above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "JBIJ8XbFmvU", "reviewer": "Reviewer_jvYC", "summary": "This paper tackles semi-supervised learning for object detection, where only a small amount of the training data is labeled with bounding boxes, while the rest of the training data is unlabeled. To solve this problem, the authors present a region uncertainty quantification method that quantifies the uncertainty of regions in unlabeled images based on their sensitivity to noisy pseudo labels. In particular, a quantification metric for the uncertainty is proposed for positive region proposals according to the classification error and the assignment error. Experiments are performed on both PASCAL VOC and MS COCO, with a comparison to three competitors: DD, CSD, STAC. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper an interesting and practically important problem in semi-supervised object detection. The materials in this paper are overall well-orgnanized and well-presented. Weakness regarding the methodology and experiments are summarized in the following.\n\n**Method**\n\nAlthough the paper addresses an interesting and practically important problem in semi-supervised object detection, the writing could be improved. In particular, the method section is hard to follow, several concerns are listed below.\n\n1. The self-confidence score is estimated by averaging probability scores from different data-augmented images. However, such estimation is not directly related to data distillation in [25]. Distillation is not related to how the confidence score derives in the paper. It is therefore suggested not to mention data distillation in line 155, but instead explain why one should averaging multiple augmented images to derive the confidence score. \n2. The definition of Eq. (1) is unclear. Why does it quantify the assignment errors in region proposals?\n3. The motivation of Eq. (2) is also unclear. Why should the self-confidence scores multiply the uncertainty of assignment I? \n4. It is unclear how the focal loss in [18] helps to promote cross-category uncertainty. It is suggested to write down the equation and analyze how the soft targets are used in the focal loss.  \n\n**Experiments**\nThe experiments are well designed to show the effectiveness on different datasets, under different use of the unlabeled data. In the following, several suggestions are given.\n1. The evaluation of the competitors (DD, CSD, STAC) is unclear. It is suggested to add a paragraph to describe these competitors in Section 5.1. In particular, a discussion about the difference of the proposed method as compared to the competitors is suggested.\n2. In the ablation study, it is suggested to analyze the necessity of bringing the different uncertainty estimators (1) uncertainty on classification vs. (2) uncertainty on assignment. It is unclear how these estimators contribute to the model. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Discussion about limitations and possible negative societal impacts are not provided in the paper. It is suggested to add an individual section at the end of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
 As mentioned by multiple reviewers, the ideas in this work and especially analysis of errors/failure modes were found to be interesting and novel. However, the reviewers raised significant legitimate empirical and experimental deficiencies that were surprising. This includes lack of standard experimental settings (e.g. varying amount of labels from 1-10%) and comparisons to recent state of art that were published before the NeurIPS deadline (Unbiased Teacher, Instant-Teaching, and Humble Teacher). Such adherence to standard practices and fair comparisons to related work are necessary to move the field forward and should have been included initially. 

  During the rebuttal the authors provided these results and comparisons and addressed the empirical concerns. The authors should add these results, in addition to descriptions of how the proposed method differs from ideas presented in those papers. 

  Unfortunately, this did not leave much room for discussion of other aspects of the work. For example, there is a question of novelty (mentioned by 123Z) and furthermore there are several uncertainty-based pseudo-labeling methods (in the context of classification or even segmentation, e.g. [A, B]) that have come out that are not even discussed. The authors should clearly address this and discuss how their methods differ (besides utilizing a similar idea in a new task, object detection). 

  Finally, several reviewers pointed out several specific writing improvements that should be addressed. These should be included in the final version.
  
  Overall, the paper did provide interesting analysis and addressed the experimental concerns, and so can be accepted. However, all of the above must be addressed.

[A] In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning, ICLR 2021.
[B] Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation, IJCV 2021.
Decision: Accept (Poster)