Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: A method for semi-supervised 2D object detection is proposed. The main contribution is an uncertainty estimate of pseudo-labels that comes from a combination of classification score (s) 
and a localization score. The latter is estimated by the IoU score with the set of pseudo-labels and is normalized to values between 0 and 1 only in a certain range. The estimated uncertainty is then utilized in setting the soft target and penalizing the L1 regression loss. 
main_review: Strengths:
* Experimental results are quite impressive
* The analysis of different failure modes in section 3.1 is pretty interesting. 

Weaknesses - major:
* A large effort is devoted to supervising uncertain pseudo-labels. I wonder how important that is. What if all uncertain labels were simply ignored (a-la fix-match)? 
* I'm missing a table with a gradually increasing percentage of labeled examples. This is quite common in semi-supervised detection. 

minor:
* Please spell out the maximization of IoU with the set G. (max over all g in G, etc)
* Writing could be improved. In particular, section 3 is very wordy and could be more concise. Same goes for e.g lines 247-253 where the idea is repeated using the same terminology instead of explaining from different views or simply explaining once. Also please rephrase sentences like 215 that are vague. 
* Confidence of localization was discussed before in [B] and then used in [C] for semi-supervised 3D detection. 
 
missing references:
[A] UNBIASED TEACHER FOR SEMI-SUPERVISED OBJECT DETECTION
[B] Acquisition of Localization Confidence for Accurate Object Detection
[C] 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection
limitations_and_societal_impact: limitation could be further discussed
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: **Goal:**
This paper aims to address semi-supervised object detection, where the object is trained with a set of labeled images (fully-annotated box and object category) and a set of unlabeled images. 

**Method:**
- The authors propose to use both classification score and IoU to pseudo-box to compute the uncertainty of each proposal boxes, and this is based on their analysis that assignment errors are large when the IoU is closer to 0.5. 

- Rather than using the hard label for the classification branch, they construct the soft labels by using uncertainty-based to weigh the ground-truth label. 

- The author also proposes to use focal loss and sigmoid to alleviate the overfitting issue of incorrect categories. 

**Experiments:**
- They present the results on VOC and COCO datasets and show the improvement against CSD and STAC. 

main_review: **Pros:**
- Analyses and categorization of errors in pseudo-labels are clear and reasonable, and, based on the observation, the authors further proposed to use IoU to construct the region uncertainty.

**Cons:**
- The authors did not mention and compare to the prior works, including Unbiased Teacher [1], Instant-Teaching [2], or Humble Teacher [3], as they show the significant improvement against CSD and STAC. 

- In the prior works of semi-supervised object detection, one important experimental benchmark is to examine the effectiveness of methods under different degrees of supervision using COCO-dataset (i.e., 1%-10% of COCO images as labeled and the remaining images as unlabeled set). This important experiment is missing in the paper, and this makes the proposed method hard to be compared with the existing works on semi-supervised object detection tasks. 

- As the proposed method has more hyper-parameters (such as C in Eq.1, beta and q in Eq. 4) and some hyper-parameter scheduling than the prior works, this makes the proposed method hard to be directly applied to new datasets or environments. These hyper-parameters might need to be tuned for the new datasets. 


[1] "Unbiased Teacher for Semi-Supervised Object Detection", ICLR 2021

[2] "Instant-Teaching: An End-to-End Semi-Supervised Object Detection Framework", CVPR 2021

[3] "Humble Teachers Teach Better Students for Semi-Supervised Object Detection", CVPR 2021
limitations_and_societal_impact: The authors did not mention the potential negative societal impact in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles semi-supervised learning for object detection, where only a small amount of the training data is labeled with bounding boxes, while the rest of the training data is unlabeled. To solve this problem, the authors present a region uncertainty quantification method that quantifies the uncertainty of regions in unlabeled images based on their sensitivity to noisy pseudo labels. In particular, a quantification metric for the uncertainty is proposed for positive region proposals according to the classification error and the assignment error. Experiments are performed on both PASCAL VOC and MS COCO, with a comparison to three competitors: DD, CSD, STAC. 
main_review: The paper an interesting and practically important problem in semi-supervised object detection. The materials in this paper are overall well-orgnanized and well-presented. Weakness regarding the methodology and experiments are summarized in the following.

**Method**

Although the paper addresses an interesting and practically important problem in semi-supervised object detection, the writing could be improved. In particular, the method section is hard to follow, several concerns are listed below.

1. The self-confidence score is estimated by averaging probability scores from different data-augmented images. However, such estimation is not directly related to data distillation in [25]. Distillation is not related to how the confidence score derives in the paper. It is therefore suggested not to mention data distillation in line 155, but instead explain why one should averaging multiple augmented images to derive the confidence score. 
2. The definition of Eq. (1) is unclear. Why does it quantify the assignment errors in region proposals?
3. The motivation of Eq. (2) is also unclear. Why should the self-confidence scores multiply the uncertainty of assignment I? 
4. It is unclear how the focal loss in [18] helps to promote cross-category uncertainty. It is suggested to write down the equation and analyze how the soft targets are used in the focal loss.  

**Experiments**
The experiments are well designed to show the effectiveness on different datasets, under different use of the unlabeled data. In the following, several suggestions are given.
1. The evaluation of the competitors (DD, CSD, STAC) is unclear. It is suggested to add a paragraph to describe these competitors in Section 5.1. In particular, a discussion about the difference of the proposed method as compared to the competitors is suggested.
2. In the ablation study, it is suggested to analyze the necessity of bringing the different uncertainty estimators (1) uncertainty on classification vs. (2) uncertainty on assignment. It is unclear how these estimators contribute to the model. 


limitations_and_societal_impact: Discussion about limitations and possible negative societal impacts are not provided in the paper. It is suggested to add an individual section at the end of the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This submission addresses the semi-supervised object detection task. Compared to previous methods, the authors propose to quantify the uncertainty degree and apply it into the model training. Besides, the authors modify the classification error to allow multi-peak in probability distribution from soft label targets. The experiments on four settings of semi-supervised object detection show the model achieves good performances. 
main_review: ### Strengths
* The motivation and technical novelties are clearly explained. The design of the uncertainty from noisy labels is reasonable and can be easily plugged into other two-stage semi-supervised object detectors' training

* The authors provide full details of hyper-parameter ablation study in both submission and supplementary materials, which is much appreciated. 

### Weakness
* The novelty is incremental in engineering. The design of noisy labels' uncertainty can be only applied into the scenario of two-stage object detectors' training. For other one-stage detectors or anchor-free detectors, there may be no $\Delta_b$ calculated from IOU, thus limited the scope of the novelty. 

* The experiment results are not significant to show the effectiveness of the design. The compared the method is not the state-of-the-art performance. Before the NeuRIPS submission deadline, there is an ICLR 2021 paper [R1]. The reported performance on Table 1 VOC+coco-20cls is 50.34 $AP_{50:95}$, which is higher than the proposed model's performance. In Table 2, the reported performance on coco-115 + coco-120 is 41.3 $AP_{50:95}$, with only 0.4 decrease compared to that from the proposed model. 

[R1]: Unbiased Teacher For Semi-supervised Object Detection, ICLR 2021. 
limitations_and_societal_impact: Please check the "weakness" part mentioned above
review_title: null

Meta-review: 
 As mentioned by multiple reviewers, the ideas in this work and especially analysis of errors/failure modes were found to be interesting and novel. However, the reviewers raised significant legitimate empirical and experimental deficiencies that were surprising. This includes lack of standard experimental settings (e.g. varying amount of labels from 1-10%) and comparisons to recent state of art that were published before the NeurIPS deadline (Unbiased Teacher, Instant-Teaching, and Humble Teacher). Such adherence to standard practices and fair comparisons to related work are necessary to move the field forward and should have been included initially. 

  During the rebuttal the authors provided these results and comparisons and addressed the empirical concerns. The authors should add these results, in addition to descriptions of how the proposed method differs from ideas presented in those papers. 

  Unfortunately, this did not leave much room for discussion of other aspects of the work. For example, there is a question of novelty (mentioned by 123Z) and furthermore there are several uncertainty-based pseudo-labeling methods (in the context of classification or even segmentation, e.g. [A, B]) that have come out that are not even discussed. The authors should clearly address this and discuss how their methods differ (besides utilizing a similar idea in a new task, object detection). 

  Finally, several reviewers pointed out several specific writing improvements that should be addressed. These should be included in the final version.
  
  Overall, the paper did provide interesting analysis and addressed the experimental concerns, and so can be accepted. However, all of the above must be addressed.

[A] In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning, ICLR 2021.
[B] Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation, IJCV 2021.
Decision: Accept (Poster)