Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper focuses on the calibration of both two-stage and one-stage object detectors. Moreover, the author conducts studies on in-/out-domain distribution.
Specifically, an auxiliary loss function (TCD), which is based on the distance between class accuracy and bounding box IOU, is introduced.
In addition, a quantification mechanism is proposed for self-training of the detectors.
strengths_and_weaknesses: [Strengths]
1. The proposed method is simple and reasonable.
2. The paper is clearly presented and well-organized.
3. The proposed method outperforms all baselines.

[Weaknesses]
In Table 1 and 2, it raises my attention that post-hoc perform worse than all the rest including the baselines for both in- and out-domain.
Does that suggest post-hoc is not suitable for detection tasks for some reasons?
questions: see Weaknesses
limitations: see Weaknesses
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose two methods to improve the calibration of confidence scores of object detection models based on deep neural networks. A first method, TCD, introduces a specific cost function that aims to constrain the model to reduce the difference between the predicted confidence and the prediction accuracy (d_cls) and to reduce the difference between the confidence and the IoU of the detected regions compared to the ground truth (d_det).
This cost function is used for supervised training of the models. A second method, ICT, aims to calibrate the predictions on a set of out-of-domain tests in an unsupervised (or self-supervised) manner. This method uses a MC-Dropout to generate multiple hypotheses and calculates a combined variance on the confidence, position and aspect ratio of predictions of the same classes among the hypotheses. This combined variance is then used to generate pseudo-targets for self-supervised adaptation. 

These methods are evaluated on 5 object detection databases, using different detection models (FCOS, Faster-RCNN, EPM, SSAL, SWDA). The experiments show that the proposed methods improve both the detection expected calibration error (D-ECE) and the accuracy of the models, on all the bases and for all the models tested. An ablation study completes the evaluation. 

strengths_and_weaknesses: The paper is well written, with clear description of the methods and very thorough experiments. The results are convincing. It is shown that the proposed methods can be applied to different types of models and on different databases.

However, the article has a small weakness in the comparison with other calibration methods: only the simple temperature scaling method is used as a basis for comparison. More complex methods could have been compared.  However, given the space constraints, it seems difficult to hold this against the authors.


questions: * Fig1: explain what is dark red and light red
* L125 avoid foot note in equation, could be read as exponent 1[IoU(bˆ, b∗) ≥ γ2]1[yˆ = y∗].  
* Eq 7: sl,r[k]  is not defined
* L196: J is not defined
limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new well-calibrated detection algorithm for both in-domain and out-of-domain scenarios. The author brought the concepts from calibration for classification into detection, and proposed a calibration loss (TCD) for object detection by considering both classvcaition and localisation predictions. In addition, a uncertainty quantification mechanism is proposed to further boost the performance. The proposed method shows clear improvement over the baselines, and the experiments are comprehensive.
strengths_and_weaknesses: Pros:
- The idea of using the concepts of calibration for detection is important and novel to me.
- The paper is easy to understand, and the experiments are very comprehensive. 
- The results are competitive and show clear improvements over baselines.

Cons:
- For Table 4 R4-6 C2-3, I feel a bit strange that both TCD and ICT can reduce the D-ECE score, however, when combining them together, the overall AP reduces with even lower D-ECE value. The similar results also appear in Table 2 R4-6 C8-9 (Supp), where the TCD variant is even worse than the baselines under AP metric, while still it achieves lower D-ECE score.
- Again For Table 2 (Supp), the results seem not promising when applying the proposed module in transformer-based detectors. The generalisation of the proposed modules becomes questionable.
- I learn the method can improve both in-domain and out-of-domain performance of object detection, so I wonder can we apply the proposed module into the existing SOTA detectors to reach a higher performance.
- Though it's commonly used in ood scenario, I am curious the performance of applying COCO trained detector into VOC datasets directly.

Overall speaking, I think it is a good paper.

questions: Please refer "Strengths And Weaknesses"
limitations: I do not see any potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This works identifies the calibration in object detection under domain shift. The author proposes a new plug-and-play loss formulation, termed as train-time calibration for detection (TCD). It can be used in some task-specific loss functions during training phase and acts as a regularization for detections. 

strengths_and_weaknesses: - The author uses mathematical deduction to analyze the domain shift in detection. The Calibration including TCD, Quantifying and uncertainty-guided soft pseudo-targets can easily be plugged into many detection frameworks.
questions: - Ablation study of the work in some well-known datasets like COCO and VOC.
limitations: - Ablation study of the work in some well-known datasets like COCO and VOC.
- Not very practical for general object detection, although it has some theoretical analysis.
review_title: null

Meta-review: metareview: This paper proposes a calibration loss function and uncertainty quantification mechanism to improve object detection calibration for both in-domain and out-of-domain detections. The reviewers have found this paper novel and clearly written with strong empirical results. Given this, I am happy to recommend this paper for acceptance at NeurIPS. 
confidence: Certain
Decision: Accept