Towards Improving Calibration in Object Detection Under Domain Shift
Authors: Muhammad Akhtar Munir
Muhammad Haris Khan
M. Saquib Sarfraz
Mohsen Ali
Venue: NeurIPS 2022 Accept (2022)
Keywords: Domain Shift
Uncertanity
Calibration
Object Detection
Out-of-domain calibration

Abstract: With deep neural network based solution more readily being incorporated in real-world applications, it has been pressing requirement that predictions by such models, especially in safety-critical environments, be  highly accurate and well-calibrated. Although some techniques addressing DNN calibration have been proposed, they are only limited to visual classification applications and in-domain predictions. Unfortunately, very little to no attention is paid towards addressing calibration of DNN-based visual object detectors, that occupy similar space and importance in many decision making systems as their visual classification counterparts. In this work, we study the calibration of DNN-based object detection models, particularly under domain shift. To this end, we first propose a new, plug-and-play, train-time calibration loss for object detection (coined as TCD). It can be used with various application-specific loss functions as an auxiliary loss function to improve detection calibration. Second, we devise a new implicit technique for improving calibration in self-training based domain adaptive detectors, featuring a new uncertainty quantification mechanism for object detection. We demonstrate TCD is capable of enhancing calibration with notable margins (1) across different DNN-based object detection paradigms both in in-domain and out-of-domain predictions, and (2) in different domain-adaptive detectors across challenging adaptation scenarios. Finally, we empirically show that our implicit calibration technique can be used in tandem with TCD during adaptation to further boost calibration in diverse domain shift scenarios.

Towards Improving Calibration in Object Detection Under Domain Shift
Muhammad Akhtar Munir Muhammad Haris Khan?, M. Saquib Sarfraz Mohsen Ali Information Technology University of Punjab, Mohaaedd bin Zayed University of Artificial Intelligence, Karlsruhe Institute of Technology. "Mercedes-eene Tech Innovation
Abstract
With deep neural network based solution more readily being incorporated in real- world applications, it has been pressing requirement that predictions by such models, especially in safety-critical environments, be highly accurate and well- calibrated. Although some techniques addressing DNN calibration have been proposed, they are only limited to visual classification applications and in-domain predictions. Unfortunately, very little to no attention is paid towards addressing calibration of ANN-based visual object detectors, that occupy similar space and importance in many decision making systems as their visual classification coun- terparts. this work, we study the calibration of DNN-based object detection models, particularly under domain shift. To this end, we first propose new, blug-and-play train-time calibration loss for object detection (coined as TCD). It auxiliary loss
be used with various application-speeific loss functions function to improve detection calibration. Second, we devise a new implicit tech- nique for improving calibration in self-training based domain adaptive detectors, featuring a new uncertainty quantification mechanism for object detection. We demonstrate TCD is capable of enhancing calibration with notable margins (1) across different DNN-based object detection paradigms both in in-domain and out-of-domain predictions, and (2) in different domain-adaptir challenging adaptation scenarios. Finally, we empirically show that our implicit calibration technique can be used in tandem with TCD during adaptation to further boost calibration in diverse domain shift scenarios.
detectors across
Introduction
Owing the success of deep neural network in last decade, they have been progressively becoming part of different safety-critical applications, including autonomous driving [5). healthcare [2]. and legal research [40] In such high-stake applications, is of paramount importance that the model predictions are not only correct, but also well-calibrated. For a calibrated model, probability of being correct across all confidence levels and the predictions-coffidence should be aligned 66 35 It is desired that the incorrect predictions have low-confidence, however, a poorly calibrated model is prone to have confident but incorrect predictions.
We have witnessed tremendous effort towards improving the predictive accuracy of models, however, considerably less attention is devoted to model calibration. Among few works addressing model calibration, the majority of them focus on classification tasks [6] 17
35 A dominant class of methods address calibration by proposing different post-hoc approaches 28 6). The core idea to transform model outputs by a single parameter which is optimized on a validation set, with
20
22
*Corresponding author, Intelligent Machines Lab, Department of Computer Science, Information Tech- nology University of the Punjab, Lahore, Pakistan. Email: akhtar a.httarmunir@i........... cdu. pk Project Page: http://im. itu, edu, wtw.aaaaww--mpooooooooaaeeeeeeee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

an objective to improve the calibration of in-domain predictions. Such methods involve limited parameters while calibrating the outputs of the models. Further, they are restrictive, since in many real-world scenarios, a validation set is not always available. To involve all model parameters, some methods propose train-time calibration techniques [17 20 22 by constraining that for a given sample, the predicted class confidence and likelihood for that class should be minimized. The model could remain poorly calibrated for the classes with non non-maximum prediction confidences 77].
Surprisingly, little to no attention has been paid towards addressing the calibration of deep learning based visual object detection methods, that form an important part of many decision making systems. Also, most current efforts aim at improving model calibrations only for in-domain predictions. However. in many real-world scenarios, owing to domain shift, the distribution of the data received by deployed model can be very different to the distribution of its training data. Therefore for several practical scenarios, a model should be well-calibrated for both in-domain and out-of-domain predictions. Besides carrying scientific value, well-calibrated object detectors, especially under domain drift, will substantially improve overall trust many vision-based safety- applications and will be of great value to industry practitioners.
In this paper, we study the calibration of object detection models for both in-domain and out-of- domain detections. We observe that: (1) detection models demonstrate poor calibration for in-domain and out-of-domain detections (see Fig. T. and (2) unsupervised domain adaptive detection models are rather miscalibrated when compared to their predictive accuracy in target domain.
Towards developing well-calibrated object detection models, for both in-domain and out-of-domain scenarios, we propose a new plug-and-play loss formulation, termed as train-time calibration for detection (TCD). It can be used with task-specific loss functions during training phase and acts as a regularization for detections. In addition, we develop an implicit calibration technique for self-training based domain adaptive detectors. Finally, we empirically show that this technique is complementary to our loss function and they both can be utilized during adaptation to further boost calibration under challenging domain shift detection scenarios. We validate the effectiveness of our loss function towards improving calibration of different DNN-based object detection paradigms and different domain adaptive detection models under challenging domain shift scenarios.
2
Related Work
Below we survey different techniques in literature aimed at addressing the calibration of deep neural networks. Broadly, the existing calibration techniques can be divided into post-hoc and train- time calibration methods. The former estimates a transformation through a hold-out data after training, whereas the latter involves model parameters during the training. Further, there are methods that achieve implicit calibration through using model's uncertainty or learning to reject OOD samples.
Post-hoc calibration methods: Post-hoe calibration methods are post processing methods that leverage hold-out validation data to optimize the calibration parameters for transforming the outputs of trained deep neural networks. A popular post-hoc calibration approach is temperature scaling (TS) [6] (an extension of Platt Scaling (28) that re-scales the logits by an optimal temperature (T") parameter which is estimated on a validation set. The overall accuracy of the model remains unaffected by temperature scaling, but the logits values are modulated in such way that confidence scores are reduced. Temperature scaling along with other methods like vector scaling and matrix scaling extends Platt scaling to multiclass setting. [II introduced bin wise temperature scaling that incorporates different temperatures for different confidence intervals. To mitigate miscalibration under long-tailed distribution scenarios, Islam et al. [10] used class frequency information. Specifically, the model is calibrated using class distribution-awan aware temperature scaling and label smoothing. The Dirichlet calibration (DC) [15] is derived from Dirichlet distributions and extends the beta calibration method 16 to multi-class setting. This is achieved by training an extra neural network layer, that inputs log-transformed class probabilities, over the hold-out validation data. Recently, (35] generalizes existing post-hoc calibration methods, to address the poor calibration of out-of-domain predictions, by applying post-hoc calibration step after transforming validation set. In general, post-hoc methods for model calibration require hold-out validation set. Also, the majority of post-hoc methods perfrom model calibration for in-domain predictions.
Train-time Calibration methods: Negative log-likelihood (NL.)) the most popular approach to train a DNN based classifier. Recent work shows that NLL-based training results in overconfident
2

---

Expected Calibration Error (ECE)
20
In-domain Detections
Out-of-domain Detections
10
5
Ours (TCD)
Decection
Ours (TCD)
(TCD)
loss
Sim10k
KITTI
8DD
(a) Calibration performance (ECE) of DNN-based detector (FCOS 34 loss and method (task-speeific detection loss+TCD), where TCD the proposed auxiliary loss.
trained using task-specific detection
Sim10k 22 C5
Poggycs
ETT
CS EcoL00
Actual
Actual
Actual
Oap
Gap
Gap
Pectzin
Confidence
Confidence
Confidence
Conficence
Sim10k to CS
CS FaggyCS
KTTT CS
CS BCD100K
Actusl
Actusl
Achual
Achuai Gop
Gap
Gop
Gop
P1032
Cantiderre
Cariitenre
Confiteore
Confiterre
(b) Reliability diagrams. Top row: DNN-based detector (FCOS 34 trained using task-specific loss. Bottom
row: Ours, trained with adding the proposed TCD loss.
Figure 1: DNN-based object detectors trained with our proposed calibration loss results better calibration.
scores 66). To this end, various methods presented additional loss formulations that can be used with task-specific losses during the training procedure. For instance, (27] penalized overconfident predictions by adding the negative of entropy to the loss function. Similarly, Liang et al. [20] improved calibration by proposing the difference of accuracy and confidence as an auxiliary loss to the cross-entropy loss. Whereas, [17] proposed an auxiliary loss, for obtaining improved calibration, which is quantified with the help of reproducing kernel in a Hilbert space. Recently, (77] developed an auxiliary loss formulation for calibrating non- preppreddeted labels along with the predicted one. It has been shown that methods which attempt to reduce overconfidence by modifying the hard-labels can achieve better model calibration. Mukhoti et. al. (23), hypothesized that minimizing focal loss 21 increases the predicted distribution's ss entropy while trying to minimize KI divergence between the predicted and ground-truth distribution. Replacing eross entropy with Focal loss results in well-calibrated classification model. (24) empirically showed that label smoothing (LS), originally proposed by [33], helps in improving calibration. [14 calibrated uncertainty by relating accuracy and uncertainty, that yields accurate predictions to be more certain while inaccurate predictions to be more uncertain.
Other relevant methods: A recent work [37] based on empirical results proposed a unified framework that includes training and post-hoe calibration for better calibration. Hein et al. [8] identified that ReLU activation (or similar piece-wise linear functions) is core reason behind the overconfident predictions of DNNs far away from the training data. To overcome this, the data augmentation incorporated using adversarial learning. [3] detected OOD samples by performing spectral analysis over initial CNN layers, resulting in much better calibrated model. Finally, some works report implicitly calibrating DNNs via some loss function 24 or selecting pseudo-labels during self-training based model's uncertainty 25
Note that, almost all aforementioned techniques for analyzing and improving DNN calibration focus on classification tasks. There is little to no literature studying the calibration of DNN-based object

---

detectors, including under domain shift. Toward this aim, inspired by the train-time calibration methods, we propose a new auxiliary loss formulation (TCD) for DNN- based object detection calibration that can be integrated with any task-specific loss function while training. It is capable of significantly calibrating in-domain and out-of-domain object detections. an implicit calibration technique, that is also complementary to our loss formulation, to boost the calibration self-training based domain adaptive object detectors.
addition, we develop
Improving Calibration in Object Detectors
3.1 Definitions of Calibration
Calibration for classification: Let D = ((x,, y1 )) be a dataset N image and ground-truth pairs from a joint distribution D(X, Where x; E RBxx xc an image, and yf E E = =,=1,2 ...,. K. is the associated ground-truth class label. H, W, and C represent the width, height and the number of channels of an image respectively. For a classification model Fas, that predicts a label ğ with confidence score s, can define perfect calibration [
PPÚ = y"18
Vs [0.
(1)
Where PÝÿ = y'Ia = s) s)
is the accuracy for a particular confidence score 8. For the model to be calibrated, this accuracy should match with the predicted confidence.
Calibration for object detection: For training an object detection model, the ground-truth an- notations include object's localization information along with their associated categories. Let b' E B [0, 1]$ be the bounding box annotation of the object and y" be the corresponding class label. We assume that an object detection model Fdet predicts the object location and the class label ý with confidence s. This prediction is accurate, if the Intersection-over-Unnee
(loU) between b and b" is larger than some threshold 7, and ý is same as y". i.c. [l, b*) 2 >) = y²]. Then, for object detection, perfect calibration can be expressed [19
IPU 1/$ = s) =s Vs E [0, 1].
Where U denotes a correctly classified prediction.
Expected calibration error for classification and object detection: For a classification model, the absolute difference between the confidence score and the accuracy corresponding to that confidence score (eq. (2) is considered Calibration Error. The miscalibration of the model is quantified by computing the expectation of calibration error over predicted confidence [26 18.
Eg [[ = - s) =s]
(3)
The continuous confidence space of & is divided into M equally spaced bins to approximate ECE:
M
[(mm)
ECE
Σ
aacc(m) con]
D
where (mm) is the set of samples in m bin, and [D] is the total number of samples. acc(m) and conf(m) denote the average accuracy and average confidence mh bin, respectively, Along similar lines, ECE for the object detection (D-ECE) can be defined as the expected deviation of precision from the predicted confidence [9]
Eg [P(U 1|$ s) s|]
(5)
As in classification ECE, confidence space is divided into M equally spaced bins to approximate D-ECE: Note that, () also be extended location-dependent calibration.

---

[[m)
ECE Σ
prec(m) - conf(m)),
(6)
where prec(m) denotes the average precision in a bin.
3.2 Train-time Calibration for Detection: TCD
this section we detail our proposed train-time calibration mechanism for object detection. It features novel auxiliary loss function that involves both the classification and localization components. DNN- based object detectors predict a bounding box and corresponding class confidences for a detected region. In order to achieve detection calibration, it is imperative to take into account both of them. Our core idea is to jointly calibrate the estimated (class cclass-wise) confidences and predicted bounding boxes. To achieve this, propose compute (1) the difference between the classification accuracy and the confidence, and (2) the deviation between the predicted bounding box overlap and the predicted class confidence. Specifically, there are two components in our loss formulation. Inspired by the train-time calibration techniques for classification, we adapt the confidence calibration loss 77 and develop the first component dels- measures the absolute difference between the average confidence and average accuracy.
quantities
mini-batch during training:
K Σ
L ΣV s
L
R Drr [A]
dels
(7)
K L
LXR
1=1==1
1=1
Where R is the number of locations in the output class confidences channel map and L. is number of images in mini-batch. [k-[k]] = label k is the ground-truth label for sample location and 0 otherwise. st [k] denotes kes. class confidence for Tth location in sample t. The second component daet computes the mean of the absolute difference between the bounding box overlap (with the ground-truth/pseude ground-truth) and the confidence of the predicted class over all the positive regions. Let No denotes the number positive regions the sample 1. The loss is defined
pos
NZ pus
deet
DoU(bn bū) 8n]
(8)
NI pos
1 (dels ddet)
LTCD
(9)
2
The dets component in the proposed loss capable of not only calibrating the confidence of the predicted class but also for the non -oeddccedt classes. It penalizes the model, if for given class k. the average confidence across mini-batcl samples and possible output locations deviates from the average occurrence across mini-batch of this class. On the other hand, the dast component penalizes the deviation between the loU score (computed between the ground-truth bounding box and predicted one) and its corresponding predicted confidence for positive regions. It explicitly forces the object detector to match the confidence, of the predicted class, with the tightness of the predicted bounding-bie box over the detected object. Since above components not based on binning, like Eqs. 46.). as such, avoids the non-differentibiliy issue
Note that, both loss components, ddes and dels, operate over the mini-batch constructed during training and so the LTCD can be used an auxiliary train-time calibration loss for calibrating object detectors, including domain- domain-adaptive ones, in conjunction with various task-specific loss functions (secs. 4.1 4.2). Further, we show that proposed loss formulation can also be used with the pseudo- labels used in self-training based unsupervised domain-adaptive detection algorithms for improving calibration in target domain (sec 4.2). Finally, we show that our loss formulation complementary to implicit calibration techniques and can be deployed to further enhance calibration (sec 4.3
5

---

3.3 Implicitly Calibrating Self- Seff-Trainng based Domain Adaptive Detectors
We propose technique aimed at implicitly improving the calibration of self-training based domain adaptive detectors. Mostly, self-training based domain adaptive detectors involve constructing pseudo instance-level labels (termed pseudo-labels thereafter) corresponding detections in target domain. Based on these pseudo-labels, pseudo-targets are formed for computing (classification) loss during the adaptation phase. We observe that, these pseudo-targets are constructed as one-hot encoded channels, and so they struggle to reflect the predictive confidence or uncertainty of detections. As a result, the adaptation model fails account for the noise in detections, mostly prevalent under domain shift, and could inadvertently learn to make overconfident predictions, hence negatively affecting calibration. To this end, we first present new uncertainty quantification mechanism for object detection and then leverage this to modulate the one-hot encoded pseudo-targets.
Quantifying uncertainty in object detection: To quantify model's uncertainty for detections, follow three-step process [25]. First, given an arbitrary image, we perform N stochastic forward passes (inferences) over the one-stage object detector using Monte-Carie dropout [.. Specifically, we apply spatial MC-dropout [36] over the convolutional filters after the feature extraction layer 225 Let Dn.m I be the mihh predicted bounding box during Tech inference, Sn.m. be the corresponding confidence vector, and Gn be the predicted class corresponding to the highest confidence fn.m E 8n.m- We define ên.m = (ba,m. Cn,m) be the prediction pair. Second, we identify and group the detections corresponding 2n.m across the inference space that have the same predicted class and an overlap with its bounding box greater than a certain threshold. Specifically, we create set An,m for each mm This consists of all predictions 2 such that C,,m Cx,1 and
IoU between bn.m and bR. is larger than some threshold y (set to 0.5 throughout experiments) 25 Here f and is an arbitrary detection in kis. MC forward pass.
An.m = (Vign U (b Cx,)), [oU bx.)) > r. Ck.l = Cn.m
(10)
Finally, Anm is utilized to estimate the uncertainty in 2n.m. We aim to capture uncertainty in predicted confidences and localization. So, we first estimate the variance predicted (class) con- fidences, center x, center and aspect ratio of the bounding boxes predictions in An,m and later aggregate them to construct a joint measure of uncertainty. Let {w)} and {V) be the vectors (J denotes the length) containing variances and means of predicted confidences, center-x, center-y. and aspect ratio of the predicted bounding boxes in An,m. respectively. Let L agg be the combined mean, computed V Vagg 1 Σラ= 1 Ví. Then, the combined variance representing (inglejoiim)
measure of uncertainty is computed as:
1 Σ1v + -V/ Maga) 1=1
Un.m
(11)
Uncertainty-guidee soft aseudo-targetss We leverage (joint) uncertainty to modulate the one-hot encoded terudtttrggets, formed according to (selected) pscudo-labels, to account for the entropy in object detections under target domain. Let H be it location in the class & one-hot encoded channel map corresponding to a detected bounding box, with uncertainty u² and (pseudo) class label k. The imcertainty-guide soft pseudo-targe constructed as:
i
(1- ut) H st.
if K s <m
(12)
w
R2
Where 3 = = Σ, 8 is the mean (class) confidence of the predicted bounding boxes in As. K1 is threshold for ranking between highly confident and relatively less confident detections, and Kig the confidence threshold below which there is no detection considered.

---

Methods
Sim10k
KITTI
Cityccapes
4
AP@05
D-ECE
AP00.5
D-RCE. 4
AP@0.
In-domain Detections
Single-stage
16.1
79.2
15.1
94.1
15.1
44.9
Single-stage post-hoc
23.4
78.7
22.8
94.1
25.3
44.3
Single-stage
83.4
12.6
48.3
Out-of-domain Detections
Sim I0K
CS
KITTI-CS
CS CS--Cgy
CS-BBDIOOK
Single stage
12.2
37.6
13.0
37.4
18.4
20.4
19.5
19.5
Single stage post hoc
20.7
38.1
23.2
37.5
22.7
20.3
27.4
19.3
Single stngle-saage stage
42.4
22.4
22.0
Table I: Calibration performance and test accuracy with single-staee detector (FCOS (34)) trained with its application specific losses. Calibration with post-hoe temperature scaling and with proposed TCD loss.
4 Experiments
Cityscapes dataset [1] consists images of road and street scenes with bounding box annotations of following object categories: person, rider, car, truck, bus, train, motorbike, and bicycle. KITTI dataset [4] offers images of road scenes, captured from different viewing angle than Cityscapes and encompass wide-view of the area. Foggy Cityscapes dataset [31 is constructed by simulating Doggy-weather, using depth maps, over the images in Cityscapes dataset. From the available three levels of fog, per the norm, we use the one with most dense fog. Sim 10k dataset [2] a collection of 10K images, synthesized from Grand Theft Auto V, and corresponding bounding box annotations of cars in those images. Follow prior works, while adapting from KITTI or Sim OOK, only car-class considered. Following prior works, car-class BDD100K dataset (39 contains 100k annotated images with bounding boxes and category labels. 70k images are used for training and 30k are used for validation. Following [3 we only consider daylight images create a training subset
Datasets:
of 36.7k images and a validation subset of 5.2k images. Note that, the validation subset is used as an evaluation set.
Evaluation and implementation details: We report calibration performance using detection expected calibration error (D-ECH) and also report test accuracy. Further, we also plot reliability diagrams for visualizing calibration. Our TCD loss is developed to be used with the task-specific loss of DNN-based object detectors, including the SOTA domain-adaptive ones. For instance, the task specific losses for single-stage detectors use Focal loss (21) and IoU loss. Similarly Smooth LI loss and Cross-Entropy are used training the two-stage detectors. Let Lp be the task-specific loss, then the total loss in our method is computed as: L = LD + LTCD. For further implementation details, refer the supplementary.
Considered object detectors: Most current state-of-the-an (SOTA) object detectors are built on either single-stage two-stage backbone. To include representative detector models, from single-stage detectors we consider a state-of-the-ar detector FCOS 34 and from multi-stage detectors we choose Faster-RCNN 229 Our choice is partially motivated by the fact that the same backbones also used in the current SOTA Nomain-adaptive object detectors. For the domain adaptive detectors we include EPM 9 and also SSAL (25) which are among the best performing single-stage domain adaptive detectors built on FCOS as source model. For the two-stage domain adaptive variant we include SWDA 30 that is built on FasterRCNN.
4.1 Experiments with One-stage and Two-stage Detectors
In this setting, we train DNN-based object detector (1) with its task-specific loss functions and (2) after adding our TCD auxiliary loss function. For comparison we also include post-hoe calibration technique based on temperature scaling. The temperature parameter T is obtained using a hold-out validation set perform temperature scaling at inference time in the target domain. We measure the performance and calibration errors in two settings. Here we measure the performance on the test set belonging to the same domain from which training dataset was sampled In-domaín Detection: Here we measure the performance over the test-set partition of same dataset from which training set was extracted and Out-of-domain Detection: where we test it on an unseen target domain i.e., a different dataset.
7

---

Methods
SimT0K
Table 2: Calibration results with two-stage detector (Faster- RCNN (29]) trained with its task-specifir loss, ap- plying post-hoe temperature scaling on pre-trained two- stage detector, and training two-stage detector after adding our TCD loss.
D-ECE
A.
D-ECE
In-domaim
Delections
Iwo-stage
23.7
66.2
16.2
38.3
Two-stage post-hoe
15.0
66.3
13.6
3800
Two-stage TCD
14.5
66.7
11.0
39.9
Out-of-domain Detections
SimlUK
+ CS
CS + CS-logyy
CS - BDDDOUK
Two-stage
13.9
33.9
22.7
16.3
23.3
Two-stage post-boe
11,9
34.1
6.3
22.7
13.7
23.3
Twa-singe + TCD
11.1
33.9
3.7
25.2
9.6
23.5
Methods
SimT0k CC
CS-+CS CS-Toggy
Table 3: Calibration results with two different single-stage Nomain-adaptive detectors, EPM [ and SSAL(UGPL) [25]. and a two-stage detector SWDA 33). We report both calibration perfor- mance (ECE) and test accuracy (AP@0.5).
AP805
D-ECE
AP&0.5
stagle-stage
Domain Adaptive Detectors
EPM 2
15.7
EPM TMD TCD
9.9
47.7
14.8
39.9
SSALUGPL 25
35.0
SSAL(UGP) TCD
855
514
19.1
35.2
Two-stage Domain Adaptive Detectors
Siml0K
+CS
CS
3 CCS Huggy
SWDA 50]
14.6
40.9
11.2
36.6
SWDA TCD
13.8
40.0
9.3
37.5
Table D reports results with a one-stage detector (FCOS (34] trained with its task-specific losses L.e. Focal loss and IoU loss. We report calibration errors (D-ECE) and performance of the model. We show the impact on calibration after adding and training with the proposed auxiliary loss TCD, For comparison we include post-hoe temperature scaling based calibration method. For in-domain detections, we see that our proposal, single-stage detector trained after adding TCD, achieves the best D-ECE score across all datasets. Furthermore, our proposal allows boosting the detection performance by notable margins compared to a single-stage detector in all datasets. For out-of-domai detections, we significantly improve the calibration performance over baselines in all domain shift scenarios. For instance, it decreases the D-ECE score by 12.9% and 7.1 % compared to the single-stage detector in CS CS-ggy- and CS 1S=+BDDIOOK respectively. At the same time, it provides notable gains in detection performance over single-stage detector e.g., a 4.8% gain (AP@0.5) in Sim1OK-<SS
We report results with two-stage detector RCNN 29] trained with a task-specific loss, taking a pre-trained two-stage detector and applying post-hoc temperature scaling, and two-stage trained with task-specific loss and our TCD loss (Table 2 For both in-domain and oft-o--domain detections, we observe that our proposal, two-stage detector trained after adding TCD, delivers the best D-ECE score across all datasets.
4.2 Experiments with Domain-adaptiv Detectors
In this setting, we incorporate our TCD auxiliary loss in the recent SOTA domain-adaptive detectors and observe impact calibration performance. In single-stage paradigm, we chose EPM 99, which accounts for pixel-wise centerness and objectness for target domain images in an adversarial alignment framework. Further, we also choose method from the line of self self-training based domain-adaptie detectors. Particularly, we select uncertainty- guidde pscudo-labelling (UGPL) from SSAL [25]. In two-stage paradigm, we choose SWDA (30] which proposes strong local alignment and weak global alignment in an adversarial alignment framework.
Table reports results with two single-stage domain domain-adaptive detectors: EPM 9. UGPL from 25 After adding our proposed loss TCD, the calibration performance for both domain-adaptiv detectors significantly improved. In Sim WomIOK-SS. both EPM+TCD and SSAL(UGPL) score by 6.1 % and 5.1%, 1,, respectively. Further, we observe that after adding our auxiliary loss (TCD), there are notable gains in AP@0.5.
reduce ECE
We show the impact of our TCD loss on the calibration performance of two-stage domain- adaativ detector (SWDA (30) in Table 3 Upon adding our loss TCD in SWDA, we observe that the calibration of SWDA is improved in both adaptation scenarios.
4.3 Experiments with Implicit Calibration Technique
We validate the effectiveness of our implicit calibration technique (ICT) (sec. 3.3 by integrating in self-training based mneerainty-guideer pseudo-labelling baseline (UGPL.) from (25). UGPI.
is strong baseline in terms of benchmarking model calibration because it leverages uncertainty
8

---

Method/Shift scenarios
Sim10k
CS
CS CS-foggy
D-FCE
AP@0.
D-ECE
mAP@0.5
SSAL(UGPL) 225
13.6
49.5
22.1
35.0
(SAL(UGPL)ICC
12.7
51.3
19.5
34.2
SSALIUGPLHHTCD
8,5
51.4
19.1
35.2
SSALIUGPLACCHIIC
7.9
50.7
16.7
36.9
Table 4: Calibration results with implicit calibration technique (ICT).
C5 Noeets Bootl
Ch Pooners Nones
ROeTOOe Nowe
Ansul
Artual
Carlenen
Datehara's
Chts honee's Nores
srenees Nost
Adaad
Aladd
Acail
Adad
Corkiens
Gréeene
Figure 2: -lis reliability diagrams. Top row: One-stage detector trained with fask-specific loss. Bottom One-stage detector trained with task-specific loss and our loss TCD.
row:
to select pseudo instance-level labels in an unlabelled target domain for self-training Table reports results for following methods: SSAL(UGPL) [25], SSAL(UGPL)HICT, SSAL(UGPL)+TCD, SSAL(UGPL)+ICT+TCDD We see that (UAL(UG decreases D-ECE by 0.9% in Sim10-CS shift. Further, adding TCD shows significant improvement in calibration performance (D-ECE). Finally, using both ICT and TCD achieves the best calibration scores in both SimIOK-CCS and CS CS-foggy shifts, thereby revealing that ICT and TCD are complementary.
4.4 Ablation Study and Analysis
For ablation experiments with TCD, we use stage detector (FCOS 34 For ICT ablation
experiments, we use the UGPL component of SSAL 25 which is a domain-adaptive detector.
Mitigating Under/OOer--ooffddenne Fig. 2 shows that our proposal, task-specific loss in conjunction with proposed TCD loss, facilitates a one-stage detector in mitigating both under-confident and over- confident detections in two different multi-class shift scenarios.
Impact on test accuracy: We observe the impact on test accuracy after adding our TCD loss with the task-specific loss of one-stage detector in Table 5 Our proposal is capable of providing gains in test accuracy with visible margins over higher IoU thresholds and over the spectrum of large, medium and small objects.
Shift scenario
Method
AP(mean)
AP@905
APG0,75
AP@S
AP@M
L
One stage
17.8
37.6
14.9
3.8
19.7
36.6
Sim10K CS
One- One-singe+TCC TCD
22.5
42.4
21.4
4.2
24.8
46.5
CS
BDD1
One-stage
9.5
19.5
7.9
3.5
11.7
20.0
One-stage- TCD
10.7
22.0
9.0
Table 5: Test accuracy after adding our TCD loss with the task specifcii loss of one-stage detector (FCOS
34
Without detection component d Table shows the impact calibration performance without dett component in TCD. We see that without datt, the calibration performance (D-ECE) degrades.
Scenarios
Simf0k CS
CS-foggy
KITTI
CS BDD
OOD
In-domain
OOD
In-domain
OOD
In-domain
00D
In-dumain
D-ECE
AP@0.5
D-ECE APer0.5 D-ECE AP60.
D-ECE
APG50,
D-ECE
APe83
D-ECE
AP@0.5
D-ECE
AP80.5
D-ECE
APQ85
wio d del
10.3
44.9
15.2
82.3
8.1
23,8
1333
4488
11.4
38.7
13.0
94.3
16.5
193
13.3
44.8
with det
9.6
42.4
14.
5.5
22.4
9.4
48.3
40.3
94.7
22.0
9.4
Table 6: Impact on calibration performance without dast component of TCD in four domain shift scenarios.
On different uncertainty quantification methods for ICT: Table reports the calibration perfor- mance of ICT with different methods of quantifying uncertainty. We use variances across: predicted confidences only 255 Uconf. predicted confidences and center-x,center-y Ucomf,xxy' predicted confi- dences and center-x,center-yy aspect-ratio (ours) Ueonf.xxy.ar. Among different methods, our proposed technique of estimating detection uncertainty in ICT provides the lowest D-ECE score.

---

Methods (Sim10k -CS)
D-ECE
AP
Table 7: Calibration performance of ICT when ing different ways quantifying detection uncer- tainty. See text for details.
SSALZUGPL]
13.6
49.5
SSALUGGLLHE (ccont.)
13.0
50.3
SSAT/UGPL ) ICT (leonf
12.8
51.1
SSALUUGPLH+CC HIC (Reont ..y.or)
12.7
51.3
Limitation: Current study does not explore the impact on calibration with respect to objects sizes. Further, our train-time calibration loss could potentially result in less improvement in classes with relatively less instances.
Conclusion
In this paper, we approached the challenging problem of calibrating DNN-based object-detectors for in-domain and out-of-domain detections and improving their calibration in the domain-adaptation context. We proposed an auxiliary train-time calibration loss TCD that jointly calibrates the class-wise confidences and localization performance. Further. develop implicit calibration technique (ICT) for self-training based domain adaptive detectors. Results show that TCD loss is capable of improving calibration of both one-stage and two-stage detectors. Finally, show that ICT and TCD together results in well-calibrated domain-adaptive detectors.
References
[1] Cordts, M., M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele (2016). The cityscapes dataset for semantic urban scene understanding. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[2] Dusenberry, M. W., D. Tran, E. Choi, J. Kemp, J. Nixon, G. Jerfel, K. Heller, and A. M. Dai (2020). Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference Health, Inference, and Learning, pp. 113.
[3] Gal, Y. and Z. Ghahramani (2016). Dropout bayesian approximation: Representing model uncertainty in deep learning. international conference on machine learning, pp. 1050-1059, PMLR.
[4] Geiger, A., P. Lenz, and R. Urtasun (2012). Are we ready for autonomous driving? the kitti vision benchmark suite. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pp. 3354-3361. IEEE.
[5] Grigoreseu, S., B. Trasnea, T. Cocias, and G. Macesanu (2020). A survey of deep learning techniques for autonomous driving. Journal of Field Robotics 37(3). 362-386.
[6] Guo, C., G. Pleiss, Sun, and K. Q. Weinberger (2017). On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR.
[7] Hebbalaguppe, R., J. Prakash, N. Madan, and C. Arora (2022). A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. arXiv preprint arXiv:2203.13834
[8] Hein, M., Andriushchenko, away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 41-50.
J. Bitterwolf (2019). Why relu networks yield high-confidence predictions
[9] Hsu, C.-C., Y.H. Tsai, Y.Y. Lin, and M.-H. Yang (2020). Every pixel matters: Center-aware feature alignment for domain adaptive object detector. In European Conference Computer Vision, pp. 733-748.
Springer.
[10] Islam, M., L. Seenivasan, H. Ren, and B. Glocker (2021). Clas--iistrbuttion-awan calibration for long- tailed visual recognition. arXiv preprint arXiv:2109.0510
[11] Ji, B., H. Jung, J. Yoon, K. Kim, and Y. Shin (2019). Bin-wise temperature scaling (bts): Improvement in confidence calibration performance through simple scaling techniques. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 4190-4196.
[12] Johnson- Roberson, M., C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan (2017). Driving in the matrix: Can virtual worlds replace human generated annotations for real world tasks? In 2017 IEEE International Conference Robotics and Automation (ICRA). pp. 746-753. IEEE.
10

---

[13] Karimi, D. and A. Gholipour (2022). Improving calibration and out-of- detection in deep models for medical image segmentation. IEEE Transactions on Artificial Intelligence.
[14] Krishnan, R. and O. Tickoo (2020). Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems.
[15] Kull, M., M. Perello Nieto, M. Kängsepp. T. Silva Filho, H. Song. and P. Flach (2019). Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems 32.
[16] Kull, M., T. Silva Filho, and P. Flach (2017). Beta calibration: well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statisties, pp. 623 631. PMLR.
[17] Kumar, A., S. Sarawagi, and U. Jain (2018). Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning, pp. 2805-2801111 PMLR.
[18] Kuppers, F., J. Kronenberger, A. Shantia, and A. Haselhoff (2020). Multivariate confidence calibration for object detection, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 326-327.
[19] Küppees,, F,, J. Kronenberger, A. Shantia, and A. Haselhoff (2020, June). Multivariate confidence calibration for object detection, In The JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
[20] Liang, G., Y. Zhang, X. Wang, and N. Jacobs (2020). Improved trainable calibration method for neural networks on medical imaging classification. arXiv preprint arXiv:2009, 04050
[21] Lin, T.-Y., Goyal, R. Girshick, K. He, and P. Dollár (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988.
[22] Maroñas, J., D. Ramos, and R. Paredes (2021). On calibration of mixup training for deep neural networks. In Joint IAPR International Workshops Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pp. 67-76. Springer.
[23] Mukhoti, J., V. Kulharia, A. Sanyal, S. Golodetz, Torr, and P. Dokania (2020). Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems 33, 15288-15299.
[24] Müller, R., S. Kornblith, and G. E. Hinton (2019). When does label smoothing help? Advances in neural information processing systems 32.
[25] Munir, M. A., M. H. Khan, M. Sarfraz, and M. Ali (2021). Ssal: Synergizing between self-training and adversarial learning for domain adaptive object detection. Advances in Neural Information Processing Systems 34.
[26] Nacini, M. P., G. Cooper, and M. Hauskrecht (2015). Obtaining well calibrated probabilities using bayesian binning. In Twemty-Nimth AAAI Conference on Artificial Intelligence.
[27] Pereyra, G., G. Tucker, J. Chorowski, L. Kaiser, and G. Hinton (2017). Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv: 701.06548.
[28] Platt, J. al. (1999). Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large margin classifiers 10(3), 61-74.
[29] Ren, S., K. He, R. Girshick, and Sun (2015). Faster r-cun: Towards real-time object delection with region proposal networks. In Advances in neural information processing systems, pp. 91-99.
[30] Saito, K., Y. Ushiku, T. Harada, and K. Saenko (2019). Strong-weak distribution alignment for adaptive object detection. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pp- 6956-6965.
[31] Sakaridis, C., D. Dai, and L. Van Gool (2018). Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision 126(9), 733-922.
[32] Sharma, M., O. Saha, A. Sriraman, R. Hebbalaguppe, L. Vig, and S. Karande (2017). Crowdsourcing for chromosome segmentation and deep classification. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 31.

---

[33] Szegedy, C., V. Vanhoucke, S. loffe. J. Shlens, and Z. Wojna (2016). Rethinking the inception architecture for computer vision. In 2016 IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 2818-2826.
[34] Tian, Z., C. Shen, H. Chen, and T. He (2019). Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE international conference on computer vision, pp. 9627-9636.
[35] Tomani, C., S. Gruber, M. E. Erdem, D. Cremers, and F. Buettner (2021). Post-hoc uncertainty calibration for domain drift scenarios, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10132.
[36] Tompson, R. Goroshin, A. Jain, LeCun, and Bregler (2015). Efficient object localization using convolutional networks. In Proceedings of the IEEE conference on computer vision and puitern recognition, pp. 648-656.
[37] Wang, D.-B., L. Feng, and -L. Zhang (2021). Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. In M. Ranzato, A. Beygelzimer, Y. Dauphin. P. Liang, and J. W. Vaughan (Eds.). Advances in Neural Information Processing Systems, Volume 34, pp. 1809-11820 Curran Associates, Inc.
[38] Xu, C.-D., X.-R. Zhao, X. Jin, and X.-S. Wei (2020). Exploring categorical regularization for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 11724 11733.
[39] Yu, F., W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, and T. Darrel (2018). Bdd100k: A diverse driving vidco database with scalable annotation tooling. arXiv preprint
046687 2(5), 6.
[40] Yu, R. and G. S. All (2019). What's inside the black box? ai challenges for lawyers and researchers. Legal Information Management 19(1), 2-13.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Since our work belongs the fundamental problem machine learning, it inherits all the benefits of object detection.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [No] Our work is not theoretical.
(b) Did you include complete proofs of all theoretical results? [No] Our work is not theoretical.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In supplementary.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In supplementary
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We are using implementations of existing published works, and those are cited.
(b) Did you mention the license of the assets? [No] We are citing the previous work whose publicly shared code we use for reporting baselines.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "u3q8jcr8Z2Q", "reviewer": "Reviewer_GHp5", "summary": "The paper focuses on the calibration of both two-stage and one-stage object detectors. Moreover, the author conducts studies on in-/out-domain distribution.\nSpecifically, an auxiliary loss function (TCD), which is based on the distance between class accuracy and bounding box IOU, is introduced.\nIn addition, a quantification mechanism is proposed for self-training of the detectors.", "questions": "see Weaknesses", "limitations": "see Weaknesses", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n1. The proposed method is simple and reasonable.\n2. The paper is clearly presented and well-organized.\n3. The proposed method outperforms all baselines.\n\n[Weaknesses]\nIn Table 1 and 2, it raises my attention that post-hoc perform worse than all the rest including the baselines for both in- and out-domain.\nDoes that suggest post-hoc is not suitable for detection tasks for some reasons?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iprp3VCYqN9", "reviewer": "Reviewer_kwna", "summary": "In this paper, the authors propose two methods to improve the calibration of confidence scores of object detection models based on deep neural networks. A first method, TCD, introduces a specific cost function that aims to constrain the model to reduce the difference between the predicted confidence and the prediction accuracy (d_cls) and to reduce the difference between the confidence and the IoU of the detected regions compared to the ground truth (d_det).\nThis cost function is used for supervised training of the models. A second method, ICT, aims to calibrate the predictions on a set of out-of-domain tests in an unsupervised (or self-supervised) manner. This method uses a MC-Dropout to generate multiple hypotheses and calculates a combined variance on the confidence, position and aspect ratio of predictions of the same classes among the hypotheses. This combined variance is then used to generate pseudo-targets for self-supervised adaptation. \n\nThese methods are evaluated on 5 object detection databases, using different detection models (FCOS, Faster-RCNN, EPM, SSAL, SWDA). The experiments show that the proposed methods improve both the detection expected calibration error (D-ECE) and the accuracy of the models, on all the bases and for all the models tested. An ablation study completes the evaluation. \n", "questions": "* Fig1: explain what is dark red and light red\n* L125 avoid foot note in equation, could be read as exponent 1[IoU(bˆ, b∗) ≥ γ2]1[yˆ = y∗].  \n* Eq 7: sl,r[k]  is not defined\n* L196: J is not defined", "limitations": "NA", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written, with clear description of the methods and very thorough experiments. The results are convincing. It is shown that the proposed methods can be applied to different types of models and on different databases.\n\nHowever, the article has a small weakness in the comparison with other calibration methods: only the simple temperature scaling method is used as a basis for comparison. More complex methods could have been compared.  However, given the space constraints, it seems difficult to hold this against the authors.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_WTPeC2Pnew", "reviewer": "Reviewer_B3RR", "summary": "This works identifies the calibration in object detection under domain shift. The author proposes a new plug-and-play loss formulation, termed as train-time calibration for detection (TCD). It can be used in some task-specific loss functions during training phase and acts as a regularization for detections. \n", "questions": "- Ablation study of the work in some well-known datasets like COCO and VOC.", "limitations": "- Ablation study of the work in some well-known datasets like COCO and VOC.\n- Not very practical for general object detection, although it has some theoretical analysis.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The author uses mathematical deduction to analyze the domain shift in detection. The Calibration including TCD, Quantifying and uncertainty-guided soft pseudo-targets can easily be plugged into many detection frameworks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8wObCWQ3F5Z", "reviewer": "Reviewer_ETNZ", "summary": "The paper proposes a new well-calibrated detection algorithm for both in-domain and out-of-domain scenarios. The author brought the concepts from calibration for classification into detection, and proposed a calibration loss (TCD) for object detection by considering both classvcaition and localisation predictions. In addition, a uncertainty quantification mechanism is proposed to further boost the performance. The proposed method shows clear improvement over the baselines, and the experiments are comprehensive.", "questions": "Please refer \"Strengths And Weaknesses\"", "limitations": "I do not see any potential negative societal impact of their work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The idea of using the concepts of calibration for detection is important and novel to me.\n- The paper is easy to understand, and the experiments are very comprehensive. \n- The results are competitive and show clear improvements over baselines.\n\nCons:\n- For Table 4 R4-6 C2-3, I feel a bit strange that both TCD and ICT can reduce the D-ECE score, however, when combining them together, the overall AP reduces with even lower D-ECE value. The similar results also appear in Table 2 R4-6 C8-9 (Supp), where the TCD variant is even worse than the baselines under AP metric, while still it achieves lower D-ECE score.\n- Again For Table 2 (Supp), the results seem not promising when applying the proposed module in transformer-based detectors. The generalisation of the proposed modules becomes questionable.\n- I learn the method can improve both in-domain and out-of-domain performance of object detection, so I wonder can we apply the proposed module into the existing SOTA detectors to reach a higher performance.\n- Though it's commonly used in ood scenario, I am curious the performance of applying COCO trained detector into VOC datasets directly.\n\nOverall speaking, I think it is a good paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept