Cal-DETR: Calibrated Detection Transformer
Authors: Muhammad Akhtar Munir
Salman Khan
Muhammad Haris Khan
Mohsen Ali
Fahad Khan
Venue: NeurIPS 2023 poster (2023)
Keywords: Model Calibration
Object Detection
Detection Transformers
Uncertainty
TL;DR: Uncertainty-guided logit modulation and mixing for calibrating vision transformer based object detectors

Abstract: Albeit revealing impressive predictive performance for several computer vision tasks, deep neural networks (DNNs) are prone to making overconfident predictions. This limits the adoption and wider utilization of DNNs in many safety-critical applications. There have been recent efforts toward calibrating DNNs, however, almost all of them focus on the classification task. Surprisingly, very little attention has been devoted to calibrating modern DNN-based object detectors, especially detection transformers, which have recently demonstrated promising detection performance and are influential in many decision-making systems. In this work, we address the problem by proposing a mechanism for calibrated detection transformers (Cal-DETR), particularly for Deformable-DETR, UP-DETR, and DINO. We pursue the train-time calibration route and make the following contributions. First, we propose a simple yet effective approach for quantifying uncertainty in transformer-based object detectors. Second, we develop an uncertainty-guided logit modulation mechanism that leverages the uncertainty to modulate the class logits. Third, we develop a logit mixing approach that acts as a regularizer with detection-specific losses and is also complementary to the uncertainty-guided logit modulation technique to further improve the calibration performance. Lastly, we conduct extensive experiments across three in-domain and four out-domain scenarios. Results corroborate the effectiveness of Cal-DETR against the competing train-time methods in calibrating both in-domain and out-domain detections while maintaining or even improving the detection performance. Our codebase and pre-trained models can be accessed at \url{https://github.com/akhtarvision/cal-detr}.

Cal-DETR: Calibrated Detection Transformer
Muhammad Akhtar Munir Salman Khan 1.3, Muhammad Haris Khan', Mohsen Ali", Fahad Shahbaz Khan 1.4
Mohhmedd bin Zayed University of AI, 'Information Technology University, Australian National University, "Linköping University
akhtar. muni muunir@mber zuai ac.
Abstract
Albeit revealing impressive predictive performance for several computer vision tasks, deep neural networks (DNNs) are prone to making overconfident predictions. This limits the adoption and wider utilization of DNNs in many safety-critical applications. There have been recent efforts toward calibrating DNNs, however, almost all of them focus on the classification task. Surprisingly, very little attention has been devoted to calibrating modern DNN-based object detectors, especially detection transformers, which have recently demonstrated promising detection per- formance and are influential in many decision-making systems. In this work, we ad- dress the problem by proposing a mechanism for calibrated detection transformers (Cal-DETR), particularly for Cifformmale-DDTTT UP-DETR and DINO. We pursue the train-time calibration route and make the following contributions. First, we propose simple yet effective approach for quantifying uncertainty in transformer- based object detectors. Second, we develop an uncertainty-guide mechanism that leverages the uncertainty to modulate the class logits. Third, we develop a logit mixing approach that acts as a regularizer with detection-specific losses and is also complementary to the uncertainty-guu nique to further improve the calibration performance. Lastly, we conduct extensive experiments across three in-domain and four out-domain scenarios. Results corrob- orate the effectiveness of Cal-DETR against the competing train-time methods in calibrating both in-domain and out-domain detections while maintaining or even improving the detection performance. Our codebase and pre-trained models can be accessed at whtttps: /ttpt/gtttu//tte/ com/ http//aaaaaviiii...o.. cal detr
logit modulation
logit modulation tech-
Introduction
Deep neural networks (DNNs) have enabled notable advancements computer vision, encompassing tasks such as classification 29
51, and semantic Due to their high predictive performance, DNNs are increasingly adopted in many real-world applications. However, it has been shown that DNNs often make highly confident predictions even when they are incorrect [10). This misalignment between the predictive confidence and the actual likelihood of correctness undermines trust in DNNs and can lead to inreversible harm in safety-critical applications, such as medical diagnosis
11, object detection 2
53
5 38
segmentation 47
42 3
41
autonomous driving (9), and legal
decision-makie making [49]
Among others, an important factor behind the overconfident predictions of DNNs is the training with zero-entropy target vectors, which do not encompass uncertainty information [1]. An early class of methods aimed at reducing the model's miscalibration proposes various post-hoe techniques [10 15 [6] 37 21 Typically, single temperature scaling parameter is used to re-scale the logits of a trained model which is learned using held-out validation set. Despite being effective and simple, they are architecture-depende
and require held-out set which is unavailable in many real-world scenarios 288) Recently, we have seen an emergence of train-time calibration methods. They are
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

typically based on an auxiliary loss that acts as a regularizer for predictions during the training
process [2 28 25 19 33.. For instance, (2 imposes a margin constraint on the logit distances in the label smoothing technique, and [1] develops an auxiliary loss that calibrates the confidence of both the maximum class and the non-maximum classes. These train-time approaches reveal improved calibration performance than various existing post-hoe methods.
We note that majority of the DNN calibration methods target the classification task 10
15
16
37
25 [9] Strikingly, much little attention has been directed towards studying the calibration of object detectors [33. Object detection is fundamental task in computer vision whose application to safety-critical tasks is growing rapidly. Furthermore, most of the existing methods focus on calibrating in-domain predictions. However, deployed model could be exposed to constantly changing distributions which could be radically different from its training distribution. Therefore, in this paper, we aim to investigate the calibration of object detection methods both for in-domain and out-domain predictions. Particularly, we consider the recent transformer-bases object detection methods (5 [1 as they are the current sfate-of-thh-arr with a simpler detection pipeline. Moreover, they are gaining increasing attention from both the scientific community and industry practitioners.
21 12 28,
In this paper, inspired by the train-time calibration paradigm, we propose an uncertainty-guide logit modulation and a logit mixing approach to improve the calibration of detection transformers (dubbed as Cal-DETR). In order to modulate the logits, we propose a new and simple approach to quantify uncertainty in transformer- based object detectors. Unlike existing approaches [8] (23), it does not require any architectural modifications and imposes minimal computational overhead. The estimated uncertainty is then leveraged to modulate the class logits. Next, we develop mixing approach in the logit space which acts as a regularizer with detection-specifer losses to further improve the calibration performance. Both uncertainty-guica as a regularizer are complementary to each other. We perform extensive experiments with three recent transformer-basee detectors "Deformable-DETR (D-DETR) [53], UP-DETR (5) and DINO (5T) on three in-domains and four out-domain scenarios, including the large-scale object detection MS-COCO dataset [27]. Our results show the efficacy of Cal-DETR against the established train-time calibration methods in improving both in-domain and out-domain calibration while maintaining or improving detection performance.
logit modulation and logit mixing
2 Related Work
Vision transformer-basee object detection; Vision transformers enabling new state-of-the of-the-art
performances for image classification 66 46), semantic segmentation [47 42), and object detection 53 5 (1). The work of DETR 2 introduced the first-ever transformer- -based object detection pipeline to the vision community by eliminating post-processing steps like non-maximum suppression with object matching to particular ground truth using bipartite matching. However, DETR very slow in convergence and performs poorly on small objects, and to address these issues. Deformablc-DETT [53) was proposed that is based on a deformable attention module as opposed vanilla self-attention in DETR (2.
The work of [5) further enhanced the performance of this object detection pipeline by
pretraining the encoder-decoder parameters. Recently, DINO [51] improved detection performance with contrastive denoising training, look forward twice approach and integrated effective initialization
of anchors.
Model calibration: Model calibration requires the perfect alignment of model's predictive confidence with the actual likelihood of its correctness. One of the earliest techniques for model calibration is known as post-hoc temperature scaling (TS) [10], which is a variant of Platt scaling (37). The idea to re-scale the logits with single temperature parameter (T) which is learned using a held-out validation set. However, a limitation of TS is that the selection of temperature parameter T value reliant on the network architecture and data 28). And, TS has been shown to be effective for calibrating in in-domain predictions. To extend TS to domain-drift scenarios, [45] proposed to perturb the held-out validation set before the post-hoc processing step. Dirichlet calibration (DC) 20 was extended from beta calibration [2I] to multi-class setting. It uses an extra neural network layer that log-transforms the miscalibrated probabilities, and the parameters for this layer are learned using the validation set. Despite being effective and computationally inexpensive, post-hoc calibration methods demand the availability of held-out validation set, which is a difficult requirement to be met for many real-world use cases.

---

Decoder
Calibrated
Decoder
Unisilen orafee
NNO
Unoo Unoorfattty-... Burd-er didedd Logits
+
Encoder
Decoder Output
Bupay Loo
Lax
Calizated
Object Querles
Postiionll Encoding
Main architecture: An image passed through feature extractor CNN and unmodified transformer's encoder structure, whereas the decoder is modified to get the model well calibrated. Logits are modulated based on uncertainty quantification that exploits the inbuilt design of the decoder without extra computational cost (Sec. B.3.1). This module is followed by logit mixing mechanism which is comprised of regularizer depicted on the right side of figure. The baseline detector (D-DETR [ outputs wrong dog prediction with relatively high confidence, however, our Cal-DETR provides accurate cat prediction with a high confidence. Green boxes are accurate predictions whereas Red (dashed) box is inaccurate prediction. Blue bounding boxes represent ground truths for corresponding detections. Ldl and Lise are classification and localization losses respectively [S] 53 ST].
Figure I:
(Cx) obtain calibrated predictions (Sec. (.3.2). <calegory:confudence>
Train-time calibration methods tend to engage model parameters during training by proposing an auxiliary loss function that is optimized along with the task-specific loss function [12 28). Some methods aim at maximizing the entropy of the predictive distribution either explicitly implicitly. For instance, 35 formulated an auxiliary term, that is based on negative of entropy, which is minimized during training alleviate the impact of overconfident predictions. Along similar lines,
32 investigated that the label smoothing (LS) technique also capable reducing miscalibration, and [31] demonstrated that the focal loss (FL.) can improve the model calibration. These methods are effective, however, their logit-distance constraints can often lead to non-informative solutions. To circumvent this, (28) proposed margin-based generalization of logit-distance constraints to achieve a better balance between discriminative and calibration performance. [12) proposed to calibrate the confidence
both predicted and non-predicted classes by formulating an auxiliary loss term known as the multi-class difference of confidence and accuracy. All aforementioned calibration techniques target classification task, recently 33 developed train-time calibration loss for object detection which imposes constraints on multi-class predictions and the difference between the predicted confidence and the IoU score.
Uncertainty estimation and model calibration: Owing to the computational intractability of Bayesian inference, different approximate inference methods have been proposed, such as variational inference I 30), and stochastic expectation propagation [14] The work of (34) evaluated predictive uncertainty of models under distributional shift and investigates its impact the accuracy-uncertaing trade-off. 19 leveraged the predictive uncertainty of model and proposed to align this with accuracy, and (18) induced soft bins and makes calibration error estimates differentiable. to quantify uncertainty by ensemble learning that takes into account empirical variance of model predictions. It allows better discriminative performance and improved predictive uncertainty estimates leading to well-calibrated models. Ensembles can emerge with difference in weights initializations and random shuffling of training examples (23 and Monte Carlo (MC) dropout [8]. However, ensemble- bhaeb uncertainty estimation techniques could be computationally expensive, especially with complex models and large-scale datasets. In this work, we propose a simple and computationally efficient technique to quantify uncertainty in transformer- object detection pipeline.
also possible
Method
This section begins by discussing the preliminaries, including the problem settings and the background knowledge relevant to calibrating object detectors. We then detail our proposed approach, Cal-DETR, encompassing uncertainty-guicae logit modulation and logit mixing for calibrating the recent vision transformer-base (ViT) object detection methods (Fig.

---

3.1 Preliminaries
Assume joint distribution VV), where X' is an input space and y the corresponding label space. Let D = (x),, be the dataset sampled from this joint distribution consisting of pairs of input image x; E X' and corresponding label y; E y. The input image belongs to x; E TR xxxxwx2 where H the height, W is the width and Z is the number of channels. For each input image X;, we have the corresponding ground truth label, ý; E R,, and it consists of class label and the bounding box coordinates (2, 2, G. w, h). Here è the class label E f1, 2...... .... C} and êt G. dē, h are the bounding box b E I coordinates. Next, we define the calibration of model in classification and object detection tasks.
A model is said to be calibrated when its accuracy perfectly aligns with the predictive confidence. An uncalibrated model is prone making over-confident and under-confident predictions. To mitigate these problems, for an image, a model should be able to predict the class confidence that matches with the actual likelihood of its correctness. Let Hats be a classification model which predicts a class label with the confidence score Σ. Now, the calibration for the classification task can be defined as: P(c- = z|z = 2) = 2, s.t., 2 E [0, 1] [22]. This expression shows that, to achieve the perfect calibration in classification, the predicted confidence and accuracy must match.
Let Hdet be the detection model, which predicts a bounding box b and the corresponding class label č with the confidence score z. The calibration for detection task is defined as I 1 2) 2, V2 E [0. 1] 22 Where F denotes the true prediction of the object detection model which occurs when the predicted and the ground truth classes are the same, and the Intersection over Union (loU) of both the predicted and ground truth boxes are above a pre-defined threshold Le. [[LoUs (6, b) 2 k] = aj. F = 0 denotes the false prediction and together with F = 1, the precision is determined. In object detection, to achieve perfect calibration, the precision of model must match the predicted confidence. In the next subsection, we revisit the formulae for measuring calibration.
3.2 Measuring Calibration
Classification: For classification problems, miscalibration is measured by computing Expected Calibration Error (ECE) [10. It computes the expected divergence of accuracy from all possible confidence levels as: Eg PC a|= = 2) -21]. In practice, the ECE metric is approximated by (1) first dividing the space of confidence scores into B equal-sized bins, (2) then computing the (absolute) difference between the average accuracy and the average confidence samples falling into a bin, and finally (3) summing all the differences after weighting each difference by the relative frequency of samples in a bin.
B Σ
|S(b) [D]
ECE
|accuracy (b) confidence(t)),
where, accuracy(b) and confidence(b) represent the average accuracy and the average confidence of samples in the 8th bin, respectively. S(b) is set of samples in the 8tth bin and [D] is the total number of samples.
Object Detection: Similar to ECE, the Detection Expected Calibration Error (D-ECE) is measured as the expected divergence precision from confidence, at all confidence levels (22). It can be
expressed as [[P(F 1|2 z) - =I]. As in Eq. confidence is continuous variable, therefore
we can approximate D-ECE:
S(b)
D-ECE Σ b=1 |D]
precision(b) confidence(b))
where S(b) contains set of object instances in 8th bin. Note that, the D-ECE considers only the precision vs. confidence, and not the mean Average Precision (mAP). The mAP requires the computation of recall for which detections are not present, and so there are no associated confidences.
'Thi calibration definition of object detectors be extended after including box information.

---

3.3 Cal-DETR: Uncertainty-Guidee Logit Modulation & Mixing
Modern object detection methods occupy a central place in many safety- safety-critical applications, so it is of paramount importance to study their calibration. Recently, vision transformer based (ViT) object detectors (e.g.. Deformable-DETR (D-DETR) (53), UP-DETR [5] and DINO (5T) are gaining importance for both industry practitioners and researchers owing to their state-of-the-arr performance and relatively simple detection pipeline. Therefore, we aim to study the calibration of ViT-based object detectors. Towards improving calibration in transformer-basee object detectors, we first describe our uncertainty-guiaa guided logit modulation approach (Sec. (.3.1) I) and then propose a new logit mixing regularization scheme (Sec 3.3.2).
3.3.1 Uncertainty-Guidee Logit Modulation
We first briefly review the DETR pipeline 2 which is popular ViT-based object detection pipeline.
largely borrow the architecture of DETR pipeline [2]. Then, we describe our new uncertainty quantification approach and how it is leveraged to modulate the logits from the decoder to eventually calibrate confidences.
Recent transformer-based detectors 5, [1
Revisiting DETR pipeline: DETR was proposed as the first ViT-based object detection pipeline with the motivation of eliminating the need for hand-designed components in existing object detection pipelines. It consists of a transformer encoder-decoder architecture with a set-based Hungarian loss which ensures unique predictions against each ground truth bounding box. Assuming the existence of feature maps f E TR. # CCCC which are generated from ResNet [11 backbone, DETR relies on the standard transformer encoder-decoder architecture to project these feature maps as the features of a set of object queries. A feed-forward neural network (FFN) in a detection head regresses the bounding box coordinates and the classification confidence scores.
Quantifying uncertainty in DETR pipeline: Some recent studies have shown that the overall (output) predictive uncertainty of the model is related to its calibration [23 19]. So estimating a model's 's uncertainty could be helpful aiding the confidence predictions to align with the precision of a detection model. To this end, we propose a new uncertainty quantification approach for DETR pipeline after leveraging the innate core component of transformer architecture i.e. the decoder. It is simple technique that requires no changes to the existing architecture and yields minimal computational cost, unlike Monte Carlo dropout [8] and Deep Ensembles (23.. Specifically, let On E I 1 XQxC and Of E R M be the outputs of all decoder layers and final decoder layer of the detection transformer, respectively. Where L is the number of decoder layers, M the mini-batch, Q is the number of queries, and C' represents the number of classes. To quantify the uncertainty for every class logit Ue we compute the variance along the dimension which consists of multiple outputs of decoder layers. In other words, the variation in the output of decoder layers reflects the uncertainty for a class logit u.. Overall, after computing variance along the 4 dimension in Op, we obtain a tensor R MXQC which reflects the uncertainty for every class logit across query and minibatch dimensions.
Logit modulation: We convert the uncertainty u into certainty using: - tanh(u), where tanh is used to scale the uncertainty values to [0, 1] from [0, inf], and use this certainty to modulate the logits output from the final layer as:
Ö of @ (1- ta)h(u)):
where Ö is the function. tanh(u) indicates that if an output in logit space shows higher uncertainty, it will be down-scaled accordingly. other words, a higher uncertainty implicitly implies likely poor calibration of the output logit value (i.e. the pre-confidence score).
modulated decoder output and then this becomes the input the loss
3.3.2 Logit Mixing for Confidence Calibration
In addition to sicertainty-guideer logit modulation, we develop new logit mixing approach which acts as a regularizer with the detection-speciff losses for further improving the confidence calibration. We propose logit mixing as the mixing in the logit space of an object detector which is inspired by the input space mixup for classification [50] In DETR pipeline, this logit space is of dimensions Q x C, where Q is the number of object queries and C' is the number of classes. We only consider positive queries for performing logit mixing as they responsible for the detection of objects. In
5

---

object detection, the instance-leved object information is important as we consider the confidence scores associated with the positive class assigned to the bounding boxes.
In our logit mixing scheme, we first build prototypical representation by computing the mean across all positive queries, which is then used to achieve the mixing for any given positive query. For example, given three positive queries, let Qp = {Q1, Q2, Q33 C Q (total number of queries) belonging to the object classes and let Q be prototypical representation of all the positive queries, we formulate the expression as:
i = + (1 (1- aQQ.
Q2: E Up
Qs} are the mixed versions of {Q1, Q2 Q3} in the logit space. a is the mixing weight for logits. For the selection of a, more details are in Sec. 4.2
Now, {Ộ1, (2.,02.0
We employ regularization loss by leveraging the proposed logit mixing output, denoted as (x.). which is expressed as: (n( 4), where & contains the labels after smoothing with mix- ing values. Instead of one-hot labels. a determines the weightage of classes for object instances. Specifically a value for a given positive query contributes to smoothing the label while remaining positive queries that contribute to forming a prototypical representation share I-a value to smooth corresponding labels uniformly. The loss L is used together with the task-specific classification loss Lets (p(ā (x;))) é.) to obtain joint loss formulation as: Ldt + XLn, where (xx, denotes the non-mixed logits and controls the contribution of regularization loss. The proposed regularizer loss tends to capture the information from the vicinity of object instances in the logit space by mixing logits and constructing a non-zero entropy target distribution. This penalizes the model for producing overconfident predictions by possibly reducing overfitting [52 36.. Our empirical results (Sec. 4. show that this not only improves model calibration but also improves the overall detection performance in most scenarios and it is also complementary to our uncertainty-guide logit modulation technique.
Experiments & Results
Datasets: To perform experiments, we utilize various in-domain and out-domain benchmark datasets. Details are as follows: MS-COCO (27] is a large-scale object detection dataset for real images containing 80 classes. It splits into 1 118K train images, 41K test images, and 5K validation images. The train set (train2017) is used for training while the validation set (val2017) is utilized for evaluation. CorCOCO contains similar set of images as present in val2017 of MS- but with the corrupted version 13 Random corruptions with random severity levels are introduced for evaluation in an out-domain scenario. Cityscapes [4] an urban driving scene dataset contains classes person, rider, car, truck, bus, truin, motorbike, and bicycle. The training set consists of 2975 images while the 500 validation images are used the evaluation set. Foggy Cityscapes (39] is a foggy version of Cityscapes, 500 validation images simulated with severe fog are used for evaluation in this out-domain scenario. BDD100k (48) contains 100k images from which 70K are training images, 20K are for the test set and 10K for the validation set. Categories are similar to Cityscapes and for evaluation, we consider the validation set and make a subset of daylight images as an out-domain scenario, reducing the set to 5.2K images. Sim10k [17] contains 10k synthetic images with car category from which the training split is 8K images and for the evaluation set, the split 1K images.
Datasets (validation post-hoc): For the given scenarios, we opt separate validation datasets for the post-hoc calibration method (temperature scaling). In the MS-COCO scenario, utilize the Object365 (40 validation dataset reflecting similar categories. For the Cityscapes scenario, a subset of the BDD100k train set is used as a validation set. And for the Siml0k scenario, we use the remaining split of images.
Detection Calibration Metric:
For the calibration of object detectors, we report our results with the evaluation metric, which is the detection expected calibration error (D-ECE) 22 It is computed for both in-domain and out-domain scenarios.
state-of-art-deteetor Deformable-DFi (D- DETR), UP-DETR, and DINO as baselines respectively, which are uncalibrated. With the respective in-domain training datasets, we train D-DETR with our proposed method and compare it with baseline, post-hoc approach [10), and train time calibration losses [28) 12 33 integrated with D- D-DETR. We
Implementation Details:
We use transformer-

---

Datasets
Table 1:
Defarmable-DETR (D-
In-Domain (COCO)
Out-Domain (CarCOCO)
Methods
DETR) Calibration: AIS-COCO & Cor- COCO. Calibration results along with the detection performance. Cal-DETR improves calibration compared to baseline, other train-time losses, and
D-ECE 4 AP
APo.5
D-ECE
+ AP
APo.5 T
Baseline
53
12.8
44.0
62.9
10.8
23.9
35.8
Temp. Scaling 10
14.2
44.0
62.9
12.3
23,9
35.8
MDCA
11.1
35.3
TS methods.
MbLS
15.7
44.4
12.4
35.3
TCD 33
11.8
44.1
62.9
10.4
23.8
35.6
Cal-DETR (Ours)
8.4
44,4
64.2
8.9
24.0
36,3
Datasets
Table UP-DETR Calibration: MS- COCO & CorCOCO. Results are re- ported on Epoch150 setting and Cal- DETR shows improvement in calibra-
In-Domain (COCO)
Out-Domain (CarCOCO)
Methods
D-ECE
AP1
APo.s
D-ECE
AP
APp.5 f
Baseline
25.5
37.1
57.3
27.5
19.9
32,5
tion compared to other methods in-
Temp. Scaling
26.4
37.1
57.3
28.2
32,5
cluding baseline.
26.5
28.6
19.4
31.8
MbLS 28
25.4
36.3
57.0
26.5
20.1
32,9
TCD K3
25.2
36.4
57.2
26.8
19.1
31,5
Cal-DETR (Ours)
23.9
57.8
25.5
20.0
32.6
find empirically over the validation set and use the logit mixing parameters O₁2 = 0.9, 02 = 0.1 and 0.5 (see Sec. 4.2).. For the classification loss, the focal loss incorporated, and for localization, losses are generalized IoU and L1 as described in [26 2]. For more implementation details, we refer
readers D-DETR [53] UP-DETR 5 and DINO 51
4.1 Main Results
We evaluate the calibration performance on large-scale datasets (MS-COCO as in-domain and its image corrupted version out-domain, CorCOCO). The CorCOCO evaluation set is constructed by incorporating random severity and random corruption levels from [33. Our method not only provides lower calibration errors but also shows improved detection performance. For D-DETR, in comparison to the baseline uncalibrated model, our method improves calibration of 4.4%) (in-domain) and 1.9%] (out-domain). In this dataset, observe that classification based train time calibration losses (MbLS 28 & MDCA [2]) are sub-optimal for the detection problems and worse than the baseline in some scenarios. We show that our method outperforms train time losses and achieves better calibration, e.g. in comparison with detection based calibration loss (TCD) 3.4%, (in-domain) and 1.5%. (out-domain). For UP-DETR, method improves calibration from baseline by 1.6% (in-domain) and 2.0% (out-domain). For more results, we refer Fig. Tab. [ & Tab. We also provide the results COCO (in-domain) and CorCOCO (out-domain) with the DINO model, as shown in Tab. 3 Our Cal-DETR improves the calibration performance of this strong baseline for both in-domain and out-domain settings.
Real and Corrupted:
33 improvements
20
14
0.402
12
14
10
12
10
(CVPR'22)
(0VPR222) (NeuriP522)
(CVPR'221
(CVPR222)
(NeuriP8'22)
(CO0O)
(DerCOCO)
Do
(FogerCS)
Figure 2: D-DETR Calibration: MS-COCO & Cityscapes as in-domains and respective Cor-COCO & Foggy Cityscapes out-domains. Our proposed method (Cal-DETR) effectively decreases D-ECE compared to methods: baseline, post-hoc, and train-time losses.

---

Datasets
Table 3: DINO Calibration: MS- COCO & CorCOCO. Results are re- ported on the Epoch12 (4-scale) setting and Cal-DETR shows improvement in calibration as compared to other meth- ods including baseline.
In-Domain (COCO)
Out-Domain (CorCOCO)
Methods
D-ECE 4
AP 1
APo.s
D-ECE + AP
APo.5 1
Baseline
51
15.5
49.0
66.6
13.2
27.3
39.2
Temp. Scaling
10
15.1
49.0
66.6
14.3
27.3
39.2
MDCA 12
16.3
66.1
14.0
26.7
38.4
MbLS
19.1
66.0
15.9
26.9
38.4
TCD 33
15.6
48.5
66.1
12.9
26.8
38.5
Cal-DETR (Ours)
11.7
49.0
66.5
10.6
27.5
39.3
Table 4: D-DETR Calibration: Results are shown in comparison with baseline, post-hoc, and train-time loss methods. Cal-DETR shows improvement in detection calibration evaluation sets of the in-domain dataset (Cityscapes) and out-domain datasets (Foggy Cityscapes & BDD100k). AP and APo.s are also reported.
Datasets
In-Domain (Cityscapes)
Out-Domain (Foggy Cityscapes)
Out-Domain (BDD100K)
Methods
D-ECE
AP
APo.5
D-ECE
AP
APo.s
D-ECE.
AP t APo.5
Baseline P-DETR 53
13.8
26,8
49.5
19.5
17.3
29.3
11.7
10.2
21.9
Temp. Scaling 10
12.6
26.8
49.5
14.6
17.3
29.3
24.5
10.2
21.9
MDCA 12
13.4
27.5
49.5
17.1
17.7
30.3
14.2
10.7
22.7
MbLS 28
10.5
22.7
TCD 33
11.9
28.3
50.8
14.3
17.6
30.3
12.8
10.5
22.2
Cal-DETR (Ours)
28.4
51.4
11.9
17.6
29.8
11.4
11.1
23.9
Our method improves calibration over the baseline, post-hoc, and train-time losses with significant margins. Also, with our method, detection performance improves or is competitive with better calibration scores. Over baseline, we achieve an improvement of 5.4%.J (in-domain) and 7.6%. (out-domain). Tab, 4 and Fig. show further results.
Weather:
Urban Scene: We show that our method improves calibration and/or competitive, especially in the out-domain. For example, with MDCA [1), we improve calibration by 5.0%. (in-domain) and 2.8% (out-domain). Detection performance with our method is also improved. With comparison 33 our method improves calibration by 1.4%J. For more results, see Tab. 4
This scenario evaluates Siml0k in-domain dataset and BDD 100k subset reflecting same category with Siml0k as out-domain dataset. Over baseline, improvement of calibration for in-domain is 4.1 1. and for out-domain 1.0%.. Compared to MbLS [28] our method shows better calibration with the improvement of 16.3%. (in-domain) and 10.5%, (out-domain). Tab. showss further results.
Synthetic and Real:
Reliability diagrams: Fig. plots reliability diagrams on selected MS-COCO classes. It visualizes the D-ECE, and perfect detection calibration means that precision and confidence are aligned.
4.2 Ablations & Analysis
We study ablation and sensitivity experiments with our proposed method. This section presents the impact of the components involved in our proposed method. For the choice of hyperparameters and sensitivity experiments, split the training set of Cityscapes into train/validation sets (80%/20%)
Datasets
Table 5: D-DETR Calibration: Cal- DETR results in improved calibra- tion performance over both in-domain and out-domain datasets, Sim and
In-Domain (Sim 10k)
Out-Domain (BDD100K)
Methods
D-ECE
AP APo.s
D-ECE_
AP
APo.s
Baseline p-DEIR
5
10.3
65.9
90.7
7.3
23.5
46.6
BDDI00K respectively. The evaluation set contains car category. AP and APu.s are also reported.
Temp. Scaling 110
15.7
65.9
90.7
10.5
23.5
46.6
MDCA 12
10.0
64.8
90.3
8.8
22.7
45.7
MbLS 28
22.5
63.8
90.5
16.8
23.4
47.4
TCD 33
90.4
Cal-DETR (Ours)
6.2
65.9
90.8
6.3
23.8
46.5

---

Baseline
Car
CHI-DETR (Duss
Ort
Figure 3: Reliability Diagrams: Baseline
VS. Cal-DETR (Ours): Reliability plots on MS-COCO dataset.
Methods
In-Domain (COCO)
Out-Domain (CorCOCO)
Table 6: Impact of Components: Calibration re- sults MS-COCE & CorCOCO along with the detection performance are shown component-wis
Lnnd Luix
D-ECE
AP
APo.5
D-ECE AP 1 APo.s 1
12.8
44.0
62.9
10.8
23.9
35.8
Logit Modulation (Lmoa) and Logit Mixing (Lmix).
11.0
44.1
62.8
9.9
23.6
35.5
9.5
43.5
63.3
9.2
23.3
35.3
V
8.4
44.4
64.2
8.9
24.0
36.3
Table 7: Impact of logit mixing and loss weightage with Cityscapes split (train/validation))
Cal-DETR
In-Domain
Cal-DETR
In- Dh-Domain
D-ECE
AP
APo.5
D-ECE
APo.5
0.25
10.4
24.6
46.1
0.9, 0.1
8.1
25.0
46.8
À 0.325
10.4
24.7
46.8
61 0.8, 02 0.2
8.0
24.6
46.5
À 0.5
8.1
25.0
46.8
01 0.7, ag 0.3
10.1
25.0
46.7
À 0.75
8.9
24.6
47.0
01 0.6, 02 0.4
10.3
24.3
46.0
A 1.0
8.7
24.4
46.4
(a) Impact of logit mixing 0 and 02 = ai values
(b) Loss weightage values impact CaI-DETR. observe little improvement in detection performance (AP) on À = 0.5 value. With the same value, we that calibration performance is competitive.
Cal-DETR. We observe improvement in detection performance by raising values. We see that calibra- tion performance is competitive with high values.
Impact of Components: We separately investigate the function of logit modulation (Imos) and logit mixing (Lmix) to see the impact calibration performance. In Tab. 6 the calibration performance is improved over the bascline while incorporating individual components in our Cal-DETR. When we combine both, it further improves calibration performance along with the detection performance. We believe that this improvement could be due to greater diversity in logit space, since beyond mixed logits, we have access to non-mixed logits as well.
We study the logit mixing weightage used in our proposed method on a separate validation set. Keeping the range of a E [0.6, 0.9). we observe in the experiments that the best trade-off of detection performance and model calibration is obtained at a; = 0.9. We notice that calibration performance is competitive for higher on values but an overall improvement is observed for a1 = 0.9. Tab. 7 shows the behavior of a in Cal-DETR.
Logit Mixing weightage:
Choice of single at Several mixup strategies operate to mix an input image using another randomly selected image, so usually involves two samples in the process. In our approach, we perform query instance/object level mixup in the logit space. We first build prototypical representation using all positive queries which is then used to achieve a mixup for a given positive query. Owing to this difference from conventional mixup strategies operating, our experiment with conventional mixup leads to suboptimal results. The subpar performance is potentially because the logit mixing suppresses the object representation with lower arbitrary simultaneously having dominant pro- totypical representation from multiple positive queries. to be noted that the sampling strategy (Cal-DETRg) still works better when compared to other calibration methods. We report the results in

---

Datasets
Table 8: Choice of singe Cal-DETR outperforms its vari- ant with sampling strategy (Cal- DETRs) but it still works better when compared to other calibra- tion methods.
In-Domain (Cityscapes)
Out- Domain (Foggy Cityscapes)
Methods
D-ECE
AP
APo.s
D-ECE
AP
APo.s f
Baseline F-DEER
13.8
26.8
49.5
19.5
17.3
TCD AB
11,9
28.3
90.8
14.3
17.6
30.3
Cal-DE De- (Ours)
11.7
28.2
52.3
13.2
17.3
29.6
Cal-DETR (Ours)
8.4
28.4
51.4
11.9
17.6
29.8
the Tab. We empirically find hyperparameter on beta distribution using the split of Cityscapes dataset. For our reported result, we choose /30.3 using this procedure.
Regularizer Loss weightage: The impact of loss weightage is studied over the validation set, which shows that À 0.5 is a better choice to use with the regularizer function. Detec- tion performance is increased with this setting along with the calibration performance. The trend shows that for higher value of X. calibration is more competitive (see Tab. (b).
To investigate the role un- certainty calibration which complements confidence calibration, we analyze the detection expected uncertainty calibration error (D-UCE) 24 in Fig. The formulation described in the
D-UCE complements D-ECE:
25
20
supplementary material.
Limitation: Our approach could struggle with the model calibration for long-taile tailet object detection and a potential future direction.
10
D-UCE
Basalins m0un
Conclusion
Figure 4: D-UCE for MS-COCO eval-
uation set. Uncertainty calibration reduced as compared the baseline.
In this paper, we tackled the problem of calibrating recent transformer-based object detectors by introducing Cal-DETR. We proposed new technique for estimating the uncertainty in transformer-basse based detector, which is then utilized develop an sncertainty-guidees logit modulation technique. To further enhance calibration performance, we developed new logit mixing strategy which acts regularizer on detections with task-speeific loss. Comprehensive experiments on several in-domain and out-domain settings validate the effectiveness of our method against the existing train-time and post-hoc calibration methods. We hope that our work will encourage new research directions for calibrating object detectors.
Acknowledgement
The computational resources were provided by the National Academic Infrastructure for Super- computing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725, and by the Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Center.
References
[1] Blundell, C., J. Cornebise, K. Kavukcuoglu, and D. Wierstra (2015). Weight uncertainty in neural network. In International Conference Machine Learning, pp. 1613-1622.
[2] Carion, N., F. Massa, Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020). End-to-end object detection with transformers. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp. 213-229. Springer.
[3] Chen, L.-C., G. Papandreou. 1. Kokkinos, K. Murphy, and A. L. Yuille (2018). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions pattern analysis and machine intelligence 40(4), 834 848.
10

---

[4] Cordts, M., M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U, Franke, S. Roth, and B. Schiele (2016). The cityscapes dataset for semantic urban scene understanding. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[5] Dai, Z., Cai, Lin, and J. Chen (2021). Up-detr: Unsupervised pre-training for object detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1601-1610.
[6] Dosovitskiy, A., L. Beyer, A. Kolesnikov, D. Weissenbom,, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021). An image is worth 16x 16 words: Trans- formers for image recognition at scale. In International Conference on Learning Representations.
[7] Dusenherry, M. W., D. Tran, E. Choi, J. Kemp, J. Nixon, G. Jerfel, K. Heller, and A. M. Dai (2020). Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 204-213.
[8] Gal, Y. and Z. Ghahramani (2016). Dropout bayesian approximation: Representing model uncertainty in deep learning. international conference on machine learning. pp. 1050-1059, PMLR.
[9] Grigorescu, S., B. Trasnea, T. Cocias, and Macesanu (2020). A survey of deep learning techniques for autonomous driving. Journal of Field Robotics 37(3),
[10] Guo, C., G. Pleiss, Y. Sun, and K. Q. Weinberger (2017). On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1-21-1330. PMLR.
[1I] He, K., X. Zhang, S. Ren, and J. Sun (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778.
[12] Hehhalaguppe, R., J. Prakash, N. Madan, and C. Arora (2022, June). A stitch in time saves nine: train-time regularizing loss for improved neural network calibration. In Proceedings of the IFFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16081-16090.
[13] Hendrycks, D. and T. Dictterich (2019), Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR).
[14] Hernández-Lobato. J. M. and R. Adams (2015). Probabilistic backpropagation for scalable learning of hayesian neural networks. In International Conference on Machine Learning, pp. 1861-1869.
[15] Islam, M., L. Seenivasan, H. Ren, and B. Glocker (2021). Class-distrbutiinn-aaar calibration for long- tailed visual recognition. arxiv preprint or arXi::2109.05263
[16] Ji, B., H. Jung, J. Yoon, K. Kim, and Y. Shin (2019). Bin-wise temperature scaling (bts): Improvement in confidence calibration performance through simple scaling techniques. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 4190 4196.
[17] Schnson-Roberson M., C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan (2017). Driving in the matrix: Can virtual worlds replace gunenngeeerrated annotations for real world tasks? 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 746-753. IEEE
[18] Karandikar, A., N. Cain, D. Tran, B. Lakshminarayanan, J. Shlens, M. C. Mozer, and B. Roelofs (2021). Soft calibration objectives for neural networks. Advances in Neural Information Processing Systems 34, 29768-29779)
[19] Krishnan, R. and O. Tickoo (2020). Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems.
[20] Kull, M., M. Perello Nieto, M. Kängsepp. T. Silva Filho, H. Song, and P. Flach (2019). Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems 32.
[21] Kull, M., T. Silva Filho, and P. Flach (2017). Beta calibration: well-founded and casily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statisties, pp. 623-631. PMLR.
[22] Kuppers, F., J. Kronenberger, A. Shantia, and A. Haselhoff (2020). Multivariate confidence calibration for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 326-327.
11

---

[23] Lakshminarayanan, B., A. Pritzel, and C. Blundell (2016). Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arX.v:1612.01474.
[24] Laves, M.-H., S. Ihler, K.-.. Kortmann, and T. Ortmaier (2019). Well-calibrated model uncertainty with temperature scaling for dropout variational inference. arXiv preprint arXiv: 1355.
[25] Liang, G., Y. Zhang, X. Wang. and N. Jacobs (2020). Improved trainable calibration method for neural networks medical imaging classification. In British Machine Vision Conference (BMVC).
[26] Lin, T.-.,, P. Goyal, R. Girshick, K. He, and P. Dollár (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988.
[27] Lin, T.-Y,, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014). Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer.
[28] Liu, B., I. Ben Ayed, A. Galdran, and Dolz (2022, June). The devil is the margin: Margin-based label smoothing for network calibration. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 80-88.
[29] Liu, Zz,, Lin, Y. Cao, H. Hu, Y. Wei, Zhang, S. Lin, and B. Guo (2021, October). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012-10022.
[30] Louizos, C. and M. Welling (2016). Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference on Machine Learning, pp. 1708-1716.
[31] Mukhoti, J., V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania (2020). Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems 33, 15288-15299.
[32] Müller, R., S. Kornblith, and G. E, Hinton (2019). When does label smoothing help? Advances in neural information processing systems 32.
[33] Munir, M. A., M. H. Khan, M. S, Sarfraz, and M. Ali (2022). Towards improving calibration in object detection under domain shift. In Advances in Neural Information Processing Systems.
(34] Ovadia, Y., E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snock (2019). Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems 32.
[35] Pereyra, G., G. Tucker, J. Chorowski, F.. Kaiser, and G. Hinton (2017). Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:/701 12X.vv/700.0654:
[36] Pinto, F., H. Yang, S.-N. Lim, P. Torr, and P. K. Dokania (2022). Using mixup as regularizer can surprisingly improve accuracy & distr--tiitiiino robustness. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), Advances Neural Information Processing Systems.
[37] Platt, J. al. (1999). Probahilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers 10(3), 61-74.
[38] Ren, S., K. He, Girshick, and Sun (2015). Faster r-cnn: Towards real real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99.
[39] Sakaridis, C., D. Dai, and L. Van Gool (2018). Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision 126(9), 973-992.
[40] Shao, S., Z. Li. T. Zhang, Peng. G. Yu, Zhang, Li, and J. Sun (2019). Objects365: large-scale, high-quality dataset for object detection. In 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pp- 8429-8438.
[41] Sharma, M., O. Saha, A. Sriraman, R. Hebbalaguppe, L. Vig, and S. Karande (2017). Crowdsourcing for chromosome segmentation and deep classification. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 34-11.
[42] Strudel, R., R. Garcia, I. Laptev, and C. Schmid (2021). Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7262-7272.
[43] Tian, Z., C. Shen, H. Chen, and T. He (2019). Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE international conference on computer vision, pp. 9627-9636.
12

---

[44] Tolstikhin, O., N. Houlsby. A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung. A. Steiner. D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy (2021). Mip-mixer: An all-mip architecture for vision. In M. Ranzato, A. Beygelzimer, Dauphin. Liang, and J. W. Vaughan (Eds.), Advances Neural Information Processing Systems, Volume 34, pp.
[45] Tomani, S. Gruber. M. Erdem, D. Cremers, and F. Buettner (2021). Post-hoc uncertainty calibration for domain drift scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10132.
[46] Touvron, H., M. Cord, M. Douze, Massa, Sablayrolles, and H. Jégou (2021). Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-1080 10357. PMLR.
[47] Xie, E., W. Wang, Z. Yu, A. Anandkumar, M. Alvarez, and P. Luo (2021). Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34, 12077-12090.
[48] Yu, F., H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell (2020). Bdd100k: A diverse driving dataset for heterogencous multitask learning. In Proceedings the IEEF/CVF conference on computer vision and pattern recognition, pp. 2636-2645.
[49] Yu, and G. Ali (2019). What's inside the black box? challenges for lawyers and researchers. Legal Information Management 19(1), 2-13.
[50] Zhang, H., M. Cisse, Y. N. Dauphin, and Lopez-Paz (2018), mixup: Beyond empirical risk minimization.
In International Conference on Learning Representations.
[51] Zhang. H., Li, S. Liu, L. Zhang, H. Su, J. Zhu. Ni. and H.-Y. Shum (2023). DINO: DETR with improved denoising anchor hoxes for end-to-end object detection, The Fleventh International Conference on Learning Representations.
[52] Zhang, L., Z. Deng, K. Kawaguchi, A. Ghorbani, and J. Zou (2021). How does mixup help with robustness and generalization? In International Conference on Learning Representations.
[53] Zhu, X., W. Su, L. Lu, B. Li, X. Wang, and J. Dai (2021). Deformable (detr): Deformable transformers for end-to-end object detection, In International Conference on Learning Representations.
13


Reviews:
[{"review_id": "wJeyMgtEJH", "reviewer": "Reviewer_wX9Z", "summary": "This paper proposes a calibrated detection transformer model, which equips the DETR variants with an uncertainty-guided logit modulator and a mixup augmentation for the classification branch of the detector. Specifically, the authors first quantify the uncertainty with the variance among the predicted logits from different decoder layers. Then logits with more uncertainty are suppressed. For mixup, it is similar to the common mixup used in detectors except the mix is between the average of the batch and each sample. Through experiments on various benchmarks, the authors demonstrate the effectiveness of the proposed method.", "questions": "Please refer to the weakness section.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies an important but underexplored problem, calibration in object detectors.\n\n2. The proposed method is lightweight and does not require any network modifications, which makes it more likely to be generalized to more architecture and application scenarios.\n\n3. The authors perform very comprehensive experiments and the results show the effectiveness of the proposed method.", "weaknesses": "1. It is unclear why the variation in the output of decoder layers can measure the uncertainty of the class prediction. Is it based on intuition? And is there any experimental support?\n\n2. As one of the two major contributions, logit mixing for confidence calibration is not technically novel enough. First, applying mixup for the classification branch of the object detector is considered a well-known technique [1][2]. Second, in [1], the mixup is considered a trick instead of a technical contribution. Finally, the difference in mixup between [1] and this paper is subtle.\n\n[1] Zhang, Zhi, et al. \"Bag of freebies for training object detection neural networks.\"\n[2] Ge, Zheng, et al. \"Yolox: Exceeding yolo series in 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: It is unclear why the variation in the output of decoder layers can measure the uncertainty of the class prediction…**\n\n**A1:** Decoder layer contains multiple dropout layers that make it stochastic in nature and so allows capturing variation in logit space to estimate uncertainty. Logits are the raw output of the model, and with logits of all decoder layers, uncertainty is computed to modulate/scale logits of the final layer. \n\n---\n\n**Q2: As one of the two major contributions, logit mixing for confidence calibration is not technically novel enough…**\n\n**A2:** `High-level intuition:` To our best of knowledge, our work is the first one that strives to improve the calibration performance of recent SOTA DETR-based pipelines by proposing an uncertainty-guided logit modulation and a logit mixing approach during training time without any hold out dataset (Reviewer Gc9J appreciates the significance of our approach). In [1], the aim is to apply complex spatial transforms to introduce occlusions and spatial signal perturbations in an effort to improve model generalization.  [2] applies the mixup from [1] and also leverages Mosaic as data augmentations for object detection to enhance detection performance. On the contrary, our main goal is to improve the calibration of  object detectors by developing a mixing approach in the logit space along with a regularizer loss that can be used with a detection-specific loss function. \n\n`Formulation and Technical Details:` [1] operates in input pixel space and performs a geometry preserved alignment of mixed images for object detection, where image pixels are mixed up and object labels are merged as a new array. Also, the rest of formulation details are same as the original mixup.  On the other hand, we operate in logit space, first, where we leverage positive queries to build a prototypical representation and then use this to achieve logit mixing for a given query. Furthermore, we employ a regularizer loss by leveraging the proposed logit mixing output. Overall, we fundamentally differ from [1] and [2] in terms of high-level intuition, formulation, and technical details.  \n\n[1] Zhang, Zhi, et al. \"Bag of freebies for training object detection neural networks.\" \n\n[2] Ge, Zheng, et al. \"Yolox: Exceeding yolo series in 2021.\n"}, {"review_id": "GLEuxlQNQk", "reviewer": "Reviewer_95V6", "summary": "This paper focuses on performing calibration for DETR, particularly for Deformable-DETR and UP-DETR. The authors first propose an approach for quantifying uncertainty in DETRs, which is built from the variation in the output of decoder layers. Then they develop an uncertainty-guided logit modulation mechanism and a logit mixing approach as regularizers. Experiments on three in-domain and four out-domain scenarios show the effectiveness of the proposed method in calibrating.", "questions": "1. I am confused that what benefits can we get when the D-ECE is lower. It seems that the box AP does not show clear differences between Cal-DETR and the baseline model in both in-domain and out-domain scenarios. If the calibration can not bring benefits in detecting objects, then what benefits can we get by performing calibration?\n2. It seems that the proposed uncertainty-guided logit modulation and the logit mixing approach only perform on the output logits of the last decoder layer. Do the authors try to implement them into all decoder layers' output logits?\n3. In Table 1 and Table 4, it seems that the D-ECE of the out-domain is lower than the D-ECE of the in-domain scenarios. Why? Does this phenomenon have a further explanation?", "limitations": "See the above sections.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The results of D-ECE are good in both in-domain and out-domain scenarios. \nThe experiments are extensive across various datasets and settings.", "weaknesses": "1. The proposed method has limited novelty. Although the authors claim that they do calibrate on object detection, the proposed method also performs the logits and shares large similarities with the methods in classification. \n2. The way to quantify uncertainty in the DETR pipeline using the variation in the output of decoder layers is a straightforward idea, which may not be propriety to claim as a contribution. Moreover, the relationship between the proposed uncertainty-guided logit modulation mechanism and the logit mixing approach is not clear. It is more like a combination of two techniques, lacking deep insights.\n3. When introducing D-ECE in Sec3.2, it would be better to cite [22] to indicate that the used measures follow previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The proposed method has limited novelty. Although the authors claim…**\n\n**A1:** To the best of our knowledge, this is the first work that strives to improve the calibration performance of recent SOTA ViT-based detectors by proposing an uncertainty-guided logit modulation and a logit mixing approach. We also refer to Reviewer Gc9J, who acknowledges the significance of our proposed approach, and Reviewer co8a appreciates no performance drop while improving model calibration. We would like to highlight that the relevant mixup strategies also target improved calibration, but majorly for the classification task [I, II], using image-level mixup in the input space. In our approach, we devise an uncertainty mechanism that does minimal changes to the model architecture and is also computationally efficient to get uncertainty-guided logits, which is followed by a logit mixing built with prototypical representations of positive queries. Both our methods are effective and complementary to each other in calibrating in-domain and out-domain predictions. Reviewer 95V6 comment does not provide details on what methods in classification lead to the concern about limited novelty. However, if specific methods are provided, we will be happy to explain the specific technical differences. \n\n[I] Zhang, Linjun, et al. \"When and how mixup improves calibration.\" International Conference on Machine Learning. PMLR, 2022.\n\n[II] Thulasidasan, Sunil, et al. \"On mixup training: Improved calibration and predictive uncertainty for deep neural networks.\" Advances in Neural Information Processing Systems 32 (2019).\n\n------------------------------------------\n\n**Q2: The way to quantify uncertainty in the DETR pipeline using the variation in the output of decoder layers is a straightforward idea…**\n\n**A2:** Although quantifying uncertainty using variation in the output of decoder layers is a straightforward idea, and simple to implement, note that:\n\n`(a)`\tIt is effective for scaling the logits during train time to calibrate DETR-based object detectors both for in-domain and out-domain scenarios, as shown in our experiments.\n\n`(b)` Requires no modifications to the underlying architecture and is computationally efficient thereby revealing its plug-and-play nature (L:189-192).\n\n`(c)`\t Finally, to our knowledge, it has not been explored in quantifying uncertainty and calibrating modern DETR-based object detection pipelines. \n\nAkin to uncertainty-guided logit modulation, our logit mixing technique also operates in the logit space. It is designed to further enhance the calibration performance by introducing the logit mixing and a regularizer loss which allows model training using non-zero-entropy supervisory signal.\n\n------------------------------------------\n\n**Q3: When introducing D-ECE in Sec3.2, it would be better to cite…**\n\n**A3:** We will add the reference [22] in Sec 3.2.\n\n------------------------------------------\n\n**Q4: I am confused that what benefits can we get when the D-ECE is lower…**\n\n**A4:** The scope of model calibration is to reduce model miscalibration, which is quantified using the D-ECE metric while preserving the (object detection) performance. In other words, a calibrated model should be able to predict the class confidence that matches the actual likelihood of its correctness. It is of great value as this not only improve the overall trust in model predictions but could also increase the adoption of detector in several safety-critical applications (L:27-30).   \n\n------------------------------------------\n\n**Q5: It seems that the proposed uncertainty-guided logit modulation and the logit mixing approach only perform on the output logits of the last decoder layer…**\n\n**A5:** With minimal modification and keeping the DETR-based architecture intact, we implement the logit modulation and logit mixing to the last decoder layer only. DETR takes logits from the last layer only for probability output distribution. We provide results when uncertainty-guided logit modulation and logit mixing are applied to all decoder layers (**Cal-DETR$_{allDec}$**) on Cityscapes (in-domain) and Foggy Cityscapes (out-domain). The calibration performance of **Cal-DETR$_{allDec}$** is inferior to **Cal-DETR**, potentially due to less refined object queries (updates iteratively) as compared to the last layer.  \n\n\n|          \t\t\t| In Domain  |      |     \t| Out Domain |      |      |\n|---------------------------|-------|------|------|-------|------|------|\n| Methods                   | D-ECE | AP   | mAP  | D-ECE | AP   | mAP  |\n| Baseline$_{D-DETR}$                    | 13.8  | 26.8 | 49.5 \t| 19.5  | 17.3 | 29.3 |\n| TS                       \t| 12.6  | 26.8 | 49.5 \t| 14.6  | 17.3 | 29.3 |\n| MDCA                      | 13.4  | 27.5 | 49.5 \t| 17.1  | 17.7 | 30.3 |\n| MbLS                      \t| 12.1  | 27.3 | 49.7 \t| 20.0  | 17.1 | 29.1 |\n| TCD                       \t| 11.9  | 28.3 | 50.8 \t| 14.3  | 17.6 | 30.3 |\n| **Cal-DETR$_{allDec}$** | 13.0  | 27.5 | 51.0 \t| 12.8  | 17.3 | 29.7 |\n| **Cal-DETR**                 | 8.4   | 28.4 | 51.4 \t| 11.9  | 17.6 | 29.8 |\n\n------------------------------------------\n\n**Q6: In Table 1 and Table 4, it seems that the D-ECE of the out-domain…**\n\n**A6:** Modern deep neural networks are usually overconfident because they are trained with zero entropy signal with entropy minimization objective. We train the detection model with in-domain data only with modulation and logit mixing modules, that largely improves calibration for in-domain. Since our modules act as a regularizer for calibrating in-domain detections, it facilitates learning domain generalizable features [I], which also helps in calibrating out-domain detections and improving out-domain detection performance.  \n\n[I] Wald, Yoav, et al. \"On calibration and out-of-domain generalization.\" Advances in neural information processing systems 34 (2021): 2215-2227.\n"}, {"review_id": "twndqny7Xn", "reviewer": "Reviewer_co8a", "summary": "This paper proposing a mechanism for calibrated detection transformers (Cal-DETR), particularly for Deformable-DETR and UP-DETR, which consists of quantifying uncertainty, an uncertainty-guided logit modulation and a logit mixing approach. Results show the method improves the baselines in calibrating both in-domain and out-domain detections while maintaining or even improving the detection performance.", "questions": "Will the method still be effective when using a strong or even a sota model?", "limitations": "The authors may try more baselines and report results. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I'm familiar with object detection and DETR, but not Model calibration.  It seems that the model calibration is a valuable problem, and the proposed method does improve it without performance drop.", "weaknesses": "It seems the performance of selected baselines is relatively low, e.g. the performance of Deformable-DETR is only 44.0 in COCO as mentioned in Table 1. Will the method still be effective when using a strong or even a sota model, like [DINO](https://github.com/IDEA-Research/DINO)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Will the method still be effective when using a strong or even a sota model, like DINO?**\n\n**A1**: As suggested by the reviewer, we provide the results on COCO (in-domain) and CorCOCO (out-domain) with the DINO model, as shown below.  Our **Cal-DETR** improves the calibration performance of this strong baseline for both in-domain and out-domain settings.\n\n|          \t\t| In Domain |      |      \t\t| Out Domain |      |      |\n|----------|-----------|------|------|------------|------|------|\n| Methods  \t| D-ECE     | AP   | mAP  \t| D-ECE      | AP   | mAP  |\n| Baseline$_{DINO}$     \t| 15.5      | 49.0 | 66.6 \t\t| 13.2       | 27.3 | 39.2 |\n| TS       \t| 15.1      | 49.0 | 66.6 \t\t| 14.3       | 27.3 | 39.2 |\n| MbLS     \t| 19.1      | 48.6 | 66.0 \t\t| 15.9       | 26.9 | 38.4 |\n| MDCA     \t| 16.3      | 48.6 | 66.1 \t\t| 14.0       | 26.7 | 38.4 |\n| TCD      \t| 15.6      | 48.5 | 66.1 \t\t| 12.9       | 26.8 | 38.5 |\n| **Cal-DETR** \t| 11.7      | 49.0 | 66.5 \t\t| 10.6       | 27.5 | 39.3 |\n\n\nDINO Calibration: MS-COCO & CorCOCO. Results are reported on the Epoch12 (4-scale) setting and Cal-DETR shows improvement in calibration as compared to other methods including baseline"}, {"review_id": "sA40Hy5yyq", "reviewer": "Reviewer_Gc9J", "summary": "The paper proposes a method to improve the calibration performance of transformer-based object detectors. In their approach, they first present a way to quantify the uncertainty of each logit using the variance of the outputs of different transformer decoder layers. Then, with the motivation that a higher uncertainty generally implies a poor calibration, they downscale the logits with high uncertainty. Second, they introduce a mixup based approach that is applied to the queries that match with the objects. Specifically, they create a prototype as the mean of the logits of the positive queries and mix it up with the logits of the foreground objects. They incorporate their method on Deformable DETR and UP-DETR, and observed notable improvements across different datasets.", "questions": "- Specifically for example for Table 1, how many detections per image on average do Baseline D-DETR, temp scaling, TCD (as the closest counterpart) and Cal-DETR output (which are then used as the inputs to estimate Detection ECE)? Or is there any thresholding of the top-100 detections while estimating the DECE?\n\n- Can you please confirm that you obtain the prototypical representation query for each iteration during training? And I'd recommend making this explicit in the paper (L217-221). Also I'd recommend making it more explicit that you compute the mean of the logits across all positive queries (L217-218). \n\n- In constrast to several mixup strategies that sample mixing coefficient (\\alpha in Eq.(4)) from a distribution, why did you choose a single \\alpha value?\n\n- How do you obtain the labels after smoothing ( `c_i in L225). I think this should be explicitly defined.", "limitations": "Briefly mentioned", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is a training time approach, hence does not require an extra hold-out validation set.\n- The improvement in the calibration performance is notable and outperforms existing approaches consistently in several datasets.\n- While the reilabiility of the estimated uncertainties are not investigated, the proposed approach to do so is intuitive and does not introduce an additional burden on the detection architecture.\n- The paper is written clearly and it is easy to follow.", "weaknesses": "- I think the logit mixup strategy that the authors introduce is quite related to a cited work [36]. The main differences are that the authors apply mixup in the logit space (which is again special case of manifold mixup), they use a fixed mixing coefficient instead of sampling it and the method is applied for object detection. I haven't seen in the paper (especially in related work or in Section 3.3.2) such a clarification. Therefore, I'd see the contribution of the authors in logit mixup as the extension of the regularized mixup [36] to object detection. I think this is still an important contribution but I'd like to see an explicit discussion with the prior related work.\n\n- Recently [A] showed that the number of detections that are input to the calibration measure has a clear effect on the calibration performance. Specifically, such measures are easy to be misleading due to the maximum number of allowed detections, which is for example 100 for COCO dataset. For example, a detector that outputs 100 detection has more advantage to achieve lower calibration error compared to a detector that outputs less and does not fill the quota of 100. So, I wonder whether such a case exists here. Specifically for example for Table 1, how many detections per image on average do Baseline D-DETR, temp scaling, TCD (as the closest counterpart) and Cal-DETR output (which are then used as the inputs to estimate Detection ECE)? Or is there any thresholding of the top-100 detections while estimating the DECE? If so, how? Related to this, a minor suggestion (that I am not considering in my rating as a weakness as [A] came out in CVPR in June) can be to include a comparison in terms of Localisation-aware ECE in the way that [A] suggests to avoid such a doubt in the final version if the paper is accepted.\n\n- The authors claim that they propose a method to quantify the uncertainty for each logit. However, further insight on these uncertainties are not provided. For example, there is no evidence that the estimated uncertainties are reliable and can be used for different purposes.\n\n[A] Towards Building Self-Aware Object Detectors via Reliable Uncertainty Quantification and Calibration, CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: I think the logit mixup strategy that the authors introduce is quite related to a cited work [36]… I think this is still an important contribution but I'd like to see an explicit discussion...**\n\n **A1:** Thanks for finding the logit mixup strategy an important contribution. Our approach can be seen as an extension of regularized mixup [36] from classification to object detection with important differences in the studied setting, the space in which mixup is applied, and the way mixup-based regularization is performed. We explain the differences and relationships between both approaches below:\n\n•\t`High-level motivation:` In [36], the goal is to improve the uncertainty estimation of mixup without sacrificing accuracy, especially under severe distribution shifts for the task of image classification. The goal of our logit mixing is to further improve the calibration performance of DETR for the task of object detection (Cal-DETR).\n\n•\t`Technical Details:` The mixup in [36] operates in input pixel space, studies the effect of hyperparameter beta of the sampling distribution for mixing coefficient and introduces a regularizer. Besides improving accuracy, it also helps in improving the quality of predictive uncertainty estimates. Ours operates in logit space, leverages positive queries to build a prototypical representation, and then uses this to achieve logit mixing for a given query. We also employ a regularizer loss, but it leverages the proposed logit mixing output. Our logit mixing complements the logit mixing and further improves the detection calibration performance of DETR-based pipelines.   \n\nWe will include the discussion with prior related work [36] in the final version.\n\n---\n\n**Q2: In constrast to several mixup strategies that sample mixing coefficient (\\alpha in Eq.(4)) from a distribution,...**\n\n**A2:** We empirically find $\\beta$ hyperparameter on beta distribution using the split of Cityscapes dataset (L:298). For our reported result, we choose $\\beta$=0.3 using this procedure. \n\n| $\\beta$ (sampling) | D-ECE | AP   | mAP  |\n|-------------------|-------|------|------|\n| 0.3               | 11.4  | 24.5 | 46.4 |\n| 0.5               | 13.0  | 24.5 | 46.2 |\n| 1.0               | 13.6  | 24.7 | 45.7 |\n\nSeveral mixup strategies operate to mix an input image using another randomly selected image, so it involves two samples in the process. In our approach, we perform a query instance/object level mixup in the logit space. We first build a prototypical representation using all positive queries which is then used to achieve a mixup for a given positive query. Owing to this difference from conventional mixup strategies operating, our experiment with conventional mixup leads to suboptimal results.  The subpar performance is potentially because the logit mixing suppresses the object representation with lower arbitrary $\\alpha$ simultaneously having dominant prototypical representation from multiple positive queries. It is to note that sampling strategy (Cal-DETR$_{sample}$) still works better when compared to other calibration methods. We report the results on Cityscapes and Foggy Cityscapes in the table below:\n\n|          \t\t| In Domain |      |      \t| Out Domain |      |      |\n|---------------------|-------|------|------|-------|------|------|\n| Methods             | D-ECE | AP   | mAP  | D-ECE | AP   | mAP  |\n| Baseline$_{D-DETR}$              | 13.8  | 26.8 | 49.5 \t| 19.5  | 17.3 | 29.3 |\n| TS                  \t    | 12.6  | 26.8 | 49.5 \t| 14.6  | 17.3 | 29.3 |\n| MDCA                | 13.4  | 27.5 | 49.5 \t| 17.1  | 17.7 | 30.3 |\n| MbLS                | 12.1  | 27.3 | 49.7 \t| 20.0  | 17.1 | 29.1 |\n| TCD                  | 11.9  | 28.3 | 50.8 \t| 14.3  | 17.6 | 30.3 |\n| **Cal-DETR$_{sample}$** | 11.7  | 28.2 | 52.3 \t| 13.2  | 17.3 | 29.6 |\n| **Cal-DETR**         | 8.4   | 28.4 | 51.4 \t| 11.9  | 17.6 | 29.8 |\n\n---\n\n**Q3. Recently [A] showed that the number of detections that are…? Or is there any thresholding...?**\n\n**A3:** We thank the reviewer for mentioning useful insights about how the number of detections affects the calibration error. Note that in our implementation, we calibrate the detection model having prediction scores >=0.3. Specifically, we use a repository [22], where we set the threshold on scores prior to estimating the calibration which leads to a reliable and robust D-ECE measure in our case. \n\n---\n\n**Q4. The authors claim that they propose a method to quantify the uncertainty for each logit. However,…**\n\n**A4:** [I] uses layer ensemble and some other works that use output space ensemble for the estimation of uncertainty (e.g., MC dropout/Deep Ensembles). In a similar spirit to these works, we exploit the logit space from different decoder layers (containing multiple dropout layers) for the estimation of uncertainty. Results show that our method is more effective in improving model calibration for both in-domain and out-domain scenarios.\n\n[I] Kushibar, Kaisar, et al. \"Layer ensembles: A single-pass uncertainty estimation in deep learning for segmentation.\" International Conference on Medical Image Computing and Computer-Assisted Intervention, 2022.\n\n---\n\n**Q5: Specifically for example for Table 1, how many detections… ? Or is there any thresholding...**\n\n**A5:** We explicitly use threshold on scores i.e. >=0.3 while estimating D-ECE\n\n---\n\n**Q6: Can you please confirm that you obtain the prototypical representation query…**\n\n**A6:** Yes, we obtain the prototypical representation of queries for each iteration during training. We surely cater to the recommendations of the reviewer in the final version and add more details.\n\n---\n\n**Q7: How do you obtain the labels after smoothing…**\n\n**A7:** $\\alpha$ value for a given positive query contributes to smooth the label while remaining positive queries that contribute to forming a prototypical representation share 1-$\\alpha$ value to smooth corresponding labels uniformly. We will add this detail in the final version.\n"}]

Author rebuttal:
We thank the reviewers (wX9Z, 95V6, co8a, Gc9J) for the positive and thoughtful feedback, and we appreciate the comments to improve our work.

**Reviewer Gc9J:** "The proposed method does not require an extra hold-out validation set, as it is a training time approach. Improvement in the calibration performance is notable. The proposed approach is intuitive and does not introduce an additional burden. Paper is easy to follow."

**Reviewer wX9Z:** "An important and underexplored problem, calibration in object detectors. The proposed method is lightweight and does not require any network modifications."

**Reviewer co8a:** "Model calibration is a valuable problem, and the proposed method does improve it without a performance drop."

**Reviewer 95V6:** "Calibration results are good in both in-domain and out-domain scenarios. Extensive experiments."

We summarize the main points presented in our response and we kindly hope that we have addressed all the questions.

+ We address the questions and provide clarifications and details that have improved our work. 
+ We include comparisons with another DETR-based pipeline as well to show the effectiveness of our method.
+ We include more variants of our proposed approach that bring further insight into our choice.
+ We will update the final version based on the recommendations of the reviewers (i.e. additional results with a DETR-based DINO model to show scalability, provide insights and clarifications, and more discussions with close works in literature).

Meta-review: The authors propose a method to calibrate DETR-based detection systems. The method applies to different variant (Deformable-DETR, DINO). 
The authors addressed the concerns of all the reviewers during the rebuttal period, and no issues seem to remain as all reviewers are leaning towards acceptance.
Decision: Accept (poster)