SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We employ a regularization loss by leveraging the proposed logit mixing output, denoted as  $\hat{\psi}(\mathbf{x}_i)$ , which is expressed as:  $\mathcal{L}_R(p(\bar{c}|\hat{\psi}(x_i), \acute{c}_i)$ , where  $\acute{c}_i$  contains the labels after smoothing with mixing values. Instead of one-hot labels,  $\alpha$  determines the weightage of classes for object instances. Specifically  $\alpha$  value for a given positive query contributes to smoothing the label while remaining positive queries that contribute to forming a prototypical representation share  $1 - \alpha$  value to smooth corresponding labels uniformly. The loss  $\mathcal{L}_R$  is used together with the task-specific classification loss  $\mathcal{L}_{cls}(p(\bar{c}|\psi(x_i)), \hat{c}_i)$  to obtain a joint loss formulation as:  $\mathcal{L}_{cls} + \lambda \mathcal{L}_R$ , where  $\psi(\mathbf{x}_i)$  denotes the non-mixed logits and  $\lambda$  controls the contribution of regularization loss. The proposed regularizer loss tends to capture the information from the vicinity of object instances in the logit space by mixing logits and constructing a non-zero entropy target distribution. This penalizes the model for producing overconfident predictions by possibly reducing overfitting [52, 36]. Our empirical results (Sec. 4.1) show that this not only improves model calibration but also improves the overall detection performance in most scenarios and it is also complementary to our uncertainty-guided logit modulation technique.
4 Experiments & Results
Datasets: To perform experiments, we utilize various in-domain and out-domain benchmark datasets. Details are as follows: MS-COCO [27] is a large-scale object detection dataset for real images containing 80 classes. It splits into 118K train images, 41K test images, and 5K validation images. The train set (train2017) is used for training while the validation set (val2017) is utilized for evaluation. CorCOCO contains a similar set of images as present in val2017 of MS-COCO but with the corrupted version [13]. Random corruptions with random severity levels are introduced for evaluation in an out-domain scenario. Cityscapes [4] is an urban driving scene dataset contains 8 classes person, rider, car, truck, bus, train, motorbike, and bicycle. The training set consists of 2975 images while the 500 validation images are used as the evaluation set. Foggy Cityscapes [39] is a foggy version of Cityscapes, 500 validation images simulated with severe fog are used for evaluation in this out-domain scenario. BDD100k [48] contains 100k images from which 70K are training images, 20K are for the test set and 10K for the validation set. Categories are similar to Cityscapes and for evaluation, we consider the validation set and make a subset of daylight images as an out-domain scenario, reducing the set to 5.2K images. Sim10k [17] contains 10k synthetic images with car category from which the training split is 8K images and for the evaluation set, the split is 1K images.
Datasets (validation post-hoc): For the given scenarios, we opt separate validation datasets for the post-hoc calibration method (temperature scaling). In the MS-COCO scenario, we utilize the Object365 [40] validation dataset reflecting similar categories. For the Cityscapes scenario, a subset of the BDD100k train set is used as a validation set. And for the Sim10k scenario, we use the remaining split of images.
Detection Calibration Metric: For the calibration of object detectors, we report our results with the evaluation metric, which is the detection expected calibration error (D-ECE) [22]. It is computed for both in-domain and out-domain scenarios.
Implementation Details: We use transformer-based state-of-art-detectors Deformable-DETR (D-DETR), UP-DETR, and DINO as baselines respectively, which are uncalibrated. With the respective in-domain training datasets, we train D-DETR with our proposed method and compare it with baseline, post-hoc approach [10], and train time calibration losses [28, 12, 33] integrated with D-DETR. We
Datasets Methods	In-Domain (COCO)	Out-Domain (CorCOCO)
D-ECE ↓	AP ↑	AP0.5 ↑	D-ECE ↓	AP ↑	AP0.5 ↑
Baseline D-DETR [53]	12.8	44.0	62.9	10.8	23.9	35.8
Temp. Scaling [10]	14.2	44.0	62.9	12.3	23.9	35.8
MDCA [12]	12.2	44.0	62.9	11.1	23.5	35.3
MbLS [28]	15.7	44.4	63.4	12.4	23.5	35.3
TCD [33]	11.8	44.1	62.9	10.4	23.8	35.6
Cal-DETR (Ours)	8.4	44.4	64.2	8.9	24.0	36.3
Datasets Methods	In-Domain (COCO)	Out-Domain (CorCOCO)
D-ECE ↓	AP ↑	AP0.5 ↑	D-ECE ↓	AP ↑	AP0.5 ↑
Baseline UP-DETR [5]	25.5	37.1	57.3	27.5	19.9	32.5
Temp. Scaling [10]	26.4	37.1	57.3	28.2	19.9	32.5
MDCA [12]	26.5	36.9	57.5	28.6	19.4	31.8
MbLS [28]	25.4	36.3	57.0	26.5	20.1	32.9
TCD [33]	25.2	36.4	57.2	26.8	19.1	31.5
Cal-DETR (Ours)	23.9	37.3	57.8	25.5	20.0	32.6
Table 1: Deformable-DETR (D-DETR) Calibration: MS-COCO & CorCOCO. Calibration results along with the detection performance. Cal-DETR improves calibration as compared to baseline, other train-time losses, and TS methods.
Table 2: UP-DETR Calibration: MS-COCO & CorCOCO. Results are reported on Epoch  ${}_{150}$  setting and Cal-DETR shows improvement in calibration as compared to other methods including baseline.
find empirically over the validation set and use the logit mixing parameters  $\alpha_{1} = 0.9$ ,  $\alpha_{2} = 0.1$  and  $\lambda = 0.5$  (see Sec. 4.2). For the classification loss, the focal loss is incorporated, and for localization, losses are generalized IoU and L1 as described in [26, 2]. For more implementation details, we refer readers to D-DETR [53], UP-DETR [5] and DINO [51].
Real and Corrupted: We evaluate the calibration performance on large-scale datasets (MS-COCO as in-domain and its image corrupted version as out-domain, CorCOCO). The CorCOCO evaluation set is constructed by incorporating random severity and random corruption levels from [13]. Our method not only provides lower calibration errors but also shows improved detection performance. For D-DETR, in comparison to the baseline uncalibrated model, our method improves calibration of  $4.4\% \downarrow$  (in-domain) and  $1.9\% \downarrow$  (out-domain). In this dataset, we observe that classification-based train time calibration losses (MbLS [28] & MDCA [12]) are sub-optimal for the detection problems and worse than the baseline in some scenarios. We show that our method outperforms train time losses and achieves better calibration, e.g. in comparison with detection-based calibration loss (TCD [33]) improvements are  $3.4\% \downarrow$  (in-domain) and  $1.5\% \downarrow$  (out-domain). For UP-DETR, our method improves calibration from baseline by  $1.6\% \downarrow$  (in-domain) and  $2.0\% \downarrow$  (out-domain). For more results, we refer to Fig. 2, Tab. 1 & Tab. 2. We also provide the results on COCO (in-domain) and CorCOCO (out-domain) with the DINO model, as shown in Tab. 3. Our Cal-DETR improves the calibration performance of this strong baseline for both in-domain and out-domain settings.
Figure 2: D-DETR Calibration: MS-COCO & Cityscapes as in-domains and respective Cor-COCO & Foggy Cityscapes as out-domains. Our proposed method (Cal-DETR) effectively decreases D-ECE compared to methods: baseline, post-hoc, and train-time losses.
Datasets
Methods	In-Domain (COCO)	Out-Domain (CorCOCO)
D-ECE ↓	AP ↑	AP0.5 ↑	D-ECE ↓	AP ↑	AP0.5 ↑
Baseline DINO [51]	15.5	49.0	66.6	13.2	27.3	39.2
Temp. Scaling [10]	15.1	49.0	66.6	14.3	27.3	39.2
MDCA [12]	16.3	48.6	66.1	14.0	26.7	38.4
MbLS [28]	19.1	48.6	66.0	15.9	26.9	38.4
TCD [33]	15.6	48.5	66.1	12.9	26.8	38.5
Cal-DETR (Ours)	11.7	49.0	66.5	10.6	27.5	39.3
Table 3: DINO Calibration: MS-COCO & CorCOCO. Results are reported on the Epoch12 (4-scale) setting and Cal-DETR shows improvement in calibration as compared to other methods including baseline.
Table 4: D-DETR Calibration: Results are shown in comparison with baseline, post-hoc, and train-time loss methods. Cal-DETR shows improvement in detection calibration on evaluation sets of the in-domain dataset (Cityscapes) and out-domain datasets (Foggy Cityscapes & BDD100k). AP and  $\mathrm{AP}_{0.5}$  are also reported.
Datasets Methods	In-Domain (Cityscapes)	Out-Domain (Foggy Cityscapes)	Out-Domain (BDD100k)
D-ECE ↓	AP ↑	AP0.5 ↑	D-ECE ↓	AP ↑	AP0.5 ↑	D-ECE ↓	AP ↑	AP0.5 ↑
Baseline D-DETR [53]	13.8	26.8	49.5	19.5	17.3	29.3	11.7	10.2	21.9
Temp. Scaling [10]	12.6	26.8	49.5	14.6	17.3	29.3	24.5	10.2	21.9
MDCA [12]	13.4	27.5	49.5	17.1	17.7	30.3	14.2	10.7	22.7
MbLS [28]	12.1	27.3	49.7	20.0	17.1	29.1	11.6	10.5	22.7
TCD [33]	11.9	28.3	50.8	14.3	17.6	30.3	12.8	10.5	22.2
Cal-DETR (Ours)	8.4	28.4	51.4	11.9	17.6	29.8	11.4	11.1	23.9
Weather: Our method improves calibration over the baseline, post-hoc, and train-time losses with significant margins. Also, with our method, detection performance improves or is competitive with better calibration scores. Over baseline, we achieve an improvement of  $5.4\% \downarrow$  (in-domain) and  $7.6\% \downarrow$  (out-domain). Tab. 4 and Fig. 2 show further results.
Urban Scene: We show that our method improves calibration and/or competitive, especially in the out-domain. For example, with MDCA [12], we improve calibration by  $5.0\% \downarrow$  (in-domain) and  $2.8\% \downarrow$  (out-domain). Detection performance with our method is also improved. With comparison to [33], our method improves calibration by  $1.4\% \downarrow$ . For more results, see Tab. 4.
Synthetic and Real: This scenario evaluates Sim10k as in-domain dataset and BDD100k subset reflecting same category with Sim10k as out-domain dataset. Over baseline, improvement of calibration for in-domain is  $4.1\% \downarrow$  and for out-domain  $1.0\% \downarrow$ . Compared to MbLS [28], our method shows better calibration with the improvement of  $16.3\% \downarrow$  (in-domain) and  $10.5\% \downarrow$  (out-domain). Tab. 5 shows further results.
Reliability diagrams: Fig. 3 plots reliability diagrams on selected MS-COCO classes. It visualizes the D-ECE, and a perfect detection calibration means that precision and confidence are aligned.
4.2 Ablations & Analysis
We study ablation and sensitivity experiments with our proposed method. This section presents the impact of the components involved in our proposed method. For the choice of hyperparameters and sensitivity experiments, we split the training set of Cityscapes into train/validation sets (80%/20%).
Datasets
Methods	In-Domain (Sim10k)	Out-Domain (BDD100k)
D-ECE ↓	AP ↑	AP0.5 ↑	D-ECE ↓	AP ↑	AP0.5 ↑
Baseline D-DETR [53]	10.3	65.9	90.7	7.3	23.5	46.6
Temp. Scaling [10]	15.7	65.9	90.7	10.5	23.5	46.6
MDCA [12]	10.0	64.8	90.3	8.8	22.7	45.7
MbLS [28]	22.5	63.8	90.5	16.8	23.4	47.4
TCD [33]	8.6	66.4	90.4	6.6	23.1	46.0
Cal-DETR (Ours)	6.2	65.9	90.8	6.3	23.8	46.5
Table 5: D-DETR Calibration: CalDETR results in improved calibration performance over both in-domain and out-domain datasets, Sim10K and BDD100K respectively. The evaluation set contains car category. AP and  $\mathrm{AP}_{0.5}$  are also reported.
Figure 3: Reliability Diagrams: Baseline [53] vs. Cal-DETR (Ours): Reliability plots on MS-COCO dataset.
Methods	In-Domain (COCO)	Out-Domain (CorCOCO)
Lmod	Lmix	D-ECE ↓	AP ↑	AP0.5 ↑	D-ECE ↓	AP ↑	AP0.5 ↑
12.8	44.0	62.9	10.8	23.9	35.8
✓	11.0	44.1	62.8	9.9	23.6	35.5
✓	9.5	43.5	63.3	9.2	23.3	35.3
✓	✓	8.4	44.4	64.2	8.9	24.0	36.3
Table 6: Impact of Components: Calibration results on MS-COCO & CorCOCO along with the detection performance are shown component-wise. Logit Modulation  $(\mathrm{L}_{\mathrm{mod}})$  and Logit Mixing  $(\mathrm{L}_{\mathrm{mix}})$ .
Cal-DETR	In-Domain
D-ECE ↓	AP	AP0.5
α1 = 0.9, α2 = 0.1	8.1	25.0	46.8
α1 = 0.8, α2 = 0.2	8.0	24.6	46.5
α1 = 0.7, α2 = 0.3	10.1	25.0	46.7
α1 = 0.6, α2 = 0.4	10.3	24.3	46.0
Table 7: Impact of logit mixing and loss weightage with Cityscapes split (train/validation).
Cal-DETR	In-Domain
D-ECE↓	AP	AP0.5
λ = 0.25	10.4	24.6	46.1
λ = 0.325	10.4	24.7	46.8
λ = 0.5	8.1	25.0	46.8
λ = 0.75	8.9	24.6	47.0
λ = 1.0	8.7	24.4	46.4
(a) Impact of logit mixing  $\alpha_{1}$  and  $\alpha_{2} = 1 - \alpha_{1}$  values in Cal-DETR. We observe improvement in detection performance by raising  $\alpha$  values. We see that calibration performance is competitive with high  $\alpha$  values.
(b) Loss weightage  $\lambda$  values impact in Cal-DETR. We observe little improvement in detection performance (AP) on  $\lambda = 0.5$  value. With the same value, we see that calibration performance is competitive.
Impact of Components: We separately investigate the function of logit modulation  $(\mathrm{L}_{\mathrm{mod}})$  and logit mixing  $(\mathrm{L}_{\mathrm{mix}})$  to see the impact on calibration performance. In Tab. 6, the calibration performance is improved over the baseline while incorporating individual components in our Cal-DETR. When we combine both, it further improves calibration performance along with the detection performance. We believe that this improvement could be due to greater diversity in logit space, since beyond mixed logits, we have access to non-mixed logits as well.
Logit Mixing weightage: We study the logit mixing weightage used in our proposed method on a separate validation set. Keeping the range of  $\alpha_{1} \in [0.6, 0.9]$ , we observe in the experiments that the best trade-off of detection performance and model calibration is obtained at  $\alpha_{1} = 0.9$ . We notice that calibration performance is competitive for higher  $\alpha_{1}$  values but an overall improvement is observed for  $\alpha_{1} = 0.9$ . Tab. 7a shows the behavior of  $\alpha$  in Cal-DETR.
Choice of single  $\alpha$ : Several mixup strategies operate to mix an input image using another randomly selected image, so it usually involves two samples in the process. In our approach, we perform a query instance/object level mixup in the logit space. We first build a prototypical representation using all positive queries which is then used to achieve a mixup for a given positive query. Owing to this difference from conventional mixup strategies operating, our experiment with conventional mixup leads to suboptimal results. The subpar performance is potentially because the logit mixing suppresses the object representation with lower arbitrary  $\alpha$  simultaneously having dominant prototypical representation from multiple positive queries. It is to be noted that the sampling strategy  $(\mathrm{Cal - DETR}_S)$  still works better when compared to other calibration methods. We report the results in
Datasets
Methods	In-Domain (Cityscapes)	Out-Domain (Foggy Cityscapes)
D-ECE ↓	AP ↑	AP0.5 ↑	D-ECE ↓	AP ↑	AP0.5 ↑
Baseline D-DETR [53]	13.8	26.8	49.5	19.5	17.3	29.3
TCD [33]	11.9	28.3	50.8	14.3	17.6	30.3
Cal-DETRS (Ours)	11.7	28.2	52.3	13.2	17.3	29.6
Cal-DETR (Ours)	8.4	28.4	51.4	11.9	17.6	29.8
Table 8: Choice of singe  $\alpha$  : Cal-DETR outperforms its variant with sampling strategy (Cal-DETR  ${}_{S}$  ) but it still works better when compared to other calibration methods.
the Tab. 8. We empirically find  $\beta$  hyperparameter on beta distribution using the split of Cityscapes dataset. For our reported result, we choose  $\beta = 0.3$  using this procedure.
Regularizer Loss weightage: The impact of loss weightage is studied over the validation set, which shows that  $\lambda = 0.5$  is a better choice to use with the regularizer function. Detection performance is increased with this setting along with the calibration performance. The trend shows that for a higher value of  $\lambda$ , calibration is more competitive (see Tab. 7b).
D-UCE complements D-ECE: To investigate the role of uncertainty calibration which complements confidence calibration, we analyze the detection expected uncertainty calibration error (D-UCE) [24] in Fig. 4. The formulation is described in the supplementary material.
**Limitation:** Our approach could struggle with the model calibration for long-tailed object detection and is a potential future direction.
Figure 4: D-UCE for MS-COCO evaluation set. Uncertainty calibration error is reduced as compared to the baseline.
In this paper, we tackled the problem of calibrating recent transformer-based object detectors by introducing Cal-DETR.
We proposed a new technique for estimating the uncertainty in
transformer-based detector, which is then utilized to develop an uncertainty-guided logit modulation technique. To further enhance calibration performance, we developed a new logit mixing strategy which acts a regularizer on detections with task-specific loss. Comprehensive experiments on several in-domain and out-domain settings validate the effectiveness of our method against the existing train-time and post-hoc calibration methods. We hope that our work will encourage new research directions for calibrating object detectors.
The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725, and by the Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Center.
[1] Blundell, C., J. Cornebise, K. Kavukcuoglu, and D. Wierstra (2015). Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613-1622.
[2] Carion, N., F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020). End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pp. 213-229. Springer.
[3] Chen, L.-C., G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille (2018). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence 40(4), 834-848.
[44] Tolstikhin, I. O., N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy (2021). Mlp-mixer: An all-mlp architecture for vision. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems, Volume 34, pp. 24261–24272.
[45] Tomani, C., S. Gruber, M. E. Erdem, D. Cremers, and F. Buettner (2021). Post-hoc uncertainty calibration for domain drift scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10132.
[46] Touvron, H., M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou (2021). Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR.
[47] Xie, E., W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo (2021). Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34, 12077-12090.
[48] Yu, F., H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell (2020). Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2636-2645.
[49] Yu, R. and G. S. Ali (2019). What's inside the black box? ai challenges for lawyers and researchers. Legal Information Management 19(1), 2-13.
[50] Zhang, H., M. Cisse, Y. N. Dauphin, and D. Lopez-Paz (2018). mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
[51] Zhang, H., F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. Ni, and H.-Y. Shum (2023). DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In The Eleventh International Conference on Learning Representations.
[52] Zhang, L., Z. Deng, K. Kawaguchi, A. Ghorbani, and J. Zou (2021). How does mixup help with robustness and generalization? In International Conference on Learning Representations.
[53] Zhu, X., W. Su, L. Lu, B. Li, X. Wang, and J. Dai (2021). Deformable  $\{\mathrm{detr}\}$ : Deformable transformers for end-to-end object detection. In International Conference on Learning Representations.