A Benchmark Study on Calibration
Authors: Linwei Tao
Younan Zhu
Haolan Guo
Minjing Dong
Chang Xu
Venue: ICLR 2024 poster (2024)
Keywords: Calibration
Primary area: datasets and benchmarks
TL;DR: This research explore the calibration property by analyzing 117,702 unique models and answering questions on calibration's generalizability, robustness and etc.

Abstract: Deep neural networks are increasingly utilized in various machine learning tasks. However, as these models grow in complexity, they often face calibration issues, despite enhanced prediction accuracy. Many studies have endeavored to improve calibration performance through the use of specific loss functions, data preprocessing and training frameworks. Yet, investigations into calibration properties have been somewhat overlooked. Our study leverages the Neural Architecture Search (NAS) search space, offering an exhaustive model architecture space for thorough calibration properties exploration. We specifically create a model calibration dataset. This dataset evaluates 90 bin-based and 12 additional calibration measurements across 117,702 unique neural networks within the widely employed NATS-Bench search space. Our analysis aims to answer several longstanding questions in the field, using our proposed dataset: (i) Can model calibration be generalized across different datasets? (ii) Can robustness be used as a calibration measurement? (iii) How reliable are calibration metrics? (iv) Does a post-hoc calibration method affect all models uniformly? (v) How does calibration interact with accuracy? (vi) What is the impact of bin size on calibration measurement? (vii) Which architectural designs are beneficial for calibration? Additionally, our study bridges an existing gap by exploring calibration within NAS. By providing this dataset, we enable further research into NAS calibration. As far as we are aware, our research represents the first large-scale investigation into calibration properties and the premier study of calibration issues within NAS.

Published as a conference paper at ICLR 2024
A BENCHMARK STUDY ON CALIBRATION
Linwei Tao
Younan Zhu, Haolan Guo University of Sydney
University of Sydney
www.ttaoolssyooo.. wwrpee..edd.coom edu.au
(yzhu0986, hguo4 658) (u @uni sydney. .acbueeeeeu.... edu, au
Minjing Dong
Chang Xu
City University of Hong Kong min jinjjongecice edu. hk
University of Sydney
xu@sedney.... edu au
ABSTRACT
Deep neural networks are increasingly utilized various machine learning tasks. However, these models grow in complexity, they often face calibration issues, despite enhanced prediction accuracy. Many studies have endeavored to improve calibration performance through the use of specific loss functions, data prepro- cessing and training frameworks. Yet, investigations into calibration properties have been somewhat overlooked. Our study leverages the Neural Architecture Search (NAS) search space, offering an exhaustive model architecture space for thorough calibration properties exploration. We specifically create model cali- bration dataset. This dataset evaluates 90 bin-based and 12 additional calibration measurements across 117,702 unique neural networks within the widely employed search space. Our analysis aims to answer several longstanding questions in the field, using our proposed dataset: (i) Can model calibration be generalized across different datasets? (ii) Can robustness be used as calibration measurement? (iii) How reliable are calibration metrics?
(iv) Does a post-hoc calibration method affect all models uniformly? (v) How does calibration interact with accuracy? (vi) What the impact of bin size on calibration measurement? (vii) Which architectural designs beneficial for calibration? Additionally, study bridges an existing gap by exploring calibration within NAS. By providing this dataset, enable further research into NAS calibration. As far we aware, our research represents the first large-scale investigation into calibration properties and the premier study of calibration issues within NAS, The project page can be found at alhttps: www. taol inwei. com www.ca/aa/aaaiiiii..ioooooooc study
INTRODUCTION
Despite their widespread success across various domains, deep neural networks (DNNs) are not immune to producing miscalibrated predictions, leading either overconfidence underconfidence. This concern becomes particularly salient for safety-critical applications such as autonomous driv- ing (Feng et al. 2019) and medical diagnosis (Thiagarajan et 2022), where reliance on accurate prediction probabilities is paramount. In these contexts, miscalibrated predictions may give rise to potentially catastrophic consequences.
A myriad of attempts (Mukhoti et al. 2020 Kumar et al. 2018 Tao et al. 2023a Karandikar et al.
202T, Krishnan & Tickoo] 20200 Zhang al. 2022; Hendrycks al. 2019 Müller et al. 2019
Deng & Zhang] [02ng Kim et al. 2021: Tao et al. (202b)) has been made to mitigate the issue of miscalibration, primarily focusing on loss functions or training frameworks. However, the calibration
properties of the neural network architectures themselves have received comparatively less attention.
Guo et al. 2017) were among the first to investigate the relationship between neural architectures and calibration performance, but their work
restricted to the effects of varying depth and width in ResNet(He et al. 2016). More recent studies (Minderer et al. 202T) have extended this exploration to modern neural networks, such as the non-convolutional MixeMixer (Tolstikhin al.) (202]) and Vision Transformers (Ranttl et al. 2021). However, the limited diversity and quantity of model architectures have constrained the depth of calibration property studies.

---

Published as a conference paper at ICLR 2024
NAS (Liu al.
2018 Dong & Yang 2019b Xu al. 2019 Su et al. 2022) has brought
about a revolution in the field of deep learning by automating the discovery of neural architectures that outperform traditionally hand hand-designed architectures like BlexNet(KKizzevaaa ResNet (He et al. 2016). NAS has achieved advancements in part due to its predefined comprehensive
et al. 2009) and
model architecture space. NAS benchmark search spaces (Ying et al. 2019, Dong & Yang 2020
Siems et al.] 2020; Dong et al.] (02T) have provided an extensive set of unique convolutional
architectures for NAS research. While most prior works on search space focus on the topological architecture of models, the taking into account models of different sizes. This comprehensive search space can serve as a potent tool to bridge the gap in previous calibration studies.
(Dong et al. 202I) provides a more expansive search space,
In our work, to exam the calibration property and address calibration- research questions, such as the reliability of calibration metrics, one approach is to assess the consistency of different metrics based on a substantial number of well well-trained models. However, collecting such substantial dataset is often challenging due to the associated training costs. Fortunately, NATS- (Dong et al. 2021 provides access to 117.9K well-trained models with various architectural designs, enabling to conduct comprehensive and generalisable study. Specifically, evaluate all 117,702 unique CNN architectures concerning topology and model size, and benchmark them on multiple calibration metrics of different types. We also include the results 11 different Vision Transformers to generalize our findings. The specifics about architectures and metrics are discussed in section g This comprehensive dataset has served as a benchmark for our subsequent studies and in-depth analysis on the calibration properties. In this study, we try to answer the following questions that are longstanding in this field:
1. Can model calibration be generalized across different datasets? 4.1
2. Can robustness be used as a calibration measurement?
3. How reliable are calibration metrics?
4. Does a post-hoc calibration method affect all models uniformly?
4.4
5. How does calibration interact with accuracy?
6. What is the impact of bin size on calibration measurement? 4.6
7. Which architectural designs are beneficial for calibration? 4.7
This exploration aims to shed light on the often-overlonn overlooked aspect of calibration, thereby contributing to more holistic understanding of deep learning model performance.
RELATED WORKS
Calibration Metrics Extensive research has been done calibration metrics, which is crucial for measure the reliability of predictions. The Expected Calibration Error (ECE) is a widely used metric introduced by Nacini et al. (015), which quantifies the absolute difference between predicted confidence and empirical accuracy. However, ECE is susceptible to estimator bias, complicating its estimation (Nixon et al., 2019. Vaicenavicius al., 2019. Gupta & Ramdas 2021). To address the bias-variance trade-off, adaptive binning techniques such as Equal Mass Binning calibration error (ECEem) have been proposed by Kumar al. (219) and Nixon et al. (019). In multi-class settings, class-wise calibration errors, including Classwise Equal-Width Binning calibration error (cwCE) and Classwise Equal-Mass Binning calibration error (cwCEem), offer potential solutions (Kumar et al. 2019). Alternatives to ECE encompass likelihood- based measures, the Brier score (Bierr 1950). and Bayesian methods (Gelman & Shalizi 2013). Zhang et al. (2020) proposed the
Top-l Label calibration error using Kernel Density Estimation (KDECE) to avoid binning schemes. calibration error (KSCE), which uses the test to compare empirical cumulative distribution functions. Additionally, Kumar et al. (018)) suggested the Maximum Mean calibration error (MMCE), a differentiable
Cupta et al. (2020) introduced the Kolmogorov-Smime
Kolmogorov-Sminna
estimator that compares the Top-Label confidence with the conditional accuracy.
Empirical Study on Calibration There have been limited studies on evaluating the calibration of uncertainty estimates in deep learning models. Guo et al. (201)) discovered that many models with deeper or wider architectures are poorly calibrated and suggested a simple post- processonss method called temperature scaling that can substantially improve calibration. In a recent study, Minderen
2

---

Published as a conference paper at ICLR 2024
et al. (021) > revisited the calibration of modern neural networks, such MLP-mixe (Tolstikhin
et al.
2021) and ViT (Dosovitskiy et al. 2020b), and found that these new, larger models outperform
CNNs. They also examined the impact of training frameworks like SimCLR (Chen et al., 2020) and CLIP (Radford et al. 2021). However, these studies are based on a limited set of model architectures, and more research is needed to generalize their findings.
NAS The search space in NAS serves as a critical component for exploring calibration properties. Various search spaces, such as fully connected, convolutional, recurrent, and attention-based architec- tures, have been proposed in existing literature. NAS-Bench-101 (Ying et al., 2019), the inaugural public architecture dataset for NAS research, comprises 423k unique convolutional architectures. NAS-Bench-201 (Dog & Yang, 2020) expands on NAS-Bench- 1e--BBncc---- 101 by offering different search space. NAS- BeSSBennh--31 (Siems et al. 2020) addresses the issues found in tabular NAS benchmarks. Most prior studies on search spaces concentrate on the topological architecture of models. Recently, NATS-Bench (Dong et al. 202T) was introduced as an extension to NAS-Bench-2D (Dong Yangl 2020), providing an expanded search space for models of varying sizes. This expansion marks sigmificant advancement in the field, creating new possibilities for the exploration and improvement of deep learning models.
DATASET GENERATION
In this section, we detail the metrics and model architectures from NATS- (Dong et al. 2021 that are involved in our study. To ensure the generality of our findings, we also incldde 11 vision transformers of different architecture design. Each unique architecture is pretrained for 200 epochs on three benchmark datasets: CIFAR-10 10 (Krizhevsky et al. 2009). CIFAR- (Krizheskyy et al. 2009), and MoageNee11-1222 @Chrabaszez al. 2017). Note that ImageNet16- 120 is flown-sampled variant of ImageNet. widely used in NAS literature (Ying et al. 2019 Dong & Yang 2020 Dong al. 202T Patel al. 2020). We choose this dataset because
reduce computational costs while maintaining comparable results as ImageNet (Chrabaszez et al., 2017). For simplicity, we refer to ImageNet16 120 as ImageNet in the following discussion. To evaluate post temperature scaling, we create a validation set by splitting the original test set into 20%/80% for validation/test.
We use extensive bin-based metrics and a range of other calibration measures in this study, which provides a thorough assessment of model calibration performance across different scenarios. We evaluate all calibration metries discussed in and more details about metrics can be found in Appendix !] Since prediction performance on Dut-of-Distribution (OoD) datasets is also strong indicator calibration (Mukhoti et al. 2020), we include the Area Under the Curve (AUC) for the TSS models, evaluating on two OoD datasets, CIFAR- (corrupted with Gaussian noise) (Hendrycks & Dietterich 2019h and SVHN (Street View House Numbers) (Netzer et al. 201 We evaluate these metrics across wide range of bin sizes, including 5, 10, 15, 20, 25, 50, 100, 200, and 500 bins. These metrics are assessed both before and after temperature scaling. Overall, this results in the assessment of 102 different measurements, providing comprehensive evaluation of model calibration.
3.1 ARCHITECTURES EVALUATED
We evaluate calibration properties on 117,702 unique architectures in NATS- (Dong et al. 202T), which is cell-based neural architecture search space consisting of two search spaces: Topol- ogy Search Space (TSS) and Size Search Space (SSS). Referring Figure D each cell is represented by densely-connected directed acyclic graph with four nodes and six edges. Within this structure, nodes denote feature maps, while edges correspond to selected op- erations from a predefined operation set O. The operation set O encompasses the following operations: O = (1 x convolution, 3 x 3 convolution, 3 x 3 average pooling, skip, zero}. The search space contains 15,625 architectures, but only 6,466 are unique due to isomorphic cells resulting from the skip and zero operations.
560
Tepring Calls
Figure I: the macro skeleton of each candidate architecture at the top, cell representations at the bottom-left, and opera- tion candidates at the bottom- right. The candidate channels for SSS are 8, 16. 24, 32, 40, 48, 56, and 64.
In SSS, each architecture has different configuration for the num- ber of channels in each layer. In this space, "stacks" refer to the aggregation of multiple cells. Each stack is constructed using the cell structure that has demonstrated the best performance in the TSS on the CIFAR-100 dataset. A total of 32,768 architecture candidates evaluated the same three
3

---

Published as a conference paper at ICLR 2024
datasets for 90 epochs. For our evaluations, assess all architectures within both search spaces and evaluate them on different test splits of their corresponding datasets. In total, we evaluate a total of 3 x 6466 19398 networks on TSS and 3 32768 98304 on SSS.
To ensure the generality of our findings, we also include 11 vision transformers of different architec- ture design including T2T-ViT-7, T2T ViT-V0.- T2T- 12, T2T- 11T--,,, ViT-V4 (Yuan et al.
202T) ViT-b-16 (Dosovitskiy et al., 2020a), Swin-T (Liu et al., 202T). Deit-T (Touvron et al. 2021a
Cait-XXSS (Touvron et al. b). PvTv2-T (Wang et al. 2022) and Poolformer-S12 (Yu et al.
2022) Each transformer is fine tuned 60 epochs on CIFAR- 10, CIFAR-100 and ImageNet-16-1
based on the pretrained weights on ImageNet-Ik.
EXPERIMENTS AND DISCUSSION 4
Our research endeavors to investigate the influence of several factors, including the dataset, calibration metrics, post-hoc calibration methods, accuracy and robustness on the calibration performance of neural networks. Additionally, we seek to analyze the impact of the architecture designs on calibration. With the comprehensive evaluation results in hand, we have made several empirical observations and derived insightful conclusions, which could be summarized to answer the following questions.
4.1 CAN MODEL CALIBRATION BE GENERALIZED ACROSS DIFFERENT DATASETS?
We're particularly interested understanding if there are noticeable variations in model calibration when different datasets are used for training. Additionally, we aim to investigate how the complexity and diversity of the dataset might impact calibration performance.
800
023
63
myy pr0e
Experimental setup: We examine two subsets of ECE measurement from our TSS dataset: the first subset consists of architectures with good prediction accuracy in each dataset (ranked top 1000 by accu- racy), while the second subset includes all architectures available in the dataset. The aim of our investigation is to probe the model calibration performance correlation between datasets. To achieve this, we resort to the Kendall ranking correlation coefficien(Keenall] 1938). This non-parametric statistic is a potent tool for measuring the degree of correspondence between two rankings, with its values ranging from -1 (indicating perfect disagreement) +1 (indicating perfect agreement). A Kendall coefficient of zero would signify a lack of correlation. With the help of Kendall ranking correlation co- efficient, we build a correlation matrix for ECE on CIFAR-10 (ECE_CIO), CIFAR-100 (CO),, ImageNet (ECE_IN) for both models with high accuracy (left) and all models (right) as displayed in Figure
Figure 2: Kendall ranking cor- relation matrix of ECE for dif- ferent TSS architecture sub- sets. The left subplot cor- responds to the top 1000 ar- chitectures based on accuracy, while the right subplot repre- sents the entire set of models.
Upon examining the correlation across datasets, we observed substantial variation in the ranking of calibration metrics. Notably, the calibration performance CIFAR- 10 exhibited little correlation with the calibration performance on CIFAR- 100, despite their images being the same. Furthermore, the calibration performance CIFAR-10 10 exhibited no correlation with the performance on ImageNet and even negatively correlated for top 1000 models. In contrast, the correlation of calibration performance between the more complex datasets CIFAR-100 and ImageNet was relatively higher. This observation suggests that the relationship between calibration performance of a certain model and the evaluation dataset may be weak or non-existent. Thus, the calibration property of a certain architecture can not generalize well to different datasets, researchers cannot rely on evaluation results on different datasets when selecting a calibrated architecture for downstream datasets. We draw the similar conclusion among Transformer architectures as shown in Appendix A Our observation is also consistent among other calibration metrics, as detailed in the Appendix K
Discussion:
4.2 CAN ROBUSTNESS BE USED AS A CALIBRATION MEASUREMENT?
Previous research Thulasidasan et al. 2019) suggests that well-calibrated model should perform well on OoD datasets. Several studies, including Mukhoti et al. 2020), have utilized this notion to evaluate their calibration methods. Given that accuracy corruption datasets is a robustness indicator and have a strong correlation with adversarial robustness, a question arises: can other robustness metrics serve as measure of calibration?
4

---

Published as a conference paper at ICLR 2024
Experiment Setup: To address this question, we focus on the Kendall ranking correlation matrix between ECE and various metrics for models ranked in the top 1000, 2000, and so on, based on their accuracy. Re- cently, a dataset based on NAS-Bench- 201 Gung et al. 2023) was proposed, which evaluates the robustness perfor- mance on all TSS models available in NATS-Bencch and enables us to eval- uate the relationship between robust- ness and calibration. Thus, the consid- ered metrics include accuracy, calibration metries such as bin-based ECE statistics- bases KSCE, and semellbased MMCE, as well as robustness metrics like adversarial robustness (PGD attack and square attack) and accuracy corruption datasets (forest, shot noise, snow, and glass blur), and Area under the Curve (AuC) on CIFAR-I0-C and SVHN. We depicted the correlation coefficients for different populations using line plot, shown in Figure The experiment was conducted on CIFAR- 10, and experiments for other datasets are provided in Appendix N
Mondaa Ranknn Comern
Comeation
001. II Jost
0.
Figure 3: Kendall ranking correlation of various metrics against ECE different top-ranked model population.
Upon analyzing the top 1000 ranked models, we found that all robustness metries, including adversarial robustness and accuracy on corruption datasets, exhibited a strong correlation with ECE (Jung
Discussion:
2023). Additionally, even prediction accuracy had high correlation with ECE, However, we inc uudd worse- pensee-erforingg models, such as top 4000 accuracy models, the correlation was not observed for most metries such as accuracy and snow corruption accuracy. Nonetheless, some robustness metrics, such PGD attack robustness accuracy, still displayed high correlation with ECE. On the other hand, bin-based ECE statistics-bass -aaed KSCE, and kernel-based MMCE had high correlation with ECE regardless of the model's 's performance prediction accuracy.
The study reveals that calibration performance can be measured not only by the robustness accu- racy on the corruption dataset, but also by other robustness metries only among models with high prediction accuracy. However, when considering models with varying prediction performance, there seems to be no correlation between the AuC OoD datasets and ECE, with most other robustness metrics. This suggests that including AuC on OoD datasets in robustness metrics may not reliably measure calibration performance for models of varying prediction performance, is worth more caution when using AuC as a metric for evaluating the calibration of neural networks.
4.3 How RELIABLE ARE CALIBRATION METRICS?
We aim to explore whether certain metrics are particularly effective in specific scenarios, or if their assessments generally align. A critical aspect of this investigation involves identifying if there is a substantial disparity in the results when employing different calibration metrics.
Experimental Setup. We provide a Kendall ranking correlation matrix between calibration metrics of different types within the CIFAR-10 and ImageNet datasets on TSS demonstrate ranking correlations between these metrics.
Discussion: Despite the availability of numerous calibration metrics, a theoretical gap exists between bin-based, kernel-based, and statistics-based metrics. Our evaluation reveals the correlation between metries, as illustrated in Figure 4a] a consistent trend in the ranking of most calibration perfor- mance regardless of metric type. Although multiple works (Nixon et al. 2019 Kumar et al. 2019) Roclots al., 2022 Gupta et al. 2020) point out the demerits of ECE, the ECE shows consistent results with most other metrics. Our analysis of the correlation coefficients of NLL and Brier score showed consistency across various metrics. We extended our correlation analysis to all models in SSS and observed similar trends. It is worth noting that the classwise-based calibration error metrics, such as cwCE and cwCEem, exhibited lower correlation with other metrics, where cwCE is even negatively correlated with other widely accepted metrics on ImageNet. This indicates that cwCE..m may not be a reliable metric for calibration measurement. We postulate that this discrepancy could stem from inherent limitations associated with equal-mast -msss binning. Specifically, equal- mass binning might yield a bin that covers a broad range of high uncertainty. This could prioritize smaller quantization errors for low-ranked logits over focusing high-ranked logits, which could subse-
5

---

Published as a conference paper at ICLR 2024
quently undermine calibration performance. We observed that this degradation caused by equal-mass binning tends to diminish when datasets have fewer classes. For instance, in the case of CIFAR- 10, the higher class prior appears to mitigate the negative effects of equal-mass binning. We draw the similar conclusion among Transformer architectures as shown in Appendix A
GIAR-11
Imagehiet
L04
0C2
93
2.02
2-92
EC2
0.36
0.32
0.36
0.75
E911
(L:1
492,5
0.22
046 0.68
5.00
0.70
C74
CXC2
0:33
0.33
0.23
06C5
36
2.29
CMCC
1:15
64
KSC5
0147
0:35
0.46
0.25
100,00
812
KOACS
<D:08-1
201
0.65
6:33
511
6:7
1:0
-0.75
62
A22
POKCE
New
100
(a) Kendall ranking correlation between diverse calibration metrics. The metrics are evaluated across the entire set of TSS models. The analysis spans each of the CIFAR-10(eft) and ImageNet(right).
(b) Kendall Ranking Correlation Ma- trix of ECE before and after tempera- ture scaling on CIFAR-IO.
Figure 4: Explore the properties of calibration metrics.
4.4 DOES A POST-HOC CALIBRATION METHOD AFFECT ALL MODELS UNIFORMLY?
The research conducted by Wang et (22TT) suggests that regularized models typically generate more accurate and well-calibrated predictions, but may have lower calibratability. In other words, regularized models have lower calibration space for post-hoc calibration techniques, such as tempera- ture scaling and histogram binning. However, this observation is based on a limited number of cases.
Our dataset allows for more comprehensive exploration of this issue.
To investigate this issue thoroughly, we computed the Kendall ranking correlation matrix between wwwppeetture-scaallllea ECE (pre-ECE) and temperature-scalngg ECE (post- ECE) on CIFAR- 10 with different bin sizes for all models in TSS dataset.
Experiment Sctup:
Discussion: As shown in Figure 4b the correlation coefficient between pre-ECE and post-ECE is nearly zero, suggesting that well-calibrated model may not maintain its ranking of calibration performance after undergoing post-hoc calibration methods. On the other hand, a less well-calibrated model may improve its calibration performance after such methods. Our findings expand the conclusion for regularized models and indicate that well-calibrated models do not necessarily exhibit better calibration performance after post-hoc calibration techniques. This observation is align with (Ashukha et al., 2020). which indicates the comparison of calibration performance between different methods without post-calibration might not provide a fair ranking. Furthermore, we note that the selection of bin size has greater influence on post-hoc calibration measurements. Additionally, we observe similar results present on ImageNet and CIFAR-100 for both top 1000 models and all models. We draw the similar conclusion among Transformer architectures as shown in Appendix Additional results are included in the Appendix
4.5 How DOES CALIBRATION INTERACT WITH ACCURACY?
Extensive research in the robustness literature has investigated the trade-off between accuracy and robustness such Zhang et al. 2019). However, the interplay between accuracy and calibration has not been as thoroughly explored. While some prior studies hinted at a potential trade-off between accuracy and calibration (Mukhoti et al. 2020 Karandikar et al. (02T). these findings seem limited or not universally consistent. Therefore, further investigation into this topic using our dataset could yield valuable insights.
We created a scatter plot for all models and models with high accuracy on ECE versus Accuracy. The experiment was conducted CIFAR-10. Experiments for other datasets are included in the Appendix D
Experiment Setup:
Discussion: Figure 5 illustrates that when considering only models with accuracy 90% (as depicted in the left plot), a distinct pattern emerges: higher accuracy is associated with better calibration performance, whereas lower accuracy results in poorer calibration performance. However,

---

Published as a conference paper at ICLR 2024
when analyzing all models in TSS, this pattern is not evident. Therefore, it can be concluded that the positive correlation between accuracy and calibration exists only among architectures with good prediction performance, challenging the previously hinted trade-off.
4.6 WHAT IS THE IMPACT OF BIN SIZE ON CALIBRATION MEASUREMENT?
A pivotal factor in bin-based calibra- tion measurement is the choice of bin size (number of bins). Larger bin sizes encapsulate wider range of confi- dence levels. which can potentially mask variations and result in an un- derestimation of miscalibration. Con- versely, very small bin sizes may lead to overfitting or unstable estimates due to insufficient data in each bin. Therefore, the study bin size choice essential for accurate calibration measurements.
99%
ECE(S)
ECE
Figure 5: Scatter plots depict the ECE versus Accuracy of model with accuracy larger than 90% (left) and all TSS mod- els (right) on CIFAR-10. The color-coded markers represent CIFAR-IO-C AUC scores.
Experimental Setup: To address this issue, focus the evaluation of the bin-based calibration metrics with different bin sizes on before and after temperature scaling. We also explore the calibration performance on both 12 and 200 epochs to examine the effect of bin size on uuder-fitting models.
12
25
20
25
100
700
1933
10
25
20
25
50
100
700
503
30
255
20
25
50
100
200
593
Bi Sizee
(a) Boxplot of different size before temperature scaling on 200 epochs.
(b) Boxplot ECE different bin size after temperature scaling on 200 epochs.
(c) Boxplot of different size before temperature scaling on 12 epochs.
Figure 6: Explore the impact of bin size on calibration before and after temperature scaling.
Discussion:
The results from Figures 6a and 6b demonstrate that both pre- and Post-ECE increase in correlation with bin size. However, the PCS showcases more pronounced rise-doubling at bin size of 100 and quintupling at bin size of 500. This observation further substantiated by Figure 4b where post-ECE measurements with a bin size of 500 reveal barely any correlation when compared to those
bin size of 100. Figures 6c and lo display the variation of ECE across diverse bin sizes at different stages of model training. Remarkably, for models trained for only 12 epochs (shown in Figure (6). the choice of bin size significantly influences calibration performance more than fully trained models that have trained 200 epochs.
These findings suggest that bin size has a more substantial impact on post-ECE,, with a negligible correlation between Post-ECE measurements at varying bin sizes. Conversely, pre-ECE seems more resistant to changes in bin sizes. Therefore, for holistic comparison, it recommended to assess post-hoc calibration performance across range of bin sizes. Moreover, the influence of bin size is more conspicuous for underfitting models, which implics that choosing the correct bin size is particularly crucial for underfitting models. These trends were also observed when evaluating other bin-based calibration metrics and in the SSS, details of which are provided in the Appendix

---

Published as a conference paper at ICLR 2024
4.7 WHICH ARCHITECTURAL DESIGNS ARE BENEFICIAL FOR CALIBRATION?
The focus now shifts towards comprchending the influence of architecture design on calibration per- formance. Specifically, the number of kernels and the topologiaal architecture design investigated. Previous research (Guo et al. 2017) has suggested that complex neural networks tend to be poorly calibrated. However, our findings only consistent with this observation in limited cases.
IC0 1525
82
89-229 289-230
Model Size
Mode
Sie
(a) Box-plots of ECE across all ar-
(b) Boxplots illustrating ECE and Accuracy across all architectural within SSS subset, segregated by different size brackets. Each plot represents the performance evaluation CIFAR-10. The "model size" is defined as total number of kernels cach layer.
chitectures within SSS subset on ImagenNet, segregated by different size brackets.
Figure 7: Explore the impact of model size on calibration performance.
Experiment Setup: In this study, we focus on our SSS dataset for ImageNet and CIFAR-10 10, segmented by different size brackets based on the total number kernels in each layer, as shown in Figure 7 Additionally, the performance of models of different width on CIFAR-10 10, CIFAR- 100, and ImageNet was plotted to compare the impact of model width on different datasets, as shown in Figure ? We also choose top-ranked model in terms of our proposed metric analyze the architecture design preference in Figure
-70
Cad Karrad Daanemalens
Calibration performance with cell kernel parameters. (Left) BCE distribution across different cell kernel parameters; (Middle) Scatter plot of all models on ECE and Accuracy; (Right): Scatter plot of all models HCS and Accuracy. 3 is set to 1.
Figure 8:
Discussion: As shown in Figure the results show that larger models do not necessarily have worse calibration performance, as it initially declines for the first few size brackets, but subsequently improves with increasing model size for CIFAR-IO, while it worsens for ImageNet. Therefore, calibration performance may depend on both model size and dataset complexity. Additionally, the study found that fewer parameters do not necessarily lead to better calibration performance, as larger kernel size does not necessarily result in worse calibration performance but instead results in a smaller calibration performance variance as depicted in the left plot of Figure g The conclusion of a prior study that calibration decreases with model size may be due to the survivorship bias in calibration, where architectures with worse performance are often ignored, and the literature overlooks the fact that less parameters do not necessarily lead to better calibration performance. Following our examination of the overall model size, we next direct our attention towards understanding the impact of the model's width, specifically the number of filters per layer, its calibration properties. As depicted in Figure S on the CIFAR- 10 dataset, the width of the model appears to have modest impact calibration. However, the dataset complexity increases, particularly with ImageNet,
8

---

Published as a conference paper at ICLR 2024
observe dramatic surge in ECE, rising from 3% to substantial 8%. This suggests that wider model can worsen calibration performance, especially when dealing with complex datasets.
CIRAR-10
CIFAS-100
ImageNet
23
10
60
84
88
92
32
35
40
45
50
55
60
05 70
15
20
25
30
35
Accurncy!9
Accurnoy
Accurncy(a1
Figure 9: ECE measured on CIFAR-IO, CIFAR-I00 and ImageNet datasets before and after applying temperature scaling. Marker size represents model size progression from 8:8:8:8:8, 16: 16:11::::: up to 64:64:64:64:64, where number indicate the number of kernels in certain layer.
To analyze the best calibrated model, it is not appropriate to directly choose the model with the lowest ECE, since it may have unsatisfactory prediction accuracy performance. To address this issue, we propose combined metric that takes into account both calibration and accuracy, called the
(Ace-(1-BCE)) Harmonic Calibration Score (HCS), denoted as HCS, = (1 + 8) (1-As+(1-ECE)) where 3 controls
the balance between accuracy and calibration with larger 3 prefer more calibration. Figure
May1
bdye Free
Freed
3,995
3.1
11443
110
69455
1083
13754
Figure 10: Top 20 HCS architectures out of the topology search space according to the ECE on CIFAR-10 10 dataset. 3 is set to 1.
illustrates the distribution of models with respect ECE and accuracy, where the best ECE is not always obtained at the same time as the highest accuracy. However, proposed HCS provides a good trade-off between accuracy and calibration, and the model with the highest HCS indicates better balance between calibration and prediction accuracy. After applying the HCS metric and replotting, a clear trend can be observed where the HCS increases as the cell kernel parameters increase. We subsequently selected the top 20 models based their HCS scores from TSS and performed an architectural analysis of each edge, as shown in Figure TO This analysis shows that well-calibrated models tend to prefer either and 6. Intriguingly, we observed a unanimous preference for the skip connection for edge 4 among these well-calibrated models. These observations offer valuable insights into the design preferences of well-calibrated models, which can guide the development of future neural network architectures.
convolution 3 convolution for edges 1, 5,
5 CONCLUSIONS
While our investigation provides significant insights into calibration studies, it is essential to note its specific scope. Our findings predominantly relate image classifications, implying that extrapolation to other domains should be approached with caution. Within our analysis, we emphasized the post-hoc Temperature Scaling technique; however, the vast domain of post-hoc calibration houses numerous other techniques that may present divergent perspectives or different effects model calibration. Importantly, our conclusions are primarily empirical. A theoretical exploration remains outside the purview of this work. Future research could remedy these confines by widening the spectrum of evaluated tasks and architectures and diving deeper into theoretical dissections.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Yang You and Yifeng Gao for valuable discussions and feedbacks. This work was supported in part by the Australian Research Council under Projects DP240101848 and FT230100549.
REFERENCES
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling deep learning. arXiv preprint arX14:2002.06470, 2020.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950.
Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR. 2020.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as alternative the cifar datasets. CoRR, abs/1707.0883 2017.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Iecc, 2009.
Xiang Deng and Zhongfei Zhang. Learning with retrospection. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7201-7209, 2021.
Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3681-3690, 2019a.
Xuanyi Dong and Yang. Searching for a robust neural architecture in four gpu hours. Proceedings of the IFEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1761-1770, 2019h.
Xuanyi Dong and Yi Yang. Nas-bench-2011 Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.0033 2020.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench: Benchmarking nas algorithms for architecture topology and size. IEEE transactions on pattern analysis and machine intelligence, 44(7): 3634-3646, 2021.
Dosovitskiy, L Beyer, A Kolesnikov, D Weissenborn, X Zhai, and T Unterthiner. Transformers for image recognition at scale. arXiv preprint arXh::2010 1929, 2020a.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 20t.::2000.1192., 1929, 2020b.
Di Feng, Lars Rosenbaum, Claudius Glaeser, Fabian Timm, and Klaus Dietmayer. Can we trust you? on calibration of probabilistic object detector for autonomous driving. arXiv preprint 1909, 12358, 2019.
Andrew Gelman and Cosma Rohilla Shalizi. Philosophy and the practice of hayesian statistics. British Journal of Mathematical and Statistical Psychology, 66( 8)88888 2013.
Chuan Guo, Gcoff Plciss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference machine learning, pp. 1321-1330. PMLR, 2017.
Chirag Gupta and Aaditya Ramdas. Distribution-free calibration guarantees for histogram binning without sample splitting. In International Conference Machine Learning, pp. 3942-3952. PMLR, 2021.
Kriti Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink. Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. In International Conference Learning Representa- tions, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint orXiv: 12261, 2019.
10

---

Published as a conference paper at ICLR 2024
Dan Hendrycks, Norman Mu, Ekin Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: simple data processing method improve robustmess and uncertainty. arXiv preprint arXiv:1912.02781, 2019.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEF. conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Steffen Jung, Jovita Lukasik, and Margret Keuper. Neural architecture design and robustness: dataset. In The Eleventh International Conference on Learning Representations, 2023.
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshninarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roclofs. Soft calibration objectives for neural networks. Advances in Neural Information Processing Systems, 34:29768 299-7799-29779, 2021.
Maurice G. Kendall. A new measure of rank correlation. Biometrika, 20(1/2):81-93, 1938.
Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6567-6576, 2021.
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems, 33:18237-18248, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. arXiv preprint 1,05055 2009.
Meelis Kull, Mare Perello Nieto, Markus K" <"angsepp. Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems, volume 32, 2019.
Ananya Kumar, Percy S. Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, volume 32, 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kemel mean embeddings. International Conference on Machine Learning, pp. 2805 2814. PMLR, 2018.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence, pp. 367-377. PMLR, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.00 09055. 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34: 15611-1621164, 2021.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht, Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference Artificial Intelligence, pp. 2901-2907. AAAI Press, 2015. ISBN 0262511 11290.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. arXiv preprint arXiv:1903.12261. 2011.
Jeremy Nixon al. Measuring calibration deep learning. In CVPR workshops, volume 2, 2019.
Kanil Patel, William Beluch, Bin Yang, Michael Pfeiffer, and Dan Zhang. Multi-class uncertainty calibration via mutual information maximization-hast binning, arXiv preprint arXiv:2006. /3092, 2020.
11

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7748-8763. PMLR, 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the uaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, Michael C Mozer. Mitigating bias calibration error estimation. In International Conference on Artificial Intelligence and Statittiss, pp. 4036-4054. PMLR, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Christoph
Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278- 25294, 2022.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter, Nas-bench-301 and the for surrogate benchmarks for neural architecture search. CoRR, (9bb/2008.097777, 2020. URL https://arxiv..r http://wwwwv..rr///////0000// 09777
Xiu Su. Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vitas: Vision transformer architecture search. In European Conference on Computer Vision, pp. 139-157. Springer Nature Switzerland Cham, 2022.
Linwei Tao, Minjing Dong, Daochang Liu, Changming Sun, and Chang Xu. Calibrating deep neural network with its predecessors. arXiv preprint 40Xiv:2302.06245, 2023a.
Linwei Tao, Minjing Dong, and Chang Xu. Dual focal loss for calibration, arXiv preprint arX 13665, 2023b.
Jayaraman Thiagarajan, Kowshik Thopalli, Deepta Rajan, and Pavan Turaga. Training valibration-basee
counterfactual explainers for deep learning models in medical image analysis. Scientific reports, 12(1):597, 2022.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 10cm100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, al. Mlp-mixer: An all-mlp architecture for vision. Advonces neural information processing systems, 34:24261-24272, 2021.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning. pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pp- 32-42, 2021b.
Julius Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3459-3467. PMLR, 2019.
Deng-Bao Bao Wang, Lei Feng, and Min-L Zhang. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Advances in Neural Information Processing Systems, 34:11 11809-18820 2021.
12

---

Published as a conference paper at ICLR 2024
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3); 415 424, 2022.
Colin White, Nolen, and Yash Savani. Exploring the loss landscape neural architecture search. In Uncertainty in Artificial Intelligence, pp. 654-664. PMLR, 2021.
Hongyan Xu, Xiu Su, Yi Wang, Huaiyu Cai, Kerang Cui, and Xiaodong Chen. Automatic bridge crack detection using a convolutional neural network. Applied Sciences, 9114)2887, 2019.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas-bench-1011 Towards reproducible neural architecture search, arXiv preprint arXiv:1902.06633 2019.
Weihao Yu, Luo. Zhou, Chenyang Yichen Zhou, Xinchao Wang. Jiashi Feng, Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10819-108222 2022.
Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Zi- HanHang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference computer vision, pp. 558 567, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprini arXiv:160500744, 2016.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing. Laurent Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp.
7482-7482. PMLR, 2019.
Jiayi Zhang, Bhavya Kailkhura, and Tony Y.-. Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International Conference on Machine Learning, pp. 11127-11128. PMLR, 2020.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. In International Conference on Machine Learning, pp. 26135-26166 PMLR, 2022.
13


Reviews:
[{"review_id": "zj4P7thrt3", "reviewer": "Reviewer_S1Jc", "summary": "This paper conducts several investigations about the calibration problem of the deep neural networks. This paper introduces a calibration dataset based on the NATS-Bench for generating massive CNNs with different topologies or model sizes. This paper adopts the calibration dataset and evaluate the different calibration metrics to analyze the calibration properties in deep neural networks. This paper provides extensive explorations and conclusions through the benchmarks", "questions": "See the weaknesses above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThis paper raises 7 initial questions for exploring the calibration properties in deep neural networks.\n2.\tThis paper builds a benchmark based on models generated by NAS for evaluating the calibration metrics.\n3.\tThis paper conducts extensive experiments to analyze and answer the questions.", "weaknesses": "1.\tThe proposed calibration benchmark might be limited since it contains only convolution neural networks for image classification. I’m concerned about how about the calibration properties for other tasks, e.g., object detection or NLP tasks. It’s more convincing when extending the benchmark for more architectures and more tasks.\n2.\tMost architectures and networks are generated from the similar search space, which might have similar effects and are limited for the conclusions. Varying the search spaces and using human designed architectures are necessary.\n3.\tDifferent architectures on different benchmarks/dataset might require different hyper-parameters. Though it’s hard to search optimal/sub-optimal parameters for different models, it will affect the experimental results.\n4.\tThis paper explores the calibration properties in neural networks, but I’m much concerned about how those findings impact the future research or guide the designing process for both accuracy and calibration performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zgk9HYSD4Q", "reviewer": "Reviewer_Xzoe", "summary": "The paper presents a study that analyzes the relationship between NAS and calibration powers of neural networks. The paper combines CIFAR-10, CIFAR-100, and ImageNet as the dataset in which multiple architectures are tested and measure its calibration powers. The study of the paper thus focuses only on image classification problems using small- and medium-scale datasets.", "questions": "Please see the Weaknesses stated above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. I really value the topic of calibration as I believe it is a good feature to have in many classification tasks and systems. I think the paper tackles an important problem.\n\nS2. The clarity of the paper is good, the narrative flows well, and is easy to understand and follow.", "weaknesses": "W1. The motivation about why NAS + Calibration is important is missing in the paper. Unfortunately, the paper lacks a clear justification for studying NAS + Calibration. It is not clear intuitively why this is a good direction to explore. It is not clear why a wholistic approach is not worth exploring over NAS + Calibration. Unfortunately, the paper makes the reader believe that the analysis was done just because it has not done before. I think the paper really needs to justify why NAS + Calibration is a good angle to study.\n\nW2. The study uses small- and medium- scale datasets for analyzing the relationship between NAS + Calibration. In particular, the study uses CIFAR-10, CIFAR-100, and ImageNet datasets for the analysis. While I understand that small datasets are easier to handle given the NAS component of the study, it is questionable the conclusions one can get from these small datasets. While ImageNet is larger, compared to modern large-scale datasets, such as, LAION, its use is also questionable. Modern image classification methods are trained using foundation models that use really large-scale datasets (e.g., LAION) and those are the worth studying in my opinion since they are the ones adopted in industry and are making an impact. I think the paper needs to justify the use of these small- and medium-scale datasets. Otherwise, the conclusions drawn from the studies are not solid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hw5vRYq2tc", "reviewer": "Reviewer_qRJE", "summary": "The paper studies the calibration properties of deep neural networks (DNNs) using neural architecture search (NAS) search space. The motivation stems around the observation that, calibration properties of DNNs have not been thoroughly studied in the past, and NAS search space allows to create a comprehensive dataset of neural network architectures, which can be evaluated to study calibration properties. The dataset encompasses several bin-based and other calibration measurements across 117,702 unique neural network architectures. Particularly, the NATS-Bench has been used to curate the proposed dataset as it allows more broader search space, comprising models of various sizes. The study also includes eleven recent vision transformer architectures. The proposed analyses aims to answer seven different questions, including the interplay between accuracy and calibration, if calibration performance generalizes across datasets, the impact of bin sizes on calibration measurement, and which architectures are better for calibration. Post-hoc temperature scaling method is used as a calibration technique.", "questions": "- How the proposed study and dataset would be helpful toward the development of new calibration methods for classification?\n\n- It is a bit hard to grasp how the current analyses addresses the question of how reliable are calibration metrics?\n\n- Would the finding that the bin size has a more substantial impact on Post-ECE be relevant for any train-time calibration technique too?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The study of calibration properties of deep neural networks is an important research direction as it could allow developing well-calibrated architectures.\n\n- The paper develops a comprehensive benchmark of neural network architectures that are then evaluated on different datasets to answer various questions. Further, recent vision transformer architectures have also been included as part of evaluation.\n\n- Some questions included in the study are interesting and important: such as the Impact of bin sizes on calibration measurement and can model calibration be generalized across different datasets.\n\n- Overall, the paper is well-written and it is not difficult to read and understand.", "weaknesses": "- Overall, the new questions posed and studied by the paper boils down to 1), 3) and 6) which are:\n\n-- Model Calibration across different datasets\n\n-- Reliability of calibration metrics\n\n-- Impact of bin size on calibration metrics\n\n- Other questions are mostly expansion of existing studies. This seems to undermine the overall contributions of the paper to some extent.\n\n-Only post-hoc temperature scaling is used as a calibration technique to evaluate pre- and post calibration performance of a large chunk of models. There have been several new calibration methods, especially in train-time calibration paradigm, such as [A], [B] and [C].\n\n-In 4.3: 1) What are the possible reasons of ECE showing consistent results with other metrics, although some other metrics are theoretically different? 2) Also, it is not clear that which calibration metric should be preferred over others in the scope of studies?\n\n- What is the significance of 4th question (i.e. does a post-hoc calibration affect all models uniformly) in terms of advancing the research and algorithmic development in calibration?\n\n- The abstract mentions ‘data pre-processing’ methods for improving calibration but such methods are discussed nowhere in the paper, including the related work.\n\n\n[A] Liu, B., Ben Ayed, I., Galdran, A. and Dolz, J., 2022. The devil is in the margin: Margin-based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 80-88).\n\n[B] Patra, R., Hebbalaguppe, R., Dash, T., Shroff, G. and Vig, L., 2023. Calibrating deep neural networks using explicit regularisation and dynamic data pruning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1541-1549)\n\n[C] Hebbalaguppe, R., Prakash, J., Madan, N. and Arora, C., 2022. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16081-16090).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)