Robust Calibration with Multi-domain Temperature Scaling
Authors: Yaodong Yu
Stephen Bates
Yi Ma
Michael Jordan
Venue: NeurIPS 2022 Accept (2022)
Keywords: uncertainty quantification
calibration
distribution shift

Abstract: Uncertainty quantification is essential for the reliable deployment of machine learning models to high-stakes application domains. Uncertainty quantification is all the more challenging when training distribution and test distribution are different, even if the distribution shifts are mild. Despite the ubiquity of distribution shifts in real-world applications, existing uncertainty quantification approaches mainly study the in-distribution setting where the train and test distributions are the same. In this paper, we develop a systematic calibration model to handle distribution shifts by leveraging data from multiple domains. Our proposed method---multi-domain temperature scaling---uses the heterogeneity in the domains to improve calibration robustness under distribution shift. Through experiments on three benchmark data sets, we find our proposed method outperforms existing methods as measured on both in-distribution and out-of-distribution test sets. 

Robust Calibration with Multi-domain Temperature Scaling
Yaodong Yu University of California, Berkeley yyuêeecs. berkeley.. edu
Stephen Bates University of California, Berkeley stephenbates@cs. berkeley edu
Ma University of California, Berkeley yima@eecs. berkeley.
Michael I. Jordan University of California, Berkeley j jordan@cs berkeley. edu
edu edu
Abstract
Uncertainty quantification is essential for the reliable deployment of machine learning models to high-stakes application domains. Uncertainty quantification is all the more challenging when training distribution and test distribution are different, even if the distribution shifts are mild. Despite the ubiquity of distribution shifts in real-world applications, existing uncertainty quantification approaches mainly study the in-distribution setting where the train and test distributions are the same. In this paper, we develop systematic calibration model to handle distribution shifts by leveraging data from multiple domains. Our proposed temperature scaling-uses the heterogeneity the domains to improve calibration robustness under distribution shift. Through experiments on three benchmark data sets, we find our proposed method outperforms existing methods as measured on both in-distribution and out-of-distribution test sets.
Introduction
To make learning systems reliable and fault-tolerant, predictions must be accompanied by uncertainty estimates. A significant challenge to accurately codifying uncertainty is the distribution shift that typically arises over the course of a system's deployment (Quiñonero-Candela et al., 2008). For example, suppose health providers from 20 different hospitals employ a model to make diagnostic predictions from fMRI data. The distributions across hospitals could be quite different differing patient populations, machine conditions, and so on. such a setting, it is critical to provide uncertainty quantification that is valid for every hospital-not just average across all hospitals. Going even further, our uncertainty quantification should be informative when a new 21st hospital goes online, even if the distribution shifts from those already encountered. As another example, centralized model is trained on training data from existing clients in federated learning. is important for the central server provide uncertainty quantification for every client. Similar to the fMRI example, the centralized model should still produce valid uncertainty quantification for unseen new clients. Another example is applying animal recognition models images in wildlife monitoring, where one set of camera traps corresponds to one domain, and the model will be deployed under distribution shift, i.e., new camera traps. In this work, we study calibration in the multi-domain setting. We find that by requiring accurate calibration across all observed domains, our method provides more accurate uncertainty quantification on unseen domains.
result of
Calibration is a core topic in learning [Platt et al., 1999; Naeini et al., 2015; Gal and Ghahramani, 2016; Lakshminarayanan al., 2017; Guo et al., 2017; Bates et al., 2021]. but most techniques targeted at settings with no distribution shift. To see this, we consider simple experiment on the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ECE on pooled data
Acc
Acc
Accuracy
Accuracy
Count
ECE (evaluated on each domain!
(a) Evaluated on pooled data.
(b) Evaluated on domain.
(c) ECE evaluated on every domain.
Figure 1: Reliability diagrams and expected calibration error histograms for temperature scaling with ResNet-50 on ImageNet-C. We use temperature scaling to obtain adjusted confidences for the ResNet-50 model. (a) Reliability diagram evaluated on the pooled data of ImageNet-C. C. (b) Reliability
diagram evaluated on data from one domain (Gaussian corruption with severity 5) in ImageNet-C. (c)
Calibration evaluated on every domain in ImageNet-C as well as the pooled ImageNet-C (measured
in ECE, lower is better).
ImageNet-C [Hendrycks and Dietterich, 2019] dataset, which consists of 76 domains. Here, each domain corresponds to one type of data corruption applied with a certain severity. We apply the temperature scaling technique [Guo et al., 2017] the pooled data from all domains. In Figure ((a) and I(b), we display the reliability diagrams for the pooled data and for one individual domain. We find that even under a relatively mild distribution shift-ic... subpopulation shift from the mixture of all domains to the single domain domain--temprratte temperature scaling does not produce calibrated confidence estimates the stand-alone domain. This behavior is pervasive; in Figure Icc), calibration on individual domains is much worse than the the reliability diagram from the pooled data would suggest.
that the
To address this issue, we develop a new algorithm, multi-domain temperature scaling, that leverages multi-domain structure in the data. Our algorithm takes base model and learns calibration function that maps each input to a different temperature parameter that is used for adjusting confidence the base model. Empirically, we find our algorithm significantly outperforms temperature scaling on three real-world multi-domain datasets. particular, in contrast to temperature scaling, our proposed algorithm is able to provide well-calibrated confidence each domain. Morcover, algorithm largely improves robustness of calibration under distribution shifts. This is expected, because if the calibration method performs well every domain, it is likely to have learned some structure that generalizes to unseen domains. Theoretically, we analyze the multi-domain calibration problem in the regression setting, providing guidance about the conditions under which robust calibration is possible.
Contributions. The main contributions of our work are as follows: Algorithmically, we develop new calibration method that generalizes the widely used temperature scaling concept from single- domain multi-domain. The proposed new method exploits multi-domain structure in the data distribution, which enables model calibration on every domain. We conduct detailed experiments on three real-world multi-domain datasets and demonstrate that our method significantly outperforms existing calibration methods on both in-distribution domains and unseen Theoretically, we study multi-domain calibration in the regression setting and develop theoretical understanding of robust calibration in this setting.
domains.
Related Work
Calibration methods. There is large literature on calibrating the well-trained machine learning models, including histogram bimning [Zadrozny and Elkan, 2001], isotonic regression [Zadrozny and Elkan, 2002], conformal prediction Vovk et al. [2005], Platt scaling [Platt et al., 1999], and temperature scaling [Guo et al., 2017]. These calibration methods apply a validation set and post- process the model outputs. As shown in Guo al. [2017], temperature scaling, simple method that uses single (temperature) parameter for rescaling the logits, performs surprisingly well on calibrating confidences for deep neural networks. We focus on this approach in our work. More broadly, there has been much recent work develop methods to improve calibration for deep learning models, including

---

mugmentation-aasee training [Thulasidasan et al., 2019; Hendrycks et al.. 2019b], calibration for neural machine translation [Kumar and Sarawagi. 2019], neural stochastic differential equation [Kong et al., 2020], self-supervised learning [Hendrycks et al., 2019a], ensembling [laaksmminrrayanan
et al., 2017], and Bayesian neural networks [Gal and Ghahramani, 2016; Gal et al., 2017], as well as statistical guarantees for calibration with black-box models Angelopoulos et al. [2021].
Calibration under distribution shifts. Ovadia al. [2019] conduct empirical study on model calibration under distribution shifts and find that models are much less calibrated under distribution shifts. Minderer et al. [2021] revisit calibration of recent sate-of-the-ar image classification models under distribution shifts and study the relationship between calibration and accuracy. Wald et al. [2021] study model calibration and out-of-distributioo generalization. Other works consider providing uncertainty estimates under structured distribution shifts, such as covariate shift (Tibshhirani et al., 2019; Park et al., 2021], label shift [Podkopaev and Ramdas, 2021], and f-divergence balls [Cauchois et al., 2020]. Another line of work studies calibration in the domain adaptation setting [Wang et al., 2020; Park al., 2020], which require unlabeled samples from the target domain.
2 Problem setup
Notation. We denote the input space and the label set by X C IRd and (1....,. We let [=]; denote the i-th element of vector z. We use P(X) denote the marginal feature distribution on input space X, P. X) to denote the conditional distribution, and P(X. Y) to denote the joint distribution. For the multiple domains scenario, we let (X) and PA ( denote the feature distribution and conditional distribution for the k-th domain. We let ʃ X + R denote the base model, e.g., deep neural network, where is the total number of classes. We assume I returns an (unnormalized) vector of logits. Throughout the paper, the base model is trained with training data and will not be modified. The class prediction of model ff on input x E X is denoted by ÿ = je
[f (x; 0)}, We use } to represent the indicator function. We use h(:; f. B) X + [0. 1] to denote a calibration map (parameterized by 3) that takes an input E X' and returns confidence score this is post-pree post-processing of the base model f. We let ी = h(2; f. 3) E [0. 1] denote the confidence estimate for sample when using model f. For instance, if we have 100 predictions {ύ) (ür....> (1oo) with confidence मं = ...... 1o00 = 0.7, then the accuracy of f is expected to be 70% on these 100 samples (if the confidence estimate is well calibrated). Data from the domains P ,,,...,?? K are used for learning the calibration models, and we call the in-distribution (InD) domains. We use P to denote the unseen oul-of-distribution (OOD) domain which is not used for calibrating the base model. Our goal is to learn calibration map h that is well calibrated the OOD domain P. To do this, we will learn a calibration map that does well on all InD domains simultaneously.
PP
To measure calibration, we first review the definition of approximate expected calibration error.
Definition 2.1 (ECE). For a set of samples DD = {[x,,, v)) with (æ,, y1) Lid. P(X,Y), the (empirical) expected calibration error (ECE) with M bins evaluated on D is defined as
M
Bml
ECE(D, = Σ
Acc Bm)) - Conf( Bmm)
(1)
m=1
and Bms B, (onf(Bm) are defined as
Bm = (i [n] t: E ((m 1)/M,
Acc( = (m/)
Σ 1 = v.), Conf(Bm.) = (1/
Σ #1 i
ieBm.
where i; and is are the confidence and predicted label of sample xy--
The empirical ECE defined in Eq. (1) approximates the expected calibration error (ECE) Ellp-P(8 g/ = []]] with bin size equal to M Naeini et al. [2015]; Guo et al. [2017]; see Lee et al. [2022] for statistical results about about the empirical ECE as estimator. The perfect calibrated map corresponds to the case when FΦO = . = p) = p holds for all p E [0.
Multi-domain calibration. Although the standard ECE measurement in Eq. (1) provides informative evaluations for various calibration methods in the single-domain scenario, it does not provide fine- grained evaluations when the dataset consists of multiple domains, PI...., Pr. It is possible that the
3

---

ECE evaluated on the pooled data Di D2 UDx is small while the ECE evaluated on one of the domains is large. For example, shown in Figure [cc), there may exist domain, & E [K]. such that the ECE evaluated on domain & is much higher than the ECE evaluated on the pooled dataset, i.e., ECE(D2) > ECE(DR K ). In the fMRI application mentioned in Section 1, producing well-calibrated confidence on data from every hospital is a more desirable property compared to only being calibrated on the pooled data from all hospitals. Therefore, it is natural to consider the ECE evaluated on every domain, which we refer to as "per-domain ECE." " Next, we introduce the notion of Multi-domain ECE formalize per-domain calibration.
Definition 2.2 (Multi-domain ECE). For a dataset Di D1 U.K consisting of samples from K domains, where D3 = {(TT,K k yi,k) J and (Zi,k) yi,k) iid Px(X,Y), the (empirical) multi-domain expected calibration enror (Multi- domain ECE) with M bins evaluated on D is defined as MDE pool, K E
Remark 2.3. In Definition 2.2, we weight each domain equally balance across domains, which could better reflect how the calibration method performs on each individual domain. Furthermore, in our experiments, we also visualize the ECE measured on each domain provide additional information on model performance on every domain.
Compared with the standard ECE evaluated on the pooled dataset, multi-domain ECE provides information about per-domain model calibration. In the multi-domain setting, we aim to learn a calibration map h that can produce calibrated confidence estimates on every InD domain. Intuitively, if the unseen OOD domain D is similar one multiple InD domains, h can still provide reliable confidence estimates on the new domain. We formally study the connection between "well-calibrated on each InD domain" and "robust calibration the OOD domain" in Section 5.
Temperature scaling. Next, we review simple and effective calibration method, named temperature scaling (TS) [Platt et al., 1999; Guo et al., 2017], that is widely used in single-domain model calibration. Temperature scaling applies a single parameter T 0 and produces the confidence prediction for the base model f as
h <x; (x; f,T)=
max [Softax(((())TT)) jE(L....))
where [Softmax(z)]; = =xxp(<,,)// exp([=]i). The parameter T is the so-called temperature, with larger temperature yielding more diffuse probability estimates. To learn the temperature parameter T from dataset DD { (xi, y;) Guo et al. [2017] propose to find T by solving the following convex optimization problem,
min T
:= E 11 j} loog (o(oofaaax(((()))T)))
(2)
which optimizes the temperature parameter such that the negative log likelihood is minimized. We use TS-Alg denote the temperature scaling learning algorithm; given inputs dataset and base model f, TS-Alg outputs the learned temeperature parameter by solving Eq. (2), e.g., T = TS-AAl(D,,
3 Multi-domain temperature scaling
We propose our aggoritnn-muli-domaaa
temperature scaling-thal aims improve the calibration on each domain. One key observation is that if we apply temperature scaling to each domain separately, then TS is able produce calibrated confidence every domain. Therefore, the question becomes how to "aggregate" these temperature scaling models and learn one calibration model, denoted by h, that has similar performance to the k-thh calibration model h2 evaluated on domain k for every l [K].
At a high level, we propose to learn a calibration model that maps samples from the input space X' to the temperature space R., To start with, we learn the temperature parameter 7j for the base model on every domain & by applying temperature scaling on Dz. Next, we apply the base deep model to compute feature embeddings of samples from different domains,' and label feature embeddings from We use the penultimate layer outputs of model f as the feature embeddings by default.

---

the k-th domain with Ty. In particular, we construct K new datasets, Di..... Dx. where each dataset contains feature embeddings and temperature labels from one domain, i... Dx = (1, R). Ti) ) Finally, we apply linear regression on these labeled datasets. detail, our algorithm is as follows:
Learn temperature scaling model each domain. For every domain k, we learn temperature 7 by applying temperature scaling on validation data D2 = [(Tak) 31.k) from k-th domain, i.e., T. TS-Alg(D and ASg denotes the TS algorithm.
2. Learn linear regression of temperatures. Extract the feature embeddings of the base deep model f on each domain. Use (z,,x) R" denote the feature embedding of the i-th sample from k-th domain. Then we learn 0 by solving the following optimization problem.
nA
0 argmin ΣΣ (((()))))) -
3. Predict temperature unseen test samples. Given unseen test sample ह, we first compute the predicted temperature T' using the learned linear model T 2 O). Then we output the confidence estimate for sample as
= max [Sotmmax ((())
We denote our proposed method by MD-TS (Mult-Domain Temperature Scaling). presentation of the algorithm in pseudocode can be found in Algorithm 1, Appendix A.
We pause to consider the basic concept in more detail. The goal of our proposed algorithm is to predict the best temperature for samples from different several domains. In an ideal setting where the learned linear model 0 results in good calibration on every InD domain, will continue to yield good calibration on the OOD domain J when J is close to one several InD domains. For example, JP will work well if a mixture of the domains, i..., T = Σk= og and a E K-1 Regarding the algorithmic design, linear regression is one of the simplest models for solving the regression problem. It is computationally fast to learn such linear models as well as make predictions on new samples, making attractive. We test alternative, more flexible, regression algorithms in Section but do not observe significant gains over linear regression.
can expect that 0
To illustrate how our proposed algorithm MD-TS per- forms differently from standard TS, we return to the ImageNet-C dataset. We compare the predicted tem- perature of our algorithm with the temperature that results from running TS on domain & alone. The results are summarized in Figure 2, where each circle corresponds to the mean predicted temperature domain, we also visualize the standard deviation of the predicted temperatures for samples from that domain (the horizontal bar around each point). We find that our algorithm predicts the temperature quite well. Note that it does not have access to the domain index information of the fresh samples. By contrast, TS always uses the same temperature, regardless of the input point.
y=x
Temperature ft on domain
samples from domain
one InD domain. For each
10-
16 Predicted temperature on domain
100
Figure 2: Compare the predicted temperature to the learned temperature Tx on the lc-th domain.
Experiments
In this section, we present experimental results evaluating proposed method, demonstrating its effectiveness on both in-distribution and out-of-distribution calibration. We focus on three real- world datasets, including ImageNet-C [Hendrycks and Dietterich, 2019]-a widely used robustness
5

---

Table I: Per-domain ECE (%) comparison three datasets. We evaluate the per-domain ECE on InD and OOD domains. We report the mean and standard error of per-domain ECE on one dataset. Lower ECE means better performance.
Datasets
Architectures
InD-domains
OOD-domains
MSP
TS
MD-TS
MSP
TS
MD-TS
ResNet-50
7.36::0.28
5,80=0,10
3.84+0.05
6.87:0016
5.70=00.06
4.55::0.04
Efficientnet-bl
6.78++0.07
6, 6.12+0.15
3.99
6.54++0.06
4.87++0.05
0.05
ImageNet-C
BiT-M-R50
6.93:10.27
6.9910.25
3.8610.06
6.32:10.16
6.5020.16
4.30:00.04
ViT-Base
4.77+0.11
4,34+0,12
3,76++0007
4.09+0.06
4,01
3,86++0.44
ResNet-50
26.22:10.38
9,8310,57
2.85110.17
26.22:10.38
13,7810043
5,2510.11
WILDD-RRRRI
ResNext-50
25.30==0.76
9,39+0058
3.13++0.19
20.71+0.30
11800+0.37
5.07+00099
DenseNet-121
32.37
8.9110060
2.94.10.18
24.49.10.35
13.08.1044
5.38
ResNet-50
12.56=0.088
11.61+0.09
9.90+0.00
11.36=0.15
10.75+0.14
9.76=00112
GLDv2
BiT-M-R50
14.86.10.12
11.31.10.07
9.78.10.66
13.9110.211
9.8310.11
9.161
ViT-Small
12.4420.11
11.12:0.07
9.75:10.55
11.00#0.18
9.65#0.11
9.01=10.10
benchmark image classification dataset, WILDS-RxRX [Koh al., 2021]-an an image of cells (by fluorescent microscopy) dataset in the domain generalization benchmark, and GLDv2 [Weyand et al., 2020] a landmark recognition dataset in federated learning. Additional experimental results and implementation details can be found in Appendix C. Our code is available at https: // thub. com/yaodongyu/lii
Datasets. We evaluate different calibration methods on three datasets, ImageNet-C, WILDS-RXRXY and GLDv2. ImageNet-C contains 15 types of common corruptions where each corruption includes five severity levels. Each corruption with one severity is one domain, and there are 76 domains in total (including the standard ImageNet validation dataset). We partition the 76 domains into disjoint in-distribution domains and out-of of-distribution by severity level or corruption type. WILDS-RxRX is domain generalization dataset, and we treat each experimental domain as one domain. We adopt the default val/test split in Koh et al. [2021]: use the four validation domains as in-distribution domains and the test domains as the duist--distriuution domains. We also provide experimental results of other random splits in Appendix C. For GLDv2, each client corresponds to one domain, and there are 823 domains in total. We randomly select 500 domains for training the model, and then use the remaining 323 domains for evaluation denoted by validation domains. We further screen the validation domains by removing the domains with less than 300 data points. There are 44 domains after screening, and we use 30 domains as in-distribution domains and the remaining 14 domains as out-of-distribution domains. For all datasets, we randomly sample half of the data from in-distribution domains for calibrating models and use the remaining samples for InD ECE evaluation. We use all the samples from OOD domains for ECE evaluation.
Models and training setup. We consider multiple network architectures for evaluation, including ResNet-50 [He al., 2016], ResNext-50 [Xie et al., 2017], DenseNet- 121 [Huang al., 2017], BiT- M-50 (Kolesnkkov et al., 2020], Efficiinttet-bbl [Tan and Le, 2019], ViT-Small, and ViT-Base [Doso- vitskiy et al., 2020]. To evaluate on ImageNet-C, we directly evaluate models that are pre-trained on ImageNet [Deng et al., 2009]. For WILDS-RXXX and GL.vv,, we use the ImageNet pre-trained models as initialization and apply SGD optimizer to training the models on training datasets.
Evaluation metrics. We use the Expected Calibration Error (ECE) as the main evaluation metric. We set the bin size as 100 for ImageNet-C, and set bin size as 20 for WILDS-RxRxI and GLDv2. We evaluate ECE on both InD domains and OOD domains. Specifically, we evaluate the ECE of each InD/OOD domain. Meanwhile, we also evaluate the ECE of the pooled InD/OOD domains, i.e. the ECE evaluated on all samples from InD/OOD domains. We use unseen samples from the InD domain to measure the per-domain ECE. We also measure the averaged per-domain ECE results (i.e.. per-domain ECE averaged across domains).
4.1 Main results
We summarize the ECE results of different methods on three datasets in Table 1 and Figure 3. We use TS to denote temperature scaling [Guo et al., 2017], and use MSP to denote applying the maximum softmax probability [Hendrycks and Gimpel, 2016] of the model output (i.c., without calibration).

---

Compane ECE on InD domains
Compare on ODD domains
Compare ECE on 000 domains
...: ymx
yex
y-x
025
LC22 (MI--18)
6.3
ECE (MP--703
0.39
626
682
042
6:14
<30 Mile Clts orm clo oin
kis> 0389 120
305
<30
315
05
(TSI
(a) (InD) ImageNet-C.
(b) (InD) WILDS-
(c) (InD) GLDv2.
Compare ECE on OOD domains
Compare ECE on OOD domains
Comparc ECE OOD domains
0.2-
0.2-
(M0-75
ECE (4D3753
0.0-
046
045
644
045-
18:
610
085
<30
xis
650
ECE (TS)
ECE
(TS)
ECE (TS)
(d) (OOD) ImageNet-C.
(c) (OOD) WILDS- RxRx1.
(f) (OOD) GLDv2.
Figure 3: Per-domain ECE of MD-TS and TS on both in-distribution domains and out-of-distribution domains. Each plot is shown with ECE of TS (X-axis and ECE of MD-TS (Y-axis). Top: per- domain ECE evaluated on InD domains. Bottom: per-domain ECE evaluated on OOD domains. Lower BCE is better.
In Table 1, we use the ImageNet validation dataset and ImageNet-C datasets with severity level s E {1,5} as the InD domains and use the remaining datasets as OOD domains. We present the averaged per-domain ECE results in Table 1, and visualize the ECE of cach domain in Figure 3. As shown in Table and Figure 3(a)-3(c), we find that our proposed approach achieves much better InD per-domain calibration compared with baselines. Also, TS does not significantly improve over MSP on ImageNet-C InD domains in Table 1, but our proposed method largely improve the ECE compared with MSP and TS. For instance, the ECE results of MSP and TS on Efficientnet-bl are 6.93 and 6.99, and our method achieves 3.84. Intuitively, when there are a diverse set of domains in the calibration dataset, single temperature cannot provide well-calibrated confidences. In contrast, our proposed method is able to produce much better InD confidence estimates by leveraging the multi-domain structure of the data.
Next we study the performance of different methods on out-of-distribution domains. From Table I, we find that MD-TS achieves the best performance on OOD domains arcoss all the settings. On ImageNet-C with BiT-M-R50, MD-TS improves the ECE from 6.54 (MSP) to 4.05, while the performance of TS is similar to MSP. Moreover, MD-TS significantly outperforms MSP and TS on WILDS-RXRx1, where MD-TS improves over TS by around 5.00 measured in ECE. Figure 3(d)-3(f) display the per-domain ECE performance on more than half of the domains in all three datasets. For the remaining domains, MD-TS performs slightly worse than TS. Furthermore, on those domains that TS performs poorly (ECE > 8), MD-TS largely improves over TS by large margins. Further comparisons in Appendix C.6 show that these improvements continue to hold when relative to two other calibration techniques: MC dropout [Gal and Ghahramani, 2016] and deep ensembles [Lakshminaray anan et al., 2017].
out-of-distribution domains. MD-TS improves over TS
4.2 Predicting generalization
Suppose a model can produce calibrated confidences on unseen samples, in which case we could leverage the calibrated confidence to predict the model performance. Specifically, based on the definition of ECE in Eq. (1), when the model is well-calibrated, the average of the calibrated confidence is close to the average accuracy, i.e., CDn ~ Acc( Meanwhile, predicting model
²C) denotes the average (calibrated) confidence on dataset D, and Acc(1) denotes the average accuracy on dataset D)
7

---

Table 2: Model performance prediction comparison results of different methods on three datasets. Lower MAE indicates better performance.
Datasets
Architectures
InD-domains MAE
OOD-domains MAE
MSP
TS
MD-TS
MSP
TS
MD-TS
ResNet-50
5.88
4.74
1.28
5.15
3,96
1.70
ImageNet-C
BiT-M-R50
6.08
6.16
1.33
4.97
5.23
1.66
ResNet-50
33.65
9.61
1.61
26.20
13.66
4.76
WILDS-R<RXI
ResNext-50
25.32
8.55
1.39
20.72
12.88
4.78
ResNet-50
9.60
9.17
7.11
9.72
9.40
8.08
GLDv2
BiT-M-R50
12.67
7.18
4.64
12.30
7.34
6.37
Accurary pred chinn performance 000 domains
Accuracy prediction performance on 000 domairs
Accurary pred etton performance on 000 domains
y-x
y-x
y-x
T5
TS
Aecuracy
Accurscy
St
de
à
41
65
64 6
Prodicted Accuracy
Prodicted Accuracy
Predicted Accuracy
(a) (OOD) ImageNet-C.
(b) (OOD) WILDS--RRXI.
(c) (OOD) GLDv2.
Figure 4: Predicting accuracy performance of MD-TS and TS on both duitr--dist domains. Each plot is shown with predicted accuracy (X-axis) and accuracy (Y-axis). Each points corresponds to one domain. The network architecture ResNet-50 for three datasets. Point closer to the Y X dashed line means better prediction performance.
performance accurately is an essential ingredient in developing reliable machine learning systems, especially under distributional shifts [Guillory et al., 2021]. As shown in Table 1, we find that proposed method produces well-calibrated confidence values on both InD and OOD domains. We now measure its performance on predicting model performance and compare with existing methods. We measure the performance using mean absolute error (MAE), MAE = (1/K) Σk (Dx) Acc(DR)) where S₃ is the dataset from the k-th domain.
We show the predicting model accuracy results Table 2. MD-TS significantly improves over existing methods on predicting model performance across all three datasets. For example, on ImageNet-CC calibrated confidence of MD-TS produces fairly accurate predictions on both InD and OOD domains (less than 2% measured in MAE), which largely outperforms MSP and TS. In Figure 4, we compared the prediction performance of TS and MD-TS on every OOD domain. We find that MD-TS achieves better prediction performance compared to TS on most of the domains. Refer to Appendix C.1 for more results in which other architectures are tested.
4.3 MD-TS ablations
To learn a calibration model that performs well per-domain, we apply linear regression on feature representations ((xx) such that (*(x))) 0) R Tk. where Iy is from domain A: and Tj is the tem- perature parameter for domain k. We investigate other methods for learning the map from feature representations to temperatures in regression framework. Specifically, beside the ordinary least squares (OLS) used in Algorithm I, we consider ridge regression (Ridge), robust regression with Huber loss (Huber), kernel ridge regression (KRR), and K K-nearest neighbors regression (KNN). The implementations are mainly based on scikit-lears [Pedregosa et al., 2011]. We use grid search (on InD domains) to select hyperparameters for Ridge, Huber, KRR, and KNN.
We summarize the comparative results for different regression algorithms in Table 3. Compared to OL.S other regression algorithms do not achieve significant improvement. Specifically, KRR achieves slightly better performance on OOD domains, while other algorithms have similar performance compared to OLS. Moreover, there are hyperparameter in OLS, which makes it more practical in
8

---

Table 3: Per-domain ECE (%) results of MD-TS ablations WILDS-RXRXI. We evaluate the per-domain ECE on InD and OOD domains, and report the mean and standard error per-domain ECE. Lower ECE means better performance.
Architectures
InD-domains
O0D-domains
OLS
Ridge
Huber
KRR
KNN
OLS
Ridge
Huber
KRR
KNN
ResNet-50
2.85
2.88
2.90
2.85
3.00
5.25
5.26
5.29
4.99
5.44
ResNext-50
3.13
3.14
3.11
3.07
3.03
5.07
5.06
5.02
4.94
5.36
DenseNet-121
2.94
3.03
2.92
2,90
3.04
5.38
5.42
5.36
5.20
5.47
real-world problems. Meanwhile, the results suggest that our proposed MD-T'S is stable the choice of specific regression algorithms.
5 Theoretical analysis
In this section, we provide theoretical analysis to support our understanding of our proposed algorithm in the presence of distribution shifts. We use (-) h(; f. Bt) X - [0. 1] to denote the best calibration map for the base model f on the k-th domain; this map minimizes the expected calibration error (ECE) Ellp Pl y/ p}]] over distribution Px- We also call hg a hypothesis in the hypothesis class H. Next, given the fixed base model f. we aim to learn h(-) = f B) such that eh,, PA.x) ) = Ex~Px.x [[ng h[X)]] is small for every domain k. where E (h) denotes the risk of h w.r.t. the the best calibration map hig under domain Pg- In addition, we are interested in generalizing to new domains: suppose there is unseen OOD domain P and its marginal feature distribution is different from existing domains, i.e., Px PA.X for l [K].
Our goal is understand the conditions under which å can have similar calibration OOD domains as the InD domains. For example, if the OOD domain is similar to the mixture distribution of InD domains, we would expect h performs similarly on InD and OOD domains. Previous work [Krueger al., 2021] consider a similar multiple training domains setting the context of out-of-distribution generalization, where they show that minimizing the differences in training risks (w.r.t. different domains) can lead to good OOD performance. To quantify the distance between two distributions, we first introduce the HC-divergence [Ben-Davis et al., 2010] to measure the distance between two distributions:
Definition 5.1 (H--iivegeecc). Given an input space X and two probability distributions x and P'x on X. let HC be hypothesis class on X, and denote by A the collection of subsets of X which are the support of hypothesis h E H. i.e., Ag = =hh (h- E HQ.. The distance between T x and Pl is defined
doc(Pxx
sup AEAx
Pre (A) Pry (A)).
The "[-divergence reduces to the standard total variation (TV) distance when H( contains all mea- surable functions on X, which implies that the "-ddivereence upper bounded by the TV-distance. i.e. (3 x, ) < dTv(px (Tv("x,,''' .P')) On the other hand, when the hypothesis class HC has a finite VC the H--divereecce can be estimated using finite samples from Px and P'x [Ben-Davia David et al., 2010]. Next, we define the mixture distribution of the K in-distribution domains Po on input space X' as follows:
dimension or pseudo-diment
K
Pa
KXX
Σ PE.x
where Σ az =a==1 and a 2 O.
k=1
Given multiple domains (PI,..., Px.. we can optimize the combination parameters such that Po x minimizes the H-divergence between Po R. and Px. More specifically, we define à
à = argmin dy (PR (PR.x, ) x(Pxx.,x)) A0P Fx) = e(h", (,rx)x + eehh", Px), (3)
aeA

---

where h" := (e }x)} and HC is defined as H := (s t) I1,, N'' E H,, S S S 1). We now give upper bound on the risk on the unseen OOD domain. This result follows very closely those of Blitzer et al. [2007]; Zhao et al. [2018], instantiated in our calibration setup. Details can be found in Appendix D.
Theorem 5.2. Let H. be a hypothesis class that contains functions X - 10 with pseudo- dimension Pd) = d. Let (DR,x 11 denote the empirical distributions generated from {Px.X] I k=1. where D contains i.i.d. samples from the marginal feature distribution Pk.x of domain k. Then for å E (0, 1), with probability at least 1 - 8, we have
1 (h, āk E(h, Dx,x) dor xX A(P k.x. Px) + Ō ( Vvkk (Pdim(())
(4)
where â and X(P kxx, Px) are defined in Eq. (3), Px denotes the murginal distribution of the OOD domain, Pdim((H) is the pseudo-dimension of the hypothesis class H, and 8(h, D2.x) ) is the empirical risk of the hypothesis h on DR,x.
Remark 5.3. As shown in Theorem 5.2, even if the OOD domain is very different from the in- distribution domains, the Eq. (4) still implies that could decrease the risk upper bound the OOD domain if we perform multi- domain calibration. More specifically, if we could achieve good calibration performance on each individual domain by using multi-domais calibration (which that the first term in the RHS of Eq. (4) small), then the term (Pi, + X(P Px) is always smaller or equal to d (P'PPx) X(P', Px), where D' is the pooled distribution or any individual domain distribution.
Remark 5.4. As suggested by Eq. (4) of Theorem 5.2, larger risks on in-distribution domains will lead to larger upper bound for the risk evaluated the OOD domain. On the other hand, as shown Figure 1, a universal temperature is not sufficient to achieve good calibration performance on each individual in-distribution domain. Therefore, even in the mixture of n--iitributtinn domain setting, universal temperature suboptimal and applying multi- domain temperature scaling could be better than using universal temperature.
This result means that if we can learn hypothesis i that achieves small empirical risk ē(h, Dk.x) on every domain, then h is able to achieve good performance on the OOD domain if distribution of the OOD domain is similar to the mixture distribution of InD domains measured by H-divergence. In this case, if the learned calibration map is well-calibrated on every domain Px. then is likely to provide calibrated confidence for the OOD domain P. Recall from Section 4, proposed an algorithm that performs well across InD domains. The upper bound in Eq. (4) provides insight into understanding why this algorithm is effective.
Discussion
We have developed an algorithm for robust calibration that exploits multi-domain structure in datasets. Experiments on real-world domains indicate that multi-domain calibration is an effective way to improve the robustness of calibration under distribution shifts. Our proposed algorithm still needs validation domains to achieve strong calibration performance on OOD domains, one interesting direction for future work would be to extend our algorithm to scenario where no domain information available. We hope the multi-domain calibration perspective in this paper can motivate further work to close the gap between in in-distribution and out-of distiiibibtt calibration.
Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for their constructive suggestions and comments. Yaodong Yu acknowledges support from the joint Simons Foundation-NSs -NFF DMS grant #2031 1899, Stephen Bates was supported by the Foundations of Data Science Institute and the Simons Institute. Yi Ma acknowledges support from ONR grants N00014-20-1-208 and 200014-22-1-2000 and the joint Simons Foundation-NSF DMS grant #2031199, Michael Jordan acknowledges support from the ONR Mathematical Data Science program.
10

---

References
Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I,, and Lei, L. (2021), Learn then test: Calibrating predictive algorithms achieve risk control. arXiv preprint. a1Xiv:2110.01052.
Barocas, S., Hardt, M., and Narayanan, A. (2019). Fairness and Machine Learning. fairmlbook.org http:// htttp://ww..e http://wwww...mbbo.k/kook org.
Bates, S., Angelopoulos, A., Lei, L.,, Malik, J., and Jordan, M. I. (2021). Distribution-free, risk- controlling prediction sets. Journal of the ACM, 68(6).
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F,, and Vaughan, J. W. (2010). theory of learning from different domains. Machine learning, 79(1):151-175.
Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Wortman, J. (2007), Learning bounds for domain adaptation. Advances in neural information processing systems, 20.
Brier, G. W. al. (1950). Verification of forecasts expressed terms of probability. Monthly weather review, 78(1):1-3.
Cauchois, M., Gupta, S., Ali, A., and Duchi, J. C. (2020). Robust validation: Confident predictions even when distributions shift, arXiv preprint arXiv:2008.0422 042667.
Deng, J., Dong, W., Socher, R., Li, L.-J, ,, Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 255. leee.
Dosovitskiy, A., Beyer, I.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. (2020). An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Gal, Y. and Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR.
Gal, Y., Hron, J., and Kendall, A. (2017). Concrete dropout. Advances in neural information processing systems, 30.
Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T., and Schmidt, L. (2021). Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 113--1144.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. (2021). The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV.
Hendrycks, D. and Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint
12261..
Hendrycks, D. and Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1rX1v:1610
Hendrycks, D., Mazeika, M., Kadavath, S., and Song. D. (2019a), Using self-supervised learning can improve model robustness and uncertainty. Advances Neural Information Processing Systems, 32.
Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. (2019b). Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781.
11

---

Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected con- volutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708.
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2021). Wilds: benchmark of in-the-wild distribution shifts. In International Conference Machine Learning, pages 5637-5664. PMLR.
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. (2020). Big transfer (bit): General visual representation learning. In European conference on computer vision, pages 491-507, Springer.
Kong, L., Sun, J., and Zhang, C. (2020). Sde-net: Equipping deep neural networks with uncertainty estimates, In International Conference on Machine Learning, pages 5405-5415, PMLR.
Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A. (2021). Out-of-distribution generalization via risk extrapolation (rex), In International Conference on Machine Learning, pages 5815-5826. PMLR.
Kumar, A. and Sarawagi, (2019). Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv: 00802.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
Lee. D.. Huang, X., Hassani, H., and Dobriban, E. (2022). T-cal: An optimal test for the calibration of predictive models. arXiv preprint arXiv:2203.01850
Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., and Lucie, M. (2021). Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. (2018). Foundations of machine learning.
Nacini, M. P., Cooper, G., and Hauskrecht, M. (2015). Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence.
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayannan, B., and Snoek, J. (2019). Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32.
Park, S., Bastani, O., Weimer, J., and Lee, I. (2020). Calibrated prediction with covariate shift via unsupervised domain adaptation. In International Conference on Artificial Intelligence and Statistics, pages 3219-3229. PMLR.
Park, S., Dobriban, E., Lee, I., and Bastani, O. (2021). Pac prediction sets under covariate shift. arXiv preprint arxiv:2 098408
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (201 1). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Platt, al. (1999). Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large murgin classifiers, 10(3):61-74.
Podkopaev, A. and Ramdas, A. (2021). Distribution- free uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence, pages 844 853. PMLR.
QuifioneroCandelaa J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D. (2008). Dataset shift in machine learning. Mit Press.
Tan, and Le, Q. (2019). Efficientnet:
Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-6114 14. PMLR.
12

---

Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhattacharya, T., and Michalak, S. (2019). On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32.
Tibshirani, R. J., Foygel Barber, R., Candes, E., and Ramdas, A. (2019). Conformal prediction under covariate shift. Advances in neural information processing systems, 32.
Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorithmic learning in a random world. Springer Science & Business Media.
Wald, Y., Feder, A., Greenfeld, D., and Shalit, U. (2021). On calibration and out-of-domain generalization. Advances in Neural Information Processing Systems, 34.
Wang, X., Long, M., Wang. J., and Jordan, M. (2020). Transferable calibration with lower bias and variance domain adaptation. Advances Neural Information Processing Systems, 33:19212- 19223.
Weyand, T., Araujo, A., Cao. B., and Sim, J. (2020). Google landmarks dataset v2-a large-scale benchmark for instance-level recogmition and retrieval. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2575-2584.
Xie, C., Tan, M., Gong, B., Wang. J., Yuille, A. L., and Le, Q. V. (2020). Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pages 819-828.
Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. (2017). Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500.
Zadrozny, B. and Elkan, C. (2001). Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pages 609-616. Citeseer.
Zadrozny, B. and Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699,
Zhao, H., Zhang, S., Wu, G., Moura, J. M., Costeira, J. P., and Gordon, G. J. (2018). Adversarial multiple source domain adaptation. Advances in neural information processing systems, 31.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Theo- rem 5.2.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix D.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix A.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Section 4.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "vpovcWJjAMb", "reviewer": "Reviewer_oWGL", "summary": "The paper proposes a simple method for calibration that is robust to distributional shifts in the data. The proposed method extends the temperature scaling approach by learning to predict a suitable temperature value on an unseen domain. \nThe authors show empirically that their approach (MD-TS) can outperform other approaches (including vanilla temp scaling) on both in- and out-of-distribution test sets. \nAlong with this, the authors also provide a theoretical justification for their proposed method. \n\nCalibration methods generally fail to remain robust on out-of-distribution examples and this paper works towards solving an important problem\n", "questions": "Why was the temperature prediction model chosen to be linear? \nPrior work has experimented with methods to learn to predict a temperature value (https://arxiv.org/pdf/1903.00802.pdf): they use a 2 layer MLP to predict the temperature. \n\nThe ECE metric can be very sensitive when the number of examples in the test set are small and are not distributed over all bins. In such cases it might be interesting to also compare on other more standard metrics (apart from MAE) like the Brier Score.\n", "limitations": "The authors do not have a section detailing potential societal impacts, but they talk about how calibration can be useful in critical applications in the introduction. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method is both simple and effective at calibrating several models over multiple domains (an important problem) (+ originality, +quality + significance)\n- The paper provides an interesting theoretical justification for their proposed approach which gives a better intuition for regarding calibration over multiple domains (+originality,  + quality, +significance)\n- The analysis seems comprehensive; the authors also provide ablations for their approach\n- The paper is well explained and easy to follow (+clarity)\n\nWeaknesses:\n- I felt that the chosen baselines were not very strong. [1] show that temperature scaling by itself isn't very robust on out-of-distr. examples. Perhaps the authors could compare against some of the methods shown in [1] to be more robust such as deep ensembles and MC dropout. \n\n\n[1] Ovadia Y, Fertig E, Ren J, Nado Z, Sculley D, Nowozin S, Dillon J, Lakshminarayanan B, Snoek J. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems. 2019;32.\n\n\nOverall, I feel this is a worthy contribution and vote for acceptance. Willing to increase my score if there's a more comprehensive comparison to previous methods. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "uGzvs24tUfF", "reviewer": "Reviewer_xbg7", "summary": "This paper propose a multi-domain temperature scaling method for distribution calibration with distribution shifts. The heterogeneity of temperature is used to improve the robustness. Numerical experiments and theoretical results are presented to validate the method.", "questions": "Like TS, MD-TS should also adopt a validation dataset, right? Can we do multi-domain without the validation dataset?", "limitations": "1. The baselines only include TS and max-softmax. Can you compare with the more recent method, such as node, sde-net and others?\n2. How is the optimization goes? Is MD-TS still efficient, compared with TS?\n3. In the introduction, only an example of multi-domain uncertainty? I suggest a more detailed elaboration on this motivation, since it’s the main contribution and claim.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n\n1. well-motivated\n2. clear presentation and good writing\n3. extensive results to support\n\nweakness\n\n1. more rigorious  experimental results (more baselines and efficiency, detailed in the limitations)\n2. more introduction to highlight the motivation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "HXaWa_JkgYe", "reviewer": "Reviewer_4JGV", "summary": "This paper proposes a type of temperature scaling that is more robust on out-of-distribution dataset. The work first learns K temperatures for K in-domain datasets. Then a linear regressor is learned to predict the K temperatures given feature embeddings, hence learning to predict a suitable temperature given a sample. The linear regressor is then applied to out-of-distribution samples, calibrating a model’s output. The paper verifies the idea on three datasets and illustrates improved calibration results by the proposed method.", "questions": "Questions\n1.\tIn Definition 2.2, is the number of pooled data identical in each domain? If not, taking the average should not be enough. It should reflect the number of examples in each domain as well. Hence, taking expectation over K domains should be modified.\n", "limitations": "-\tThe concept of the proposed method is tested on image domain. It would be interesting to see the method on other domains, such as NLG (translation).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n-\tThis paper asks an important question: How should one choose temperature for predictions from out-of-distribution inputs. \n-\tThe empirical result shows improved calibration score with the proposed method over the baselines.\n\nWeakness\n-\tThe assumption in the Theorem 5.2 is strong. The assumption starts by assuming that Out of distribution is similar to a mixture of in-domain distributions. This assumption, however, is too strong without proper citations or empirical results. If this assumption holds, then simply learning a universal temperature with data collected from the in-domains may be enough. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept