Where are we with calibration under dataset shift in image classification?
Authors: Mélanie Roschewitz
Raghav Mehta
Fabio De Sousa Ribeiro
Ben Glocker
Venue: Accepted by TMLR ()

Abstract: We conduct an extensive study on the state of calibration under real-world dataset shift for image classification. Our work provides important insights on the choice of post-hoc and in-training calibration techniques, and yields practical guidelines for all practitioners interested in robust calibration under shift. We compare various post-hoc calibration methods, and their interactions with common in-training calibration strategies (e.g., label smoothing), across a wide range of natural shifts, on eight different classification tasks across several imaging domains. We find that: (i) simultaneously applying entropy regularisation and label smoothing yield the best calibrated raw probabilities under dataset shift, (ii) post-hoc calibrators exposed to a small amount of semantic out-of-distribution data (unrelated to the task) are most robust under shift, (iii) recent calibration methods specifically aimed at increasing calibration under shifts do not necessarily offer significant improvements over simpler post-hoc calibration methods, (iv) improving calibration under shifts often comes at the cost of worsening in-distribution calibration. Importantly, these findings hold for randomly initialised classifiers, as well as for those finetuned from foundation models, the latter being consistently better calibrated compared to models trained from scratch. Finally, we conduct an in-depth analysis of ensembling effects, finding that (i) applying calibration prior to ensembling (instead of after) is more effective for calibration under shifts, (ii) for ensembles, OOD exposure deteriorates the ID-shifted calibration trade-off, (iii) ensembling remains one of the most effective methods to improve calibration robustness and, combined with finetuning from foundation models, yields best calibration results overall.

Published in Transactions on Machine Learning Research (10/2025)
Where are we with calibration under dataset shift in image classification?
Mélanie Roschewitz Imperial College London
m.roschewitz21 00mprtral..mmmmmmmmcc..mo
Raghav Mehta Imperial College London
maghau. Dimpeniall...m.i...c...
Fabio De Sousa Ribeiro Imperial College London
f. pttp:/ssss..reeeeooimeieeeereled uk
Ben Glocker Imperial College London
b. b.glocker uk
Reviewed on OpenReview: https: // openreview. net/ forum? id= 1NYXXURU2H
Abstract
We conduct an extensive study on the state of calibration under real-world dataset shift for image classification. Our work provides important insights on the choice post-hoc and in- training calibration techniques, and yields practical guidelines for all practitioners interested in robust calibration under shift. We compare various post-hoc calibration methods, and their interactions with common in-training calibration strategies (c.g., label smoothing), across a wide range of natural shifts, on eight different classification tasks across several imaging domains'. We find that: (i) simultaneously applying entropy regularisation and label smoothing yield the best calibrated raw probabilities under dataset shift, (ii) post-hoc calibrators exposed to a small amount semantic out-of-distribution data (unrelated to the task) are most robust under shift, (iii) recent calibration methods specifically aimed at increasing calibration under shifts do not necessarily offer significant improvements over simpler post-hoc calibration methods, (iv) improving calibration under shifts often comes at the cost of worsening in-distribution calibration. Importantly, these findings hold for randomly initialised classifiers, well as for those finetuned from foundation models, the latter being consistently better calibrated compared to models trained from scratch. Finally, we conduct an in-depth analysis of ensembling effects, finding that (i) applying calibration prior to ensembling (instead of after) more effective for calibration under shifts, (ii) for ensembles, OOD exposure deteriorates the ID-shifted calibration trade-off, (iii) ensembling remains one of the most effective methods to improve calibration robustness and, combined with finetuning from foundation models, yields best calibration results overall.
1 Introduction
Reliable uncertainty estimation paramount the development of trustworthy machine learning models. Models should accurately estimate their level of uncertainly, without being over- under-confident in their predictions. Model calibration precisely measures how well predicted confidence scores reflect the likelihood of model predictions to be correct (Murphy, 2023; Guo et al., 2017). Calibrated probabilities are more interpretable and enable reliable performance estimation post-deployment. Improving calibration has been
'Our code publicly available httpa://github. http://.cmmmmmmmiianiireee ibration, under shifts

---

Published in Transactions on Machine Learning Research (10/2025)
INITIALISATION
ECE BRIER SCORE
RANDOM
TEMPERATURE SCALING
SINGLE MODEL
LABEL SMOOTHING
ISCTONIC REGRESSION
ENTROPY REGULARISATION
ENERGY-BASED
LIVINE-7 WIL Con WILDS Cam RETINA
ENTITY3C EMBED DomainNet CXR
TOCALLOSE
DYNAMICALYY-NN LY-AWRR
BRATION
ENSEMBLE
ID SHIFTED TEST
FOUNDATION
CALIBRATION EFFECT
the post-hoc calibration strategy
under shifts?
F50064
140
02: How do in-training and post-hoc calibration methods interact?
Q3: Is there trade-off between ID and shilted calibratio??
findings hold classifiers fine-tuned from foundation models?
Q5: How about model ensembbees?
611
028
Figure 1: Overview of our calibration study.
Foundation (best)
Model
post-hac calibration?
YES (best)
Use ensembling?
No in-training calibraiion 000
exposure needed
(best)
ID-shifted calibration trade-off
Random
Go-to baseline: TS
ND
Exposure to semantic CCD key in-training calibraiin needed ID-shifted calibration trade-off
Train with ER+LS for best calbrrtion under shift
Go-to baseline: TS-OOD, EBS
Figure 2: Practical guidelines for improved calibration under dataset shifts.
the topic of many papers over the last few years, across various tasks (Guo al., 2017; Szegedy et al., 2016; Müller al., 2019 Mukhoti et al., 2020; Zadrozny & Elkan, 2002; Gupta et al., 2021; Zhang al., 2020: Tomani et al., 2023; 2021; Seligmann et al., 2023; Kim & Kwon, 2024).
Dataset shift is an eminent issue for model calibration (Ovadia et al., 2019). Yet, for long, most of the field has concentrated on evaluating new calibration methods in in-distribution (ID) settings, where calibration and test sets come from the same distribution (Guo et al., 2017: Zhang et al., 2020; Percyra et al., 2017: Mukhoti al., 2020; Meister et al., 2020). While calibration under shifts still largely under-studied, recently, an increasing number of studies specifically focus on this issue (Tomani et al., 2021; 2023; Kim & Kwon, 2024; Yuksckgonul et al., 2023). However, most of those exclusively rcly limited synthetic corruption datasets for evaluation, i.e. CIFAR-C, ImageNet-C (Hendrycks & Dietterich, 2018; Krizhevsky, 2009; Deng et al., 2009). Unfortunately, it is known that observations made synthetic distribution shifts may not always hold on more realistic shifts (Taori et al., 2020; Koh et al., 2021). Hence, this paper aims to better understand existing calibration methods and identify which specific factors contribute to improved calibration under realistic dataset shifts, across various imaging domains.
A large portion of the calibration literature has focused on developing post-hoc methods, allowing to correct model
after training. These, for example, include temperature scaling (Guo et al., 2017) isotonic regression (Zadrozny & Elkan, 2002; Zhang et al., 2020). Others have instead proposed to address miscalibration directly during training, developing specific loss functions and training strate- gies (Mukhoti et al., 2020; Müller et. al., 2019; Pereyra et al., 2017). Prominent examples include label smoothing, entropy regularisation, and focal loss, which are now widely used and part of the standard im- plementation of most deep learning frameworks. However, there need for more in-depth look at their behaviour under natural dataset shifts. Moreover, some studies suggest that such in-training calibration methods may suffer from 'negative calibration' i.e. their calibration worsens when combined with posthoc
2

---

Published in Transactions on Machine Learning Research (10/2025)
calibration methods (Wang et al., 2021; Zhang et al., 2022). This prompts us to not only study in-training and post-hoc calibration methods in isolation, but also to carefully examine their interaction across wide range of shifts.
Meanwhile, foundation models have emerged as a promising avenue to improve model robustness (Oquab et al., 2023; Zhang et al., 2025; Zhou et al., 2023; Shi et al., 2024; Mehta et al., 2022). Hence, we also investigate the calibration of classifiers finetuned from these large foundation models. Specifically, we aim to understand whether: (i) the findings for randomly initialised classifiers generalise to classifiers finetuned from foundation models, (ii) calibration under dataset shifts is indeed improved compared to models trained from scratch.
Finally, perhaps the most common approach to increase robustness under shifts, when the computational budget allows, is ensembling (Lakshminarayanan et al., 2017). Many studies showed that ensembling can significantly help with model robustness and calibration (Seligmann al., 2023; Kumar et al., 2022). How- ever, there is also some contradictory evidence suggesting that deep ensembles are not necessarily better calibrated than individual ensemble members (Rahaman & Thiery, 2021; Kumar et al., 2022). Hence, we conclude our study by analysing ensemble calibration, comparing the effect of various calibration methods, and in particular, the impact of applying calibration before or after ensembling.
To summarise, as illustrated in Fig. 1, we comprehensively study existing in-training and post-hoc calibration methods, specifically investigating their impact on calibration under dataset shifts. Our gool is not to propose a new method, but rather to find practical guidelines on which methods to choose for obtaining well- colibraddd probability under shifts. Our study covers eight datasets and tasks, wide range of imaging modalities from satellite images to mammography and histopathology images. We focus on realistic distribution shifts, from geographical location shifts, medical device shifts, sub-population shifts, experimental al protocols shifts, and more. study the effect of five different in-training calibration methods and their interactions with ten post- hoc calibration methods. We evaluate models trained from random initialisation as well as finetuned from foundation models, across various CNN- and transformer- based architectures, training 420 models in total. Finally, we analyse the effect of model ensembling on calibration under shifts, compare various ensemble calibration strategies and study interactions between ensemble calibration and calibration of individual ensemble members.
The breadth of our analysis enables us to derive clear practical guidelines for practitioners interested in building classification systems with robust calibration under real-world shift, which we summarise in Fig. 2 and detail below, we find that:
(i) Classifiers fine-tuned from foundation models offer significantly better calibration (shifted and ID) compared classifiess trained from scratch, regardless of the calibration strategy ($4.2.2);
(ii) In-training calibration is not needed when post-hoc calibration is used (64.1.2);
(ifi) Ensembling improves both ID and shifted calibration regardless of the calibration strategy ($4.3.1);
(iv) Without ensembling, the most effective strategy to improve calibration under shifts consists of adding small amounts of OOD data (unrelated to the task) to the calibration set, regardless of the choice of post-hoc calibrator ($4.1.1);
(v) For model ensembles, the best strategy to obtain robust calibration under shifts consists of applying post-hoc calibration prior to ensembling, and without OOD exposure (64.3.2);
(vi) Improving calibration under shifts often comes at the cost of ID calibration (64.1.3, $4.3.3).
We make all of our code available to the research community to facilitate further developments in this space.
2
Related work
Model calibration has been a prolific area of research. Proposed approaches notably include post-hoc calibra- tion methods (Guo et al., 2017; Kim & Kwon, 2024; Zhang et al., 2020), alterations to loss functions (Szegedy
3

---

Published in Transactions on Machine Learning Research (10/2025)
et al., 2016; Pereyra et al., 2017: Mukhoti et al., 2020), model ensembling (Lakshminarayanan et al., 2017) and Bayesian Neural Networks (BNN) (Detommaso et al., 2022; Maddox et al., 2019). In this study, we focus on popular calibration methods, which can be applied within standard training pipelines and for any model architecture. In particular, we consider the comparison of various BNN training strategics for calibration robustness out-of-scope. However, the reader interested such a comparison is invited to consult Seligmann et al. (2023).
2.1 Model calibration metrics
We consider a classification task with C classes. Let p E [0, 1] denote the model predicted probabilities with arg max; Pe the predicted class and y the associated label. A model is said to be well calibrated if the probability associated with the predicted class (a.k.a. confidence score) matches the true probability of being correctly classified (Guo et al., 2017). Several metrics have been proposed to evaluate the calibration of a given model. The most popular one is the Expected Calibration Error or ECE (Guo et al., 2017), approx- imating the expected difference between predicted confidence and prediction accuracy (see Appendix A.2). Despite its popularity, a major limitation of ECE (and its variants like AdaECE, SCE (Nixon et al., 2019)) resides in the necessity of introducing binning scheme, which may influence the calibration measure and make the metric sensitive to small test set sizes (Nixon et al., 2019; Gruber & Buettner, 2022). An alternative metric is, for example, the Brier Score (Brier, 1950), which does not require any binning, is a proper score, and measures total calibration (as opposed to only top-level calibration like ECE). This score computes the average mean squared error between probabilities and one-hot labels (details in Appendix A.2).
2.2 In-training model calibration techniques
Several works have proposed specific loss functions and regularisation schemes tailored to improve the cali- bration of model outputs during training, without requiring any architectural changes. These include: label smoothing (LSS) (Szegedy et al., 2016; Müller et al., 2019), entropy regularisation (ER) (Pereyra et al., 2017: Meister et al., 2020), and focal loss (Mukhoti et al., 2020). In LS, labels are softened to explicitly discourage the network from being too confident in its predictions, i.e. ý = (1 - A)y + +yg. with the one-hot encoded labels, and X the hyperparameter controlling the amount of smoothing (typically ranging in 0.1).. ER adds the negative entropy of the predictions to the cross-entropy loss, where the strength of the regularisation is controlled by a multiplicative constant a (with a = 0.1 working well across lasks (Pereyra et al., 2017)). Another popular approach is the focal loss (Lin et al., 2020), defined as E (1 ((1-)x)' log(fx), where y is an hyperparameter. Mukhoti et al. (2020) show that focal loss can improve model calibration, in particular when using 7 = f E [0, 0.2) else 7 = 3. Note that other variants of these losses have been subsequently proposed (Meister et al., 2020; Liu et al., 2022); however, here, we focus on LS, ER, and focal losses as these remain the most popular for image classification (Gao et al., 2020; Wei et al., 2022; Wang et al., 2022; Carse et al., 2022; Fuentes et al., 2023).
2.3 Post-hoc model calibration techniques
Modifying the loss function to improve calibration is not always practical solution, as one may not always have full control over the training process, and loss function alterations may have inadvertent side effects, such as performance reduction. Hence, many have focused on developing post-hoc recalibration techniques to address mis-calibration after training. This paradigm has the advantage of being independent of the model architecture or the choice of the loss function during training and is widely used in practice. One example of such post-hoc calibration method is temperature scaling (TS) (Guo et al., 2017) (or Platt-scaling), where the logits are scaled by constant T, where T is tuned the validation set such that the negative likelihood is minimised. Calibrated probabilitics are then obtained with p softmax(z/T), Adjusting the temperature controls the 'peakiness' of the distribution, allowing to correct for model over/under-con Another approach consists of using isotonic regression (IRM) (Zadrozny & Elkan, 2002) between the predicted probabilitios and the expected accuracy man-maaramtriic. Gupta et al. (2021) have alternatively proposed to use splines for calibrating outputs (SPL). The synergies between various parametric and non non-parametric post- hoe methods have been explored by Zhang et al. (2020), who notably proposed ETS, an ensemble formulation
4

---

Published in Transactions on Machine Learning Research (10/2025)
of TS to allow for increased expressivity. They also proposed IROVaTS, combining IROVa (IRM in one- versus-all setup for multiclass problem) with ETS. Recently, Kim & Kwon (2024) proposed energy-based calibration (EBS), utilising the energy of the logits, i.e., F(z) = -logsumexp(z) to derive a sample-wise temperature. Importantly, this method utilises both an ID validation set and an out-of-distribution (OOD) set, semantically unrelated to the lask, to improve calibration on shifted data (contrary to previous methods, which do not explicitly require access to external OOD data for calibration). More details on EBS can be found in Appendix A.5.1. Importantly, TS, ETS, IRM, and EBS are all accuracy-preserving, ie., the final class prediction is unchanged after model calibration.
2.4 Calibration and the problem of dataset shift
An increasing number of studies have recently focused on increasing the robustness of model calibration under dataset shift. An early study from Tomani et al. (2021) proposed to apply random augmentations to images in the validation set used to fit post-calibrators, for better results under shifts. However, the generalisation of such calibrators is bounded by the ability of these random augmentations to faithfully capture shifts encountered at test-time. Yu et al. (2022) have, on the other hand, proposed to train a commaintoottempeatuure regressor. However, this requires access to multiple domains at validation time and for them to caplure test-time shifts reliably a highly impractical assumplion. Instead Tomani al. (2023) proposed Density-Aware- Aranityy-AvareCCalbrttttoo Calibration (DAC), leveraging nearest neighbours distances in the embedding space (at different layers throughout the network) to modulate the temperature for each sample, without requiring access to multiple domains at calibration time; this idea of measuring the sample's 'atypicality' to calibrate the network is also adopted in concurrent. work from (Yuksekgonul et al., 2023). Importantly, DAC is a plug-in method that can be used in conjunction with any other post-hoc calibration method. Finally, Kim & Kwon (2024) demonstrated stato-of-tho-ane results with their EBS calibrator against a wide range of post-hoc calibrators on CIFAR1O-C and ImageNet-CC C (Hendrycks & Dietterich, 2018), with best results obtained when EBS is used in conjunction with DAC. Yet, more analyses are needed to determine the robustness of these calibration methods on a broader set of tasks and, crucially, on real-world shifts beyond synthetic image corruptions.
For the convenience of the reader, a comparative overview of prior work on calibration under shifts in image classification and our study presented in Table A.2.
2.5 Model ensembling and calibration
Another popular approach to improve calibration is model ensembling (Lakshminarayanan et al., 2017). Kumar et al. (2022) showed that ensembling diverse calibrated models yields more robust models and better calibration across various tasks. Similarly, recent study from Seligmann et al. (2023) on the calibration of Bayesian neural networks showed that ensembling was more efficient than any single Bayesian calibration method in terms of robustness under shifts. However, others report contradictory evidence as well. Wu & Gales (2021) provided theoretical insights on why calibration of individual ensemble members is not sufficient for their aggregate predictions to be well calibrated, and Rahaman & Thiery (2021) showed that deep ensembles are not necessarily better calibrated than individual ensemble members. These contradictory findings prompt us to include ensembling effects in our study.
3 Experimental setup
3.1 Datasets and studied shifts
We analyse calibration robustness across different image classification tasks and real-world natural distri- bution shifts. First, we investigate robustness against geographic and sub-population shifts, in natural image classification using the Living17 and Entity30 datasels (Santurkar et al., 2020), as well as Wilds- iCam (Koh et al., 2021). Next, we analyse realistic shifts specific to medical imaging, a high-stakes domain where robustness and calibration are of particular importance. We study calibration robustness against: (i) scanner changes in breast density assessment models using the EMBED (Jeong et al., 2023) mammog- raphy dataset, (ii) scanner, population and prevalence changes in chest X-ray classification (No Finding /
5

---

Published in Transactions on Machine Learning Research (10/2025)
Dataset
Type of shift(()
Task
N classes
Living17
Population
Balanced
17
Entity30
Population
Balanced
30
CameLyon
Staining
Balanced
2
CXR
Scanner, Prevalence, Population
Imbalanced
2
RETINA
Scanner, Prevalence, Population
Imbalanced
5
EMBED
Scanner
Imbalanced
4
iCam
Geographic, Prevalence Image type, Prevalence
Imbalanced
182
DomainNet
Imbalanced
345
Table 1: Datasets and shifts used in this study.
Diseased) using CheXpert (Irvin et al., 2019) and MIMIC-CXR (Johnson et al., 2019) (CXR), (iii) equip- ment, prevalence, and geographic location changes for diabetic retinopathy assessment models, combining multiple public fundus imaging datasets (Karthik & Sohier, 2019; Decencière et al., 2014; Dugas et al., 2015) (RETINA), and (iv) staining protocols changes in histopathology. using WILDS-CameLyy 2021). Finally, we test. against hard modality shifts natural image classification using DomainNet (Peng et al., 2019) with 'Real' images as the ID domain. Details about datasets and ID DD/shifted domain definitions can be found in Appendix A.1, including visual examples of samples from both ID and shifted test sets for each dataset.
(Koh et al.,
3.2 Models
Experiments are performed in two settings. First, for models trained with randomly initialised weights, where for each dataset, trained classificrs with six different architectures: ResNet18/50 (He et al., 2016), Effi cientNet (Tan & Le, 2019), MobileNet (Howard et al., 2017), ViT-Base (Dosovitskly et al., 2020), ConvNext- tiny (Liu et al., 2022), keeping the checkpoint with the highest balanced accuracy on the validation set. We additionally investigate the calibration of models finetuned from pretrained foundation models, using the vision encoders from Dino- Bino-V2-BBl 1a-Base (Oquab et al., 2023), CLIP (Radford et al., 2021), SiG-LIP (Zhai et al., 2023), MAE-Base (He al., 2021) and, for medical datasets only, BioMedClip (Zhang el al., 2025). Addi- tional details regarding our training setup and hyperparameters can be found Appendix A.4. All detailed training configurations are also available in our codebase to ensure reproducibility.
3.3 Benchmarked methods
In this study, we aim to analyse the robustness of existing post-hoc calibration methods and their interactions with in-training calibration strategies, on a large variety of shifts and tasks. In terms of post-hoc methods, we compare TS (Guo et al., 2017), IROVaTS (Zhang et al., 2020), IRM (Zadrozny Elkan, 2002), and EBS (Kim & Kwon, 2024). EBS has been shown to achieve state-of ofate-of-thearr the-art performance on calibration under synthetic corruptions for CIFAR and ImageNet. (Kim & Kwon, 2024). However, one crucial difference separates EBS from other calibrators: it requires exposure to a small 'semantic OOD' set for calibration. This set is wnrelated to the classification task and independent of the tested modality hiimty/iiiinnnggg domain. Following Kim & Kwon (2024), we use samples from the Textures (Cimpoi et al., 2014) dataset for this purpose. However, we consider that comparing 'standard' post-hoc calibrators, such as TS, IRM, etc., with EBS is 'unfair' since EBS gets some outlier exposure in the calibration step, and, by default, the others don't. Hence, to better understand whether any potential calibration gains from EBS should be attributed to the energy- based calibration approach the use of semantic OOD data, we also compare with augmented versions of these standard post-hoc calibrators where the same semantic OOD set is used along with the ID calibration set in the calibration stage. This yields additional calibrators TS + OOD, IRM + OOD, IROVaTS + OOD (see Appendix A.5.2 for implementation details). Conversely, we also report results of cnsrgy-based calibration the absence of semantic OOD exposure, referred as EBS- Finally, we investigate the impact of DAC (Tomani et al., 2021), designed to improve calibration under shift, in combination with various calibrators.

---

Published in Transactions on Machine Learning Research (10/2025)
LIVING17
RETINA
CAMELYON
ICAM
.500
.280
.440
.80
210
.300
ECE SHIFTED
.350
180
.70
.250
*
180
080
.00
250
.000
.100
.10
2000
100
.100
.150
080
.050
.026
.850
.78
140
180
000
240
280
026
.050
875
050
100
.100
ID
ID
EMBED
CXR
ENTITY
DOMAINNET
.180
.40
.50
.110
.090
ECE 3HFFED
.080
.20
*
400
0070
.100
200
300
150
200
100
.060
.100
.040
.050
000
050
100
160
220
080
0550
050
1980
.180
.100
250
250
300
(a) ECE
LIVING17
RETINA
CAMELYON
ICAM
1,100
1.020
.880
170
.870
BRIER SHIFTED
165
.840
.900
.820
640
155
800
.850
150
.70
.820
145
.700
.000
.10
.740
.800
.135
23
.720
/1
320
340
380
410
A00
5e0
910
916
028
580
580
600
830
.000
EMBED
CXR
ENTITY30
DOMAINNET
.460
480
.455
460
1,280
BRIER SHIFTED
.450
.40
860
*
.M00
840
1,200
820
400
.120
800
.410
380
780
.000
.360
7600
.900
320
340
A20
140
1880
280
440
400
520
600
550
600
822
870
ID
ID
Calibrator
PROBAS
T8
@
RM
IROVASS
ESS
T3 000
JI
00D
IROWATS + 0C0
EBS
Loss
CE
ER
x
LS
EReLS
Fecal
(b) Brier Score
Figure 3: Calibration under shifts in function of in-distribution calibration for classifters with random weight initialisation: For each dataset, we report average ECE (top) and Brier score (bottom), averaged over different models (6 architectures)
7

---

Published in Transactions on Machine Learning Research (10/2025)
In terms of training strategies, compare standard expected risk minimisation training with cross-entropy loss (CE), label smoothing (LS), entropy regularisation 1 and focal loss. Moreover, given the symmetric effects of label smoothing and entropy regularisation, we also compare with a training strategy using both label smoothing and entropy regularisation simultancously (ER+LS). We then analyse interactions between training strategies and post-hoc calibration methods. In terms of regularisation strength, we chose to fix the hyperparameters for the strength of label smoothing and entropy regularisation across all datasets and experiments (A 0.05 for LS, a - 0.1 for ER, was deliberate choice (chosen based on the best average ID calibration results across all datasets) to assess the generalisability of a given training strategy to new scenarios. Indeed, we don't have access to the shifted test set in advance, we would not be able to tune those precisely for improving shifted calibration.
0.05, a 0.1 for ER+LS). Fixing these hyperparameters
4 Results
We evaluate calibration using Expected Calibration Error (ECE) as well as the Brier Score. Whereas ECE focuses on top-label calibration, the Brier score mcasures full calibration and has the advantage of also taking model performance into account, offering two complementary measures of model calibration (see Section 2.1). In Fig. 3 (resp. Fig. 6), we report both ID and shifted calibration results for every combination of training loss and post-hoc calibration methods, for every dataset, for models trained from scratch (resp. finetuned from foundation models). In the following, we summarise the main take-aways from these experiments.
4.1 Calibration under shifts for models trained from scratch
4.1.1 Post-hoc calibration: exposure to semantic OOD significantly improves calibration under shift.
In Fig- 3, calibrators without semantic OOD exposure appear in lighter colours whereas their counterparts with exposure appear in darker colours. First, we notice that after post-hoc calibration, there is consistent difference across the various tested training losses - no calibration-specific training loss consistently beats the standard cross-entropy loss. Most strikingly, across all datasets, we observe that calibrators with OOD exposure offer the best results in terms of calibration under shifts (y-axis), for both ECE and Brier scores, regardless of the training loss used. We, for example, find that surprisingly, simple temperature scaling with semantic exposure (TS + OOD) performs the most consistently across datasets and is at least as good as EBS, which was specifically designed for tackling calibration under shifts. Note that this improvement in calibration occurs even though the OOD data used is entircly unrelated to the task, and fixed across all evaluated dalasets (Textures dalasel, see Section 3.3). This shows that semantic OOD data does not need to be close to the ID images to benefit calibration. Some datasets, like Living17, are closer to images in the Textures dataset (RGB photos) while medical image datascts are further. In Fig. A.3, we confirm this by analysing differences between samples from the ID test set, from the shifted test sets and from the semantti OOD Textures the embedding space across all datasets, showing that for some datasets the semantic OOD samples are much closer to the 1D/shifted test sets than for others. However, results in Fig. 3 show that, regardless of the distance of the semantic OOD set to the ID data, adding exposure to these OOD images benefits calibration under shifts.
After statistical analysis, we confirm that calibrators with semantic OOD exposure have a significantly different effect over those without OOD exposure, however there are no significant differences across cali- brators once controlling for semantic OOD exposure (see Appendix A.7 for details on statistical analysis). Regarding model performance, it important to highlight that most post-hoc calibrators tested here are eecuracy-preservinn (TS, IRM, EBS), i.e. they do not affect the class prediction. Only IROV&TS affects model predictions and we note that it tends to worsen balanced accuracy both ID and under shifts, across datasets (sce Fig. A.4).
Regarding the amount of semantic data to use for calibration, authors of EBS used pre-defined numbers of OOD images for their experiments on number on new datascts. We found that the number of OOD samples has to be dependent on the size of the ID calibration set. In Fig. 4, we provide a detailed investigation of the sensitivity of various calibrators the relative size of the OOD set. Results show that the ideal OOD size depends on the dataset, heless,
but no rationale was given on how to choose this
8

---

Published in Transactions on Machine Learning Research (10/2025)
CAMELYON
ICAM
0.29
0.22
0.40
0.20
0.23
0.30
0.18
0.12
0.10
0.12
0.90
0.00
0.00
0.10
0.20
EMBED
CXR
ENTITY30
DOMAINNET
0.22
0..25
0.15
0.20
0.25
SIIFTE
SHIFTED
SHI 1HD
0.15
0.12
0.10
0.15
0.10
0.06
0.10
0.05
0.00
0.10
0.10
0.20
0.30
0.00
0.10
0.30
Calibrator with DOD
EBS
IRM
IROVATS
TS
Relative size OOD set
0.02
0.1
0.5
1.0
Figure 4: Effect of the amount of semantic OOD data used for calibration on ECE: (averaged over six model architectures). The amount of OOD data is here relative to the number of ID validation samples used to calibrate the model (varying across datasets).
find that when the semantic OOD set is about 10% the size of ID validation set (used for calibration). the calibration performance lies on the Pareto front of ID-shifted calibration for most post-hoc calibrators and datasets. We chose this parametrisation across all experiments.
Finally, we additionally evaluate the effect of complementing post-hoe calibrators with Dynamically Aware Calibration (DAC). DAC uses distances in the embedding spaces to further optimisc the sample-wise tem- perature and is compatible with many calibrators. Results in Fig. suggest that DAC has mixed effects model calibration, with substantial gains for some datasets (EMBED) and worsening calibration under shifts for others (IROVaTS) S/EBS CameLyon). Overall, it simply has little effect on many datasets (Living17, CXR, Entity30).
4.1.2 ER+LS training yields the best calibrated raw outputs on shifted data and is hard to beat with post-calibrators.
Without post-hoc calibration (black markers), models trained with both antropy-regulariiatin and label smoothing (ER+LS, diamond) are best calibrated, often by a large margin, both in terms of ECE and Brier score. For example, on Living17 and Entity30 the ECE is 50% lower for models trained with ER+T. than with CE. Overall, without post-hoc calibration, ER+LS has the lowest ECE across all tested training paradigms on 7 out of the tested datasets, and the lowest Brier in 6 out 8. Importantly, nor LS nor ER alone is able to reach the same consistent calibration robustness, combining both paradigms is key. Importantly, we find that the different loss functions perform similarly in terms of balanced accuracy across datasets (see Fig. A.4).
4.1.3 Improving calibration under shift often comes the cost of ID calibration.
In Fig. 3, we observe that on several datasets (Living17, RETINA, CameLyon, CXR, Entity30) a trade-off pattern between ID and shifted calibration appears, in particular for ECE. This particularly striking for calibrators with OOD exposure, which yield much lower calibration errors under shifts compared to other calibrators, but sometimes display substantially higher calibration error ID test sets.
9

---

Published in Transactions on Machine Learning Research (10/2025)
LIVING17
RETINA
CAMELYON
0.15
0.24
0.10
0.16
0.14
94FFED
0.08
0.14
0.13
0.20
0.12
0.00
0.05
0.11
0.04
0.03
0.04
0.05
0.05
0.10
0.15
0.00
0.02
0.04
0.06
0.06
0.04
0.00
0.08
0.10
0.12
EMBED
CXR
ENTITY30
DOMAINNET
0.07
0.18
0.40
0.07
0.25
0.00
0.80
0.06
0.14
0.29
0.05
0.12
0.20
0.16
0.04
Calibrator
IROVATS
IROVATS DOD
DAC
x Without DAC
Figure 5: Effect of DAC with various post-hoc calibrators on ECE: mixed results across datasets. Large circles show calibration with DAC; crosses denote calibration without DAC. Full comparison with Brier score can be found Appendix A.8.
4.2 Do these findings hold for classifiers finetuned from foundation models?
4.2.1 Findings from randomly initialised models generalise to models finetuned from foundation models
Fig. 6 compares calibration results both under shifts and on ID test sets, for classifiers finetuned from foundation models (averaged over four to five models, see Section 3.2). These results confirm our previous conclusions: exposure to semantic OOD improves calibration under shift, with EBS and TS+OOD yielding most consistent results in terms of DD-sifted trade-off across datasets (average ID-shifted calibration is best for TS-OOD on out of 8 datasets). In terms of base predictions (without post-hoc calibration), models finetuned with ER+LS are best calibrated under shifts compared to base predictions from models trained with other losses (black markers, y-axis) on out of datasots, however this again often comes at the cost of ID calibration (3 out 6 datasets).
4.2.2 Models finetuned from foundation models are better calibrated under dataset shifts, irrespective of the calibrator used
In Fig. 6, the red star depicts average calibration for models trained from scratch (with CE) and using robust EBS post-hoc calibration, strong baseline from Fig. 3. This comparison shows that models finetuned from foundation models offer substantially better calibration under dataset shifts, in terms of Brier score in particular (also reflecting an improvement in classification performance, see Fig. A.5), but also in terms of ECE. This conclusion may perhaps not. be very surprising per se, however, it is important to note that even with foundation models applying robust post-hoc calibration like EBS or TS with semantic OOD significantly improves calibration compared to base probabilities. Morcover, a further ablation study Fig. A.8 on the impact of model size on shifted-ID calibration demonstrates the calibration improvements are indeed attributable to the pretraining of the foundation models and not to their increased model size.
10

---

Published in Transactions on Machine Learning Research (10/2025)
LIVING17
RETINA
CAMELYON
ICAM
200
.180
.240
.400
-40
380
.140
.100
120
.120
.100
100
.100
080
.400
080
.080
.080
.060
.040
.040
.040
020
040
070
-100
000
.80
.00
.28
100
.22
.080
.78
.050
-110
-60
ID
10
ID
ID
EMBED
CXR
ENTITY30
DOMAINNET
.150
.200
280
.200
.100
.160
200
500
.080
.120
SHIFTED
.75
400
.000
.060
.800
.30
.255
.500
.400
360
.100
200
.440
.200
.200
.050
.100
020
020
040
.000
200
030
050
180
200
080
020
.40
082
100
050
.100
-120
140
(a) ECE
LIVING17
RETINA
CAMELYON
ICAM
.850
.660
.770
.150
.750
620
.730
BRIER- SHIFTED
.440
.575
130
.550
.400
.500
120
.225
.320
480
110
.500
300
A60
100
4500
280
.400
060
500
350
300
350
378
460
010
330
375
EMBED
CXR
ENTITY30
DOMAINNET
.400
440
.050
1.120
.480
330
.750
1.100
.440
BRIER SHIFTE
.580
.360
.120
.60
1.000
.300
.400
.440
340
.520
.900
.300
.300
.500
.30
.320
.480
.000
300
320
.40
200
240
060
070
080
0g0
260
225
250
275
500
650
Calibrator
PROBAS
75
RM
IROAAS
mns.
oon
BI
GDD
IROVATS 0CD
FRS
Loss
x 18
ER
FazM
(b) Brier score
Figure 6: Calibration results for classifiers finetuned from foundation models: top row ECE, bottom row Brier Score (average calibration finetuning four different foundation models, five for medical datasots). As a bascline, for each dataset, the red star depicts the average calibration of modols trained from scratch with CE loss and EBS calibration.
11

---

Published in Transactions on Machine Learning Research (10/2025)
LIVING17
RETINA
CAMELYON
ICAM
.500
.300
.080
.200
.070
ECE SHIFTED
200
.300
.75
150
.200
.25
.400
100
.100
.100
.030
.50
.066
.000
.050
-100
.150
.100
.200
.000
.020
.040
.000
.100
200
ID
EMBED
CXR
ENTITY30
DOMAINNET
400
.800
.080
.100
.500
.800
.990
.300
.400
.70
.080
200
.300
.050
.070
.100
.200
.060
.100
.050
.100
.020
.040
.100
200
.100
200
300
(a) Ensembles of classifiers trained from scratch.
LIVING17
RETINA
CAMELYON
ICAM
.300
.140
.045
.20
.120
.040
ECE SHIFTED
200
-100
.100
.030
.150
.080
.080
.05
.00
.060
.050
.040
.015
.000
.025
.050
.075
.050
.100
.000
.010
.020
.000
.000
.100
200
EMBED
CXR
ENTITY30
DOMAINNET
.400
.100
200
.360
.055
.300
.050
.080
.150
.445
.250
.040
.200
.000
.160
.030
.050
.050
-100
.020
.20
.040
.050
ID
ID
Calibrator
PROBAS
00D
EBS
ERILS
Calibration appliad
Before ensembling
After ensembing
Ensembles of classifiers finetuned from foundation models.
Figure 7: Calibration of model ensembles in terms of ECE, in function of post-hoc calibration method, Large dots denote calibration results for model ensembles, small dots the average calibration of individual ensemble members for reference. We compare the effect of (i) applying post-hoc calibration ensemble members (before ensembling), (ii) applying post-hoe calibration after ensembling predictions.
12

---

Published in Transactions on Machine Learning Research (10/2025)
4.3 Ensembles and shifted calibration
In light of conflicting results on the effect of model ensembling on calibration the literature (see Section 2.5), we here investigate whether the previous findings experimentally generalise to model ensembles. For each datasct, we create various model ensembles by sampling random combinations of three models from the pool of previously trained classifiers. We investigale two lypes of ensembles: ensembles of classifiers trained from random initialisation and ensembles of classifiers finetuned from foundation models. In Fig. 7, compare calibration results for different post-hoc calibrators when calibration is applicd: (i) to individual ensemble members before ensembling or (ii) after ensembling (as argued in (Rahaman & Thiery, 2021)). Motivated by take-away 2, we additionally compare with ensembles of models trained with ER+LSS.
4.3.1 Ensembling remains a very effective method for calibration
Fig. 7 shows that ensembles achieve better calibration under shifts compared to individual models (small versus large dots), irrespoctive of the calibration strategy. This is visible both in ECE and even more strikingly the Brier score (also reflecting, as expected (Lakshminarayanan et al.; 2017), an improvement in model performance after ensembling). Then, comparing results in Figs. 7a and 7b, we notice that calibration results lie in drastically different scales for ensembles of classifiers trained from scratch and ensembles of classifiers finetuned from foundation models. The latter being much better calibrated overall. Finally, in Figs. A.12 to A.15, ensemble n, by comparing n E {2, 5,J. We observe that calibration gains are already visible with ensembles with two members only. Gains on shifted calibration increase slightly as the number of members increases, however there is a plateau-effect with only minimal improvements when increasing from 3 to 5 ensemble members.
investigate whether these calibration gains depend on the size of the
4.3.2 Applying post-hoc calibration before ensembling yields best calibration under shifts.
Comparing post-hoc calibration pre- and post-ensembling, results in Fig. 7 show that there is a clear trade-ofi between ID and shifted calibration in terms of ECE: doing TS (or EBS) after ensembling lcads better ID calibration, whereas TS (or EBS) before ensembling leads better calibration on shifted datasets, and better overall ID-shifted calibration trade-off. For example, TS before ensembling (filled circles) systematically reaches lower calibration error shifted test sets compared to TS after ensembling (hollow circles). However, we observe the inverse pattern for ID test sets on Living17, Density, Retina, CXR, and Entity30. This finding also holds for models trained with ER+LS (see Fig. A.11). Finally, our ablation study on ensemble size in Figs. A.12 to A.15 shows that these conclusions hold for all ensemble sizes tested (n = 2, 5).
Comparing TS and TS + OOD in Fig. 7 we find that semantic OOD exposure post-hoc calibration often degrades calibration for shifted tests (y-axis, RETINA, EMBED, CameLyon) and even more so for ID test sets (x-axis). Interestingly, this is contrary to previous findings and independent of whether calibration is applied before after ensembling. Further ablation studies in Fig. A.10 confirm this finding on additional post-hoc calibrators. These results suggest that, if the goal is to ensemble models, individual ensemble members should be calibrated with standard calibration methods, not with OOD exposure.
4.3.3 Without post-hoc calibration, ensembles whose members were trained with ER+LS are substantially better calibrated under shifts.
Finally, comparing ensemble calibration without post-hoc calibration (black markers in Fig. 7), we can, again, see that ensembles whose members were trained with ER+LS offer substantially better calibration results under shifts compared to ensembles whose members were trained with CE, across all datasets, for both ECE and Brier score. However, this often comes at the cost of worse ECE on ID test sets (Living17, Entity30, EMBED, CXR, Net)
13

---

Published in Transactions on Machine Learning Research (10/2025)
5 Conclusion
We conduct a large-scale experimental evaluation of calibration methods in the presence of realistic dataset shifts. Associated findings provide practical guidelines for constructing models with robust. calibration under such shifts. First, for single models (i.e. without ensembling), find that post-hoc calibrators exposed to lask-agnostic OOD data performed best for calibration under shifts, particularly EBS and TS+OOD. This finding holds irrespective of the training paradigm, across datasets, for classifiers trained from scratch and finetuned from foundation models. Yet, this can come at the cost of worsening TD calibration, yielding an DDshifted calibration trade-off. Moreover, and perhaps surprisingly, this finding does not hold for model en- sembles, where we find that OOD exposure should not be used for calibration. Moreover, we find that models trained with entropy regularisation and label smoothing provided the best calibrated base probabilities for shifted test sets (without any post-hoe calibrator) across datascts.
Based on these findings, we derive practical guidelines for robust calibration under shifts, visually summarised in Fig. 2. First, in terms of training strategy: the best results are obtained when classifiess are initialised with foundation model weights (as opposed to random initialisation). Then, regarding the use of in-training calibration techniques, we find that they are not needed if post-hoc calibration is used. However, the aim to bypass post-hoc calibration, we find in-training calibration with ER+LS to be most effective. Regarding the choice of post-hoc calibrators, we find that applying TS+OOD (or EBS) is a robust strategy for improving calibration under shifts of single classifiers (without ensembling). Finally, regarding model ensembles, we find that simply calibrating ensemble members with TS prior to ensembling yields robust calibration results. We find that combining this ensembling strategy with finetuning from foundation models yields the most robust calibration results overall, both ID and under shifts.
In this study, we focused exclusively on robustness of calibration under dataset shifts for image classification. It is worth noting that building reliable and well-calibrated uncertainty estimates, robust across dataset shifts, is equally relevant. for other tasks. Gustafsson et al. (2023) for example studied the reliability of existing uncertainty estimates in the context of regression tasks, whereas Jorge et al. (2023) similarly studied robustness and calibration of uncertainty estimates in segnentation tasks. Morcover, the guidelines proposed here are based on our exlensive evaluation of currently commonly used post-hoc and in-training calibration methods, and these may require updating as new calibration methods appear in the future. To this end, our publicly available benchmarking codebase provides a comprehensive evaluation framework and facilitates future benchmarking efforts for calibration under shifts in image classification.
Acknowledgments
M.R. is funded by an Imperial College London President's PhD Scholarship and a Google PhD Fellowship. R.M. is funded by the European Union's 's Horizon Europe research and innovation programme under grant agreement 1010S0302. B.G. and F.d.S.R. acknowledge the support of the UKRI AI programme, and the EPSRC, for CHAI EPSRC Causality in Healtheare AI Hub (grant no. EP/ (P//028885/1)) 1). B.G. received support from the Royal Academy of Fngineering as part of his Kheiron/RAFngg Research Chair.
References
Glenn W. Brier. VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY.
Monthly Weather Review, 78(1):1-3, January 1950. ISSN 1520-0493, 0027-0644. doi: 10.1175/
200.000 URL https://journal) ametsoc www.ato..org/viee///ouua..e org/viee//ourraal/// mwre/78/1/1520- 0493 1950
____1_90_07_8 _0___78__00 Bare__001_aaa 1o00__ofeit_200 co_ xml. Publisher: American Meteorological So-
ciety Section: Monthly Weather Review. (14)
Jacob Carse, Andres Alvarez Olmo, and Stephen McKenna. Calibration of Deep Medical Image Clas- sifiers: Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, pp. 89-99, 2022. ISSN 0783031167483 doi: 10.1007/978- 8-031-16749- 9. URL http::://mmuureiiccaa.ggnmn i0/. Place: Switzerland Publisher: Springer Nature. (14)
14

---

Published in Transactions on Machine Learning Research (10/2025)
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing Tex- tures in the Wild. In 2014 TEEE Conference on Computer Vision and Pallern Recognition, pp- 1606-3613, June 2014. doi: 00.1199/CPPR22144
URL https://ieeexplorss www.:ieeee....eeeeee.o wttp.//////ocme////////6666
ISSN: 1063-6919. (16), (17)
Etienne Decencière, Xiwei Zhang, Guy Cazuguel, Bruno Lay, Béatrice Cochener, Caroline Trone, Philippe Gain, Richard Ordonez, Pascale Massin, Ali Erginay, Béatrice Charton, and Jean-Claude Klein. Feedback publicly distributed image database: the mossidor database. Image Analysis and Stercology, 33(3): 231 234, August 2014. ISSN 1854-5165. doi: 155566/aas.1155 URL httpp://wwws ier--ias--ss aaa.iss.rrg/ooa........ TAS/article/view// 1155. Number: 3. (16), (14)
Jia Deng, Wei Dong, Richard Sacher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, Miami, FL, June 2009. IEEE. ISBN 978-1-4244-3092-8. doi: B000//VVR.2009.52000 URL https: //ieeexplore ieee. org/ httg///ccmmmmt/5268488///// (12)
Gianluca Detommaso, Alberto Gasparin, Andrew Wilson, and Cedric Archambeau. Uncertainty Calibration in Bayesian Neural Networks via Distance-Aware Priors, July 2022. URL http://arxiv httpxxiivor///bbba. ab07/207. 08200. (XXi::2207.080000 [cs, stat]. (14)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Usækoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In Interna- lional Conference on Learning Represenlations, October 2020. URL https://opeireviee..nttt/ id=YicbFaNTTy. (16)
Emma Dugas, Jorge Jared, and Will Cukierski. Diabetic Retinopathy Detection Kaggle Challenge, 2015. (16), (14)
URL attps://kagge com/coo////ccccc coopetitions wet.aabeiii-rr..nppptpt--eeeeeed
Milton Friedman. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance, Journal of the American Statistical Association, 32(200):67557000 7015 December 1937. ISSN 0162- 1459. doi: 10. 10.1080 10000000621555 11012211119977972111111111919911 1050352... URL https: http:://wwwtt.dffn////e tandfonn ine. com/doi/abs//0. Publisher: ASA Website. (110)
01621459. 1937 10503522.
Miguel Fuentes, Benjamin M. Van Doren, Daniel Fink, and Daniel Sheldon. BirdFlow: Learning scasonal bird movements from eBird data. Methods in Ecology and Evolution, 14(3):923 938, 2023. ISSN 2041- 210X. doi: 111111/241-2100X44000 URL 2041-210X 14052.
wiley. hatm///i/aa/////eeaaaa 10.111111/11//////// /01t:///o//////.101111//01111110 20001222211110005533 (14)
eprint:
Yingbo Gao, Weiyue Wang, Christian Herold, Zijian Yang, and Hermann Ney. Towards a Better Un- derstanding of Label Smoothing in Neural Machine Translation. In Kam-Fai Wong, Kevin Knight, and Hua Wu (eds.), Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Pro- cessing, pp. 212 223, Suzhou, China, December 2020. Association for Computational Linguistics. doi: 10.18653/v1/ /ttp:/v//2222aaaa main.25. URL https: httpp://aclantholoog/ org/2020 aacl maallmmain.22. 25/. (14)
Sebastian Gregor Gruber and Florian Buettner. Better Uncertainty Calibration via Proper Scores for Clas- sification and Beyond. In Advances in Neural Information Processing Systems, October 2022. URL https:://penreevie net net/forrmmid=PPilk...
(14)
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 1321-1330 PMLR, July 2017. URL scttps:/procceedingg mml.ediinnnmmm......mmmmm.rmre mlr. wttp:///iiw.......ee//////////// gww1a. html. ISSN: 2640-3498. (t1), (t2), (13), (14), (16)
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of Neural Networks using Splines. In International Conference on Learning (t2), (14)
Representations, 2021. URL wtt.:p/wwriii.....t/ott///dddaaa
15

---

Published in Transactions on Machine Learning Research (10/2025)
Fredrik K. Gustafsson, Martin Danelljan, and Thomas B. Schön. How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts? Transactions on Machine Learning Research, Septem- ber 2023. doi: 10.48550/ wxXiv23302.03677 URL
(114)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770 778. IEEE, June 2016. ISBN 978-
doi:
URL http://ieeexplors leee. org/
document/7780459/ (16)
Yufan He, Aaron Carass, Lianrui Zuo, Blake E. Dewey, and Jerry L. Prince. Autoencoder based self- supervised test-time adaptation for medical image analysis. Medical Image Analysis, 72:102136, August 2021. ISSN 1361-8415. doi: m.10116j.mmeiia. aeeedia.222..1... 2021.1022288 URL attps://wwe http://wwww.ceenneeert..co com/
(t6)
Dan Hendrycks and Thomas Dictterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In International Conference on Learning Representations, September 2018 URL (t2), (15)
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, April 2017. URL http://arxiv.ot a0g/4 04861. arXiv:1704.04861 [es]. (16)
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Sttrea-lleus, Chris Chute, Henrik Mark- lund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seckins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the Thirly- Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence,
Thirl
pp- 590-597, Honolulu, doi: 10.1609 /a000/9aaii.a v33101.3301599
Hawaii, USA, Jamuary 2019. AAAI Press. ISBN 978-1-57735-809-1. URL https: //oii org/10. 10/./1..10 1609 /aa9//aai.... v33101. 33015990. (f6), (14)
Jiwoong J. Jeong, Brianna L. Vey, Ananth Bhimireddy, Thomas Kim, Thiago Santos, Ramon Correa, Raman Dutt, Marina Mosunjac, Gabriela Oprea-Ilies, Geoffrey Smith, Minjae Woo, Christopher R. McAdams, Mary S. Newell, Imon Banerjee, Judy Gichoya, and Hari Trivedi. The EMory BrEast imaging Dataset (EMBED): A Racially Diverse, Granular Dataset of 3.4 Million Screening and Diagnostic Mammographic Images. Radiology:
Artificial Intelligence, 5(2)2220 January 2023. doi: renaa rsna. org/ htg//ooi111. 1148/ryai. 220047. Publisher: Radiological Society of North America. (15), (11)
URL
Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. MIMIC-CXR, a do-identified publicly available database of chest radiographs with free-texl reports. Scientific Dala, 6(1):317, December 2019. ISSN 2052- 4463. doi: 00.10388/411597-01--333
URL https://www https://www.nature..om nature. com/ htta///acceee/11117--------22220 Publisher: Nature Publishing Group. (16), (f4)
Pau de Jorge, Riccardo Volpi, Philip Torr, and Gregory Rogez. Reliability in Semantic Segmentation: Are 11298. arXiv:2308. [cs].
We on the Right Track?, March 2023. URL (+14)
Maggie Karthik and Dane Sohier. APTOS 2019 Blindness Detection Kaggle Challenge, 2019. URL https: (16), (14)
/ //kaggle.
Mijoo Kim and Junseok Kwon. Uncertainty Calibration with Energy Based Instance-Wii Scaling in the Wild Dataset. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gäl Varol (ods.), European Conference on Computer Vision, volume 15104 of Lecture Notes in Com- puter Science, pp. 232-248, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-72952-2. doi: 10.1007/9988-3-033-729222 14. URL https:/ ptttps//arxiv...o wwwpwwwwwwww.//////.....// 1233/ 12330. (t2), (13), (15), (16), (17)
16

---

Published in Transactions on Machine Learning Research (10/2025)
Pang Wei Koh, Shiori Sagawa, Henrik Markhund Sang Michael Xie, Marvin Zhang, Akshay Balsubra- mani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Trena Gao, Tony Tee, Etienne David, lan Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A Benchmark of in-the-Wii Dis- tribution Shifts. In Proceedings of the 38th International Conference on Machine Learning, pp. 5637-5667 PMLR, July 2021. URL https: https://proceedings mlr. wtwppp//ss///ooorassoas html. ISSN: 2640-3498. (f2), (15), (16), (14)
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009. (12)
Anayy Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, pp. 1041-1051. PMLR, August 2022. URL attps://proceedinng press/ v180/kumar201 hatr///uar222.... html. ISSN: 2640-3498. (13), (15)
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncer- tainty Estimation using Deep Ensembles. In Advances in Neural Information Processing Systems, vol- nwwrips......... wwwuuip..copppppeeeeeeeeee wwtpcccppppppeeeeeeeeeeee saapprrfiles/pppeeee
ume 30. Curran Associates, Inc., 2017. URL Awsrrtatbttttt....oee html. (13), (14), (15), (113)
2017
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object De- tection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):318 327, February 2020, ISSN 1939-3539. doi: /T1009/TA....011 2.1.2M22.1...888822. URL https://ieeeepporre ieee org/documents Conference Name: IEEE Transactions on Pattern Analysis and Machine
74177776/armmmbor8444777 Intelligence. (14)
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11966-11976, New Orleans, LA, USA, June 2022. TEEE. ISBN 978-1-665546-946-3, doi: 10.1109/CVPR52666 20000PR026882202200111... URL
org/ httg//bbuumen///777755///// (14), (16)
Laurens der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 986):25579 2605, 2008. ISSN 1533-7928. URL http://jmlr. atg/papers/v//
vandermaatenOBa..taa (t8)
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A Simple Baseline for Bayesian Uncertainty Deep Learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceede
neurips cc/paper/
html. (f4)
Raghav Mehta, Vitor Albiero, Li Chen, Ivan Evtimov, Tamar Glaser, Zhiheng Li, and Tal Hassner. You Only Need Good Embeddings Extractor to Fix Spurious Correlations, December 2022. URL http: /
Drrxxi..rrg/bbs/212222 06254. arXiv:2212.0625/ [es]. (13)
Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized Entropy Regularization or: There's Nothing Special about Tabel Smoothing, May 2020. URI. http: http://xxxvv.r//aeee.// httg//bb/2005.0000 arXiv:2005.00820 [cs]. (t2), (14)
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Cal- ibrating Deep Neural Networks using Focal Loss. In Advances in Neural Information Processing Systems, volume 33, pp. 15288-15299. Curran Associates, Inc., 2020. URL https:/ cttps://proceeding wwt.igs..euriisicccm cc/ Abstract htmm. (t2), (14)
Kevin P. Murphy. Probabilistic machine learning: advanced topics. Adaptive computation and machine learning series. The MIT Press, Cambridge, Massachusetts, 2023. ISBN 978-0-262-37599-3 978-0-262.
37600-6. (t1)
17

---

Published in Transactions on Machine Learning Research (10/2025)
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In Proceedings of the 33rd International Conference on Newral Information Processing Systems, pp- 4694-4703, Red Hook, NY, USA, December 2019. Curran Associates Inc. (12), (14)
Peter Nemenyi. Distribution-free Multiple Comparisons. Princcton University, 1963. (110)
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring Calibration in Deep Learning. In Proceedings of the IEEE/CVR /CV Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 38 41, 2019. (14)
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szalraniec, Vasil Khalidov, Pierre Fernandez, Damiel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research, July 2023. ISSN 2835-8856. URL https: httpp://pppaaeeeeee.eeee/rmrrrts met.://ooeeo.rrtttaaaaee (13), (16)
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Bal- aji Takshminarayanan, and Jasper Snoek. Can you trust your model' uncertainty? Evaluating pre- dictive uncertainty under dataset shift. In Advances in Newrul Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https://proceeding neurips. www..pps...pppper////// wttppaprr////9///ees//t 8558cb408c1d7766217777 1788872227771888772bbbbbb www.de2eabddbbbaaa...abaa html. (12), (15)
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment Matching for Multi-Source Domain Adaplation. In 2019 IEEE/CVP International Conference on Compuler Vision (ICCV), pp. 1406-1415, 1415, Seoul, Korea (South), October 2019. IEEE. ISBN 978-1-72814-803-3 doi: 10. 1109/1CCV.20199 2019..0199 URL https://ieeexplore wtw.ee.rgg//ocmmee//000000
(16), (14)
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing Neu- ral Networks by Penalizing Confident Output. Distributions. In International Conference on Learning Representations, February 2017. URL https://openrevier hett//oorumdd--Hybbooot (t2), (14)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference Machine Learning, pp. 8748-8763. PMLR, July 2021. URL https: https://prooceedings mlr. prees//119/raafordd2.. html. ISSN: 2640-3498. (16)
Rahul Rahaman and Alexandre Thiery. Uncertainty Quantification and Deep Ensembles. In Advances in Newral Information Processing Systems, volume 34, pp. 20063 20075. Curran As- sociates, Inc., 2021. URL https: pttps://proceedingg htw.rcceeinnnnnnuiiiiss...... httpp/pprr.llleepppp//////////// 37o6c60976o2accee12224-77888 Aeee---bbbrtt.t.ee html. (13), (f5), (113)
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. BREEDS: Benchmarks for Subpopulation Shift. In International Conference on Learning Representations, October 2020. URL
forum? Dorum?id==QPBmryAAu (15)
Florian Seligmann, Philipp Becker, Michael Volpp, and Gerhard Neumann. Beyond Deep Ensembles: Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift. In Thirty-seventh Conference on Neural Information Processing Systems, November 2023. URL https: https://openreviieee http:///pppeeeeeeee.ee//torrriia jx491Kr6vbbnoteee-00 AMnX.U53. (f2), (f3), (14), (15)
Congzhen Shi, Ryan Rezai, Jiaxi Yang, Qi Dou, and Xiaoxiao Li. A Survey on Trustworthiness in Foun- dation Models for Medical Image Analysis, October 2024. URL http://arxiv. http://aartv.orr//bbb//// org/abs/2407. 15851. arXiv:2407.15851. (13)
18

---

Published in Transactions on Machine Learning Research (10/2025)
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In 2016 IEFE Conference on Compuler Vision and Pat- tern Recognition (CVPR), pp. 2818 2826, June 2016. doi: (.1199//VR.2000 2016.308. URL https:
//ieeexplore leee. o
ISSN: 1063-6919.
(12), (13), (14)
Mingxing Tan and Quoc Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, pp. 6105 6114. PMLR, May 2019.
URL https: / httpas///proceeeeeee pttp//eecceee.s../.o/// mlr press/v97/tane html. ISSN: 2640-3498. (16)
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Lud- wig Schmidt. Measuring Robustness to Natural Distribution Shifts in Image Classifica- tion. In Advances in Neural Information Processing Systems, volume 33, pp- 18583- 18599. Curran Associates, Inc., 2020. URL https: http::///pooccdnngg..... wwwpeeidiinnnnnuuis..... http://eeei.aaec/a///////////// 88300885711155321114444777777771 htt.:brtact---btaattt... html. (12)
Christian Tomani, Sebastian Gruber, Muhammed Ebrar Erdem, Daniel Cremers, and Flo- rian Buettner. Post-Hoc Uncertainty Calibration for Domain Drift Scenarios. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 10124- 10132, 2021. URL https://openaccess: thecvf. com/ content/(VPRR222//mmn //t////mmaammiiiii Post-Hoc_ Encertainty_aliibattioo Caw.ibaaaiiainnnomm for Domain Ceamain_Drift.._ Serrtt_Seeaario_CVVV CCVPR_20211_____00_220222222
_aapp__ppp_________ html. (12), (15), (t6)
Christian Tomani, Futa Kai Waseda, Yuesong Shen, and Daniel Cremers. Beyond In-Domain Scenarios: Ro- bust Density-Aware Calibration. In Proceedings of the 40th International Conference on Machine Learn- ing, pp. 34344 34368. PMLR, July 2023. URL https://proceediii wwwpeediiinnmm....p.eeme www. S.t..pree///202/to0 html. ISSN: 2640-3498. (12), (15)
Cheng Wang, Jorge Balazs, György Szarvas, Patrick Ernst, Lahari Poddar, and Pavel Danchenko. Calibrating Imbalanced Classifiers with Focal Loss: An Empirical Study. In Yunyao Li and Angeliki Lazaridou (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 145-153, Abu Dhabi, UAF, December 2022. Association for Computational Linguistics. doi: 10.18653/ 11/2022.emmlpp 1t0.22.emmllinndusry...... 14. URL https: http://aaaaaatllooorrrrr/e/////e
eenee-ennmnieeee industry. (14)
Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking Calibration of Deep Neural Networks: Do Not Be Afraid of Overconfidence. In Advances in Neural Information Processing Systems, volume 34, pp. 11809-11820. Curran Associates, Inc., 2021. URL 4haraa777..5542....4444--aaar.t
neurips. cc/paper/2021/
(13)
Jerry Wei, Lorenzo Torresani, Jason Wei, and Saeed Hassanpour. Calibrating Histopathology Image Clas- sifiers Using Label Smoothing. In Martin Michalowski, Syed Sibte Raza Abidi, and Samina Abidi (eds.), Arlificial Intelligence in Medicine, pp. 273 282, Cham, 2022. Springer International Publishing. ISBN 978-3 978-3-031-09342-5,
doi: 10.1007/977333-03--888---- 26. (14)
Xixin Wu and Mark Gales. Should Ensemble Members Be Calibrated?, January 2021. URL http: org/abs/2101 05397. arXiv:2101.05397 [cs]. (15)
Yaodong Yu, Yi Ma, Stephen Bates, and Michael Jordan. Robust Calibration with Multi-domain Temper- ature Scaling. Tn Advances Newral Information Processing Systems, 2022. (15)
Mert Yuksekgonul, Linjun Zhang, James Zou, and Carlos Guestrin. Beyond Confidence: Reliable Models Should Also Consider Atypicality. In Advances in Newral Information Processing Systems, November 2023. (t2), (15)
URL https::/openreeeeeee http:////o//////////ottttotttaad
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '02, pp. 694-699, New York, NY, USA, July 2002. Association for Computing Machinery. ISBN 978- 1-58113-566-1. doi: 10.1145/775047.7755511 775047.775165 (t2), (14), (16)
URL https:// /ttp:::///oo../10//// 10.////.114///
19

---

Published in Transactions on Machine Learning Research (10/2025)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre-Training, In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp- 11941- 11952, Paris, France, October 2023. LEEE. ISBN 9798350307184. doi: 10.1009/1CCV550000002...00000 URL pettps://ieexpllee leee. org/ /ttp////mmmme////55///. (16)
Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-Match Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning. In Proceedings of the 37th International Conference Machine Learning, Pp. 11117-11128, PMLR, November 2020. URL attps://proceeddings www.reeeeeeinnmmm.......m.. .ml..pesss/ v119/2haa 2019/2hang20k.. html. ISSN: 2640-3498. (12), (13), (14), (16)
Jun Zhang, Wen Yao, Xiaoqian Chen, and Ling Feng. Understanding Negative Calibration from Entropy Perspective. In Michelangelo Ceci, Sergio Flesca, Elio Masciari, Giuseppe Manco, and Zbigniew W. Raš (eds.), Foundations of Intelligent Systems, volume 13515, pp. 305-314. Springer International Publish- ing, Cham, 2022. ISBN 978-3-031-1665634 978-9--33-11566411 doi: 10.1007 10077998-3-001-1600
29. URL Series Title: Lecture Notes Com-
https://link.. springer... wt.o//1...e 10011100777-----01-6666-- 29. puter Science. (t3), (15)
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jasprect Bagga, Robert Timn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Angela Crabtree, Brian Piening, Carlo Bifulco, Matthew Lungren, Tristan Naumann, Sheng Wang, and Hoifung Poon. Multimodal Biomedical Foundation Model Trained from Texat
Fifteen Million Image Text Pairs. NEJM AI, 200):41o,2400440, January 2025. doi: 10.1056/ Aloa2400640. URL https:/ //ai ww..memmmrr....... htrg/doi/full/10 1056/ AIa2240640. ety. (13), (16)
Publisher: Massachusetts Medical Soci-
Yukun Zhou, Mark A. Chia, Siegfried K. Wagner, Murat S. Ayhan, Dominic J. Williamson, Robbert R. Struyven, Timing Liu, Moucheng Xu, Mateo G. Lozano, Peter Woodward-Court, Yuka Kihara, Andre Altmann, Aaron Y. Lee, Eric J. Topol, Alastair K. Demniston, Daniel C. Alexander, and Pearse A. Keane. A foundation model for generalizable disease detection from retinal images. Nature, 622((981):1566 1111-6666,,,666666 URL https: / /www. nature. articles/s41588 s01-eees/41566-000 -2--1556--23000000 0486-/111160-0000055-------- x. Publisher: Nature Publishing Group. (13)
October 2023. ISSN 1476-4687. doi: 10.1038/ 10.1038/s41586- (s11586-023-0655---
20

---

Published in Transactions on Machine Learning Research (10/2025)
A.1 Dataset definitions
In this work study realistic shifts in the medical imaging and natural images domain. For this we leverage multiple public datasets across various modalities and tasks. We here detail how we defined ID and OOD domains for non-standard benchmark datasets. We summarise splits and ID/shifted domains definitions in Table A.1. In Figs. A.1 and A.2 we illustrate the types of shift by showing examples of images from both the ID and the shifted sets for each dataset.
Dataset
Type (sift(s)
classes
ID domain
Shifted domains
Balanced (Y/N)
(Nirain; Nyat Neest)
(N images)
Scanner
CheXpert
MIMIC-CN random subset (25000)
CXR
Prevalence
2 classes (N)
random train/val/ val/test split (129732, 23086, 38192)
Population
Scanner
EyePACS (35126, 10715, 42861)
Messidor (1744)
RETINA
Prevalence
5 classes (N)
Aptos (3662)
Population
Senographe Essential (12218)
Senograph 2000D (13268)
Lorad Selenia (10515)
Selenia Dimension 2D (169758, 43055, 52975)
EMBED
Scanner
4 classes (N)
Senographe Pristina (498)
Clearview CSm (8128)
Selenia Dimensions
C-views (108369)
Official ID splits Hospitals 1,2,3 (272192, 30244, 33560)
Official OOD splits Hospital (34904) Hospital 5 (85054)
CameLyon
Staining
2 classes (Y)
Geographic Prevalence
Official ID splits (129809, 7314, 8154)
*OOD val' split (14961) 'OOD test' split (42791)
iCam
182 classes (N)
Source domain from official 'rand' split (37570, 6630, 1700)
Target test set from official 'rand' split (1700)
Living17
Population
17 classes (Y)
Source domain from official 'rand' split (131123, 23140, 6000)
Target test set from official 'rand' split: (6000)
Entity30
Population
30 classes (Y)
Clipart test (14604)
Official ID splits from Wilds with ID domain 'Real' (217630, 24182, 104082)
Infograph test (15582)
Image type Prevalence
DomainNet.
345 classes (N)
Painting test (21850)
Quickdraw test (51750)
Sketch test (20916)
Table A.l: Datasets and shifts used in this study, details ID and shifted splits definition.
we analyse robustness against scanner changes for brcast density assossment models in mam- mography. For this purpose, we use the public available EMBED dataset (Jeong et al., 2023). This dataset contains data from 6 different scanners (Selenia Dimensions, Senographe Essential, Senograph 2000D, Lorad Selenia, Clearview CSm, Senographe Pristina). In this work, we used Sclenia Dimensions (2D images) as the ID scanner for training and validation, and all the other scanners, as well the C-view images from Selenia Dimensions, used as shifted test sets, resulting in six shifted test sets.
EMBED

---

Published in Transactions on Machine Learning Research (10/2025)
SHIFTED ID
(a) CXR
SHIFTED ID
(b) RETINA
ID
SHIFTED
EMBED
SHIFTED ID
(d) CameLyon
Figure A.l: Illustration of each dataset and associated shifts (1/2).
2

---

Published in Transactions on Machine Learning Research (10/2025)
ID
SHIFTED
(a) iCam
SHIFTED ID
(b) Living17
ID
SHIFTED
(c) DomainNet
Figure A.2: Iustraaion of each dataset and associated shifts (2/2).
3

---

Published in Transactions on Machine Learning Research (10/2025)
RETINA analyse robustness against equipment and geographic location changes in diabetic retinopa- thy grading models by combining three publicly available datasets: Kaggle Diabetic Retinopathy Challenge (EyePACSS dataset (Dugas et al., 2015), Kaggle Aptos dataset (Karthik & Sohier, 2019) and the Messidor- v2 (Decencière et al., 2014) dataset. These datasets cover various locations (US, India and France) and different level of equipment from high-quality scanners mobile phone cameras. We use the largest, dataset. (EyePACS) as ID domain, and images from the two other datasots as OOD test sets.
we analyse robustness against scanner, location and label distribution changes in chest-xray clas- sification models (No Finding versus Discase task) using the CheXpert (Trvin et al., 2019) and MIMIC- CXR (Johnson et al., 2019) datasets. We use the CheXpert dataset as our ID domain, splitting randomly between train/val/ter test at the patient level, and we use random subset of 25,000 images from the MIMIC- CXR database as our OOD test set.
CXR
DomainNet we use the implementation and splits definition provided by the Wilds package (Koh et al., 2021) for our DomainNet (Peng et al., 2019) experiments. Specifically, we use images from the 'REAL' domain our ID images (trai/test//ees) and images from all other 5 domains (clipart, infograph, painting, quickdraw, sketch).
A.2 Details on calibration metrics computation
Expected Calibration Error, ECE (Guo al., 2017) approximates the expected difference between pre- dicted confidence and prediction accuracy. We follow the notations from the main paper: with classification task with C' classes, X the model input, and y E CI, the associated label. Let P E [0, 1J denote the model predicted probabilities for each class, ú = arg max, Ps the predicted class, and è = max P. the associated confidence score. To compute ECE, the output space [0, 1] partitioned into M bins, for each bin Bm, we compute the average confidence score in the bin Conf(B,.) - I EeeB... and the actual accuracy in this bin Acc(Bm) - E.en. 1ly -*]. The ECE then reflects the average error over all bins, weighted by the number of samples each bin:
M Bml [cc((Bm) Conf(BB.)
ECE
(1)
N
m=1
with N the total number of samples.
The Brier Score (Brier, 1950), the other hand computes the average class-wise mean squared error between
probabilities and one-hot labels. Formally, the score is computed as: N/ DE (#k - vi)
A.3 Comparison with existing studies
For the convenience of the reader, below we provide a comparative table summarising differences between previous studics on calibration under shifts in image classification and our study.

---

Published in Transactions on Machine Learning Research (10/2025)
Efffect of foundation models
Table A.2: Comparison of our and other existing studies on calibration under datasct shifts in image classification, see Section 2 for more details.
Elffett of ensembling approaches
V for BNNs
Bayesian
LS. ER+LSS, focal
ER,
In- li-training calibration methods
LS, Focal
TS, IR, IRM, TS-IR with(out) perturbation on val set
TS, ETS, IRM, SPL with(out) DAC
ETS, SPLINE,
EBS, IRM, IROVa, IROVaTS, DAC
TS (+-0DD, IROVaTS
IRM
Post-hoc methods
(+OOD), EBS
TS, MD+TS
(+OOD),
TS
TS
Syuthelic TS
+
+
Type of shifts
Synthotic Realdworld
Real Real-worlk world
WILDS datasets (iCam, CameLyon), Living17, Entity30, EMBED, CheXpert, Real-world MIMIC-CRR RETINA, DomainNet.
Synthetic
Synthetic
CIFARIO-C, CIEAR100-C, ImageNet-C Synthetic
Synthetic
Synthetic Real-world
Rotated MNIST, Translated MNIST, CIEARIO-C, ImageNet-C
Kim Kwon & CIFARIO-C, CIFAR10U-G ImageNet-C
Cropland, Landcover, CelebA, Living17, Entity Domain Net, ImageNet, iWild- Cam, Waterbirds
CIFAR100-C,
WiLDS datasets (iCAM, FMoW, RxRx1), CIFAR10-C,
CIEARIO-C, C, ImageVet-CC ObjectNet
ImageNet-C. WILDS-BxXRXL RxRx1, GLDv2
CIFARIO-C, TiyyImggNNeted
Datasots
Tomani et al. (2021)
al. (2023)
Yu et al. (2022)
al. (2022)
Seligmann et al.
Paper
Ovadia et (2019)
Tomani
(2024)
Kumar et
Zhang et (2022)
Ours (2023)
et

---

Published in Transactions on Machine Learning Research (10/2025)
Dataset
LR schedule for CNNs
LR schedule for ViT
LR schedule for ConyNext
Max epochs
Living17
Cosine 10-3 to 10-6
Fixed 10-5
Fixed 10
50
Entity30
Cosine 10-3 to 10-5
Fixed 10-5
Fixed 10-1
100
CXR
Cosine 10-1 to 10-5
Fixed 10-5
Fixed 10-1
50
Density
Cosine 10-4 to 10-5
Fixed 10
Fixed 10
50
iCam
Cosine 10 10-4 to 10-5
Fixed 10
Fixed 10
100
RETINA
Cosine 10-3 33 10-5
Fixed 10-3
Fixed 10-1
150
DomainNet
Cosine 10-3 3
10
Fixed 10-5
Fixed 10
150
CameLyon
Cosine 10-3 10-5
Fixed 10-5
Fixed 10-4
150
Table A.3: Learning rate schedules used to train models initialised with random weights, per dataset.
Learning rate schedules were chosen based on best validation performance.
A.4 Hyperparameters and training setup details
All our classification models were trained using a standard cross-entropy loss, using the Adam optimiser, with a batch size of 32. For models trained from scratch, we used a cosine learning rate schedule or a fixed learning rate schedule, depending on the architecture and the dataset, chosen based on validation performance, we detail all configurations in Table A.3. For finetuning foundation models, we used fixed learning rate of 10-5 chosen based validation performance. We performed full-finetuinng of foundation models, i.c. all weights were finetuned. For all experiments, we chose the checkpoint with the highest validation balanced accuracy for inference.
A.5 Post-hoc calibrators with semantic OOD exposure: implementation details
A.5.1 EBS: Energy-based calibration, implementation details
In this section, we detail the algorithm proposed by Kim & Kwon (2024) for their energy-based calibration method.
Algorithm 1: Energy-based calibration (Kim & Kwon, 2024)
Input: In-distribution dataset: 'Did ((x; y:))
Input: Classifier fxx
Input: Semantic OOD data: Dout = {(x,, (000))))nnnn
) Nood
Input:
Temperature value from temperature scaling: Tts /* Get energy for all samples
D Diau
z + f(x)
F(x) +- -logsumexp(()
Estimate Gaussian PDFs on correctly classified samples
Pi < fit(F(Zcorree))
/* Estimate Gaussian PDFs incorrect 00D samples
P2 + fit(F (((FF(incorreet))
/* Fit calibrator
T(z,) < Tta - P + Pa(F(z)))
Ô + arg ming EseD MSE(softmax(z, ((zz)))))) 9:)
Output: 0, Te,, P1. P2

---

Published in Transactions on Machine Learning Research (10/2025)
A.5.2 Enhancing other calibrators with OOD exposure
EBS (Kim & Kwon, 2024) requires exposure to small 'semantic OOD' set for calibration, unrelated to the task at hand. Kim & Kwon (2024) propose to use samples from the Textures (Cimpoi et al., 2014) dataset as OOD samples. To disentangle the contribution of the energy-based calibration approach and the use of semantic OOD towards improved calibration, in this study we evaluate augmented versions 'standard' post-hoc calibrators, where the same semantic OOD set. is used along the ID calibration set the calibration stage, yielding additional calibrators TS + OOD, IRM + OOD, IROVaTS + OOD. Concretely, we collect an unlabeled sample of OOD data and combine it with the ID calibration set to fit the calibrator of choice. For the calibration set, labels need to be assigned to the OOD samples, in this case we simply [I/C, 1/C] where C' is the number of classes. We
assign the uniform distribution i.c. for all samples illustrate the full process for the example of temperature scaling with OOD, in Algorithm 2. For any other calibrator (IRM, IROVTTS) the process is identical, only the fit step needs to be adapted.
Algorithm 2: Temperature scaling with semantic OOD
Imput: In-distribution dataset: Dia {(x,, v:))
Input: Classifier f(x)
Input: Semantic OOD data: Dout {x, Ver
/* Assign uniform distribution as labels for 00D samples
for i = 1 Nood do
di + _/4 -c
end
Dout +- {(x,,, ((R;;0))) Dar
Combine both datasets
D + Dout
/* Fit calibrator on the combined dataset, here TS
/. < arg minn (xw)eD CrosFantopy/ssotmmx/////)
Output: f

---

Published in Transactions on Machine Learning Research (10/2025)
A.5.3 Ablation study: embedding space analysis between ID, shifted and semantic OOD sets
LIVING17
ROTEMA
CAMELYON
TEST DD SENANTIC
TEST STTED
SORNNIIC COO
-22
-07
-50
ICAM
EMBED
CXR
197
TEST I
TEST -D SEMAATICOO
SEMANTIC 500
100
SOHENTIC ORO
33
150
118
-4
320
-15
DOMAINKET
TixT PIST-DIITTD
SEWATIC 000
75
3-
-25
-00
232
Figure A.3: Embedding space analysis across datasets with TSNE (Maaten & Hinton, 2008). For every dataset, we analyse the distribution of the ResNet-18 classifier embeddings from ID test samples, shifted test samples and semantic OOD Textures samples. The TSNE representation computed based on the entire set (ID + shifted + semantic OOD samples). Note that if either test contains more than 10,000 samples, randomly sampled subset of 10,000 samples.

---

Published in Transactions on Machine Learning Research (10/2025)
A.6 Balanced Accuracy results
LIVING17
RETINA
CAMELYON
11.5
11.0
SWIFTE
SHIFTED
32.0
91.8
BIIFTED
10.6
91A
+
30.0
10.0
28.0
80.
8.0
80.
18.0
78.5
26.0
26.0
30.0
32.0
34.0
09.4
19.5
20.0
25.0
1D
EMBED
CXR
ENTITY30
DOMAINNET
04.0
17.0
05.0
62.0
18.5
SHFFTED
67.0
84 EETEE
80.0
35.2
16.0
36.0
15.5
16.0
378
16.0
65.0
54.0
14.5
68.0
70.0
72.0
56.0
60.0
66.0
67.5
68.0
08.5
66.0
06.0
58.0
60.0
1D
Calibrator PROBAS / RWW 000
IROWATS
IROWAT3 000
Loss
CE
ER
Focal
ER+LS
LS
Figure A.4: Balanced accuracy results (in %) ID & shifted test sets, models trained from scratch. Most post-hoc calibrators balanced accuracy across datasets. We do not observe any consistent performance differences across training losses.
preeeacii,,,,,,ing, however IROVaTS is not and tends to decrease
LIVING17
RETINA
CAMELYON
94.5
81.5
58.0
34.0
52.5
94.0
12.0
12.0
80.0
$1.5
83.0
30.0
78.6
78.0
51.0
82.0
28.0
78,5
50.5
10.0
4.
26.0
96.5
96.6
96.7
05.5
48.0
47.0
48.0
69.0
10.0
99.6
88.6
99.6
40.0
45.0
500
CXR
ENTITY30
DOMAINNET
72.0
70.0
38.0
71.5
67.
SHFFTE
71.0
80.0
88
36.0
62.5
64.5
35.0
80.0
64.0
575
63.5
0,00
35.0
89.0
69.0
72.0
73.0
74.0
75.0
$5.0
60.0
65.0
70.0
35.0
95.2
$5.4
55.6
820
$5.0
84.0
ID
10
Calibrator PROBAS /TS: /TS RM/EBB/TT (T8 000
IROVATS
IRCVATS
COD
Loss
ER
Focal
ERVLS
LB
Figure A.5: Balanced accuracy results (in %) on ID & shifted test sets, models finetuned from foundation models. Most post-hoc calibrators are securacy-presorving however IROVaTS is not and tends to decrease balanced accuracy across datasets. We do not observe any consistent performance differences across training losses.
9

---

Published in Transactions on Machine Learning Research (10/2025)
A.7 Statistical significance analysis
To determine the significance of the overall training loss + post-hoc calibrator 'treatment" effect across datasets and models, we use the Friedman (Friedman, 1937) and Nemenyi (Nemenyi, 1963) test. Technically, we run this test, separately for ID and OOD test sets. In our input data matrix, each row corresponds to one architecture-datased combination for which we collect the average calibration error (across available test sets) for every treatment (loss + post-hoe calibrators). We find an overall significant effect with the Friedman test in both scenarios (ID and shifted). Hence, follow by Nemenyi post-hoc test to determine which treatment pairs have significantly different treatment effect the calibration results, results presented in Fig. A.6.
We find that on shifted test sets, calibrators with semantic OOD exposure have a significantly different effect over those without OOD exposure, however there are no significant differences across calibrators once controlling for semantic OOD exposure (take-away 1). We also find that base probabilities from ER+LS models are not are not significantly different from any post-hoc calibration with OOD exposure (take-away significant differences
2). We find that, for nearly all post-hoc calibrators, after calibration there between different training losses. For ID sets, find significant effect of semantic OOD exposure well, confirming the ID-OOD trade-off detailed in take-away 3.
PROSAS OE MADAAS BR
PROBAS CE HACUAS noal FROBAS ER HOCS
15 Feol
IS Fooa TRLER
T9.ER
PUICE
PAL SA
PO S
RONATS FRES
ROWITS ESLS
ROUNTS
ROUATS
TH OOD SALS
WITH 0O0 FRLS
TE WITH GOO
TE WITH COD
IN CSD
H cOD SLS
1
DO WITH cOn FAUR UN COOD LE THA OF
FOM THE cOn FALM AR TN OOD IS
THI
4
Foo!
Food
WET CO0 TRE
na an (00 Food
ro with aor WitH gol co he
to en com hem ta Fred
with pookinn
with 100
(a) Significant differences on shifted test sets
(b) Significant differences on ID test sets
Figure A.6: Full post-hoc comparison results the Nemenyi test for ECE values on shifted and in-distribution test sets across all datasets and models (trained from scratch). The Nemenyi-post post-hocs identifies which specific treatments are significantly different from each other. Blue indicates significant differences between the pairs of treatments in the and y axes.
10

---

Published in Transactions on Machine Learning Research (10/2025)
A.8 Detailed analysis of DAC effect
LIVING17
RETINA
ECE
Brier
ECE
Biler
0.75
0.24
0.03
0.22
0.02
0:19
220
031
5110
6.82
0.11
0.00
0.16
0.53
D04
0.05
033
0.33
0.20
0.33
0.06
0.10
0.15
0.30
0.40
0.43
0.44
0.66
ID
10
ECE
Brief
EGE
Brier
0.10
0.17
078
0.15
0.00
0.17
028
SWEFTE
0.06
0.04
SWITTE
0.54
0.7
076
0.10
0.66
6.15
075
0.44
074
0.00
0.04
0.00
0.01
0.02
0.02
0.08
0.28
0.70
0.12
0.57
0.58
0.50
1D
10
10
EDB
Brier
EGF
Brer
0.45
3.57
0.42
0.18
0.42
0.77
0.42
0.15
0.42
0:14
0.40
0.66
3.41
0.12
0.39
0.38
0.5
0.11
0.30
0.5
8.42
0.08
0.3
0.4
0.02
0.04
0.20
0.10
0.33
0.34
0.25
0.04
0.05
0.1
0.14
0.15
0.15
ID
D
D
ENTITY30
DOMAINNET
ECE
Brier
EGE
Brer
0.81
0.18
2.81
0.40
0.80
0.35
2.16
SIITTO
0.02
1:10
2.14
0.79
0
0.25
1.55
412
6.78
0.20
1.00
210
0.78
0.00
0.33
0.66
6:42
0.00
0.15
0.54
0.56
0.50
10
Figure A.7: Detailed ablation results of DAC effect in terms of average ECE and Bricr score on ID and shifted test sets.
11

---

Published in Transactions on Machine Learning Research (10/2025)
A.9
Ablation study: effect of model size on shifted calibration
CAMELYON
.900
.00
.10
400
ECE -N TED
250
.25
.300
200
.100
.50
-100
100
G25
150
EMBED
CXR
ENTITY30
NOMAINNET
000
7:30
(090
.20
400
.500
.0/00
110
.300
400
0000
.200
0000
.040
070
.100
.100
.00
628
602
10
Calibrator
PROBAS
TS + OOD
Model size
5.0
12.0
28.1
86.0
Foundation
False
True
Figure A.8: Ablation study: calibration results in function of model size. We plot the WD-sifted ECE results by model size (for models trained from scratch) and compare with VIT-B-DINOv2 (i.e. finetuned from foundation model). We can see that calibration tends to degrade as model size increases. However, findings like benefit of semantic OOD exposure on shifted calibration and ID/ /Shiffted calibration trade-off hold irrespective of model size. Comparing ViT-B trained from scratch and from DINO, we see that calibration improvements are consequence of pretraining, not of model size.
12

---

Published in Transactions on Machine Learning Research (10/2025)
A.10
Additional figures for ensembles
1.050
.170
.900
1.000
.875
900
BRIER SHIFTED
.900
880
.150
.000
825
.40
.800
.130
800
.750
575
.750
.700
.500
.10
7700
.80
250
300
.500
400
400
.500
.010
0015
.200
.500
.50
800
850
ID
1D
1,000
430
.365
.950
425
.300
.900
1.200
.355
.800
415
1.100
.800
410
.350
1.000
.900
.320
:340
.140
.145
.150
350
400
.400
.500
.600
(a) Classifiers trained from scratch
.550
.130
.500
BRIER SHIFTED
.320
.200
.500
.300
480
110
100
450
.600
.260
.90
240
440
400
.040
.050
.200
.300
.300
.005
.007
.010
.013
330
350
.885
1.000
.380
360
.600
.950
.900
.375
.300
.500
.500
.850
340
480
.800
.65
.750
.300
J60
.400
.700
.310
.320
.60
.225
ID
ID
ID
D
Calibrator
EBS
Callbration applied
Before ensembligg
After ensemblig
(b) Classifiers finetuned from foundation models
Figure A.9: Brier score of model ensembles, in function of post-hoc calibration method (3 member- ensembles). Large dots denote calibration results for model ensembles, small dots the average calibration of individual ensemble members for reference. We compare the effect of (i) applying post-hoc calibration ensemble members (before ensembling), (ii) applying post-hoe calibration after ensembling predictions.
13

---

Published in Transactions on Machine Learning Research (10/2025)
RETINA
ECE
04
099
SWFF DIFFTDD
0.50
0.55
0.180
082
0.2
0.10
0/62
0.76
0.70
0.073
0.50
0.05
0.35
0.15
0.25
0.30
0.35
0.40
0.00
0.06
0.13
0.15
0.20
0.25
0.375
0403
6.425
0.450
0.05
CAMELYON
0.40
0.85
0.16
0.99
0.90
0:15
S1FTT
0.29
0.14
0.20
0.06
0.15
0.12
0.04
0.00
0.02
0.04
0.08
0.06
0.010
0.015
0.025
0.05
0.10
0.15
0.20
0.25
0.50
0.95
0.50
0.05
10
10
EMBED
CXR
ECE
Brier
ECE
Orier
0.47
0.19
0.08
0.43
0.00
0.12
0.25
0.06
0.10
0.05
0.04
0.025
0.250
0.075
0.100
0.32
0.36
0.02
0:01
0.100
0.445
0.153
10
1D
ENTITY30
DOMAINNET
ECF
EGE
Brier
0.40
1.00
0.5
0.36
0.56
1.9
8.30
0.25
0.50
0.4
1.2
0.55
0.3
1.1
0.15
0.50
8.10
1.0
0.75
0.70
0.1
0.9
0.56
0.90
0.15
0.20
5,300
0,375
0.600
0:425
0.400
0.455
0.10
0.15
0.20
0.25
0.45
0.80
0.66
0.00
0.55
Calibrator
PROBAS
IROVATS GOD
ESS
Figure A.10: Ablation study of the effect of additional post-hoc calibrators of ensemble members (prior to ensembling), for ensembles whose members trained with CE (3 member--nsembles, Large dots denote calibration results for model ensembles, small dots denote the average calibration of individual ensemble members for reference.
14

---

Published in Transactions on Machine Learning Research (10/2025)
LIVING17
RETINA
ECE
ECE
0.30
0.16
0.25
0.550
0.14
0.826
D18ITCD
0.20
0:16
0.775
0.10
0:10
0.700
0.00
0.050
5,075
8,100
0.125
0.225
0.250
0.275
0.002
0.15
0.20
0.40
CAMELYON
ICAM
ECE
EDF
0.12
0.79
0.27
0.18
0.11
0.16
5-IfTE
0.72
0.06
0.04
0.70
0.12
0.05
0.07
0.11
0.52
0.00
0.00
0.01
0.02
0.08
0.04
0.05
0.000
0,008
0.210
0,012
0.014
6.04
0.06
0.08
0.60
0.52
0.54
0.56
EMBED
ECE
Brier
0.13
0.990
0.400
0.070
0.12
0.390
0.870
0.11
0.
B
0.10
0. 0.00
0.423
0.09
0.365
0.415
0.08
0.852
0.413
0.07
0.35
0.048
0.06
0.80
0.82
0.04
0.26
0.325
0.33
0335
0.340
0.345
0.11
0.02
0.03
0.04
0.05
0.00
0.140
0.14
0.150
10
DOMAINNET
0.275
0.202
0.82
0.20
0.175
0.90
0.192
0.54
0.74
0.002
0.72
0.52
0.025
0.50
0.10
0.15
0.20
0.35
0.31
0.42
0.10
0.15
0.20
0.45
0.50
Calibrator
00D
Loss
ER-LS
applied
erowmbrin
ensembliing
Figure A.11: Calibration of model ensembles, in function of member calibration method, for ensembles whose members were trained with ERLS. Large dots denote calibration results for model ensembles, small dots denote the average calibration of individual ensemble members for reference. We compare the effect of: (i) applying post-hoc calibration to ensemble members (prior to ensembling): (ii) applying post-hoc calibration after ensembling model predictions. We report calibration results for models trained with CE and with TS calibration post-ensembling as a baseline.
15

---

Published in Transactions on Machine Learning Research (10/2025)
LIVING17
LIVING17
LIVING17
(n=2)
(n=3)
(n=
.500
.500
.500
400
.400
ECE SHIFTED
.300
.300
.200
200
.100
.100
.050
.150
10
ID
1D
RETINA
RETINA
RETINA
(n=2)
(n=3)
(n=55
250
.250
250
.25
.25
225
ECE- SHIFTED
.200
.75
.175
-150
150
.100
125
.100
1100
.100
.075
.055
.075
.150
.100
.150
.000
.100
.200
.100
.200
CAMELYON
CAMELYON
CAMELYON
(n=2)
(n=3)
(n=5)
.080
.080
.070
.070
.080
ECE -HIFFTED
.060
.060
.070
.050
.080
.040
.030
.030
.000
.020
.040
.000
.200
.040
.000
.000
.040
ID
ICAM
ICAM
ICAM
(n=2)
(n=3)
(n=5)
.300
.300
.300
.250
.20
.50
ECE- SHIFTED
200
.150
150
150
.100
100
.100
.050
.050
.050
.000
.000
.000
.100
ID
ID
ID
Calibrator
PROBAS
TS OOD
EBS
Loss
CE
ER+LS
Calibration applied
Before ensembling
Aller ensembling
Figure A.12: Calibration of model ensembles, in function of ensemble size and member cali- bration method, measured by ECE (1/2). Large dots denote calibration results for model ensembles, small dots denote the average calibration of individual ensemble members for reference, We compare the effect of: (i) applying post-hoc calibration to ensemble members (prior to ensembling); (ii) applying host-hoc calibration after ensembling model predictions. We report calibration results for models trained with CE and with TS calibration post-ensembling as a baseline.
16

---

Published in Transactions on Machine Learning Research (10/2025)
CXR (n=2)
CXR
CXR [n=5)
(n=3)
-100
-100
SHIFTED
.080
.000
080
.080
.800
.070
.070
.070
.600
060
.000
ID
ID
ID EMBED (n=55
EMBED
EMBED
(n=2)
(n=3)
-100
.090
.090
.090
ECE- SHIFTED
.080
.080
.070
.000
1060
.060
.060
.050
.050
.050
.040
.050
.100
.050
.100
.050
110
ENTITY30
ENTITY30
ENTITY30
(n=2)
(n=3)
(n=5)
.400
.400
.400
ECE: SHIFFED
.300
200
.20
200
.100
.050
.100
.150
.100
200
.100
200
ID
DOMAINNET
DOMAINNET (n=3)
DOMAINNET
(n=2)
(n=5)
.600
.800
.500
.500
.500
ECE SHIFF
.400
400
400
.300
.300
.200
.200
.200
.100
.100
.100
.300
.100
.200
.100
.200
.300
ID
ID
ID
Calibrator
PROBAS
TS + OOD
EBS
Loss
CE
ER+LS
Calibration applied
Before ensembling
Aller ensembling
Figure A.13: Calibration of model ensembles, in function of ensemble size and member cali- bration method, measured by ECE (2/2). Large dots denote calibration results for model ensembles, small dots denote the average calibration of individual ensemble members for reference, We compare the effect of: (i) applying post-hoc calibration to ensemble members (prior to ensembling); (ii) applying host-hoc calibration after ensembling model predictions. We report calibration results for models trained with CE and with TS calibration post-ensembling as a baseline.
17

---

Published in Transactions on Machine Learning Research (10/2025)
LIVING17
LIVING17
LIVING17
(n=2)
(n=3)
(n=5)
1.050
1.050
1.050
1.000
1.030
BRIER- SHIFTED TEDT
.950
.950
900
800
.900
.850
850
850
800
800
800
.750
750
.700
.250
.250
.300
ID
10
ID
RETINA
RETINA
RETINA (n=55
(n=2)
(n=3)
.700
.700
.700
.680
.75
BRIER SHIFTED
860
625
.600
800
.55
.580
.550
.560
.400
.50
.400
.500
.350
.400
.450
CAMELYON YON
CAMELYON
CAMELYON
(n=2)
(n=3)
(n=5)
-170
1170
160
BRIER- SHIFTED
.160
150
1500
.150
.130
.10
-120
.100
.010
.015
.010
.015
.200
.005
.010
.015
.000
ICAM
ICAM
ICAM
(n=2)
(n=3)
(n=5)
.950
.950
.950
.900
.900
BRIER- SHIFTED
.850
.750
.750
.700
.650
.850
.600
.50
.500
.600
ID
ID
ID
Calibrator
PROBAS
TS 0OD
EBS
Loss
CE
ER+LS
Calibration applied
Before ensembling
Aller ensembling
Figure A.14: Calibration of model ensembles, in function of ensemble size and member cali- bration method, measured by Brier Score (1/2). Large dots denote calibration results for model ensembles, small dots denote the average calibration of individual ensemble members for reference, We com- pare the effect of: (i) applying post-hoc calibration to ensemble members (prior to ensembling): (ii) applying post-hoc calibration after ensembling model predictions. We report calibration results for models trained with CE and with TS calibration post-ensembling as a baseline.
18

---

Published in Transactions on Machine Learning Research (10/2025)
CXR
CXR
CXR
EMBED
(n=2)
(n=3)
(n=
(n=2)
.370
.370
.433
.365
.305
.365
.430
BRIER SHIFTED
BRIER SHIFTEL
.300
.360
.228
.355
.355
.425
.355
.350
.350
.423
.350
.420
.345
345
.418
.140
.140
.145
.150
ID
ID
ID
ID
EMBED
EMBED
ENTITY30 (n=2)
ENTITY30
(n=3)
(n=5)
(n=3)
1,000
1.000
430
430
.950
.950
425
425
BRIER SHIFTED
.900
420
415
.850
410
.800
.410
.750
.750
405
400
.700
.320
.320
.340
.450
.350
.450
ID
ID
ID
ID
ENTITY30
DOMAINNET
DOMAINNET
DOMAINNET (n*5)
(n=
(nF 2)
(n*3)
1.000
1.400
1,400
1.400
.50
1.300
1,300
BRIER- SHIFTED
900
1,200
1,200
860
1.100
1.100
1.100
800
.750
1.000
1.000
1.000
900
.900
.000
.300
400
.500
.600
.500
.00
.400
.500
.600
Callbrator
PROSAS
T5
T5 OOD
EBS
Loss
CE
ER+LS
Calibration applied
Before ensembling
Afler ensembling
Figure A.15: Calibration of model ensembles, in function of ensemble size and member cali- bration method, mcasured by Bricr Score (2/2). Large dots denote calibration results for model ensembles, small dots denote the average calibration of individual ensemble members for reference. We com- pare the effect of: (i) applying post-hoc calibration to ensemble members (prior to ensembling); (ii) applying post-hoc calibration after ensembling model predictions. We report calibration results for models trained with CE and with TS calibration post-cnsembling as a bascline.
19


Reviews:
[{"review_id": "lU1Z0JzVKH", "reviewer": "Reviewer_4DCR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides extensive evaluation studies on post-hoc calibration methods and their interactions with some in-training calibration strategies (e.g., label smoothing). It focuses on the natural shifts in real-world image classification tasks. The findings and observations are clearly conveyed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The evaluation motivation, pipelines are clearly described. \n\nMost of the evaluation details are clearly stated. \n\nThe implementation codes are attached, which is a plus.\n\nIn terms of the presentation, it is well structured, easy to follow. \n\nIn terms of the conveyed findings from the evaluation, the experiment can support their claims.", "audience_yn": "Yes", "audience_explanation": "The work focuses on calibration techniques in real-world image classification under natural distribution shifts. The findings can contribute to a broader context of robustness and trustworthiness of AI.", "requested_changes": "For me, the work only requires some minor changes:\n\n1. Consider adding a table to highlight the differences compared to the existing studies.\n\n2. Consider adding some examples to visualize some details of the natural distribution shifts in the applied datasets. Because synthetic distribution shifts (corrections) are quite easy to picture for readers.\n\n3. Consider increasing the size of the main result figures for better illustration.", "broader_impact_concerns": "Not applicable from my side.", "additional_comments": "In the experiments, the authors consider the pre-trained models. My additional question is do the authors have the insights will the drawn conclusions are robust to the type of backbones?", "extra_scores": {}, "extra_text": {}}, {"review_id": "wiL9sDF68L", "reviewer": "Reviewer_5gJ9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a comprehensive benchmark of methods for calibrating deep neural networks under dataset shift in image classification. Motivated by an overview of related works, the benchmark is designed to span diverse combinations of natural dataset shifts, randomly initialized and pretrained architectures, in-training and post-hoc calibration methods, as well as single models and ensembles. Performance is evaluated using the expected calibration error and the Brier score, supported by statistical tests. Based on the results, the authors analyze several (often underexplored) factors influencing calibration quality, highlighting the substantial benefits of fine-tuning foundation models over training from scratch and showing that post-hoc calibration largely eliminates the need for in-training calibration. The key findings are distilled into a straightforward pipeline that guides practitioners in selecting the most effective combination of calibration method, architecture, and training procedure, tailored to their computational budget and the availability of semantic out-of-distribution data. Moreover, the provided codebase delivers a good starting point for adopting such combinations toward specific learning tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The takeaways are well grounded in extensive benchmark results spanning numerous calibration pipelines and statistical tests.\n- Most calibration methods considered, such as label smoothing, temperature scaling, and entropy regularization, are well established in practice, which increases the practical value of the proposed decision pipeline.\n- The overview of the related work identifies underexplored aspects and, thus, demonstrates the need for the presented benchmark.\n- The dataset collections vary in class counts, domains, and shift types, providing a realistic and challenging testbed for evaluating calibration methods on both in-distribution and shifted data.\n- The code is publicly available and generally well documented, enabling researchers to reproduce results and practitioners to apply the calibration pipelines in their own settings.\n- The paper is clearly structured, with each takeaway explicitly tied to specific sections, and the graphical presentation of results is intuitive, making the authors’ claims easy to evaluate.", "audience_yn": "Yes", "audience_explanation": "- Calibration is a highly relevant and active research topic, especially in the presence of dataset shifts.\n- By considering both modern and established calibration methods and architectures, the paper offers conclusions that are valuable to the broader research community and practitioners.\n- The provided codebase is extensible and can support future studies, including the development of new calibration methods.\n- The shift-aware testbed may also benefit adjacent research areas, such as active learning under distribution shifts.", "requested_changes": "- While the authors provide a solid overview of prior work on calibration, including under dataset shift, a tabular comparison highlighting how this benchmark differs from earlier evaluations would better justify its necessity. For instance, the authors could contrast their number of calibration methods, datasets, and architectures against those considered in earlier studies.\n- The paper does not describe the training procedures and optimizer hyperparameters, which can only be inferred from the codebase. This is particularly relevant for fine-tuning foundation models, where choices such as full fine-tuning with block-wise learning rate decay versus fine-tuning only selected blocks can significantly affect results.\n- The exclusive focus on image classification is restrictive, since other modalities also require calibrated neural networks. A more detailed rationale for this scope limitation would strengthen the paper.\n- Although the Nemenyi test is still widely used, it has well-known drawbacks, such as the dependence of pairwise outcomes on which additional methods are included in the comparison. Benavoli et al. [1] recommend pairwise Wilcoxon signed-rank tests with proper correction for family-wise error. The authors should either explain their choice of test procedure or adopt more up-to-date approaches.\n- The paper would benefit from a dedicated section on current limitations and future research directions, such as extending the analysis to other modalities or integrating hyperparameter optimization.\n\n**Reference:**\n- [1] Benavoli, Alessio, Giorgio Corani, and Francesca Mangili. \"Should we really use post-hoc tests based on mean-ranks?.\" The Journal of Machine Learning Research 17, no. 1 (2016): 152-161.", "broader_impact_concerns": "The paper lacks a dedicated discussion of broader impacts. The authors could briefly address the importance of well-calibrated probabilities as a requirement in many real-world applications and explain how their practical guide can support such use cases.", "additional_comments": "- In Figure 6, some numbers on the $x$-axis overlap.\n- In scientific writing, “do not” is preferred over “don’t.”\n- Why is EBS presented prominently as Algorithm 1 in the appendix?\n- In Figure A.1, the top of the dataset name “RETINA” is cut off.\n- The first paragraph of Section 4.1.1 ends with two periods.\n- In Figure 4, should the $y$-axis label be “shifted” rather than “OOD”?\n- The references show inconsistencies, for example, in how NeurIPS papers are cited.\n- Do you mean ER+LS instead of ERLS in the caption of Figure A.9 and Appendix A.5?", "extra_scores": {}, "extra_text": {}}, {"review_id": "1GQDzI2dkU", "reviewer": "Reviewer_kehV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper conducts a large-scale experimental evaluation on model calibration under real-world dataset shifts in image classification. The authors benchmark 5 in-training calibration strategies (e.g., label smoothing, entropy regularization) and 10 post-hoc calibration methods (e.g., temperature scaling, EBS) across 8 diverse datasets (natural and medical imaging) with realistic shifts (e.g., scanner changes, population shifts). Based on some interesting findings, the authors provide practical guidelines for constructing models with robust calibration under such shifts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "## Strengths\n1.  The authors conduct a comprehensive evaluation with breadth and diversity. This study covers 420 models from both random and foundation model initializations across 8 datasets with several in-training and post-hoc calibration techniques.  \n\n2. This paper is easy to read and uses many tables and figures for an intuitive visualization, e.g., Figure 1 shows an overview of the calibration study.    \n\n3. The paper has a clear structure and enough workload. The experiments are extensive and include various models with different calibration strategies.", "audience_yn": "Yes", "audience_explanation": "Model calibration is a meaningful and trustworthy topic in many fields. This paper focuses on calibration under dataset shift in image classification, providing valuable findings and inspiring insights for many TMLR's audience.", "requested_changes": "This paper is very comprehensive and solid, so I recommend it for acceptance with a minor revision.\nRegarding the model ensembles, the ensemble size is fixed as 3 in the experiments; scalability with size (k=1…9) is unstudied, leaving unanswered whether the reported calibration gains plateau or continue to improve as more models are added. It could be better to include a figure showing the calibration varies with the ensemble size k ∈ {1,3,5,9}.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FsEnngdlSW", "reviewer": "Reviewer_NV3v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts extensive experiment on existing calibration methods under real-world dataset shift. The author uses various image recognition dataset with natural dataset shift and evaluates several in-training and post-hoc calibration methods on random initialization and on foundation model. The author then describes the practical guideline about the calibration method to be used.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The author evaluates multiple methods on several real-world image recognition benchmarks to obtain the guideline. Therefore, I think the result is reliable.\n\nOne of my concerns is the number of trials. If each setting is evaluated once, it would be better to conduct evaluation multiple times and plot the standard deviation to ensure more reliability.", "audience_yn": "Yes", "audience_explanation": "The contribution is the extensive study of existing calibration methods in realistic setting. The results would be interesting for the researchers in this area.", "requested_changes": "It would be clearer to visualize the example dataset images corresponding to Table A.1. so that we can see the dataset shift easily.\n\nUpon acceptance, it would be better to publish the trained model in the experiment so that the reader can reproduce the results easily.", "broader_impact_concerns": "I see no broader impact section, but since the contribution is the experimental evaluation of existing methods on existing datasets, the risk of ethical problem is small.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)