An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration
Authors: Hiroki Naganuma
Ryuichiro Hataya
Kotaro Yoshida
Ioannis Mitliagkas
Venue: Accepted by TMLR ()

Abstract: In out-of-distribution (OOD) generalization tasks, fine-tuning pre-trained models has become a prevalent strategy. Different from most prior work that has focused on advancing learning algorithms, we systematically examined how pre-trained model size, pre-training dataset size, and training strategies impact generalization and uncertainty calibration on downstream tasks. We evaluated 100 models across diverse pre-trained model sizes, five pre-training datasets, and five data augmentations through extensive experiments on four distribution shift datasets totaling over 120,000 GPU hours. Our results demonstrate the significant impact of pre-trained model selection, with optimal choices substantially improving OOD accuracy over algorithm improvement alone. Additionally, we find that larger models and bigger pre-training datasets not only enhance OOD performance but also improve calibration, helping to mitigate overconfidence, contrary to some prior studies that found modern deep networks to calibrate worse than classical shallow models. Our work underscores the overlooked importance of pre-trained model selection for out-of-distribution generalization and calibration.

Published in Transactions on Machine Learning Research (04/2025)
An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration
Hiroki Naganumal Ryuichiro Hataya Kotaro Yoshida', Ioannis Mitliagkas
neganama. hwwoki@mmla..ueeee.....
yoshida. (.2add.. 115330m.isct.a... ioannis@mila
Mila Quebec Al Institute, UUnseessité de Montréal, "RIKEN AIP, Instibute of Science Tokyo,
Reviewed on OpenReview: ht tp s: //op re ew.n nnt net/ /f um d= tY Sh oF
Abstract
In the field of computer vision, fino-tuning pro-trained models has become a prevalent stratogy for out-of-distributie (OOD) generalization tasks. Different from most prior work that has focused on advancing learning algorithms, we systematically examined how pre-trained model size, pre-training dataset size, and training strategies impact generalization and confidence calibration downstream tasks. We evaluated 100 models across diverse pre-trained model sizes, five pre-training datasets, and five data augmentations through extensive experiments on four distribution shift datasets totaling over 120,000 GPU hours. Our results demonstrate the significant impact of pre-trained model selection, with optimal choices substantially improving OOD accuracy over algorithm improvement alone. Additionally, we find that larger models and bigger pro-training datasets not only enhance OOD performance but also improve calibration, helping to mitigate overconfidence, contrary some prior studies that found modern deep networks to calibrate worse than classical shallow models. Our work underscores the overlooked importance of pre-trained model selection for outto--distribution generalization and calibration.
1 Introduction
distriiiiittintttion (OOD) generalization for com- puter vision models has become critical challenges in recent years. This stems from their indispensable role in developing robust and reliable artificial intelligence across diverse tasks and situations (Nagarajan et al., 2021; Geirhos et al., 2020; Shen et al., 2021; Zhou et al., 2021). Our research holds particular relevance to practical contexts requiring OOD genoralization in computer vision, such as domain generalization tasks (Arjovsky al., 2019b; Ghifary al., 2015; Li et al., 2017; Fang et al., 2013; Venkateswara et al., 2017; Peng et al., 2019).
Table 1: Comparison of OOD test accuracy of ResNet- 50 trained with ERM (Vapnik, 1991), ResNet-50 with the best algorithms (Best Alg.) reported in Gulrajaní & Lopez-Pax (2021), and larger model (ViT-base pre- trained let2eNet-2lk for VLCS and ViT-large pre- trained on ImageNet-211 for other (asks) trained with ERM. A indicates the improvement from results with the best algorithms to results with the best models. Baseline results are adopted from Gulrajani & Lopez- Paz (2021).
ERM
Best Alg.
Best Model
A
Currently, the prevalent strategy for tackling this issue relies on fine-tuning adjusting the parameters of pre-trained models to optimize performance on specified downstream tasks (Gulrajani & Lopez-Paz, 2021).
PACS
88.1
89.1
96.7
+7.71
OfficeHome
62.7
64.7
87.4
+22.7
DomainNet
58.4
59.5
77.8
+18.3
VLCS
76.4
77.5
82.2
44.7
"These authors contributed equally this work.

---

Published in Transactions on Machine Learning Research (04/2025)
PACS
OfficeHome
DomainNet
VLCS
WILDSCCmelyon
Expected Calibration Error
Expected Callbration Eroor
0.35
Expected Callbration Error
Expected Calibration Error
0.30
Expected Callbration Error
0.20
0.08
0.25
0.06
0.05
0.33
0.25
0.05
0.20
0.04
0.20
0.10
0.04
0.13
0.03
0.10
0.02
0.30
0.02
6.05
0.01
0.05
0.00
0.7 0.8
0,9
0.5
06
0.7
0.8
0.4
0.5
0.6
0.7
0.8
0.
07 Test accuracy
0.8
0.52
0.34
0,96
0.58
0OD
Test accuracy
00D
Test accuracy
OOD Test accuracy
OOD
0OD
Test accuracy
VGG
ResNet
EfficientNet
RegNet
ConvNeXt
VIT
MLP-Mixer
Figure 1: This paper investigates the out- distiiiibtiiiiiinn (OOD) generalization and confidence calibration of total of 100 ImageNet pre-trained models five datasets from DomainBed (Gulrajani & Lopez- Paz, 2021) and WILDS (Koh et al., 2021). These panels show the relationship between expected calibration error (ECE) rates (lower is better) and OOD lest accuracy (higher is better). Marker sizes are proportional to the number of model parameters, and marker shapes correspond to pre-training configurations. We can observe a general trend that larger models achieve the best of both worlds (bottom-right corner), except for VGGs and MLP-Mixers The full legend can be found in fig. D.6.
A limitation of current approaches is that existing research has predominantly focused on advancing lcarning algorithms (Arjovsky et al., 2019a; Rame et al., 2022; Krueger et al., 2021; Sun & Saenko, 2016; Sagawa et al., 2019; Yan et al., 2020; Blanchard et al., 2021; Koyama & Yamaguchi, 2020; Ahuja et al., 2021), with limited investigation into how the choice of pre-trained model affects OOD generalization. More recent studics (Li ct al., 2022; Cha et al., 2022; Arpit al., 2022; Zhang et al., 2023) have evaluated their proposod methodology on more modern architectures, like vision transformers; however, they also used a limited number of pre-trained models solely to benchmark the effectiveness of their proposed OOD generalization algorithms. While this approach has yielded significant progress, it often overlooks the crucial impact of pre-trained model selection on OOD generalization.
Recently, some works (Goldblum et al., 2023; Vishniakov et al., 2023; Zhang et al., 2022) have studied the effects of pre-trained model selection on OOD generalization; however, these papers partially answer a subset of the questions that we pose, and leave other important questions unanswered. Specifically, Goldblum et al. (2023) evaluates several model architectures and model sizes as backbones in various downstream tasks but does not evaluate and discuss the scaling laws in model size and dataset size that found. Zhang et al. (2022) focuses exclusively and ConvNext models. Our experiments are designed to answer the important questions left unanswered by these studics. What is the effect of dataset sizc, and the relevant scaling laws? Do the phenomena on which we focus persist when we study a wide class of architectures?
Vision Trainformer (ViT) models and, Vishniakov et al. (2023) focuses on both VIT
In section 3, we provide more details on the related work.
Motivated by these limitations, conduct comprchensive and systematic evaluation of pre-trained model solection for OOD generalization. We have found that, pre-trained model selection is more powerful than initially thought, leading to substantial performance improvements over algorithmic enhancements (See table 1).
Furthermore, to gain a deeper understanding of the scaling laws observed in OOD generalization, we also evaluated the models from the perspective of confidence calibration (Nacini et al., 2015; Guo ct al., 2017; Krishnan& & Tickoo, 2020), which aims to ensure that the estimated class probabilities are consistent with actual class probabilities. Calibration has been shown to have strong connection to OOD generalization (Wald et al., 2021; Ovadia et al., 2019; Immer et al., 2021; Naganuma & Kimura, 2023; Yoshida & Nagamuma, 2024), and it allows for a more detailed analysis by focusing on the model's confidence, which cannot be captured by accuracy alone. Similar to OOD generalizat ion, research on calibration in pre-trained models has been limited to specific settings, and a generalized understanding has yet to be established.
We evaluate pre-trained models along three dimensions: (1) pre-trained model parameter size and architecture, (2) pre-training dataset size, (3) model design and training strategy for pre-trained models. We assess
2

---

Published in Transactions on Machine Learning Research (04/2025)
performance downstream tasks using two key metrics: accuracy and calibration errors under distributional shift. For calibration, we employ Expected Calibration Error (ECF) (Naeini et al., 2015; Nixon et al., 2019). The few existing studies for this setting do not thoroughly explore all three axes above, an exploration that is important for achicving the best results from model selection and for investigating this factor.
Through extensive experiments on five distribution shift datasets, we provide new insights into pre-trained model selection for OOD generalization in computer vision. Specifically, we evaluated 100 pre-trained models, spanning five pre-training datasets (ImageNet-1k to 22k and JFT-300M (Sun et al., 2017)), five data augmentation strategies, and several pre-training algorithms (including DINO (Caron et al,, 2021), MAE (He et al., 2022), AdvProp (Xie et al., 2020a), Noisy Student (Xie et al., 2020b)) and CLIP (Radford et al., 2021) on downstream tasks over five distribution shift datasets. Our extensive experiments, involving over 120,000 GPU hours, consistently show that larger models and more extensive datasets improve OOD generalization and calibration. As can be seen in table 1, compared to learning algorithm improvements (up to 2% on DomainBed (Gulrajani & Lopez-Paz, 2021)), the best pre-trained model selection boosts OOD accuracy dramatically (up to 22.7% in our experiments). These results underscore that pre-trained model selection plays critical yet overlooked role in OOD generalization.
Contributions
This work highlights three key insights into model selection for OOD generalization:
We perform a broad-scope study which establishes that larger model sizes and bigger pre-training datasets improve OOD generalization (fig. 3). Our study is the first to evaluate the effect of dataset size, and model size on wide set of architectures, and it lends solid experimental support to common intuition among researchers.
A deeper analysis of OOD generalization through the lens of calibration reveals that scaling laws also hold for calibration. This suggests that the signs of scaling laws observed in OOD generalization can be explained, in part, from the perspective of calibration within the context of invariant feature learning (fig. 3).
The generic model design and training strategy produces the most transferable pre-trained models, compared to techniques that overfit on ImageNot-1K (fig. and fig. 5).
Our systematic findings advocale an increased focus on pre-trained model selection. Specifically, we highlight best practices like using the largest pre-trained models and datasets and avoiding techniques tailored narrowly for in-distribution performance. These insights could redefine approaches to OOD generalization.
2 Preliminary
Before delving into our experiments and findings, it is crucial to lay the groundwork by defining key concepts that underpin our research.
Out-of-distribution (OOD) generalization encompasses a range of concepts, including domain generalization, domain adaptation, robust generalizatio,, and adversarial training. It aims to represent model's capacity to accurately interpret and handle data that it has not encountered during training. This aspect is crucial for ensuring that models perform effectively in real-world scenarios where data can vary from the training set distribution.
OOD Generalization:
Quantifying the correctness of model's confidence is vital for understanding the reliability of the model prediction. This becomes particularly relevant in high-stakes domains like cancer detection, where a model's overconfidence can have serious consequences. Guo et al. (2017) showed that "modern" deep neural networks (in 2017), such as ResNet-110, tend to be overconfident in their predictions compared to classical neural networks such as LeNet-5, highlighting the need for reliable quantification of confidence calibration.
Confidence Calibration:
3

---

Published in Transactions on Machine Learning Research (04/2025)
Expected Calibration Error (ECE) is widely used metric for quantifying confidence calibration. It measures the discrepancy between a model's predicted confidence and its actual accuracy. The formula is given as:
M Bm
ECE
=
Jace (Bm) conf (Bm))
(1)
Accuracy acc (Bm) and confidence conf (Bm) are defined as follows
Σ acc (Bm) = [mml iEBm..
1 1 = y)), conf (Bm) [m ie
Σ Pi.
(2)
where Bm is the set of samples whose prediction scores fall into bin m, Pss is the confidence of sample i, is the true label, di is the predicted label, and n is the total number of samples in all the bins.
While ECE provides a useful overall measure of calibration, it is important to interpret ECE carefully, especially when accuracy is low. Models that make random predictions or fail to learn can still achieve low ECE. Therefore, when comparing models, is critical to consider ECE primarily in high-accuracy regions. Furthermore, ECE suffers from vulnerability to biases in model confidence due to its use of fixed confidence bins. Specifically, recent large-scale neural networks are typically highly confident, resulting in a substantial number of data points concentrated in high-confidence bins while low-confidence bins contain very few data points. This imbalance leads to variability in the accuracy of the estimated errors. To address this issue, Nixon et al. (2019) proposed ACE. ACE adaptively partitions the confidence scores into bins such that each bin contains an equal number of instances. It is defined by the following equation:
K
M
ACE
acc (Bm,,k) - conf (Bm,,)),
(3)
KM
k= m=1
where K denotes the number of classification classes, and Bm, represents the m-th bin within the model's confidence for class k. For cach class, the model confidence are sorted, and the binning is performed such that each bin contains [N/M] data points evenly.
3 Related Works
OOD Generalization Algorithms and Benchmarks: In recent years, varions algorithms have been proposed to improve OOD generalization, such as invariant risk minimization (Arjovsky et al., 2019a), group distributionally robust optimization (Sagawa al., 2019), risk extrapolation (Krueger et al., 2021), correlation alignment (Sun & Saenko, 2016), gradient variances control (Rame et. al., 2022). However, their performance depends heavily on the dataset characteristics, and optimal algorithm has been identified. Furthermore, empirical risk minimization still achieves competitive performance on covariate shift (Ahuja et al., 2020). DomainBed (Gulrajani & Lopez-Pase 2021) has become standard benchmark consisting of image classification datasets with different domain shifts for evaluating OOD generalization. Other benchmarks have also been introduced, such as WILDS (Koh et al., 2021) for various data types and distributional shifts, WOODS (Gagnon- Audet et al., 2022) focusing on time series data, and TableShift (Gardner et al., 2023) for tabular data.
There are four differences between their studies and ours. First, our protocol targets pre-trained model selection for fine- tuning tasks, whereas their protocol targets scratch training. Second, we evaluate in the context of OOD generalization, and they evaluate on robustness benchmarks such as image corruption and perturbation (We also provide the results in fig. D.9, in addition to the OOD generalization benchmarks). Third, we evaluate more than 100 models, including wide range of architoctures, training strategies, model parameter size, and training dala, as well as an exhaustive hyperparameter search, which has been done with a limited number of model families and training sets (28 at most) in the previous study, Finally, we conduct a systematic evaluation of the impact of the number of parameters in the pro-trained model and the size of the pre-training dataset on OOD generalization, with deeper analysis also examining calibration metrics. This is an attempt to explore an aspect that has not been previously investigated.
4

---

Published in Transactions on Machine Learning Research (04/2025)
Pre-trained Model Selection for Downstream Tasks: Fino-tuning pre-trained models has become the prevalent approach for OOD generalization tasks (Gulrajani & Lopez-Paz, 2021). However, most works have focused on advancing algorithms, with a limited investigation into pre-trained model selection. Some analyses (Goldblum et al., 2023; Zhang et al., 2022; Tada & Nagammma, 2023; Yamada & Otani, 2022; Andreassen et al.; 2021) have compared model architectures on robustness benchmarks (Hendrycks el al., 2021a;b; Beyer et al., 2020; Recht al., 2019; Hendrycks & Dictterich, 2019).
However, in studies that measured the effect of pre-trained model selection for OOD generalization, the number of pre-trained models for comparison is limited. For example, Carlucci et al. (2019) proposed pre-training CNNs to solve puzzles to learn features useful for domain generalization and evaluated them using ResNet18 and AlexNet as pre-trained models. Li ct al. (2022) proposed a GMoE model that replaces the FFN block of the ViT architecture with a mixtuur--ffeeperrs block and compared it with the CNN and ResNet. Zhang et al. (2023) have tested their proposed method on 12 pre-trained models (CNN-, MLP-, and Transformer-based) in the context of test time domain adaptation. Except for ResNet, there are no comparative experiments on more than three different parameter sizes pre-training data in the same model architecture, and no analysis of parameter sizes or datasets was conducted. Cha et al. (2022) and Arpit et al. (2022) conducted evaluation experiments using RegNetY-16GF as a pro-training model in addition to ResNet to evaluate the validity of their proposed method.
Yu et al. (2021) focused on only BiT, ViT, and ResNet architecture, Zhang et al. (2022) focused on only ViT, while Vishniakov et al. (2023) studied more in detail of ViT and ConvNeXt architect in the context of downstream tasks. In addition, all related works mentioned above pay no attention to ECE under distributional shift.
Trend of Larger Models and Datasets: Recent evidence (Mikami et al., 2022) suggests that larger models and bigger pre-training datasets lead to better generalization in downstream tasks of Syn2Real (Yasarla et al., 2020). Not only in downstream tasks, this trend also observed in the context of scaling laws (Kaplan et al., 2020; Zhai et al., 2022; Hestness et al., 2017). We systematically verify if this trend applies to OOD generalization and calibration scenarios.
On another note, Ruan et al. (2021) has taken on the challenge of improving OOD generalization performance in multimodal models using CLIP (Radford et al., 2021). However, as shown by Zhang et al. (2023). they point out that the images such as LAION-2B" that CLIP has trained on contain images analogous to the sketch domain of DomainNet (Peng et al., 2019), one of the evaluation datasets for OOD generalization. Since this could potentially undermine our problem sctting for evaluating OOD generalization, although we have performed evaluation experiments on CLIP models as pre-trained models, we only present these settings and results in sections A.3 and D.7.
More recent studies have also evaluated modern architectures in comparison with calibration and robustness benchmarks, and Minderer et al. (2021) argue, as do Guo et al. (2017). that in independently and identically distributed (ID) settings, the ECE degrades with larger models. Conversely, in OOD scenarios, larger models tend to exhibit improved calibration performance (i.c., lower ECE) only under specifie configurations, such as on certain models (e.g., BERT) and datasets (Tran el al., 2022; Kadavath et al., 2022; Dan & Roth, 2021). However, these findings have been limited to particular settings and lack generalizability. To address this, conducted a far more comprehensive set of experiments, yielding results with broader applicability.
Confidence Calibration Benchmark:
Link between OOD Generalization and Calibration: Recent theoretical study (Wald et al., 2021) has shown that multi-domain calibration is equivalent to invariant risk minimization (Arjovsky et al., 2019a), a popular OOD generalization technique. Some empirical observations indicate that techniques good at OOD generalization also achieve lower ECE (Ovadia et al., 2019; Immer et al., 2021; Naganuma & Kimura, 2023; Yoshida & Naganuma, 2024). We examine if this correlation holds for varying pre-trained models under distributional shift as well.
On another note, our experiments adopted pro-trained models from the Py Image Models library (Wight- man, 2019), also known as timm. It benchmarks models on ImageNet OOD test sets, such as ImageNet-
htttps//laaion http:///ssooo.ailogooooo/ as/blogglaion-Bbb
5

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Datasets from the DomainBed (Gulrajani & Lopez-Pazz 2021) and WILDS (Koh et al., 2021)
Banckmark used in our experiments. Domains with bold faces are used as out-of-distribution test domains, and others are used for (in-domain) training and validation.
Dataset
Domains
# Images
# #Clssses
PACS
art, cartoon, photos, sketches art, clipart, product, real clipart, infograph, painting, quickdraw, real, sketch VOC2007, Caltech101, LabelMe, SUN09 Hospital1, Hospital2, Hospital3, Hospital4, Hospital5
9,991
7
Officellome
15,588
65
DomainNet
586,575
345
VLCS
10,729
5
Camelyon17
455,954
2
PACS
OfficeHome
DomainNet
VLCS
1.0
0.8
OOD Test Accuracy
OOD Test Accuracy
0.8
Test T Accuracy
Accuracy
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.4
000
0.2
0.2
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
0.0
0.2
0.4
0.6
0.2
0.4
0.6
0.8
Validation Accuracy
Validation Accuracy
Validation Accuracy
ID Validation Accuracy
VGG
ResNet
RegNet
EfficientNet
ConvNeXt
VIT
MLP-Mixer
The relationship between in-domain (ID) validation accuracy and out-of-distribution (OOD) test accuracy of all experiments. When ID validation accuracy high enough, ID and OOD accuracy are highly correlated, the "training-domain validation set" scheme can select near-optimal models.
Figure 2:
R (Hendrycks el al., 2021a), ImageNet-A (Hendrycks et al., 2021b), ImageNet-v2 (Recht et al., 2019), and ImageNet-Reu Real (Beyer et al., 2020) which is beneficial for our study.
4 Experimental Setup
Our experiments evaluate the OOD generalization accuracy (or error) of 100 pre-trained models fine-tuned on five OOD image classification datasets, namely PACS, OfficeHome, DomainNet, and VLCS, from the DomainBed benchmark (Gulrajani & Lopez-Paz, 2021) (see also table 2). In addition to accuracy, we use ECE as complementary metric assess the correctness of model confidence, which cannot be captured by accuracy (error) alone. In order to address concerns regarding ECE (Nixon et al., 2019), we also mcasured the ACE and confirmed the presence of identical trends, presented in section D.1. Additionally, conducted medical image classification experiments using Camelyon17 datasets from WILDS benchmark (Koh et al., 2021) to cover other types of OOD generalization task evaluation (section D.6).
Pre-trained models are selected from the timm library (Wightman, 2019), which offers a vast collection of over thousand pre-trained image classification models. Among them, we use models pre-trained on families of ImageNet to minimize data overlap with the DomainBed and WILDS datasets, since our interest lies in OOD generalization. Specifically, we adopt pre-training datasets of ImageNet- (1k categories, 1.4M images), ImageNet-12k (12k categories, 12M images), ImageNet-211 21k (2lk categories, 14M images), and ImageNet-22kk (22k categories, 14M images). While timm also offers models pro-trained on larger datascts, such as LAION (Schuhmarn et al., 2022) consisting of billions of images, we refrain from utilizing them due to their excessive size, which raises concerns about the potential inclusion of OOD test data. In addition to the models pre-trained on image data only, we validate CLIP models, which are pro-trained on imago-text pairs (section A.3). Selected 100 pre-trained models, varying parameter sizes from 3M 600M, are listed in table A.1. Further details are explained in section A.1.

---

Published in Transactions on Machine Learning Research (04/2025)
(a)
(6)
Expected Calibration Error
(4)
Expected Calibration Error
(d)
0.10
0.30
Test Error
0.25
O0D Test Error
0.14
0.20
0.08
0.06
0.10
0.10
0.04
0.10
0.06
0.05
0.05
0.02
10r
10a
#parameters (x10%)
Dataset size (x10%)
# #parameters (x109)
Dataset size (x10%)
ResNet-So
RegNat-Y 400MF
RegNetY 8GF
BricieettietB83
ConvtleXt-tiny (IN12K, ft)
VIT-small (w/ AugReg)
ResNet-101
RegNet-Y 600MF
RegNeetY 12GF
EfricientNet-B4
ConvNeXt-smll (IN12K,
ViT-base (w/ AugReg)
RegNet-Y
RegNetY
ConvNeXt-tiny (IN22X.
ViT-tiny AugReg, IN21k,
ResNet-So (v2)
RegNat-Y 1.6GF
RegNetaY 32GF
ConvNext-tin
Convitett-smal (IN22K, Rt)
VIT-small (w/ AugReg, IN21,, f1
ResNet-101 (v2)
RegNet-Y 3.2GF
EfficientNet-00
sonlNeXt-small
ConvNexX-base (IN22K, ft)
ViT-base (w AugReg, IM21K, ft)
RosNet-152 (v2)
RegNet-Y 4.0GF
EmclentNet-a1
ConwNeXt-base
ConvNext-lagg (INZ2K,
VIT-lagge (w/ AugRog. IN21K, ft)
RegNet-Y 200MF
RegMat-Y 6.4GF
EmclentNet-a2
ConvNeXt-large
Convtext-xlarge BN22K, mt
Figure 3: (a,b) OOD test error rates (lower is better) with respect to the number of parameters (a, log-scale) and the pre-training dataset sizes (b) on the PACS dataset. In the right panel, the same model architectures with different pre-training datasets are connected by dashed lines. We can observe that the number of parameters and the pre-training dataset sizes contribute to the OOD generalization. Presented models are selected for a better view. Results other OOD datasets with all models are presented figs. D.5 to D.7. (c,d) OOD expected calibration error (lower is better) with respect to the number of parameters (c) and the pre-training datasel sizes (d) on the PACS dataset. In the right. panel, the same model architectures with different pre-training datasets connected by dashed lines. We parameters and the pre-training dataset sizes improve ECE. Presented models are selected for the belter view. Results of other OOD datasets with all models are presented in figs. D.5 to D.7.
trends that the number of
For fine-tuning, we follow DomsinBedd's experimental protocol. The key difference from the original protocol is that various pre-trained models from timm, rather than fixing the pre-trained model only to ResNet-50. To prioritize the exploration of models under fixed computational constraints, we adopt simple evaluation approach different from the original DomainBed. Namely, we select each datasct's 's first domain listed in table 2 as the OOD Lest domain and use the remaining domains as (in-domain) training and validation data.
Another difference from Bed the choice optimizer for fine-tuning. We adopt SGD with a momentum of 0.9 as a default optimizer rather than Adam (Kingma & Ba, 2015), as SGD with Momentum is more ilfective in OOD generalization (Naganuma et al., 2023) (we confirm this in figs. C.2 and C.3). In addition, fine-tune models with adaptive optimizers when they are used for pre-training: RMSProp for EfficientNets, AdamW for ConwNeXts, and Adam for ViTs.
Their hyperparameters, learning rales, and weight decay rates are selected using in-domain validation data, as we cannot access OOD data. This selection is referred to as the "training-domant validation set" scheme in the DomainBed benchmark (Gulrajani & Lopez-Paz, 2021) and capable finding near-optimal models when in-domain validation accuracy is high enough (fig. 2). Further details on hyperparameter selection are described in section A.2. We also experiments three different. random seeds, and their effects the final results negligible, shown in table C.1.
5 Results
In the remaining text, we report OOD generalization performance and ECE on the held-out test domain data of models selected by the above-mentioned validation scheme. Full results and detailed analysis for each model architecture shown in section D.4 and section D.5.
7

---

Published in Transactions on Machine Learning Research (04/2025)
5.1 Scaling Trend: OOD Generalization
Our comprehensive experiments reveal several consistent and significant trends in how pre-trained model selection impacts out-of-distribution generalization performance.
First, we find that increasing the number of parameters in pre-trained models leads to noticeable gains in OOD accuracy across almost all model architectures examined, as shown in fig. 3. Similarly, pre-trained models on larger dataset sizes yield substantial OOD performance improvements. These trends hold not just for the PACS examples shown here but also generalize to other evaluation datascts for domain generalization (figs. D.6 and D.7). Second, when comparing models with similar parameter sizes, we observe that ViT and ConvNeXt architectures tend to achieve higher OOD accuracy than conventional CNN architectures like ResNet and EfficientNet. This suggests the benefits of more recent model architectures specialized for computer vision tasks.
As a remarkable observation, we also find that OOD error decreases almost linearly on log-scale of parameters as the pre-trained model size increases as shown in fig. 3 (a). This implies the sign of scaling laws and hints at shared underlying mechanics that drive performance gains in downstream domain generalization tasks. Lastly, through targeted experiments on ConvNeXt and ViT models, we verify that increasing the pre-training dataset size significantly boosts OOD accuracy. This further underscores the importance of large-scale pre-training data for enhancing OOD generalization.
In summary, our extensive empirical results reveal consistent trends that bigger pre-trained model sizes and larger pre-training datasets lead to substantial and reliable improvements in out-of-distribution generalization performance across diverse model architectures and evaluation datasels. These insights provide practical guidance for pre-trained model selection when targeting OOD tasks.
5.2 A Deeper Analysis from the Perspective of Confidence Calibration
To uncover the mechanisms behind the scaling trends in OOD generalization observed in fine-tuning pre- trained models, we evaluate confidence calibration using ECE. Calibration is adopted for two main reasons: (1) Calibration in multi-domain settings is thcoretically equivalent to invariant fcature learning for OOD generalization (Wald et al., 2021), and it has been empirically shown to improve OOD accuracy and invariant feature learning (Ovadia et al., 2019; Immer et al., 2021; Naganuma & Kimura, 2023; Yoshida & Naganuma, 2024), whereas accuracy or error alone cannot capture the degree of invariant feature learning; (2) Other promising generalization metrics, such as curvature, are challenging to compare across model architectures, whereas calibration, based on output distributions, is more suitable for our evaluation. The results are shown in fig. 3 (c, d).
First, we find that increasing the number of parameters in pre-trained models leads to lower F.CE, mirroring the trends observed for OOD generalization. Using pre-trained models trained on larger dataset sizes also improves calibration. From these results, we infer that increasing model size or the amount of training data during pre-training helps to miligate domain-specific overconfidence in downstream lasks, which in turn promotes OOD generalization, including through mechanisms like invariant feature learning.
Furthermore, these results contrast with prior work in IID settings, found modern deep networks like ResNet- 110 to have significantly higher ECE than small networks like LeNet-5, despite their higher accuracy (Guo et al., 2017; Minderer et al., 2021). In line with Minderer et al. (2021), we demonstrate that in OOD scttings, at even larger parameter scales beyond ResNet-110, the trend reverses, and ECE starts to decrease with model parameter size. Notably, ECE reduces almost linearly as pre-trained model size increases on log scale, novel finding that parallels the scaling laws for OOD error. This strengthens the claim that, in OOD settings, increasing model size significantly supports improved calibration.
In summary, we confirm that increasing the size of the pre-trained model and pre-training data boosts OOD generalization. Additionally, a deeper investigation from the perspective of calibration reveals that a similar trend holds for ECE, showing that larger models and datasets during pre-training help mitigate overconfidence in downstream tasks. Our results also overturn prior observations in IID settings that modern deep networks
8

---

Published in Transactions on Machine Learning Research (04/2025)
PACS
OfficeHome
DomainNet
VLCS
00D Test accuracy
0.95
OOD Test accuracy
0OD Test accuracy
0.75
OOD Test accuracy
0.80
0.90
0.75
0.65
0.60
0.6
0.65
0.75
0.55
0.70
0.5
0.50
0.60
0.70
0.75
0.80
0.as
0.70
0.75
0.80
0.85
ImageNet-1k accuracy
ImageNet-1k accuracy
ImageNet-1k accuracy
ImageNet-lk accuracy
ResNet-50
RegNetYY 800MF
RegNet-Y 32GF
ConvNeXt-small
ConwNeXt-large (IN22K, ft)
ResNet-101
RegNet-Y 1.6GF
EfficientNVe-800
ConvNeXt-base
ConvNeXt-xlarge (IN22K,
ResNet-152
RegNet-Y 3.2GF
EfficientNet-81
ConvNeXt-large
VIT-small (w/ AugReg)
ResNet-50 (v2)
RegNetY 4.0GF
EfficientNet-82
ConvNeXt-tiny (IN12K,
ViT-base (w/ AugReg)
ResNet-101 [v2)
RegNet-Y 6.4GF
EfficieetNNt-83
ConvNeXt-small (IN12K, ft)
ViT-tiny AugReg, IN21K,
ResNet-152 (v2)
RegNet-Y 8GF
EfficientNet-84
ConvNeXt-tiny (IN22k, ft)
ViT-small (w/ AugReg, IN21k, ft)
RegNetY 200MF
RegNet 12GF
EfficientNet 85
ConvNeXt small (IN22k, ft)
ViT-base (w/ AugReg, IN21k, ft)
RegNetY 400MF
RegNet-Y 16GF
ConvNeXt-tiny
ConvNeXt- base (IN22k, rt)
ViT-large (w/ AugReg, IN21k, (t)
RegNetY 600MF
600MF
Figure 4: Correlation between ImtgeNet-Ik validation accuracy and out-of-distribution test accuracy. OOD
test accuracy shows strong positive correlation with IeageNet-1k validation accuracy. However, EfficientNets fall below the trend of other models, suggesting that they may be overfitting to ImageNet-11.
PACS
OfficeHome
DomainNet
VLCS
0.80
Test Accuracy
0.900
OOD Test Accuracy
0.70
OOD Test Accuracy
OOD Test Accuracy
0.75
0.875
0.5
0.850
0.4
0.60
0.825
0.55
0.65
0.800
0.2
0.775
0.1
5
5
# #parameters (x10*)
# #parameters (x10
#parameters (x10%)
# #parameters (x10*)
ResNet-50
Nes-et1101 (v2)
ResNet-152 (A1)
ResNet-50 (A3)
ResNet-101
ResNet-152 (v2)
ResNet-50 (A2)
ResNet-101 (A3)
ResNet-152
ResNet-50 (A1)
ResNet-101 (A2)
ResNet-152 (A3)
ResNet-50 (v2)
ResNet-101 (A1)
ResNet-152 (A2)
Figure 5: Out-of-distribution test accuracy ResNets (He et al., 2016) with different training schemes. The
results of the original ResNets are connected by dashed lincs. Although extensive training recipes (Wightman et al., 2021) improve ImageNet-Ik performance, their fine-tuned OOD results often underperform the original
ones.
suffer from higher ECE, underscoring the potential of large-scale pre-trained models to address calibration challenges specifically in OOD contexts.
5.3 Pre-trained Models can Overfit to Training Set
We observe that some pre-trained models are likely to overfit to training set such as ImageNet-1k. Figure 4 presents the relationship between ImageNet-lk validation accuracy and OOD test accuracy. As can be seen, the EfficientNNtt models tend to fall below the trendlines of other models, suggesting that they are more prone overfitting ImageNet- 1k and do not transfer well to OOD datasets. This overfit could be attributed EfficientNets being designed by neural architecture search (Tan & Le, 2019), where the components of networks
9

---

Published in Transactions on Machine Learning Research (04/2025)
are searched to maximize validation performance. Although improvement of pro-training methods, such using extensive data augmentation (Cubuk et al., 2019; Xie et al., 2020a), enhances OOD generalization performance (fig. D.1I), overfit trends unchanged as evident in fig. D.8. This figure also depicts that fine-tuned EfficientNets degenerates calibration on the OOD datasets. In contrast, RegNets (Radosavovic et al., 2020) seems avoid overfitting, probably because they are optimized at. a meta-level (design principles), not at a component level.
Wightman et al. (2021) reported that ResNet-50 with renewed training recipes, including additional data augmentation and longer training, attains more than 5% ImageNet- Ik accuracy improvement the original setting. Figure 5 displays the comparison of the ResNets pre-trained with the original and the updated recipes: a new training recipe introduced in torchvision referred to as v2 and others proposed in (Wightman et al., 2021) referred to as A1-A3. Although models trained with these new training recipes improve ImageNet- 1k performance, A1-A3 often perform inferior to the original ones under the same fine-tuning setting. Contrarily, the v2 models consistently outperform the original models. These results may indicate that certain aspects of pre-training strategies in Wightman et al. (2021) may lead to overfitting to ImageNet-Ik: Disent angling the specific factors causing the OOD performance degeneration is challenging, due to the various techniques employed in these recipes. However, it is possible to guess that binary cross-entropy loss and/or Adam optimizer during pre-training are attributed to the issue. It is also worth noting that extensive data augmentation, such as Randenttent (Cubuk et al., 2020) and AugMix (Hendrycks* et al., 2020), improves the OOD generalization as in fig. D.14.
These overfit phenomena of EfficientNets and enhanced ResNets cannot be observed on ImageNet OOD test sets, such ImageNet-A, presented in fig. D.10. Rather, EfficientNets and ResNets with A1-A3 recipes are aligned with the correlations between the performance of ImageNet-lk and ImageNet OOD test sets. These findings suggest that the generalization on pre-training datasets, such as ImageNet-lk, does not directly translate to the generalization after fine-tuning.
5.4 Other Findings
Above three are the major findings, but other findings of this study are briefly described. Details are given in section B. First, in the pre-training phase, supervised pre-training with AugReg as a data augmentation approach improves OOD test accuracy compared to self-supervised methods like DINO and MAE. Second, as prior work (Nagamuma et al., 2023) suggests advantages for SGD with momentum over adaptive optimizers for OOD generalization, this study confirmed SGD with momentum consistently outperforms other optimizers when fine-tuning diverse architectures, including ViT and EfficientNeet. Third, the impact of weight decay OOD performance varies across datasets. While stronger regularization improves calibration, it can hinder OOD accuracy on datasets with visually distinct domains, as shown in fig. B.2. The ablation studies for sensitivity and seed of hyperparameters are shown in section C.
Analysis of ImageNet variant datasets for robustness benchmarks suggests a positive correlation between model robustness on ImageNet variants and domain generalization performance, with improvements on sampling bias datasets showing stronger correlation compared to robustness benchmarks as shown in D.9.
Additionally, section D.6 shows the results evaluated on the Camelyon17 datasets, aligning with the trends observed in Deda Section D.7 shows the results of CLIP models, which align with other models in terms of test accuracy.
6 Discussion and Conclusion
In this work, we demonstrated the signs of scaling laws for OOD generalization and calibration of pre-trained models in the image classification setting. This conclusion is drawn from a comprehensive study involving five datasets and over 100 different combinations of pre-training model architectures, pre-training datasets, model sizes, training strategies, and hyperparameter choices.
Details descrihed in httpe://pycorch.org/////gg/ t-prinitives.
staten--ttt--o--o ofaste-o--theeeee mod--moddllllnnnn -t--uug-ooccciiiiiooooo---- torchvision-
10

---

Published in Transactions on Machine Learning Research (04/2025)
Our results are also useful to computer vision practitioners due to their prescriptive nature: systematic study on over 100 pre-trained visual models yielded that in the OOD setting with pretraining, larger architectures using larger training sets not only achieve better out-of-disriibtioo generalization but also improve calibration, further reducing overconfidence downstream tasks. We also found that non-generic training stralegies for example, the ones specifically designed to maximize in-distribution performance for ImageNet-1k on smaller models- do not achieve good OOD performance, indicating overfitting to the pre-training dataset.
In light of these results, summarize findings the following guidelines pre-trained model selection for computer vision practitioners:
Use large pre-trained models,
Use models pre-trained on large datasets,
3. Use models pre-trained using generic methodology (i.e. not specifically designed to maximize in- distribution performace specific architecture).
Following these guidelines can substantially improve OOD gencralization and calibration capabilities.
Our results can also prove to be useful to researchers, they provide guidance for future work in the area. In particular, strong basclinc. This point is important because recent work providing new, complex methodology, yields results that are inferior to the ones we provide here based on simple model selection. Future work on compatation-effiiens methods for the OOD setting is critical; we hope that our results will help researchers build those new, efficient methods on top solid baselines.
important by-product of our results is newfound appreciation of model selection
Acknowledgments
Our deepest gratitude goes out the anonymous reviewers and the Action Editor, whose invaluable insights ials enhanced the quality of this manuscript. The computational resources instrumental to this study were provided under the auspices of the "ABC1 Grand Challenge" Program, National Institute of Adyanced Industrial Science and Technology.
11

---

Published in Transactions on Machine Learning Research (04/2025)
References
Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney. Empirical invariant risk minimization? a sample complexity perspective. arXiv preprint arXiv:2010. 16412, 2020.
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Christope Gagnon-Audet, Yoshua Bengio, loannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for outtof-distribution generalization. Advances in Neural Information Processing Systems, 34:3438 345883,0,,,-00 2021.
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of- distribution robustness throughout fine-tuning. arXiv preprint arXiv:2106. 15831, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint ar arXft:1907.02899, 2019a.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprinl Xiv: 02893, 2019b.
Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Advances in Newral Information Processing Systems, 35:8265 8277, 2022.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018.
Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006. 071159, 2020.
Gilles Blanchard, Aniket Anand Deshmukh, Orun Dogan, Gyemnn Lee, and Clayton Scott. Domain gen- cralization by marginal transfer learning. The Journal of Machine Learning Research, 22(1):46-100 2021.
Fabio M Carlucei, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVV Conference on Computer Vision and Pattern Recognition, pp. 2229 2238, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sell-superiieed vision transformers. In Proceedings of the TEEE/CVF international conference on computer vision, pp. 9650 1000-9660,00 2021.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual-inffrmaatin regularization with pre-trained models. In European Conference on Computer Vision, pp. 440 457. Springer, 2022.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randangment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVE conference on computer vision and pallern recognilion workshops, pp. 702 703, 2020.
Soham Dan and Dan Roth. On the effects of transformer size on in- and out-of-domain calibration. In Mlarie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2096 (096,1101, Punta. Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1 11/2021. 1tt......niinseemmm.. findiing--mmnlp..s.. URL httt:::://cclaahhoooo..oorgg http:/222..f.ndinnn.... findings-enne emllp.18..
12

---

Published in Transactions on Machine Learning Research (04/2025)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199: 2020.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1657 1664, 2013.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https://openreview http::///pperrreieeeeeee/orrrrr id-TTmmmossr...
Jean-Christophe Gagnon-Audet, Kartik Ahuja, Mohammad-Javad Darvishi- Pooneh Mousavi, Guil- laume Dumas, and Trina Rish. Woods: Benchmarks for out-of-diistribtiio generalization in time series. arXiv preprint Xrx:::2203. 09978, 2022.
Joshua P Gardner, Zoran Popovi, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. In Thirty-seventh Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreviee http:/rreeeeeeeetttoouummai/iaae net/forrumid==X/sNlllll
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11): 115-673, 2020.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551 2559, 2015.
Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al. Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks. arXiv preprint arXiv:2810. 2023.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1333 PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFF. conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked auloencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 16000-16009, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks*, Norman Mu*, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations, 2020. URL https: h/tp:s///ppeeeviers
met//forum1ddSSgnrriit
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340 8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostola Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empiri- cally. arXiv preprint wrXie:1712.004400 2017.
Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Khan Mohammad Emtiyaz. Scalable marginal likelihood estimation for model selection in deep learning. In International Conference on Machine Learning, pp. 4563 4573. PMLR, 2021.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, al. Language models (mostly) know what they know. arXiv preprint w:Xi:2207.05222/ 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint XKi:20001.0866 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637 5664. PMLR, 2021.
Masanori Koyama and Shoichiro Yamaguchi. When is invariance useful in an dut-of-distii generalization problem? arxiv preprint arXiv:2008.0 0188,, 2020.
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. Advances in Newral Information Processing Systems, 33:18237-18248, 2020.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse mixture-of-experts are domain generalizable learners. arXiv preprint arXic:2206.00 04046, 2022.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain gencralization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet. for the 2020s. Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), 2022.
lly Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0511 2017.
Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-ichi Maeda, and Kohei Hayashi. A scaling law for syn2real transfer: How much is your pre- pre-training effective? In Joint European Conference on Machine Learning and Knowledge Discovery in Dalabases, pp. 477-492. Springer, 2022.
Matthias Minderer, Josip Djolonga, Rob Romijuders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34:15682-15694, 2021.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrocht. Obtaining well calibrated probabilities using bayesian binning, 2015.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Hiroki Naganuma and Masanari Kimura. Necessary and sufficient hypothesis of curvature: Understanding connection wetn out-of-distribation generalization and calibration. TCIR20033 Workshop on Domain Generulization, 2023.
Hiroki Naganuma, Kartik Ahuja, Shiro Takagi, Tetsuya Motokawa, Rio Yokota, Kohta Ishikawa, Ikuro Sato, and Ioannis Mitliagkas. Empirical study on optimizer selection for dutto--distiibution generalization. Transactions on Machine Learning Research, 2023.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. In International Conference on Learning Representations, 2021.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR workshops, volume 2, 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Aduances in neural information processing systems, 32, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 1406 1415, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transforable visual models from natural language supervision. CoRR, abs/2103.00020,, 2021. URL https: wttp.//gg/../....../////
Lija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10428 10436, 2020.
Alexandre Rame, Corentin Dancelte, and Matthieu Cord. Fishr: Invariant gradient variances for out-of- distribution generalization. In International Conference on Machine Learning, pp. 18347-18377. PMLR, 2022.
Benjamin Recht, Rebecca Roelols, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 1389-5400. PMLR, 2019.
Yangjun Ruan, Yann Dubois, and Chris J Maddison. Optimal representations for covariate shift, arXiv preprint ar orXi:2201.00057, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Perey Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXis:1911.08791. 2019.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Claylon Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sizth Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2022. URL //opnrev/oerrevier neet/forrum/ddeMM774aasss
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of- distribution genoralization: A survey, orXiv preprint ar arXi::2108.1988 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largo-scale image recognition. In International Conference on Learning representations, 2015.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreviiet netp:/ommm//ooomnnnnn wr1KcP.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCe 2016 Workshops: Amsterdam, The Netherlands, October 8- 8-10 and 15-16, 2016, Proceedings, Part III 14, pp. 443-450. Springer, 2016.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp- 843 052- 2017.
Keigo Tada and Hiroki Naganuna. How image corruption and perturbation affect out-of-distribution generalization and calibration. In 2023 International Joint Conference on Newral Networks (IJCNN), pp. 1-6, 2023. doi: N..1100/JCCNN555400220000 20011/11.0.22...1111000
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105 6114. PMLR, 2019.
llya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advonces in neural information processing systems, 34:24261 -44222, 2021.
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, and Balaji Lakshminarayanan. Plex: Towards reliability using pretrained large model extensions, 2022. URL https: /ttp///aavvvv/.aa///a//eeeeaaaaa wwexiiv.ogg/a.. wwgg/ba///// 07411.
Vladimir Vapnik. Principles risk minimization for learning theory. Advances in neural information processing systems, 4, 1991.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pollern recognition, pp. 5018 5027, 2017.
Kirill Vishniakov, Zhiqiang Shen, and Zhuang Liu. Convnet transformer, supervised vs clip: Beyond imagenet accuracy, 2023.
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain generalization. Advances in neural information processing systems, 34:2215 2227, 2021.
Yixuan Wei, Han Hu, Zhenda Xie, Ze Liu, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Improving clip fine-tuning performance. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5416 5426, 2023. doi:
Ross Wightman. Pytorch image models.
2019.
Ross Wightman, Hugo Touvron, and Hervé Jégon. Resnet strikes back: An improved training procedure in timm. Xrxiv preprint or Xiv:21 10.004766 2021.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoe V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVE conference computer vision and pottern recognition, pp. 819-828, 2020a.
Qizhe Xie, Minh-Thaue Thang Luong, Eduard Hovy, and Quoc Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10698. 2020b.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Yutaro Yamada and Mayu Otani. Does robustness on imagenet transfer to downstream tasks? In Proceedings of the IERE/CVE Conference on Computer Vision and Pallern Recognilion, pp. 9215-9224, 2022.
Shen Yan, Huan Song, Nanxiang Li, Linean Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXi::2001.0077, 2020.
Rajeev Yasarla, Vishwanath A Sindagi, and Vishal M Patel. Syn2real transfer learning for image deraining using gaussian processes. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2726-2736, 2020.
Kotaro Yoshida and Hiroki Naganuma. Towards understanding variants of invariant risk minimization through the lens of calibration, 2024.
Yaodong Yu, Heinrich Jiang, Dara Bahri, Hossein Mobahi, Seungyeon Kim, Ankit Singh Rawat, Andreas Veit, and Yi Ma. An empirical study of pre-trained vision models on out-ol-distribution generalization. In NeurlPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12104-12113, 2022.
Chongzhi Zhang, Mingyuan Zhang, Shanghang Zhang, Daisheng Jin, Qiang Zhou, Zhongang Cai, Haiyu Zhao, Xianglong Liu, and Ziwei Liu. Delving deep into the generalization of vision transformers under distribution shifts. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp- 7277 7286, 2022.
Yiyuan Zhang, Kaixiong Gong, Xiaohan Ding, Kaipeng Zhang, Fangrui Lv, Kurt Keutzer, and Xiangyu Yue. Towards unified and effective domain generalization, 2023.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arxiv preprint arxiv:1103. 02503, 2021.
17

---

Appendix Table of Contents
A Detailed Experimental Settings
18
A.1
Models used in the Experiments
18
A.2 Model Selection and Hyperparameter Search
21
A.3 Experimental Protocol of CLIP Model
21
A.4
Datasets..
B Other Findings
23
B.1 Factors Pre-training Phase
B.2 Factors Fine-tuning Phase
B.3 Relationship with Other OOD Datasets
24
C Ablation Study
25
C.1 The Effects of Random Seeds
................................
25
C.2 The effects of Hyperparameters
................................
26
D Additional Results
29
D.1 Expected Calibration Error Adaptive Calibration Error
................................
29
D.2 TerraIncognita Dataset Results
................................
29
D.3 Choice of Test Environments
29
D.4 Full Results of DomainBed
................................
29
D.5 Results per Model Architecture
................................
D.6 Results of Camelyon17 Datasets
................................
D.7 Results of CLIP models
42
E ECE Under Infinite Data and Near Zero Cross-Entrop Loss
42
A Detailed Experimental Settings
All experiments' codes are modifications provided by the authors who introduced the datasets (Gulrajani & Lopez-Paz, 2021; Koh et al., 2021). Licenses of the codes are MIT license for DomainBed (Gulrajaní & Lopez-Paz, 2021) and WILDS (Koh et al., 2021). The license of the pre-trained model used in the experiment is Apache 2.0 license
We performed our experiment with ABCI Cluster. For ABCI Cluster, each node is composed of NVIDIA Tesla V100x 4GPU and Intel Xeon Gold 6148 2.4 GHz, 20 Coresx2CPU, As a software environment, we use Red Hat 4.8.5, gee 7.4, Python 3.6.5, Pytorch 1.6.0, cuDNN 7.6.2, and CUDA 10.0. The total amount of computation is over 120,000 GPU hours.
Please see the code here: https: /ttt::///tttbb www./ooo///ii.iie http:///ttt/mmmm000maarrrrottto
0DD Calibration
A.1 Models used in the Experiments
Table A.1 present all models, in a total of 100, we used in the experiments, sorted by ImageNet-Il accuracy.
Table A.I: Models from timm used the experiments with ImageNet-1k validation accuracy and the number of parameters sorted by ImageNet- 1k validation accuracy. "ft" stands for finetuning on ImageNet-Iie 1k. Self-supervieed pre-trained models present "NaN" accuracy.
Model
IN-1k Acc. # Param. (x10%)
ConvNeXt-xlarge (IN22k, ft)
(Lou al., 2022)
87.33
350.20
Continued next, page
18

---

Published in Transactions on Machine Learning Research (04/2025)
Table A.1: Models from timm used in the experiments with ImageNet-Ik validation accuracy and the number of parameters sorted by ImageNet-1k validation accuracy. "ft" stands for finctuning on ImageNet-1k, Self-supervised pre-trained models present "NaN" accuracy.
Model
IN-1k Acc.
Param. (x10%)
ConvNeXt-large (IN22k, (1) (1.te
87.03
197.77
Conv-eett-bsse (1N22k, ft) (Lia et st. 2022)
86.27
88.59
ElficientNet-B35 (w/ Noisy Student, JFT, ft) (Tan & Le, 2019; Xie al, 2020u)
86.09
30.39
Vir-large (w/ AugReg, IN21k, ft) (Deaovitskiy et al., 2020; Steiner es al., 2022)
85.83
304.33
ConsNeXt-small (TN12k, ft) (1am et al,, 2022)
85.33
50.22
ConvNeXt-smaale (IN22k, ft)
at., 2022)
85.26
50. 20.22
EfficientNet-B4 (w/ Noisy Student, JFT, ft)
Xie 20205)
85.16
19.34
Conv:Nett-large
84.85
197.77 .7
ViT-base (w/ AugReg, IN21k, ft) (Dorovitalty al,, 2020; Statner al., 2022)
84.53
86.57
ConvNeXt-tiny (IN12k, [t). (Liu et al, 2022)
84.45
28.59
ConvNeXt-base (Lis at., 2022)
84.43
88.59
Efficient.Net-355 (w/ AdvProp) (Thn & Le, 2018; et al, 2020s)
84.26
30.39
Efficient.NetB33 (w/ Noisy Student, JFT, ft) (Tan Le, 2019; Xie al., 20205)
84.06
12.23
ElficientNet-B35 (w/ RandAugment) (Thn
al, 2020)
83.81
30.39
Conv>NeXt-mmall (Lia sl.,
83.70
50.22
Efficient.Net-B5 (w/ AutoAugment) (Tan & La, 2019; Cuubak at al., 2019)
83.69
30.39
ResNNet-152 (A1H) (Wightmmn al., 2021)
83.45
60.19
Efficient.Net-B34 (w/ AdvProp) (Tha & Le, 2019; Xie et ab, 2020s)
83.25
19.34
E (Thu Lc, 2019)
83.18
30.39
EfficientNee-BA. (w/ AutoAugment) (Tan Le, 2019; Cubak et al., 2019)
83.02
19.34
ResNet-101 (A1H) (Wightman 2021)
82.78
44.55
ResNet-152 (A1) (Wighrman al, 2021)
82.73
60.19
ConvNeX(-tiny (L.in 2022)
82.70
28.59
Efficient.NetBB4 (Tin 2019)
82.61
19.34
ResNet-152 (A2) (Wightman at al. 2021)
82.61
60.19
Efficient.Net-B2 (w/ Noisy Student, JFT, ft) (Tan & Lo, 2010; Xie es al. 20200)
82.38
9.11
ResNet-101 (A1) (Wightman at al, 2021)
82.32
44.55
ResNet-112 (v2)
st., 2010)
82.29
60.19
ResNet-101 (A2) (Wightman 2021)
82.24
44.55
ResNNet-101 (v2) al., 2016)
81.89
44.55
Efficient.et--B. (w/ AdvProp) (Tan & Le, 2019; Xie et at., 2020a)
81.83
12.23
ViT-base (IN21k, ft) (Dunovitokly 2020)
81.79
86.57
Efficient.eetB33 (w/ AutoAugment) (Tan & Lo, 2019; Cubak at al., 2019)
81.64
12.23
Efficient (The & Le, 2018)
81.53
13.01
Efficient.Net-B1 (w/ Noisy Student, JFT, ft) (Tan 2019; Xie al., 20200)
81.39
7.79
ViT-small (w/ AugReg, IN21k, ft) (Dosovitskly al.,
81.38
22.05
ResNet--50 (A1) (Wightman
81.21
25.56
RcsNet-50 (D)
(Wigheman al. 2021)
80.97
25.56
ResNet--50 (C1)
(Wighoman al, 2021)
80.91
25.56
Efficient.NetB33 B3
(Tan Ls, 2019)
80.88
12.23
ResNet-50 (C2)
(Wightman al., 2021)
80.87
25.56
ResNet-50 (v2)
(He at al,, 2016)
80.85
25.56
RegNet-Y 32GF (Radooravovie al., 2020)
80.81
145.05
ResNett-50 (A2)
(Wightmma
80.77
25.56
RosNett-50 (B1K)
(Wightman al, 2021)
80.71
25.56
ResNett-50 (A1H)
(Wightman et al, 2021)
80.68
25.56
ResNet-1152 (A3)
(Wightman at al. 2021)
80.55
60.19
Continued on next page
19

---

Published in Transactions on Machine Learning Research (04/2025)
Table A.1: Models from timm used in the experiments with ImageNet-Ik validation accuracy and the number of parameters sorted by ImageNet-1k validation accuracy. "ft. stands for finctuning on ImageNet-1k, Self-supervised pre-trained models present "NaN" accuracy.
Model
IN-1k Acc.
Param. (x10%)
ResNet-50 (B2K)
(Wightman ah,, 2021)
80.45
25.56
RegNet-Y 12GF (Radenvvoiic et al. 2020)
80.39
51.82
OfficientNeet-222 (w/ AdvProp) (Tan & Le, 2019; Xie et al,, 2020a)
80.31
9.11
RegNet-Y 16GF
(Radonavovic at al., 2620)
80.30
83.59
ViT-base (SAM) (Dovovitekly et al,, 2020; Pores et al, 2021)
80.24
86.57
ElficieentNet-B22 (w/ AutoAugment) (Tau Lo, 2019; Cubakk al., 2019)
80.08
9.11
ResNet-50 (AugMix) (He
Hondeycles" 2020)
79.98
25.56
RegNet-Y 8GF (Radosavoiic
79.8
39.18
RcsNet-50 (RA) (He al,, 2016; Chubalc at al., 2020)
79.
25.56
ResNett-101 (A3) (Wightman et al, 2021)
79.81
44.55
EfficienttNet-lite3 (Tha & Le, 2019)
79.81
8.20
RegNet-Y 6.4GF (Radoesvoriec at al., 2020)
79.71
30.58
EfficientNNe-B2 (Tan Le,, 2010)
79.60
9.11
ElficientNet-B11 (w/ AdvProp) (Tan Le, al, 2020s)
79.28
7.79
RegNet-Y 4.0GF (Radanuvoiic
2020)
79.23
20.65
ViT-basc (w/ AugReg) (Donovitalty at al, 2020; Statner al, 2022)
79.15
86.57
ConvNeXt-tiny (IN22k, (I). (Lia al, 2022)
78.90
28.59
RegNet-Y 3.2GF (Radonsvveetcc at al., 2020)
78.88
19.44
ViT-small (w/ AugReg) (Dosovitokiy et al., 2020; Steiner et al., 2022)
78.85
22.05
Efficient.Nt--BB (w/ AutoAugment) (Tan Le, 2019; Cubaak et al,, 2019)
78.83
7.79
EfficientNNtt-30 (w/ Noisy Student, JFT, ft.) (Thn 2019;
78.67
5.29
EfficicntNNetB11
78.56
7.79
ResNet--12 ol., 2016)
78.32
60.19
ResNet-50 (A3) (Wightman 2021)
78.05
25.56
RegNet-Y 1.6GF (Radonsvoviic at al., 2020)
77.86
11.20
ElficientNet-lite22 (Tha & Le, 2019)
77.46
6.09
ResNet-101 (1te et al, 2016)
77.38
44.55
Efficient.Net-BO0 (w/ AdvProp) (Thn Le, 2019: al., 2020s)
77.09
5.29
EfficientNet-BO0 (w/ AutoAugment) (Tan Le, Clubale 2019)
76.83
5.29
EfficientNet-litell (Tho Le,
76.64
5.42
MLP-MixereB (IN21k, ft) (Toleikhi al., 2021)
76.60
59.88
Efficient.Net-B00 (Tan Le, 2019)
76.53
5.29
RegNet- 800MF (Radonavovic et al., 2020)
76.30
6.26
ResNet-50 (Mte al., 2016)
76.13
25.56
ViT-tiny (w/ AugReg, IN21k, ft) (Droovitokly et al., 2020; Steiner al., 2022)
75.45
5.72
RegNet-Y Y 600MF
(Badoravavic 2020)
75.27
6.06
Efficient.Net-iiee
74.83
4.65
RegNet-Y 400MF (Iadoravovic al., 2020)
74.03
4.34
VGG19 (Simonyan & Zisserman, 2015)
72.38
143.67
MLP-Mixer-L (IN21K, ft) (Tiisiihin es al., 2021)
72.05
208.20 20
VGG16 (Somanyan Ziezerman, 2015)
71.59
138.36
RegNet-Y Y 200MF (Radoravavic at al., 2020)
70.28
3.16
VGG13 (Stmonyan Zisserman, 2015)
69.93
133.05
VGG11
Zizeemmn,,
69.02
132.86
ViT-basc (patch 16, MAE)
(Donocttekty
He al., 2022)
NaN
85.80
ViT-huge (patch 14, MAE)
(Deaovisskly et al., 2020; He al., 2022)
NaN
630.80
ViT-large (patch 14, MAE)
(Dreevitakly es sl., 2020; He et al., 2022)
NaN
303.30
Continued on next page
20

---

Published in Transactions on Machine Learning Research (04/2025)
Table A.1: Models from timm used in the experiments with ImageNet-Ik validation accuracy and the number of parameters sorted by ImageNet-lk validation accuracy. "ft" stands for finctuning on ImageNet-1k, Self-supervised pre- pre-trained models present "NaN" accuracy.
Model
IN-1k Acc.
# Param. (x10%)
ViT-small (DINO) (Dorovitskty ala, 2020; Caron
NaN
21.70
ViT-base (DINO) (Duvovitskiy ol., 2020; Caron es al., 2021)
NaN
85.80
CLIP (RosNet-550, Zero-shot) (Radiord at al,, 20213
59.6"
38.31
CLIP (ResNet- 101, Zero-shot) (Radford al., 2021)
62.2
56.26
CLIP (ViT-B/Se //2, Zero-shot) (Radford al., 2021)
63.2"
87.85
A.2 Model Selection and Hyperparameter Search
We adopted the most common model selection schome, "training-domain validation set" of the DomainBed paper (Gulrajani & Lopez-Paz, 2021). We adopted grid search over learning rales from {1.0 10-1,5.0 x 10-1 1.0 10-3 3, 5.0 x 10-3 1.0 10-2, 5.0 10-2, 1.0 x 10-1,5.0 5.0 10-1} and weight-docay rates from [1.0 x 10 1.0 x 10-3 3, 1.0 10-2] of SGD with a momentum rate of 0.9, because Mo SGD is known to be more effective in OOD environment (Naganuma et al., 2023). Additionally, for models trained with adaptive optimizers, such as Adam (Kingma & Ba, 2015) and AdamW (Loshchilov & Hutter, 2017), we also tried the same optimizers and searched learning rates from {1.0 x 10-4, 5.0 x 10-4, 1.0 x 10-3,5.0 x 10-3, 1.0 x 10-2,5.0 5.0 x 10-2, 1.0 x 10-1,5.0 5.0 x 10-1). The batch size and the number of training ilerations were not subjects of our exploration: the batch size was fixed to 32, and the number of training iterations was fixed to 5,000. Figures C.2 and C.3 present the effect of hyperparameter selection on in-domain validation accuracy, which was used to select hyperparameters, and oul-of-distribution test sccuracy, which was used to report the final performance.
4,00
A.3 Experimental Protocol of CLIP Model
Our research deliberately focused on fine-tuning pre-trained models, rather than zero-shot learning with large-scale models like CLIP (Radford et al., 2021). While we recognize the novelty and significance of zero-shot learning, we prioritized maintaining a specific research scope, especially as fine-tuning remains necessary for domain-specific models, even with advancements in Large Language Models and Large Vision Models, shown in (Wei al., 2023). Although is questionable whether the CLIP pre-trained model quantify true domain generalization in terms of pre-training large datasets that contain images corresponding to such domains as the sketch domain, we evaluate the CLIP model on four different datasets from DodainBed.
We conducted the experiments according to Ruan et al.. In particular, and evaluated three different vision encoders for CLIP: ResNet-50", ResNet-101", and ViT-B/32 "
For zero-shot inference, as shown in figure (3) of the paper by Radford et al., input the data to be inferred in the image encoder and photo of a {class}' in the text encoder.
A.4 Datasets
Among the datasets within the DomainBed (Gulrajani & Lopez-Paz, 2021) and WILDS benchmarks (Koh et al., 2021), we selected specific datasets detailed in table A.2 for our experiments. The initial domain
We use IN-IK accuracy reported in Radford et al. (2021)
htt.://www........ooooooooaaaaaa b6762/RK50.p peppmmiippbiic..azeeeeee..oe 16699/R8101.pt http:://ppessjpbiiccseeeeee
/ttp///////////////////////////n 950xf/ViT-B-32.0
21

---

Published in Transactions on Machine Learning Research (04/2025)
outlined in table A.2 served the test domain for each datasct, while the remaining domains were employed for training and validation.
Within DomainBed (Gulrajani & Lopez-Paz, 2021), the VLCS dataset (Fang et al., 2013) compiles natural images from various image datasets, which are treated as distinct domains. For example, in the PACS dataset (Li et al., 2017), a label such as "dog" is associated with several domains including photographs sketches, art, and cartoons, which are used for domain generalization tasks. OfficeHomme (Venkateswara et al., 2017) and DomainNet (Peng al., 2019), similarly, possess domains that analogous in nature; however, OfficeHome focuses objects typically found in offices like chairs, whereas DomainNet includes wider array of objects.
To evaluate task trends beyond visual domain generalization, we examined Camelyon17 (Bandi et al., 2018) from the WILDS benchmark as medical imaging dataset. Camelyon17 includes histopathological images sourced from five different hospitals, each treated as a separate domain.
Dataset
Domains
# Exampls
# #Ctegoories
PACS
art, cartoon, photos, sketches art, clipart, product, real clipart, infograph, painting, quickdraw, real, sketch VOC2007, Caltech101, LabelMe, SUN09,
9,991
7
OfficeHome
15,588
65
DomainNet
586,575
345
VLCS
10,729
5
Camelyon17
Hospital1, Hospital2, Hospital3, Hospital4, Hospital5
455,954
2
Table A.2: Datasets from the DomainBed and WILDS benchmark used in the experiments.
22

---

Published in Transactions on Machine Learning Research (04/2025)
PACS
OfficeHome
DomainNet
VLCS
0.85
lest Accuracy
0.950
OOD Test Accuracy
0.80
OOD Test Test Accuracy
0.75
OOD Test Accurack,
0.900
0.75
0.70
0.80
0.70
0.875
0.78
0.65
0.65
0.850
0.60
0.76
0.625
0.55
0.600
0.50
# parammeers (x10")
#parameters (x10")
paramettrs (x10%)
# pparameters (x1)))
ViT-small (w/ AugRegi
ViT-tiny (w/ AugReg, IN21K, ft)
ViT-base (patch 16, MAE)
VIT-small (DINO)
VIT-base AugReg)
VIT-small (w/ AugReg, IN21K, ft)
ViT-lagge (patch 14, MAE)
ViT-base (DINO)
VIT-base (w/ AugReg, IN21K, (t)
ViT-huge (patch 14, MAE)
VIT-base (SAM)
VIT-arge (w/ AugReg, IN21K, ft)
VIT-base (IN21K, ft)
Figure B.1: Out-of-distribuuion test accuracy of ViT (Dosovitskiy et al., 2020) with various training schemes.
Comparing the same architectures aligned in vertical lines, supervised pre-training outperforms self-supervised pre-training (DINO (Caron et al., 2021) and MAE (He et al., 2022)). Also interestingly, pre-training with the SAM optimizer (Foret et al., 2021) yields inferior performance to its SGD counterpart.
PACS
OfficeHome
DomainNet
VLCS
0.94
0.70
OOD Test Accuracy
0.92
OOD Test Accuracy
0.75
Accuracy
0.65
Test Accuracy
0.90
0.88
0.55
0.76
0.86
0.65
0.50
0.74
0.60
0.45
0.72
0.40
0.55
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
# #parameters (x10%)
# +arameteess (x10*)
#parameters (x10º)
# #parameters (x109)
wd=10-2
wd=10-3
wd=10-<
ResNet
ResNet
ResNet
ConvNexT
ConvNexT
ConvNexT
VIT
VIT
VIT
Figure B.2: OOD lest accuracy with respect to the number of model parameters with different weight decay rates of momentum SGD. For the datasets different from ImageNet, PACS, OfficeHome, and DomainNet, smaller weight-decay rates result in better performance. On the other hand, for datasets similar to ImageNet. and VLCS, larger weight-decay rates improve performance.
Other Findings
This section describes other findings of the factors in the fine-tuning and pre-training phase and correlation behavior of datasets.
B.1 Factors in Pre-training Phase
Through focused analysis, gained several insights into optimal pro-training strategics for ViT architectures. Our key findings are as follows.
For supervised learning, pre-training with data augmentation and regularization through AugReg (Steiner et al., 2022) improves OOD test accuracy. Examining self-supervised training methods, we find models pre-trained with DINO and MAE perform worse than supervised learning on OOD test accuracy. This result that using self-supervised learning in the pre-training phase degrades performance in the downstream task is
23

---

Published in Transactions on Machine Learning Research (04/2025)
in agreement with the results of (Goldblum et al., 2023). More critically, pro-trained with the optimizer of Sharpness Aware Minimization (Foret et al., 2021) exhibit overfitting on ImageNet-1k that degrades OOD generalization, as evidenced by their lower OOD versus in-distribution accuracy (fig. D.15).
B.2 Factors in Fine-tuning Phase
We also performed comprehensive experiments to evaluate optimizer selections for fine-tuning pre-trained models. Prior work by Naganuma et al. (2023) showed SGD with momentum outper- forming adaptive optimizers like Adam for OOD generalization across benchmarks, including DomainBed. However, their experiments DomainBed were limited to ResNet-50 models, leaving the question of optimal optimizers for diverse model architectures open.
Optimizer Selection:
While our main experiments use SGD with momentum, different architectures like EfficientNet and VIT were originally proposed with other optimizers for pre-training. We directly compare fine-tuning these models with their original pre-training optimizers versus SGD with momentum. Our results in fig. C.3 show SGD with momentum achieves superior OOD test accuracy in nearly all cases, with the exceptions ConvNeXt and ViT on DomainNet. The same trend holds for in-distribution validation, as seen in fig. C.2.
In summary, while adaptive optimizers can benefit from pre-training some model architectures, SGD with momentum the most effective optimization strategy for fine-tuning diverse model architectures. This aligns with and expands upon prior finding (Nagamuma al., 2023) that mamtentun-bssee SGD excels for OOD generalization. Our analysis provides practical guidance for fine-tuning optimizer selection.
Regularization: We systematically examined the role of regularization, specifically weight decay, when fine-tuning diverse pre-trained models. Our experiments evaluated OOD test accuracy across range of weight decay coefficients from 1.0 x 10-4 to 1.0 x 10-?, The results in fig, B.2 reveal differing trends between PACS, OfficeHome, and DomainNet versus VLCS.
For the former group, stronger regularization hurts OOD test accuracy while improving calibration (ECE), as shown in fig. C.1. This represents a trade-off between OOD performance and uncertainty estimates. In contrast, for VLCS, increased weight decay improves both OOD accuracy and ECE.
The discrepancy likely stems from datasct differences- former group combines visually distinct domains (art, photo, sketch), while VLCS merges similar photo datasets.
To summarize, regularization does not lead to unified conclusions across distribution shifts. While helpful for calibration, weight decay can degrade OOD accuracy on domain shifts with more separated distributions like PACS.
B.3 Relationship with Other OOD Datasets
While our main experiments use domain generalization datasets like PACS, VLCS, OfficeHome, and Do- mainNet, OOD generalization also encompasses robustness, adversarial examples, sampling bias, and other distribution shifts. Therefore, we evaluated all our pre-trained models on ImageNet OOD test sets (ImageNet- A, ImageNet-R, ImageNet-v2, and ImageNet-Real) for mcasuring robustness and sampling bias effect and analyzed how these results relate to domain generalization performance.
Notably, we find rank correlation between pre-training and fine-tuning performance. Models better ImageNet OOD test sets also achieve higher domain generalization accuracy after fine-tuning (fig. D.9). Further, the correlation exhibits distinct trends improving the last few percent on robustness datasets, i.e., TmageNet-R and ImageNet-A,, give diminishing returns for domain generalization. However, gains on sampling bias datasets, i.e., ImageNet-v2 and ImageNet-Real, contribute substantially.
These trends reveal muanced relationships between how models generalize on different OOD datasets. Perfor- mance robustness benchmarks provides valuable signals, but still differs from the tasks casted to domain generalization.
24

---

Published in Transactions on Machine Learning Research (04/2025)
C Ablation Study
C.1 The Effects of Random Seeds
Table C.1 demonstrates the mean and standard deviation of in-domain validation accuracy and out-of- distribution test accuracy of models that achieved the highest validation performance over three runs with different random seeds. Again, only selected models are presented to avoid verbose tables. Notice that the standard deviation of both in-domain validation accuracy and out-of-distribution test accuracy so small that our conclusion is valid.
Table C.1: Mean and standard deviation of in-domain validation accuracy and out-of- distribttibiton test accuracy. Models are trained with SGD with a learning rate of 1.0 x 10-1 and a moment rate of 0.9 over three different random seeds.
PACS
Model
In-domain validation accuracy
Out-of-distribattio test accuracy
ResNet-50 (3te al., 2010)
96.46 0.19
82.83 1.32
ResNet-101 (ir et al.. 2019)
0.39
88.17 0.94
ResNet-152 (Hhe al. 2019)
96.49 1 0.64
89.20 1 0.52
ConvNeXl-small
97.69 1 0.09
92.60 1 0.15
ConvNeXt--bas
at. 2022)
97.76 1 0.40
93.51 1 0.38
ViT-small (w/ AugRog) (Unsontraity et al., 2010 bretnee al., 2022)
94.72 0.18
84.94 0.08
ViT-base (w/ AugReg) (Dansviiikly en 2023, Steiie al. 2002)
95.70 0.15
86.99 0.49
OfficeHome
Model
In-domain validation accuracy
Out-of-distribatiion accuracy
ResNet-50
88.98 +: 0.38
60.49 +: 0.73
ResNet-101 (Ht 2010)
89.48 0.22
64.86 + 0.27
ResNet-152 (tr et 2019)
0.45
65.42 1.18
Con-NeXX-ssmall (Lin al. 2022)
92.28 1 0.27
76.15 1 0.86
ConvNeXt-bees at al. 9002)
92.43 + 0.12
77.20 = 0.77
ViT-small (w/ AugReg) (Danorisklly
90.22 + 0.39
63.89 = 0.97
ViT-base (w/ AngReg) (Donsditkkl os 2005, Steiims al, 2022)
90.90 0.11
69.86 0.55
NomainNet
Model
In-domain validation accuracy Out-of-distribattio test accuracy
ResNet-50 (Hte 2016)
52.12 + 0.36
56.96 1 0.31
ResNet-101
53.95 +: 0.22
58.29 i: 0.30
ResNet-152 (Ht 2010)
55.44 1 0.18
59.84 1 0.51
ConvNeXt-smas (Ein all, 2022)
63.28 0.01
68.40 0.26
ConyNeXt- base (Lin al., a002)
63.98 0.13
69.49 = 0.43
ViT-small (w/ AngReg) (Dannnitskly 9000, Stoiies al.
56.03 + 0.25
59.07 + 0.74
ViT-base (w/ AugReg) (Dansiitkii
59.80 + 0.09
64.20 = 0.19
VLCS
Model
In-domain validation accuracy
Out-of-distribttion test accuracy
ResNet-50 (Hte wl., 2016)
85.26
73.91 1.29
ResNet-101 mi as at. 2016)
83.69 0.33
72.72 + 2.00
ResNet-152
2010)
81.61 + 0.62
73.73 = 2.05
ConvNeXt-smmal
87.45 + 0.15
79.61 = 0.26
Conv-NXt-basse (1in 2022)
87.12
77.93 0.28
ViT-small (w/ AugReg) (Danovitdlly et al., 8020, Stoiies al., 8008)
86.61 0.50
77.73 + 0.22
ViT-base (w/ AugRog) (Dasovtbakly et 2020: Bcetner al. 2022)
85.68 1 0.45
77.74 1 0.48
25

---

Published in Transactions on Machine Learning Research (04/2025)
C.2 The effects of Hyperparameters
Figure C.1 shows expected calibration error rales with respect to model sizes with different weight decay
rates of momentum SGD. Unlike outto--distribution test accuracy in fig. B.2, clear trends cannot be found.
PACS
OfficeHome
DomainNet
VLCS
Expected Calibration Error
Expected Calibration Error
Expected Calibration Error
Expected Calibration Error
0.14
0.25
0.200
0.175
0.20
0.10
0.150
0.15
0.100
0.125
0.100
0.06
0.075
0.075
0.04
0.050
0.050
0.05
0.025
0.025
0.5
1.0
1.5
2.0
2.0
0.5
1.
1.5
2.0
#parameters (x10%)
#parameters (x10%)
#parameters (x10*)
# #parameters (x10%)
wd=10-2
wd=10-3
wd=10-1
ResNet
ResNet
ResNet
ConvNexT
ConvNexT
ConvNexT
VIT
VIT
Figure C.1: Out-of-distribution expected calibration error with respect to the number of model parameters with different weight decay rates of momentum SGD.
Figures C.2 and C.3 present the effect of hyperparameter selection on in-domain validation accuracy, which was used to select hyperparameters, and out-of-distribution test accuracy, which was used to report the final performance. These figures only show results of ResNet-50, ConvNeXt-base, ViT-base (w/ AugReg), EfficientNet-BB to avoid verbosity, and AdamW for ConwNeXt-base is denoted as Adam in fig. C.2 and fig. C.3 for simplicity. Aligned with fig. 2, we can observe that in-domain validation accuracy is good estimate out-of-disuributtio test accuracy, regardless of the model architectures. Also, we notice that SGD with weight-decay rate of 10-4 consistently achieves (nearly) the best validation and test accuracy regardless of learning rates.
26

---

Published in Transactions on Machine Learning Research (04/2025)
ResNet-50
PACS
OfficeHome
DomainNet
VLCS
Val Accurac)
ID Val Accuracy
0.8
Val Accuracy
0.5
Val Accuracy
0.8
0.8
0.6
0.6
0.4
0.6
0.4
0.4
0.2
0.2
0.1
0.2
0.0
10
10
10
10
learning rate
leaming rate
leaming rate
learning rate
ConwNeXt-base
PACS
OfficeHome
DomainNet
VLCS
Val Accuracy
Val Accuracy
0.8
Val Accuracy
0.6
Accuracy
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.0
10
10
10-3
10-1
10-3
10
10-3
10-1
learning rate
learning rate
leamiing rate
learning rate
ViT-base (w/ AugReg)
PACS
OfficeHome
DomainNet
VLCS
1.0
DD Val Accuracy
0.8
ID Val Accuracy
0.8
ID Val Accuracy
Val Accuracy
0.6
0.6
0.4
0.6
0.4
0.4
0.4
0.2
02
0.0
0.0
0.0
0.0
10
10
10
10-3
10"
10
learning rate
leamingg rate
leaming
learning rate
EfficientNet-B3
PACS
OfficeHome
DomainNet
VLCS
1.0
Val Accuracy
Val Accuracy
0.8
Val Accuracy
0.5
Val Accuracy
0.8
0.6
0.6
0.4
0.6
0.4
0.4
0.4
0.2
0.2
learning rate
leaming rate
leaminng rate
learning rate
SGD (wd-0.01)
SGD (wd-0.001)
SGD (wd-0.0001)
RMSPRop
Adam
Figure C.2: In-domain validation accuracy with different hyperparameters of optimizers. "wd" stands for the weight decay rate. For simplicity, we convert "NaN" value to 0.
27

---

Published in Transactions on Machine Learning Research (04/2025)
ResNet-50 DomainNet
OOD Test Accuracy
PACS
OOD Test Accuracy
OfficeHome
OOD Test Accuracy
OOD Test Accuracy
80
VLCS
80
80
60
60
40
40
20
20
10
10
10
learning rate
learning rate
learning rate
learning rate
ConvNeXt-base
OOD Test Accuracy
PACS
OOD Test Accuracy
OfficeHome
OOD Test Accuracy
DomainNet
OOD Test Accuracy
VLCS
Reg
80
80
60
80
60
40
20
10-3
10
10-3
10-1
10-3
10-1
10th
10-1
learning rate
learning rate
learning rate
learning rate
ViT-base (w/ AugReg)
OOD Test Accuracy
PACS
OOD Test Accuracy
OfficeHome
Test Accuracy
DomainNet
Test Accuracy
VLCC
80
80
60
60
60
40
40
20
20
OOD
10-1
10-J
10-3
10-3
10-1
learning rate
learning rate
learning rate
learning rate
EfficientNet-B3
OOD Test Accuracy
PACS
OOD Test Accuracy
OfficeHome
Test Accuracy
DomainNet
OOD Test Accuracy
VLCS
80
5
60
40
40
20
ODD
10
10
-0
10-4
10
10
10
10:1
learning rate
leaming rate
learning rate
learning rate
SGD (wd=0.01)
SGD (wd=0.001)
SGD (wd=0.0001)
RMSPRop
Adam
Figure C.3: Qut-of-distiibution validation accuracy with different hyperparameterss of optimizers. "wd" stands for the weight decay rate. For simplicity, we convert "NaN" value to 0.
28

---

Published in Transactions on Machine Learning Research (04/2025)
Expected Callbration Error
PACS
Expected Calibration Error
OlficeHome
Expected Calibration Error
DomainNet
Expected Callbration Error
VLCS
Adaptive Calibration Error Exported Calibration Error
WILDSCammllon
2.30
8:25-
0.030
0.025
0.09
0.29
0.020
0:15
6.15
0.09
6:35
0.055
6.9
0.000
0.05
0.5
#parameters (x10")
#parameters (x10*)
#parametters (x10*)
#parameters (x10%)
#parametters (x10*)
Adaptive Callbration Error
PACS
Adaptive Cajibration Error
OfficeHome
Adaplive Calibration Error
DomainNet
Adaptive Cal Fallbration Error
VLCS
WILDSCamelyon
0.030
0.025
0.08
6.20
0.000
0.15
0.033
0.04
0.10
0.03
2.05
0.02
0.05
3005
0.09
#parameters (x10%)
#parameters (x10%)
@parameters (x109)
#parameters (x10%)
#parameters (x10%)
RegNetal
Efkcientee-64
ConvieXt-smaal
Auglteg)
RegNetY
RegNetYY
ConvNeX: tiny (IN22K ft)
AugRog)
ResNet-152
RegNeta 800MF
RegNet-Y 16GF
ConviveXt-liny
ConvNext-smi small (IN22K,
VIT-tiny (wi Augileg, IN21K,
ResNet-S0 50 (v2)
RegNete 1. L.GG
RegNet-Y 32GF
ConvteeX--mall
ConviteXt- base (IN22K, ft
Vil-small Owr Augiteg, IN21K, Nt
ResNet-101 (v2)
RegNetY 3.2GF
EffiientNet.BB BO
ConvllcXt base
ConvNeXtilarge (N22K, Rt]
Vitbaaee twi AugRog, IN21K 10)
ResMet-
4.0GF
EffccentNeet82
Convtlext-lagge
ComWNeXt-ttagge (IN22,
ViT-large AugRog, IN23K,
RegNet/Y RegNete
ConviieXt-inn
Figure D.1: Expected calibration error (top) and adaptive calibration error (bottom) with respect to the
number of parameters.
D Additional Results
D.1 Expected Calibration Error Adaptive Calibration Error
Figure D.1 compares expected calibration error (ECE) and adaplive calibration error (ACE). From this figure, we can confirm that ECE and ACE show almost the same values within the scope of our experiments across multiple model architectures and multiple datasets.
D.2 Terralncognita Dataset Results
Figure D.2 shows OOD test error, ECE, and ACE on the Terraincognita dataset with the test environment of "L38". Similar trends to the other datasets can be observed.
D.3 Choice of Test Environments
Figures D.3 and D.4 show OOD test error and ECE evaluated on OfficeHome and PACS with different test enviromments. We can observe similar trends regardless of the choice of test environments. The reason that the behaviors of OfficeHome (test env = Product) and PACS (test env = Photo) look different from others at a glance, e.g., ConvNeXt-xlarge yields poorer performance than smaller ones on OfficeHome (test env = Product), could be attributed the saturation of the performance in these environments (see the error rates).
D.4 Full Results of DomainBed
Figure D.6 presents the relationship between the number model parameters and out-of-distribution test accuracy (top) and expected calibration error (bottom) with all modols considered in this work. This figure is the full version of fig.
Figure D.5 shows the OOD test error and expected calibration error with respect to the number of parameters all four da tasets, corresponding to the left panels of fig. 3.
29

---

Published in Transactions on Machine Learning Research (04/2025)
Terralncognita
Terralncognita
Terralncognita
Expected Calibration Error
Adaptive Callbration Error
0.6
OOD Test Test Error
0.70
0.5
0.5
0.60
0.3
0.3
0.55
# pparameters (x10*)
#parameters (x10%)
#parameters (x10*)
ResNet-50
RegNet-Y 400MF
RegNet-Y 8GF
EfficientNet-83
ConvNeXt-base (IN22K,
ResNet-101
RegNet-Y 600MF
RegNet-Y 12GF
EfficientNet-B4
ConvNext-large (IN22K, ft)
ResNet-152
RegNet-Y B0OMF
RegNet-Y 16GF
EfficientNet-85
ViT-tiny AugReg, IN21K,
ResNet-50 (v2)
RegNet-Y 1.6GF
RegNet-Y 32GF
ConvNeXt-tiny
VIT-small (w/ AugReg, IN21K, (t)
ResNet-101 (v2)
RegNet-Y 3.2GF
EfficientNet-80
ConvNext-smail (IN12K, ft)
VIT-base (w/ AugReg, IN21K,
ResNet-152 152 (v2)
YegNet-Y 4.0GF
EfficientNet-Bi B1
ConvNeXt small (IN22K, ft)
ViT-large (w/ AugReg, IN21k, ft)
RegNet-Y 200MF
RegNet-Y 6.4GF
EfficientNet-82
Figure D.2: OOD test error and calibration error metrics on Terralncognita.
OfficeHome (Art)
OfficeHome (Clipart)
OfficeHome (Photo)
OfficeHome (Product)
0.225
0.06
00D Test Error
0.40
OOD Test Error
0.40
ODD Test Error
0.200
OOD Test Error
0.05
0.175
0.150
0.30
0.30
0.125
0.03
0.100
0.02
#parameters (x10*)
# #parameters (x10*)
parrameters (x10*)
#parameters (x10*)
Expected Calibration Error
OfficeHome (Art)
Expected Calibration Error
OfficeHome (Clipart)
Expected Calibration Error
OfficeHome (Photo)
Expected Calibration Error
OfficeHome (Product)
0.25
0.25
0.20
0.20
0.10
0.08
0.020
0.06
0.015
0.10
0.10
0.04
0.010
0.005
1
#parameters (x10*)
#parameters (x10*)
# #parameters (x10²)
#parameters (x108)
ViT-small (w/ AugReg)
ConNNeXt-base
ConvNeXt-bars (IN22k, Rt)
VIT-base (IN21k, ft)
ConvNeXt-small (IN12K,
ConvNext-large (IN22k, ft)
ConvNext:tiny
ConwNeXt-small (IN22k,
ConvNeXtsxlarge (IN22k, ft)
ConvNeXt-smal
Figure D.3: OOD test error (top) and expected calibration error (bottom) evaluated on OfficeHome with different test environments.
30

---

Published in Transactions on Machine Learning Research (04/2025)
PACS (Art)
PACS (Cartoon)
PACS (Photo)
PACS (Sketch)
0.250
0.012
0OD Test Error
0.4
OOD Test Error
0.225
Error
0.010
Test Error
0.4
0.200
0.3
0.008
0.3
0.006
0.2
0.150
0.125
0.002
0.100
0.000
0.1
0.5
1.0
1.5
2.0
1.0
#parameters (x10*)
# #parameters (x10f)
#parameters (x10*)
# #parameters (x109)
Expected Calibration Error
PACS (Art)
Calibration Error
PACS (Cartoon)
Expected Calibration Error
PACS (Photo)
Expected Calibration Error
PACS (Sketch)
0.12
0.25
0.20
0.010
0.10
0.008
0.15
0.08
0.806
0.15
Expected
0.06
0.004
0.10
0.05
0.002
0.04
3
0.5
2.0
0.5
1.0
1.5
2.0
2.0
#parameters (x10*)
#parameters (x10*)
#parameters (x10*)
# #parameters (x10%)
VIT-small AugReg)
ConvNeXt-small
ConvNeXl-small (IN22k, ft)
VIT-base (w/ AugReg)
ConvNeXt-base
ConvNeXt-base (IN22K,
ViT-base (IN21k, ft)
ConvNeXt-small (IN12k, (t)
ConvNeXL-large (IN22k, ft)
ConvNeXt-tiny
Figure D.4: OOD test error (top) and expected calibration error (bottom) evaluated on PACS with different. test environments.
31

---

Published in Transactions on Machine Learning Research (04/2025)
PACS
OfficeHome
DomainNet
VLCS
0.30
0.5
0.40
Test Enror
0.25
ODD Test Test Error
0OD Test Emro
0.45
OOD Emor
0.35
0.40
0.15
0.10
0.30
0.2
0.25
107
10a
10'
10a
10'
1a'
10a
# #parameters (x10%)
#parameters (x10*)
#parameters (x10²)
# #parameters (x10*)
Expected Calibration Error
PACS
Expected Calibration Error
OfficeHome
Expected Callbration Erro
DomainNet
Expected Callbration Emor
VLCS
0.10
0.25
0.25
0.08
0.20
0.15
0.06
0.15
0.04
0.05
0.05
0.02
10²
10a
10a
10²
10'
10a
# #parameters (x10*)
#parameters (x10*)
#parameters (x10%)
#parameters (x10*)
ResNet-50
RegNet-Y
RegNet-Y
ConvNeXt-small
ConvNeXt-large (IN22K, ft)
ResNet-101
RegNet-Y 1.6GF
EfficientNet-B0
ConvNeXt-xlarge (IN22K,
ResNet-152
RegNet-Y 3.2GF
EfficientNet-l BB
ConvNeXt-large
VIT-smal (w/ AugReg)
ResNet-50 (v2)
RegNet-Y 4,0GF
EfficlentNet-B2
ConvNeXt-tiny (IN12K, ft)
VIT-base (w/ AugReg)
ResNet-101 (v2)
RegNet-Y 6.4GF
EfficientNet-83
ConvNeXt-small (IN12k, ft)
VIT-tiny (w/ AugReg, IN21k ft)
ResNet-152 (v2)
RegNet-Y 8GF
EfficientNet-B4
ConvNeXt-tiny (IN22K, tt)
VIT-small (w/ AugReg, IN21k tt)
RegNet-Y 200MF
RegNet-Y 12GF
EfficientNet-85
ConvNeXt-small (IN22k, ft)
VIT-base (w/ AugReg, IN21k, ft)
RegNet-Y 400MF
RegNet-Y 16GF
ConvNext-tiny
ConwNeXt-baae (IN22K, (t)
VIT-large (w/ AugReg, IN21K, (t)
RegNet-Y
Figure D.5: Out-of-distributiio test error (top) and expected calibration error (bottom) with respect to the
mumber of parameters. This figure plots all models used in the experiments.
32

---

Published in Transactions on Machine Learning Research (04/2025)
PACS
OfficeHome
DomainNet
VLCS
0.8
Test Accuracy
Test Test Accuracy
OOD Test Accuracy
Test Accuracy
0.8
0.6
0.75
0.7
0.80
0.70
0.75
0.65
0.70
0.60
0.0
2
4
# #parameters (x10%)
# #parameters (x10*)
parameters (x10%)
# #parameters (x10*)
Expected Calibration Error
PACS
Expected Calibration Error
OfficeHome
Expected Callbration Error
DomainNet
Expected Calibration Error
VLCS
0.35
0.10
0.20
0.08
0.25
0.15
0.25
0.20
0.15
0.10
0.04
0.15
0.10
0.05
0.10
0.02
0.05
0.00
# #parameters (x10²)
#parameters (x10%)
#parameters (x10*)
# #parameters (x10*)
Restiet:50 50
EffiientNet-82
Effic EttidientNeetlie lite
ResNet-50 (D)
(RA)
Restiet: 152
EfficientNet-84
EfficientNee--iee lite
ResNet- 50 (AugMix)
ResNet-50 (v2)
EffiiienNNet855
EffidientNe--Re4
ResNet-101 (A1)
ResNet-101 (v2)
EfficienNNet-00 (w/ AutoAugment)
RegNet-Y 200MF
ResNet-101 (A1H)
ResNot-152 (v2)
EffiiennNaa-31 (w/ AutoAugment)
RegNettY 400MF
ResNet-101 (A2)
ConviVeXt-lnn
EfficienNNe-922 (w/ AutoAugment)
RegNet-Y 600MF
ResNet-101
Conv/eett-small
EfficientNee-33 (w/ AutoAugment)
RegNet-Y 800MF
ResNet-152 (A1)
ConviVeXt-base
EfficientNet-44 (w/ Auto/ugment)
RegNet-Y 1.6GF
ResNet-152 (A1H)
lonwVeXt.large
Efficientlee-B 85 (w/ AutoAugment)
RegNet 3.2GF
ResNet: 152 (A2)
Convilettttiny (IN12K, ft)
EfficienNeet-00 (w/ AdvProp)
RegNetY 4,0GF
ResNet-152
ConvNeXt-tnn (IN22k, 1t)
EmcientNet-a11 (w/ AdvProp)
RegNet-Y 6.4GF
VIT-base (patch 16, MAE)
ConvNeett-smlll
EmfiinntNe--22 AdvFrop)
RegNet-Y
VTT-huge MAE)
ConvNeXt-small [IN12kk. R1
EmcientNat-a3 (w/ AdvProp)
RegNet-Y 12GF
VIT-large (patch 14, MAE)
Conv/eXt-base (IN22K,
EfficientNet-44 AdvProp)
RegNet-Y 16GF
(DINO)
ConvNeXt-large (IN22K, FU)
EfficienNNet-55 (w/ AdvProp)
RegNet-Y 32GF
VIT-base (DINO)
Conw/eett>xarree
EfficientNet-00 (w/ Noisy Student,
(SAM)
VIT-small (w/ AugRog)
EfficienNee-81 31 (w/ Noley Student, JFT, ft)
ResNet-50 (A1H)
VIT. base (IN21k, ft)
(wl AugReg)
(w/ Noisy Student, JFT, ft)
(A2)
ViT.tiny (wi Augieg, IN21K, ft)
EfficienNeet-83
33 (w/ Noisy Student, JFT, ft)
ResNet-50 (A3)
VGG16
VIT-small AugReg, IN21k, ft)
EMfcientNet-44 (wi Noisy Student, (t)
ResNet-50 (81K)
VGG13
VIT-base twl AugReg, IN21 (1)
EmccienNNet-55 (w/ Noisy SFudent. (t)
ResNet-50 (82K)
VGG11
VIT-large tw/ AugReg, IN21K,
EmcientNet-85 (w/ RandAugment)
ResNet-50 (C1)
MLP-MixereB (IZ2IK
EffiieetNee-B0
EffiienNNetiite0
ResNet-50 (C2)
MLP-Mixer-L (IN21K, ft)
EffiientNet-81
Figure D.6: Out-of-distiibutiio lest accuracy (Lop) and expected calibration error (bottom) with respect to the number of parameters. This figure plots all models used in the experiments.
33

---

Published in Transactions on Machine Learning Research (04/2025)
Figure D.7 presents expected calibration error rates with respect to pretraining dataset sizes, corresponding to the right panels of fig. 3.
PACS
OfficeHome
DomainNet
VLCS
00D Test Accuracy
0.96
OOD Test Accuracy
OOD Test Accuracy
Test Accuracy
0.750
0.94
0.60
0.725
0.81
0.700
0.90
0.80
0.88
0.70
0.79
0.650
0.78
0.625
0.84
0.77
10
5
10
10
10
Dataset size (M)
Dataset size (M)
Dataset size (M)
Dataset size (M)
Expected Calibration Error
PACS
Expected Calibration Error
OfficeHome
Expected Calibration Error
DomainNet
Expected Calibration Error
VLCS
0.225
0.200
0.08
0.175
0.06
0.150
0.06
0.10
0.125
0.08
0.04
0.100
0.04
0.06
0.03
0.02
Dataset size (M)
Dataset size (M)
Dataset size (M)
Dataset size (M)
VIT-small (w/ AugReg)
ConsNeXt-small
ConvNeXt-tiny (IN22k. ft)
VIT-base (w/ AugReg)
ConvNeXt-base
ConvNeXt-tiny (IN22k, ft)
VIT-small AugReg, IN21K, ft)
ConvNeXt-large
ConvNeXt-small (IN22K,
VIT-base (w/ AugReg, IN21K,
ConvNeXt-tiny (IN12k,
ConvNeXt-base (IN22k,
ConvNext-tiny
ConvNeXt-small (IN12k, (t)
ConvNeXt-large (IN22k, ft)
ConvNeXt-tiny
Figure D.7: The relationship between pro-training dataset sizes and expected calibration crror (lower is better). The same model architectures with different pre-training datasets are connected by dashed lines.
34

---

Published in Transactions on Machine Learning Research (04/2025)
Figure D.6 is the version of fig. D.5.
PACS
OfficeHome
DomainNet
VLCS
OOD Test accuracy
0.95
OOD Test accuracy
Test accuracy
Test accuracy
0.8
0.80
0.90
0.7
0.80
0.70
0.75
0.65
0.65
0.80
0.85
0.70
0.75
0.80
0.85
0.70
0.75
0.80
0.85
ImageNet-1K accuracy
ImageNet-1l accuracy
ImageNet-1k accuracy
ImageNet-1k accuracy
Expected Calibration Error
PACS
Expected Calibration Error
OfficeHome
Expected Calibration Error
DomainNet
Expected Calibration Error
VLCS
0.35
0,30
0.20
0.30
0.08
0.25
0.06
0.20
0.15
0.10
0.04
0.15
0.10
0.05
0.10
0.02
0.05
0.05
0.00
0.70
0.75
0.80
0.85
0.70
0.75
0.80
0.85
ImageNet-1k accuracy
ImageNet-1k accuracy
ImageNet-1k accuracy
ImageNet-1k accuracy
EfficientVet-liel
(D)
ResNet-101
EfficientNet-83
EffidientNet-itt2
ResNet-50 (RA)
EfficientNet-B4
EffideenNett-iee
(AugMix)
ResNett55 (v2)
EfficientNet.55
efficientNNe- lte4
ResNet-101 (A1)
ResNet- 101 (v2)
(w/ AutoAugment)
RegNet-Y
(A1H)
Restiet: 152 (v2)
EfficientNet-11 (w/ AutoAugment)
RegNetYY 400MF
ResNet: 101 (A2)
Conv/eetttiny
EfficienNNet922 (w/ AutoAugment)
RegNetoY 600MF
ResNet-101 (A3)
ConvNext-small
EfficienNNet-33 (w/ AutoAugment)
RegNet-Y 900MP
ResNet-155 (A])
ComvNeXt-basse
EfficientNet-84 (w/ AutoAugment)
RegNet-Y 1.6GF
ResNet-152 (A1H)
Comvieett-lrrge
EfficienNNet-55 (w/ AutoAugment)
RegNet-Y 3.2GF
ResNet-152
ConvileXt-tiny (IN12K, ())
EfficientNet-30 (w/ AdvProp)
RegNet-Y 4.0GF
ResNet-152 (A3)
Convilett-inn (IN22k,
EfficienNee-a11 (w/ AdvProp)
RegNety 6.4GF
VIT-base
ConvNeXt small (IN22k, ft)
Efficentbeet-B 82 (w/ AdvProp)
RegNet 8GF
VIT-huge (patch 14, MAE)
Conviiett small
EfficientNet-33 (w/ AdvProp)
RegNetY 12GF
VIT-large (patch MAE)
ConvNeXt-basee (IN22K, (t)
EmcieenNNe--A (w/ AdvProp)
RegNet-Y 16GF
VIT-small (DINO)
ConvNeett-arge (IN22k,
EffiiietNee-855 AdvFrop)
RegNetaY 32GF
(DINO)
ConlWeXt-xdlaree (IN22k, (t)
EmcieentNet-00 (w/ Noisy Student, JFT, (t)
ResNet-50 (A1)
VIT-base (5AM)
AugReg)
EfficienNNet-11 (w/ Noisy Student, JFT,
(A1H)
(IN21K,
VIT-base (w/ AugReg)
EfficientNet-22 (w/ Noisy Student, JFT, (t)
ResNet-50 (A2)
VGG19
VIT-tiny (wi AugReg, IN21K,
EfficientNet-83 (w/ Noisy Student,
(A3)
VIT-smali (w/ AugReg, IN21K, (t)
EfficcnnNee-34 84 (w/ Noley SFudent. ft)
ResNet-50 (BIK)
VGG13
VIT (wl AugRog, (N21K, ft)
EfficienNee-855 (wi Noiey Student, JFT, ft)
(82K)
VIT-large (w/ AugReg, IN21K,
EmccienNee-955 (w/ RandAugment)
ResNet-50 (C1)
MLP-MixereB (NZ2I,
EfficienNNe-B0
EmcieetNeccRtee
ResNet-50 (C2)
MLP-Mixer-L (IN21K, Rt)
BffiiienNee-a1
Figure D.8: The relationship of out-of-distribution test accuracy (top) and expected calibration error (bottom) with ImageNet-Ik 1k accuracy. This figure plots all models used in the experiments.
35

---

Published in Transactions on Machine Learning Research (04/2025)
Figure D.9 shows the relationship between out-of-distribution test accuracy after finetuning and ImageNet- OOD test sets, namely, ImageNet-A (Hendrycks et. al., 2021b), ImageNet-R (Hendryeks et al., 2021a), ImageNet-RRal (Beyer et al., 2020), and ImageNet v2 (Recht et al., 2019), before finetuning.
PACS
OfficeHome
DomainNet
VLCS
Accuracy
OOD Test Accuracy
0.8
OOD Test Accuracy
OOD Test Accuracy
0.90
0.70
0.75
0.85
0.7
0.65
0.70
0.80
0.60
0.65
0.75
0.55
0.50
0.0
0.6
ImageNet-A Accuracy PACS
ImageNet-A Accuracy OfficeHome
ImageNet-A Accuracy DomainNet
ImageNet-A Accuracy
VLCS
0OD Test Accuracy
0.95
OOD Test Accuracy
0.8
OOD Test Accuracy
0.75
0DD Test Accuracy
0.80
0.70
0.75
0.85
0.7
0.65
0.80
0.6
0.65
0.70
0.5
0.60
0.3
0.4
0.5
0.6
0.3
0.6 ImageNet-R Accuracy
0.4
0.5
0.3
0.4
0.5
0.6
0.3
0.4
0.5
0.6
ImageNet-R Accuracy
ImageNet-R Acauray
Accuracy
ImageNet-R Accuracy
PACS
OfficeHome
DomainNet
VLCS
Test Apcuracy
OOD Test Accuracy
0.8
ODD Test Accuracy
O0D Test Accuracy
0.80
0.90
0.75
0.85
0.7
0.65
0.70
0.60
0.75
0.55
0.50
0.60
0.80
0.30
ImageNet-Real Accuracy
ImageNet-Real Accuracy OfficeHome
ImageNet-Real Accuracy DomainNet
ImageNettRRal Accuracy
PACS
VLCS
Test Accuracy
0.95
OOD Test Accuracy
0.8
OOD Test Accuracy
0.75
ODD Test Accuracy
0.80
0.90
0.70
0.85
0.05
0.70
0.80
0.6
0.65
0.70
0.5
0.60
0.60
0.65
0.70
0.75
0.60
0.65
0.70
0.75 0.60
0.65 0.70 0.75
0.60
0.65
0.70
0.75
ImageNet-v2 Accuracy
ImageNet-v2 Accuracy
ImageNet-v2 Accuracy
ImageNet-v2 Accuracy
ResNet-50
RegNet-Y 800MF
RegNet-Y 32GF
ConvNeXt-small
ConvNext-large (IN22K,
ResNet-101
RegNet-Y 1.6GF
EfficientNet-B0
ConvNeXt-base
ConwNeXt-xlarge (IN22k,
ResNet-1152
RegNet-Y 3.2GF
Erficlentier-a 81
ConvNeXt-large
VIT-small (w/ AugReg)
ResNet-50 (v2)
EfficientNet-22
ConvNeXt-tiny (IN12k, (t)
VIT-base (w/ AugReg)
ResNet-101 (v2)
RegNetY 6.4GF
EfficientNet 83
ConvNeXt small (IN12k, ft)
VIT.tiny (w/ AugReg, IN21k, ft)
ResNet-152 (v2)
RegNet-Y 8GF
EfficientNet-64
ConvNeXt-tiny (INZZk, ft)
VIT-small (w/ AugReg, IN21K. ft)
200MF
RegNet-Y 12GF
EMficientNet-55
ConwNeXt-small
VIT-base AugReg,
RegNetY 400MF
RegNet-Y 16GF
ConvNeXIU-inny
ConvNeXt-base (IN22k, R0
ViT-large (w/ AugReg, IN21k, ft)
RegNet-Y
Figure D.9: Relationship between out-of-distrrbution test accuracy after finetuning and ImageNet-Ik OOD test sets before finetuning.
Figure D.10 displays the relationship between accuracy on ImageNet- validation and ImageNet OOD test sets.
36

---

Published in Transactions on Machine Learning Research (04/2025)
0.7
ImageNet Accuracy
0.6
Net-R Accuracy
0.65
0.60
ImageNet-Red Re-RRall Accuracy
0.88
ImageNet-v2 Accuracy
0.75
0.5
0.55
0.70
0.4
0.84
0.65
0.40
0.2
0.80
0.35
0.60
0.30
0.0
0.25
0.76
0.55
0.75
0.75
0.70
ImageNet-Ik Accuracy
ImageNet- Accuracy
ImageNet-Ik Accuracy
ImageNet-1k Accuracy
ResNict-50
EfficienNee-a2
Efficienthet lite
ResNet-50 (D)
ResNet-100 101
EfficientNet.833
EfficientNett lnc2
ResNet
Restict: 152
EfficientMet-84
EfficientNet lite
ResNet: 50 (AugMix)
ResNet-50
EfficienNNet-55
EffideenNet-RRd
ResNet-101
ResNet-101 (v2)
EMfcientNet-800 (w/ AutoAugment)
RegNet-Y 200MF
ResNet-101 (A1H)
ResNet-152
EmfiientNet-81 AutoAugment)
RegNet-Y 400MF
ResNet-101
ConvWeetttiny
EfficientNet-22 (w/ AutoAugment)
RegNet-Y 600MF
ResNet-101 101 (A3)
Conv/VeXt-small
EfficientNet-83 AutoAugment)
RegNet-Y
ConvNeXt-baase
EfficientNet-ad (w/ AutoAugment)
RegNet-Y 1.6GF
ResNet-152 (A1H)
ConvNexttlarge
(wi AutoAugment)
RogNett
ResNet-152 (A2)
Conv tiny (IN12K, ft)
EfficientNet-80 (w/ AdvProp!
RegNetYY 4.0GF
ResNet-151 152 (A3)
ConvNeXtttiny (IN224, (t)
EffiiienNee-91 (wi AdvProp)
RegNettY 6.4GF
VIT-base (patch 16, MAE)
ConvNeet-small (IN22K.
EmcientNet-32 (w/ AdvProp)
RegNet-Y 8GF
VTT-huge (patch 14, MAE)
ConvNext-small [IN12K,
EMfcientNet-83 (wi AdvProp)
RegNetaY
VIT-large (patch 14, MAE)
Convieett-baee (IN22K,
EfficientNet-44 (wi AdvProp)
RegNet-Y 16GF
VIT-small (DINO)
ConvNeet--lrrge (IN22K, Rt)
EfficientNet-55 (w/ AdvProp)
RegNet-Y 32GF
VIT-base (DIND)
Comvieett-xlrrge (IN22K, (U)
EfficientNet-00 (w/ Noisy Student, JFT, (t)
ResNet-50 (A1)
VIT-base (SAM)
VIT VIT-small (w/ AugReg)
Efficienteee-a 81 (w/ Noley Student, JFT, ft)
ResNet-50 (A1H)
WiT-base (IN21k,
VIT base (w/ AugRog)
EfficientNet-922 (w/ Noley Student, JFT,
ResNet-50
VGG19
VIT tiny (wi AugReg, IN21K, ft)
Efficientlet 83 (w/ Notoy Student, JFT, ft)
ResNet 50 (A3)
VGG16
VIT-small AugReg, VIT-base (w/ AugReg, IN21 (t)
EffiieetNee-84 Noisy Student.
EmcientNet-as (w/ Noisy JFT, (t)
ResNet-50 (82K)
VGG11
AugReg,
EfficienNNet-55 RandAugment)
(C1)
MLP-Mxeere
EffiieetNeet-00
Effiienteetiiee
ResNet-50 (C2)
MLP-Mixer-L (IN21K, R)
EffciietNee-81
Figure D.10: The relationship between ImageNet-lk validation accuracy and accuracy on ImageNet OOD
test sets.
37

---

Published in Transactions on Machine Learning Research (04/2025)
D.5 Results per Model Architecture
Figures D.11 and D.12 show OOD test accuracy and expected calibration error rales of ElficientNets with various training schemes. Different from out-of-disribuuton test accuracy, data augmentation (AutoAugment and AdyProp) or additional unlabeled data (NoisyStudent on JFT) do not always contribute to ECE improvement.
PACS
OfficeHome
DomainNet
VLCS
Test Accuracy
Accuracy
0.700
0 0 Test Test Accuracy
0.68
ODD Test Test Accuracy
0.750
0.725
0.66
0.650
0.625
0.600
0.62
0.650
0.575
0.60
0.625
0.550
0.58
0.600
#parameters (x10?)
#parameters (x10")
#parameters (x10?)
#parameters (x10?)
EfficientNet-B0
EfficientNet-80 (w/ AA)
EfficientNet-80 (w/ AP)
EfficlentNet-80 (w/ NS, ft)
EfficientNet-B1
EfficientNet-81 (w/ AA)
EfficientNet-81 (w/ AP)
EfficientNet-B1 (w/ NS, JFT, ft)
EfficientNet-B2
EfficientNet-82 (w/ AA)
EfficientNet-82 (w/ AP)
EfficientNet-82 (w/ NS. JFT, ft)
EfficientNet-B3
EfficientNet-B3
EfficientNet-83 AP)
EfficientNet-83 (w/ NS,
EfficientNet-BB EfficientNet-B4 (w/ AA)
EfficientNet-84 (w/ AP)
EfficientNet-B4 (w/ NS, JFT,
EfficientNet-B5
EfficientNet-85 (w/ AA)
EfficientNet-85 (w/ AP)
EfficientNet-85 (w/ NS, JFT, ft)
Figure D.11: Dustrr-distribution test accuracy of EfficientNets (Tan & Le, 2019) with various training schemes. AA, AP, and NS stand for AutoAngment (Cubuk et al., 2019), AdvProp (Xie et al., 2020a), and
Noisy Student (Xie et al., 2020b), respectively. JFT is Google's internal unlabeled dalaset consisting of 300
million images.
PACS
OfficeHome
DomainNet
VLCS
Expected Calibration Error
Expected Calibration Error
Expected Calibration Error
Expected Calibration Error
0.08
0.30
0.18
0.26
0.25
0.24
0.06
0.14
0.22
*
0.05
0.12
0.20
0.10
0.18
0.04
0.10
0.08
0.16
0.03
0.05
# #parameters (x107)
#parameters (x10")
#parameters (x10²)
# #parameters (x10')
EfficientNet-B0
EfficientNet-B0
EfficientNet-B0
EfficientNet-B0 (w/ NS, JFT,
EfficientNet-B1
EfficientNet-81 (w/ AA)
EfficientNet-81 (w/ AP)
EfficientNet-81 (w/ NS, JFT,
EfficientNet-B2
EfficlentNet-82 (w/ AA)
EfficientNet-B2 (w/ AP)
EfficientNet-B2 (w/ JFT, ft)
EfficientNet-B3
EfficientNet-83 (w/ AA)
EfficientNet-83 (w/ AP)
EfficientNet-83 (w/ NS, JFT, ft)
EfficientNet-B4 EfficientNet-B4
EfficientNet-S4
EfficientNet-B4 (w/ NS, JFT,
EfficientNet-85
EfficientNet-85 (w/ AA)
EfficientNet-85 (w/ AP)
EfficientNet-85 (w/ NS, JFT,
Figure D.12:
expected calibration error of EfficientNets (Tan & Le, 2019) with various training schemes. AA, AP, and NS stand for AutoAugment (Cubuk et al., 2019), AdvProp (Xie et al., 2020a),
and Noisy Student (Xie et al., 2020b), respectively. JFT is Google's internal unlabeled dataset consisting of 300 million images.
38

---

Published in Transactions on Machine Learning Research (04/2025)
Figure D.13 demonstrates expected calibration error rates with respect to model sizes of ResNets with different, pre-training schemes (Wightman et. al., 2021). Similar to their out-of-distributtio test accuracy in
fig. 5, extensive pre-training schemes (A1-A3) do not always contribute to the ECE improvement.
PACS
OfficeHome
DomainNet
VLCS
Expected Calibration Emor
Expected Calibration Erro
Expected Calibration Erro
Expected Calibration Erro
0.14
0.28
0.05
0.26
0.12
0.04
0.18
0:44
0.22
0.03
0.16
0.08
0.02
0.18
0.12
0.06
0.16
0.01
# #parameters (x10%)
# #parameters (x10*)
#parameters (x10²)
#parameters (x10%)
ResNet-50
ResNet-101 (v2)
ResNet-152 (A1)
ResNet-50 (A3)
ResNet-101
ResNet-152 (v2)
ResNet-50 (A2)
ResNet-101 (A3)
ResNet-152
ResNet-50 (A1)
ResNet-101 (A2)
ResNet-152 (A3)
ResNet-50
ResNet-101 (A1)
ResNet-152 (A2)
Figure D.13: The relationship between model sizes and expected calibration error (lower is better) of ResNets
with different pre-training schemes (Wightman et al., 2021). The original ResNet models are connected by dashed lines.
Figure D.14 shows the relationship between ImageNet-Ik validation accuracy and OOD Lest accuracy of ResNet- 50, pre-trained with various recipes, including Ragmmnt (Cubuk et al., 2020) and AugMix (Hendrycks* et al., 2020).
PACS
OfficeHome
DomainNet
VLCS
Test accuracy
0.90
OOD Test accuracy
OOD Test accuracy
Test accuracy
0.88
0.66
0.78
0.86
0.64
0.62
0.4
0.60
0.76
0.58
0.3
0.56
0.78
0.54
0.2
0.74
0.76
0.78
ImageNet-li 1k accuracy
ImageNet-1k accuracy
ImageNet-1k accuracy
ImageNet-1k accuracy
ResNet-50
R5 (A2)
ResNet-50 (B2K)
ResNet-50 (C2)
ResNet-50
ResNet-50 (v2)
ResNet-50 (A3)
ResNet-50 (C1)
ResNet-50 (D)
ResNet-50 (AugMix)
ResNet-50 (A1)
ResNet-50 (B1K)
Figure D.14: The relationship between ImageNet- 1k validation accuracy and distiiiiiitiituttion accuracy
(higher better) of ResNet-50 with different pre-training schemes, such as (Wightman et al., 2021).
39

---

Published in Transactions on Machine Learning Research (04/2025)
Figure D.15 shows OOD test accuracy with respect to ImageNet-lk validation accuracy of Vision Transformers, pre-trained in supervised learning.
PACS
OfficeHome
DomainNet
VLCS
0.975
0.85
list Accuracy
0.950
Accuracy
0.80
Accuracy
0:75
Test Accuracy
0.82
0.925
0.75
0.900
0.70
0.70
0.80
0.65
0.850
0.60
0.65
0.825
0.55
0.800
0.60
0.50
0.750 0.775 0.800 a. 825 0.850
0.750 0,775 0.800 0.825 8.0 ImageNet-1k Accuracy
0.750 0,775 0.500 0.825 0.850 ImageNet-1k Accuracy
0.750 0,775 0,800 0.825 0.850 ImageNet-1k Accuracy
ImageNet-1k Accuracy
ViT-small (w/ AugReg)
VIT-tiny (w/ AugReg, IN21K, ft)
ViT-base (w/ AugReg, IN21K, ft)
VIT-base (SAM)
ViT-base tw/ AugReg)
VIT-small (w/ AugReg, IN21K,
ViT-large tw/ AugReg, IN21K, ft)
VIT-base (IN21K, ft)
Figure D.15: The relationship between ImageNet-llk validation accuracy and out-of-distribution of Vision
Transformers trained in supervised learning. SAM an optimizer, standing out for Sharpness Aware Minimization (Foret al., 2021).
Figure D.16 shows the expected calibration error of Vision Transformers with various training schemes, including self-supervised learning, namely DINO (Caron al., 2021) and MAE (He et al., 2022), corresponding to fig. B.1.
PACS
OfficeHome
DomainNet
VLCS
Expected Calibration Error
Calibration Error
Expected Calibration Error
Expected Callbration Erro
0.175
0.25
0.06
0.10
0.20
0.125
0.08
0.04
0.100
0.06
0.15
Expected
0.03
0.075
0.10
0.02
0.050
0.02
0.025
#parameters (x 108)
# #parameters (x10*)
parameters (x109)
#parameters (x10*)
VIT-small (w/ AugReg)
ViT-tiny (w/ AugReg, IN21k, ft)
ViT-base (patch 16, MAE)
VIT-smail (DINO)
ViT-base (w/ AugReg)
ViT-small (w/ AugReg, IN21K, ft)
ViT-large (patch MAE)
ViT-base (DINO)
VIT-base AugReg, IN21K, ft)
ViT-huge (patch 14, MAE)
(SAM)
ViT-lagee (w/ AugReg, IN21K. ft)
ViT-base (IN21k, ft)
Figure D.16: Out-of-distribution expected calibration error of Vision Transformers (Dosovitskly et al., 2020) with various training schemes. Comparing the same architectures aligned in vertical lines, supervised pre-
training purperfrmms self-supervised pre-training (DINO (Caron et al., 2021) and MAE (He et al., 2022)). Also interestingly, pre-training with the SAM optimizer (Foret et al., 2021) yields inferior performance to its SGD counterpart.
40

---

Published in Transactions on Machine Learning Research (04/2025)
D.6 Results of Camelyon17 Datasets
Figure D.17 presents the OOD lest accuracy and the expected calibration error on Camelyon17, consisting of medical images that are completely different from the domains of the pre-training data. As the task is relatively casier than others, all models can achieve over 90% tost accuracy and near 0 ECE results, which makes difficult to differentiate the characteristics of the models. Yet, we can still observe the general trend that larger models yield better OOD test accuracy and ECE, supporting our claims.
0.98
Expected Calibration Error
Expected Calibration Error
0.98
OOD Test accuracy
0.97
0OD Test Testaace accuracy
0.97
0.05
0.05
0.96
0.95
0.04
0.04
0.95
0.94
0.93
0.02
0.02
0.93
0.01
0.01
0.92
0.00
0.91
2
0,92
0.94
0.96
0.98
0.70
0.75
0.80
0.85
# parameters (x10%)
# #parameters (x10%)
OOD
Test accuracy
ImageNet-1k accuracy
ResNot-50
ErfciinnNet-a2
EffidientNet-Reel
ResNet-50
ResNet-101
EfficienNNet-33
EffiddentNet-RRe2
ResNet-50 (RA)
EfficientNet-84
ElfideenNet--iee
(AugMix)
ResNet-50 (v2)
EfficienNNet-55
EffiddentNet-Ree
ResNet-101 (A1)
ResNet-101 (v2)
(w/ AutoAugment)
RegNetYY
(A1H)
Restitt 152 (v2)
EffcientNet-81 (w/ AutoAugment)
RegNet/Y 400MF
ResNet- 101 (A2)
Conv/eett-tiny
EficieetNet-922 (w/ AutoAugment)
RegNet-Y
(A.3)
Conv/ext-smalll
EfficienNNet-33 (w/ AutoAugment)
RegNet-Y 9O0MF
ResNet-152 (A1)
ConvNeett-bace
EfficienNNat844 (w/ AutoAugment)
RegNet-Y
ResNet-152 (A1H)
Convieett-large
EfficienNNet-55 (w/ AutoAugment)
RegNet-Y 3.2GF
ResNet-152 (A2)
Convieetttiny (IN12k, (t)
EfficientNet-00 (w/ AdvProp)
RegNet-Y 4.0GF
ResNet-152 (A3)
Convieett-nny (IN22k, (t)
efficientNet-11 (w/ AdvProp)
RegNet-Y 6 3.4GF
VIT-base (patch MAE)
Convilett small (IN22k, ft)
EfficientNee-BB 92 (wi AdvProp)
RegNet 8GF
VIT-huge (patch 14, MAE)
ConvileXt small (IN12K,
EffiientNee-983 (wi AdvProp)
RegNetYY 12GF
VIT-large (patch MAE)
Convieett base (IN22k, ft)
EfficientNet-84 84 (wi AdvProp)
RegNetYY 16GF
VIT-small (DINO)
ConvNexttlarge (IN22k,
erfcientNet-85 AdvFrop)
RegNetoY 32GF
VIT-base (DINO)
ConvNext-xxarrge (IN22k, ft]
emfiientNat-a0 (w/ Noisy Student,JFT. (t)
ResNet-50 (A1)
VIT-base (SAM)
VIT-small AugReg)
erfcientNet-81 Naley Student, JFT,
ResNet-50 (A1H)
VIT-base (IN21K,
VIT-base (w/ AugReg)
EfficienDNet-22 (w/ Noisy Student, JFT, (t)
ResNet-50 (A2)
VGG19
VIT-tiny (wl AugReg,
EfficienNNet-83 Noisy Student,
(A3)
VIT-small (w/ AugReg, IN21 fU)
EfficieniNet-a (w/ Noisy Student, JFT, (t)
ResNet-50 (B1K)
VGG13
(w/ AugRog, IN21K,
(w/ Naisy Student, JFT,
(82K)
ViTilarge (w/ AugReg, IN21K, ft)
EffiientNet--55 (w/ RandAugment)
5esNet-50 (C1)
Mix-MixereB (IN21k,
EfficienNNe-B0
EridentNeciRte
ResNet-50 (C2)
MLP-MixerLL (IN21K,
BrfciienNee-81
Figure D.17: OOD test accuracy, expected calibration error, and their relationship with the number of model parameters and ImageNet-Ik accuracy..
41

---

Published in Transactions on Machine Learning Research (04/2025)
D.7 Results of CLIP models
As reference, fig. D.18 shows the test accuracy and expected calibration error in the zero-shot CLIP models with different visual encoders. We can observe that CLIP models achieve nearly comparable accuracy to other models, while they degenerate calibration scores as the number of parameters in the visual encoder increases.
PACS
OfficeHome
DomainNet
VLCS
Expected Calibration Error
Expected Callbration Error
Expected Calibration Error
Expected Calibration Error
0.64
0.75
0.68
0.805
0.66
0.800
0.62
0.795
0.73
0.62
0.790
0.58
0.72
0.60
0.785
0.58
0.54
0.70
0.56
0.775
0.52
0.69
0.93
0.94
0.95
0.96
0.75
0.80
0.85
0.72
0.74
0.76
0.55
0.60
0.65
(OOD)
Test Accuracy
(OOD)
Test Accuracy
(OOD)
Test Accuracy
(OOD) Test Accuracy
CLIP VIT-B/32
CLIP ResNet-101
CLP ResNet-50
Figure D.18: The relationship between (out-of-distributtion) test accuracy and expected calibration error of CLIP models with different visual encoders.
E
ECE Under Infinite Data and Near Zero Cross-Entropy Loss
In this section, we provide Theorem E.1., which shows that, given an infinite dataset and a sufficiently parameterized model, the Expected Calibration Error (ECE) converges zero.
Theorem E.1 (informal). Let ply)z) denote the model's predicted probability distribution over the torget variable y given input x, and let P(ylæ)) represent the true conditional probability distribution. Suppose the following conditions hold:
The dataset is infinite, and the empirical data distribution matches the true distribution
The model achieves near-zero cross-entropy loss on this dataset.
Then, under these assumplions, the model's predictions plyla:) converge to the true distribution P(y)æ)) resulting in ECE that approaches zero.
Proof. To start, we define the cross-cntropy loss for classifier with predicted probabilitics p(y[z) and true labels drawn from the data distribution:
== - [log p(y[=))-
With an infinite datasct, the empirical data distribution aligns with the true distribution, so the empirical average of the loss matches the expected value.
Given that the eross-entropy loss nearly zero, we can infer that p(y[æ) close to P(ylæ), as cross-entropy loss minimizes when p(y|z) P(y|z). To clarify, we can express the cross-cntropy loss as:
L = H((g()) (L(P()))))))))))
where H()())z)) the entropy of the true distribution, and (I)((())))l(())>))) represents the Kullback- Leibler (KL) divergence between Pyy|z) and p(y|z). Since the KL divergence is non-negative and zero only when p(y[=) Pyy]:), near-zero cross-entropy loss suggests that p(y(x) z P(y[=))
42

---

Published in Transactions on Machine Learning Research (04/2025)
Now, considering the ECE, if p(ylz) z P(ylz), each predicted probability closely approximates the true conditional probability. This leads to the following relation:
ace (Bm)
(ū: = ys)
Bm
~ ((2,a))~ Patta (P() p(ull)) E Bml
Paaya [p()(()) ply|z) Bml
conf (Bm).
Refer to Section for notation definitions. Thus, for each bin Bm, the predicted confidence conf(Bm) and true accuracy acc( Bm) are approximately equal.
Using this in the ECE formula, we have:
M
Bmul
ECE = N
Σ m=
(acc(Bm) conf(Bmm) ~ O.
Therefore, under the conditions of infinite data and near-zero cross-entropy loss, we conclude that ECE +
0.
43


Reviews:
[{"review_id": "VZOzLmVnWw", "reviewer": "Reviewer_AK7q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Extensive and systematic experiments with clear practical takeaways\n- Bridge a gap in the literature by jointly studying the models, the pre-training datasets, and training strategies\n- Consider both performance and calibration for a fine-grained analysis of the results\n- Provide scaling laws and phenomena common to many architectures\n- Variety in the models and data to better cover the possible choice of practitioners\n- Very well written with many experiments and ablation studies to ensure impactful insights\n\n**Weaknesses**\nOverall the paper is very well written and I believe it provides a very strong contribution to model selection for OOD generalization in computer vision. I list below some questions that are not weaknesses per se but could strengthen how future work relies on the current submission.\n-I think that only conducting experiments on 5 OOD datasets is too small. Could the authors include more?\n- This study is focused on only one modality (although it is a very broad-scope work). Do the authors have any idea of how their practical insights could translate to other modalities?\n- The ECE metric has some flaws and other calibration metrics have been proposed in the literature [1], e.g., e Adaptive Calibration Error (ACE). Do the authors have any idea of how the results would translate using the ACE (or others) instead of the ECE? \n\n**Questions**\n- Do the authors have any insights from their work on how to design better (performance-wise) pre-trained models/training strategies or pre-training corpus of data?\n- Same question to improve calibration?\n- I think the term \"uncertainty calibration\" is unusual. Do the authors mean \"uncertainty quantification\" and 'calibration'?\n\n[1] Measuring Calibration in Deep Learning, Nixon et al., 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "*Idea*: The paper presents a systematic study of pre-trained model selection for fine-tuning OOD data. The scope is to gain insights on which pre-trained models to choose (especially which size), which pre-training data are the best, and which training strategy to use. In addition, the authors report both the accuracy and the ECE to gain better insights both on the performance and the calibration. The study is focused on computer vision models and data.\n\n*Method*: The authors conduct experiments with 100 models, 5 pre-training datasets (variations of ImageNet), 5 data augmentation techniques, and report performance on 5 OOD datasets from DomainBed and WILD benchmarks. The code is implemented following the official implementations of the different algorithms and mostly follows the DomainBed framework.\n\n*Experiments*: The authors perform large-scale experiments and provide guidelines for practitioners. In summary, they report that larger pre-training models and larger pre-training datasets improve both the performance and the calibration in OOD generalization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following reference is present in double in the bibliography:\n- Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization.\narXiv preprint arXiv:1907.02893, 2019b.\n\nCould the authors update that?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vurHxiVzo8", "reviewer": "Reviewer_AqLf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The authors conducted a lot of experiments.\n\nWeaknesses\n1. The contribution to academia is relatively limited. The work resembles more of a pre-trained model training manual rather than presenting generalizable or quantifiable conclusions.\n2. Could the study incorporate some newer architectures, such as Mamba or recent Vision-Language Models (VLMs)?\n3. The selection of datasets lacks diversity. OfficeHome, VLCS, and PACS belong to the same dataset series. It would be more compelling to include experiments on more diverse datasets, such as Woods.\n4. Using the first domain of each dataset as the reference for results raises questions. As far as I know, results can vary significantly depending on the target domain chosen.\n5. Given that it is already the end of 2024, the paper lacks references to more recent works, including many methods from the DomainBed library, which are not mentioned.\n6. It would be interesting to investigate how downstream fine-tuning of pre-trained models performs when combined with some out-of-distribution (OOD) algorithms, such as CLIPOOD or ADRMX.\n7. The article's presentation lacks clarity, with no unified conclusions. Additionally, some labels in the figures, such as Figure 1, are not explained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article starts with the fine-tuning of pre-trained models, exploring the impact of factors such as the size of pre-trained models, the size of pre-trained datasets, and training strategies on downstream task generalization and uncertainty calibration. Through extensive experiments, it makes some findings, including the confirmation that the choice of pre-trained models indeed affects downstream tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y248vgjORw", "reviewer": "Reviewer_kFXi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe strength of this paper is clearly its extensive experiment. There are some interesting observations in Figure 3. The conclusions are intuitive. \n\n## Weaknesses\n\n1. I disagree that ECE should be considered an uncertainty quantification metric. If the authors believe that it is \"widely used'' please include references. It is widely used in calibration literature, and calibration has a connection to uncertainty quantification, but they are still very different. For one, calibration is inherently a very frequentist concept yet most UQ works (various BNNs, MCMC, GP) have a Bayesian perspective. Please use the term \"confidence calibration\" instead.\n\n2. In fact I'd argue ECE's not even a good measure for calibration anymore. This paper didn't specify how the bins for ECE are constructed, so I assume it follows the practice of (Guo et al. 2017) which is using bins of equal width. This is particularly problematic given the nature of this paper, which compares models with very different size and accuracy. This is because ECE takes the average of predictions within a bin. \n\n    a. To see this, for simplicity assume we have a binary classification problem with 50% cat and 50% dog. A dummy classification that always predict $\\hat{p}=[0.5,0.5]$ is perfectly calibrated. Now, consider a high-accuracy classifier that tend to predict only in the bin [0.95, 1]. A 20-bin ECE, for the same reason as before, is essentially degenerate for this classifier. \n\n    b. There are various improved measurements of calibration. At the very least, please use adaptive bins (e.g. [0%,5%] percentile of the prediction).\n\n    c. On a side note, ECE only measures *confidence calibration* which is also quite outdated, as it ignores the non-maximum predictions. I suggest going through [1-3] as a start to see why ECE is problematic.\n\n3. Also, the paper uses only computer vision datasets and models, relying heavily on the timm library. Thus, I think the current title and abstract is too general. \n\n\n\n[1] Nixon, Jeremy, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. \"Measuring Calibration in Deep Learning.\" In CVPR workshops, vol. 2, no. 7. 2019.\n\n[2] Widmann, David, Fredrik Lindsten, and Dave Zachariah. \"Calibration tests in multi-class classification: A unifying framework.\" Advances in neural information processing systems 32 (2019).\n\n[3] Kull, Meelis, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. \"Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration.\" Advances in neural information processing systems 32 (2019).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the relation between model size and accuracy and calibration quality in computer vision. It runs extensive experiments across models and datasets, and summarize the ECE and accuracy as models grow. It also studies the effect of pre-training on these evaluation metrics.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please justify the language about uncertainty quantification, or change it to be the more appropriate ``confidence calibration'' throughout.\n\n2. Please address the concerns about ECE (e.g. report other measures such as adaptive ECE, kernel based ECE, class ECE, etc.)\n\n3. Please limit the scope to classification in computer vision tasks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission performs a relatively extensive empirical study of how to select the best pre-trained model for downstream transfer, enabling OOD generalization (in terms of accuracy) and calibration (in terms of ECE). The study is relevant and will be intriguing some different thoughts in the foundation model community. In particular, the joint consideration of generalization and calibration is new. Considering these aspects, the paper is recommended for (weak) acceptance.
Decision: Accept (poster)