ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization
Authors: Qishi Dong
Muhammad Awais
Fengwei Zhou
Chuanlong Xie
Tianyang Hu
Yongxin Yang
Sung-Ho Bae
Zhenguo Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Out-of-Distribution Generalization
Model Zoo
Probabilistic Models

Abstract: Recent advances on large-scale pre-training have shown great potentials of leveraging a large set of Pre-Trained Models (PTMs) for improving Out-of-Distribution (OoD) generalization, for which the goal is to perform well on possible unseen domains after fine-tuning on multiple training domains. However, maximally exploiting a zoo of PTMs is challenging since fine-tuning all possible combinations of PTMs is computationally prohibitive while accurate selection of PTMs requires tackling the possible data distribution shift for OoD tasks. In this work, we propose ZooD, a paradigm for PTMs ranking and ensemble with feature selection. Our proposed metric ranks PTMs by quantifying inter-class discriminability and inter-domain stability of the features extracted by the PTMs in a leave-one-domain-out cross-validation manner. The top-K ranked models are then aggregated for the target OoD task. To avoid accumulating noise induced by model ensemble, we propose an efficient variational EM algorithm to select informative features. We evaluate our paradigm on a diverse model zoo consisting of 35 models for various OoD tasks and demonstrate: (i) model ranking is better correlated with fine-tuning ranking than previous methods and up to 9859x faster than brute-force fine-tuning; (ii) OoD generalization after model ensemble with feature selection outperforms the state-of-the-art methods and the accuracy on most challenging task DomainNet is improved from 46.5\% to 50.6\%. Furthermore, we provide the fine-tuning results of 35 PTMs on 7 OoD datasets, hoping to help the research of model zoo and OoD generalization. Code will be available at \href{https://gitee.com/mindspore/models/tree/master/research/cv/zood}{https://gitee.com/mindspore/models/tree/master/research/cv/zood}.

ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization
Qishi Dong 2. AAwas Muhammad 3.1e, Fengwei Zhou ', Chhannong Xie 4.11, Tianyang Hu
Yongxin Yang Sung-Ho Bae 3, Zhenguo Li
Huawei Noah's Ark Lab,
Hong Kong Baptist University,
Kyung- Hee University,
4 Beijing Normal University
Abstract
Recent advances on large-scale pre-training have shown great potentials of lever- aging large set of Pre-Trainee Models (PTMs) for improving of-t--iiiiiitttinn (OoD) generalization, for which the goal is to perform well on possible unseen domains after fine-tuning multiple training domains. However, maximally exploit- ing zo0 of PTMs is challenging since fine-tuning all possible combinations of PTMs is computationally prohibitive while accurate selection of PTMs requires tackling the possible data distribution shift for OoD tasks. In this work, we propose ZooD, a paradigm for PTMs ranking and ensemble with feature selection. Our proposed met- ric ranks PTMs by quantifying inter-class discriminability and inter-domain stability the features extracted by the PTMs in a leave-one- ane-domain-out cross-valle validation manner. The top- K ranked models are then aggregated for the target OoD task. To avoid accumulating noise induced by model ensemble, we propose an efficient vari- ational EM algorithm to select informative features. We evaluate our paradigm on diverse model zoo consisting 35 models for various OoD tasks and demonstrate: (i) model ranking is better correlated with fine-tuning ranking than previous methods and up to 9859x faster than brute-force fine-tuning: (ii) OoD generalization after model ensemble with feature selection outperforms the state-of-the-art methods and the accuracy on most challenging task DomainNet is improved from 46.5% to 50.6%. Furthermore, we provide the fine-tuning results of 35 PTMs on 7 OoD datasets, hoping to help the research of model zoo and OoD generalization. Code will be available at http://ccccccooooooooooddceeeent
Introduction
Although data Independent and Identically Distributed (IID) a primary assumption behind most machine learning systems, it does not hold in many real-world scenarios due to continuous distribution shifts (39 88). Machine learning models encounter serious performance degradation 32 34 in
such Out-of-Distributie (OoD) scenarios. To alleviate the accuracy degradation caused by distribution
shifts, numerous algorithms have been proposed [3] [ 40 44
5 41
70 31
20
47 6). Recently,
Gulrajani and Lopez-Paz 29 have argued for the systematic comparisons of OoD algorithms and
introduced a standard and rigorous test bed called DomainBed Their experimental comparison has
raised concerns about the effectiveness of OoD algorithms since they often fail outperform the simple Empirical Risk Minimization (ERM).
Equal Contribution. This work was carried out Huawei Noah's Ark Lab. Correspondence li.zhenguo Cacinnguo@uawwii.coom huawei.com; clxie@ bnu.edu.cn
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

PTM Ranking
Feature Selection
Tegget Dank
Modal Zoo
Ecclodkings
Nattl
Banking
Target Data
K.Moddis
Features
Pesture Selection
Choonter
D
MI
Faaa
M2
O
Min
Selactoon
D
Data
Pre-zaized Medal
Classos
Dumaina
Inter class Disnizinshiiy
Inteedoorain Stability
Figure I: An overview of ZooD. Given a task with multiple training domains, the model ranking component evaluates and selects the top-K models that generalize well on this task. The features from selected models are then aggregated and denoised based on the feature selection component.
On the other hand, recent works [33 2
64 have shown the advantages of pre-training for improving OoD generalization, i.e., learning from multiple training domains in order to generalize to an unseen domain. The availability
large set of Trained Models (PTMs) provides a huge potential for solving various OoD tasks. However, it is challenging sufficiently cxploit the power of model zoo (a large set of PTMs). One naive approach could be fine-tuning all possible combinations of PTMs on the target dataset and choosing the perffrrnnn one, which is computationally expensive especially when the number of PT and the data size are large. Besides, tne-tuning may also require exhaustive hyper-parameter search and encounter the risk of over-fitting (91].
Recently, many ranking metrics have been proposed to estimate the transferability of models under IID unseen distribu- tion shifts is more challenging compared with the IID setting. Moreover, even if a metric can correctly evaluate the transferability of each PTM, simply using the best model will not fully utilize rich knowl- edge present in a zoo of models. But the problem is even more serious that the most transferable model will include some noise, because noise and invariant features are undistinguishable in the sense that they are all stable across domains. Previous study [87) also pointed this out and emphasized the necessity of feature denoising. Therefore, we leverage the model zoo by assembling relatively transferable models, the accumulation of noise features may increase memory use and hurt the predictive performance.
assumption & 76 58 [0] However, ranking 200 of models for generalization
To solve the aforementioned problems, we propose ZooD, paradigm to rank and aggregate Zoo of PTMs for OoD generalization. An overview our method shown in Figure Given classification task with multiple training domains, to evaluate the generalization capability of each model, we quantify both the inter-class discriminability and inter-domain stability of the features extracted from each PTM in Seave-oneedomain validation domain and each domain rotating as the validation domain, which is critical for identifying models that can extract -nvvariat features. Each PTM in the zoo is ranked by this quantification. ZooD then continues with model aggregation consisting of model ensemble and feature selection. By introducing latent masks over candidate features, an efficient EM algorithm is proposed to select informative features. To tackle the intractability of the posterior, variational approximation to the true posterior using factorizable distribution is derived. We further extend it to large-scale datasets by building a local estimator under the stochastic approximation [65]
cross- valss-vviiatiion manner, i.c., choosing one domain the
To demonstrate the efficacy of our method, we have performed extensive experiments with 35 diverse PTMs and 7 OoD datasets. First, we show that our ranking metric is strongly correlated with the fine-tuning performance of PTMs compared with existing IID metrics. Second, we illustrate the outstanding performance of ZooD on OoD datasets. For instance, on Office-Home, we get 85, % average accuracy compared with the previous SOTA of 70.6% Lastly, we show the speedup of our method compared with brute-force fine-tuning. ZooD gives a maximum speedup of == 10000x (0.27 GPU hours vs 2662.27 GPU hours), making it practical and scalable.
Finally, to speed up research and make our work more reproducible, we have devised a test bench consisting of extracted features, fine-tuning accuracy results, and ranking scores for all 35 PTMs in our model zoo. This testbed can help future research as the process of getting fine-tuning accuracy results based on DomainBed 29 for of models is computationally expensive. For instance,

---

fine-tuning 35 models on all 7 OoD datasets costs approximately 35140 GPU hours (equivalent to 1464 GPU days or 4 GPU years). Concisely, our contributions are follows:
We propose an efficient and scalable ranking metric gauge the generalization-ablliyy of PTMs for unseen domains.
Using EM, we propose method for selecting informative features and discarding invariant but noisy features in an ensemble of models.
We have established test bed for PTMs on 7 OoD datasets, including features extracted by 35 PTMs in our model zoo, fine-tuning accuracy results, and model ranking scores by different methods.
2 Related Work
Pre-training for OoD generalization. To tackle the problem of distribution shifts between training and
test data, various OoD methods 30 44 44|222) (2 3
[1
70
20
47 6 have been proposed with the aim
to learn invariant representations across different environment However, standard evaluation [29] of many OoD algorithms shows that they do not significantly outperform simple ERM. On the other hands, recent works have shown the effeectiveness of pre-trained models for OoD generalization. Yi et al. [89 theoretically showed that adversarially pre- pre-trained models also perform better for OoD generalization. Yu et al. [92] showed that the right choice of pre-trained models can achieve SOTA results. They also showed that IID performance is not good indicator of OoD performance and emphasized on the impor- tance of model selection. Albuquerque al. [2] showed the importance of feature extractor by proposing a new boDebased pretext task for self self-supervised pre-training. CLIP [64 demonstrated that large-scate pre-training on dataset of image-text text pairs results in much more robust models for downstream tasks with various distribution shifts. Our work based on these observations and we aim facilitate utiliza- tion of PTMS by proposing efficient metric as well as efficient feature ensemble and selection method.
Ranking pre-trained models by metric design. Recently, number of metrics have been introduced to estimate transferability of saarreetsss-leaannn representations for target task under IID conditions. H-score [8] estimates the transferability by finding the relationship between extracted features and target class labels. NCE (76 proposes to estimate transferability via measuring conditional entropy between source and target labels. LEEP (58] simplifies NCE by using the joint distribution of source and target labels to estimate log expected empirical prediction. LogME [91 90) estimates the maximum value of label evidence given features from pre- pre-trained models. These transferability metrics focus on determin- ing the compatibility of saarre-task-leannee representations for the target task. We, on the other hand, aim to compute stability of these features across domains in addition to source-target transferability.
Ensemble and feature selection. Early works have shown that model ensemble can significantly improve predictive performance (21). In the age of deep learning, Lakshminarayanan propose deep ensemble to measure predictive uncertainty. Similar works [60] 62) on uncertainty estimation focus on the context of outlier detection and reinforcement learning. When facing zoo of PTMs, it's natural to leverage the rich knowledge by assembling multiple PTMs. In prior works, 49 propose using PTMs as teacher models that distill knowledge to target model for Liu et al. downstream tasks. Shu et al. [71] propose Zoo- Moo-Tuning that learns to aggregate the parameters of multiple PTMs to a target model. However, these methods require the target model must have the identical architecture as the PTMs, thus sacrificing flexibility.
al. [42
Our proposed paradigm involves selecting informative features from assembled feature extractors. In the framework of Bayesian variable selection, it is common practice to identify promising features by estimating the posterior probabilities over all potential feature subsets. Here we mainly focus on Stochastic Search Variable Selection (SSVS) (59) that involves specifying priors over regression coefficients such that higher posterior probabilities will be allocated coefficients substantially different from zero. Then the features whose coefficients have higher posterior probabilities will be selected. George and McCulloch 2.5 first propose SSVS for the linear model and conduct the posterior inference using Gibbs sampling 57 Li and Zhang 45 consider SSVS for regression modeling in high-dimensional spaces incorporating structural information. Ročková and George (66) propose EMVS for efficient SSVS in high-dimentional cases with sparse estimations of posterior probabilities. Note that all aforementioned feature selection methods have inherent assumptions that observed datasets must be IID, which makes these methods difficult to use in our scenarios.
3

---

Each domain ZooD for OoD Generalization
3.1 Model Ranking
Assume that we have a domain distribution D from which we observe m domains: P2....
Dj a set label and datum pairs, i.e. D; - (1n))
j i n,) Meanwhile, we have a zoo of pre-trained feature extractors: M {0i, 02,,0..... OR. ... ,R,
((x)
}. Our objective is to select feature extractor from M (e.g., Φ&)), such that when we train a predictor f on top of it, the composed model fod; can have the best performance on both the observed domains and unseen domains from D.
yi
W
n
In this section, we propose method that facilitates model selection without carrying out the fine-tuning step. For every model in M, our method produces an associated score, by which we can rank the models, such that the higher-ranked ones have better chance deliver stronger results after fine-tuning.
1
M
n
Figure 2: A directed graphical model that represents the model assumptions (T) Here a and 3 are aaper-paramtters. The goal is inference the conditional distri-
The proposed method is combination of 1) model transferability metric and 2) a ceave-one-dommin-ou cross- validation scheme. More specifically, we evaluate each feature extractor m times, and each time we treat the data from the held-out domain as validation data (v)..zz) j== while aggregating all remaining (m-1) 1) domains' data as the training data (y), In the end, we average the m values of the model transferability metric. Finally, we rank feature extractors in descending order of the average.
bution of v/J given ((r)), Vi. and o(x;).
To simplify the notation, denote the aggregated domain's label and feature as y (yI..... " and + = (((=)))...,ó(r,,,,
E ERadd respectively. We use y' ER' and 4' e ERR xd for the held-out domain. The main idea of the designed metric is to evaluate whether the classifier fitted on (y b) also performs well on (y', &²). Hence, we formulate the problem estimating the likelihood function of (y's d') given (y,
ply', -)
(),
where p(y") ( .y, D) measures inter-class discrininability between features E' and labels y', given the aggregated training data. Meanwhile, p( measures covariate shift between features & and Φ which quantify the inter-domain stability.
Given hypothetical space F of classifiers, we - (otpplyyyppppppe p/T((()) We consider
a linear classifier
i.e. foo(x) = T(()) with Gaussian prior of w:
a~~N00,a
(I)
where and 3 are positive parameters. Figure summarizes the model assumptions in 11 with directed graphical model. We estimate â and (3 by maximizing the model evidence
p
ply (. (,:::))((w:a)ddd
SWERe
according to Algorithm 3 You al. [90] and compute the likelihood of as follows:
p(y', y y/o 1/ Φ;ū,,)
ply 4b v.
p(y (3;,,3)
'According to the Laplace approximation pansion of the log-likelihood mode log Po P f] Rs log p(gp(( (. (. [y 14 A(y 14), where
51
if 2 f) is unimodal at μ. can take Taylor ex-
A Vyr Vy Jogp(y 0 flyma The quadratic term implies that p(y|D, f) can be approximated with Gaussian distribution.

---

For measuring covariate shift, we approximate the distribution of 6(x:) with a Gaussian distribution N (p)) where üo and Eg estimated from the training data 2. Then compute the density p( ()) p(('' / [g) to quantify the covariate shift.
Finally, we compute the density at the logarithmic scale and this defines the proposed metric
log p(y' | ,) +lo p( ( (
Please refer to Appendix B.3 and B.4 for more details.
One distinctive aspect of selection process is the cross-domain validation, embodied in the first term of (2). Across different domains, there are where overfitting the latter can severely harm the OoD generalization. By evaluating on held-out domains, we are able to filter out models that fixate on domain-specific features. To provide theoretical justification, an explicit analysis in the linear regression setting is conducted, where we show that features. Despite the
iivaiininvvaiint and domain-specific features,
the model with the optimal metric is the one that selects all over- seelsiimliiiiaaion,, it does reflect the essence of our approach. Due to page limit, the technical details are presented in Appendix B.5
3.2 Model Ensemble with Feature Selection
The top-ranked PTMs in Section are preferred for solving the OoD generalization task. To further aggregate different PTMs, we consider assembling the top-ranked feature extractors and rewrite [6()) (b.,, (R) where (+)) is the feature matrix from the i-th ranked feature extractor.
As we show in experiments, in most cases, aggregating features from multiple models can outperform any single model. However, simply concatenating features inevitably introduces more noise. As found in [87], non-informative but invariant features from training domains may only bring some noise that is irrelevant to the classification problem, and the accumulation of noise hurts the learnability of the OoD generalization task while increasing the memory and computation cost. Therefore, we propose feature selection method under the Bayesian linear model framework in Section 3.1
First, we impose a binary mask z (Zz, .z,,..,Zzz) T for the weight vector w (W1, W2,..., W T where z; 1 indicates that w, is an active weight in the top linear model, i.e., W; & O, meaning the corresponding feature is informative, while w, 0 if =0, indicating noisy feature that should be screened. Therefore the Bayesian feature selection is formulated by estimating the probability n, of z, with n; = p( 1) and π = [ (2..,,A..
To facilitate the utility of the mask, we assume that the weights {w;) ) are independent of each other and each weight w; is drawn from either slab prior or spike prior [37] with the mean of zero:
{
(O.. (0
if 7 = 1; if if
p(w.)2.001100:2)=
We make the Bayesian treatment to the linear model in Section inverse variance terms:
3.1 by introducing gamma priors for all
01,1 ~ Gamma((,.1PPa)))
0: ~ Gamma(pp,.......))) B Gamma(yo,1.P0.00
and denote hyper-parameters as v = {v,y}. In addition, we denote all latent variables as & = {3, (,,(,,,,,,a 0;, 1,4,,, 2 Under certain conditions, maximizing marginal likelihood provably leads to consistent selection and obeys Occam' razor phenomenon (27 and thus screens non- informmaatteee features. To estimate n. the maximum marginal likelihood estimator of (πT is given by
D argmax log p(y ();,, -a:.v)=arrpa =argmax log f p(y (lyyg/o;,,,)d H.
However, the direct maximization of 3 is intractable due the integration over One possible solution is to use EM algorithm (66.. In the E-step, we compute the conditional expectation:
Eg [og ply & (.),,)) y (.:, old old
Notice that evaluating the expectation involves the posterior distribution of &. However in our case, it is not straightforward to obtain analytical form of the true posterior distribution. We instead
5

---

Algorithm 1 Pseudocode of Variational EM Algorithm for Bayesian Feature Selection
Step 1: Initialization:
I (w,,) and II_ (0)
Step 2 (E-Step): Approximate the posterior of &u for each &Α E & at iteration with:
= =exp log p(y - (,;," "]]]
Step 3 (M-Step): Update (T ()) at iteration / by maximizing (5
π v argmax (E) [log ply wtllyyll....... {|
Step 4: Repeat Step and Step until the convergence criterion is met.
approximate it using variational inference 12 by introducing a tractable distribution Q. Considering the following objective function:
ply ply,ElQ;T.M)) de, Q(E:T.V)
((Q)= - Q(::
which is lower bound of log p(y (,;,, has been shown the maximizer of L(Q) is the optimal approximator of p(E yy, 2,;,,, ,V) under the KL divergence. To obtain explicit solution, we consider the classical mean- mean-field family [1], where variational distribution Q can be factorized into:
(())= II (1..)
(4)
After all variational parameters in (4) are updated by running one-step coordinate gradient descent [12] in the M-step, we update new and new by maximizing:
R RE uiid) [log ply v))
(5)
By repeating the E- and M-step, the estimator (Tre p converges to an optimal solution. We then screen those variables with converged prior πT, smaller than the predefined threshold r. The pseudocode is provided in Algorithm/ to illustrate the main idea of the proposed method, where & denotes the k-th variable in the set a - is the subset of all other variables except & the E-step, the optimal approximator (E) under the mean-field family takes the tractable form of the expectation of the joint distribution and the optimization of (5) M-step is equivalent to substituting with corresponding variational parameters of (E)) from E-step. Our derivations for variational approximations and prior lyper-parameters optimization are listed in Appendix C.3
However, the proposed algorithm still suffers from heavy computational cost: (ndd). To address this problem, we propose an efficient version based on stochastic variational inference (35 A local estimator (E) is established under stochastic approximation that enjoys less computational complexity and guarantees convergence to global optimum 665. We successfully reduce the computation cost to Onn"d d?) with n² < n. Readers can refer to Appendix C.4 for more detailed discussions and the complete algorithm for feature selection.
each iteration costs
Experiments
In this section, demonstrate the effectiveness of ZooD. First, evaluate the ability of our ranking metric to estimate OoD performance and compare it with the ground-trutte truth performance and several existing IID ranking methods. Second, show that our aggregation method achieves significant improvements and SOTA results on several OoD datasets. Finally, we demonstrate that ZooD requires significantly less computation, and, therefore, is practically scalable compared with naive fine-tuning
Setup Details. We use PTMs with diverse architectures, pre-training methods, and pre-training datasets. We divide the PTMs into three groups. Group consists of models with different architectures, Group consists of models pre-trained with different training methods, and Group consists of models pre-trained on large-scale datasets. We conduct experiments on six OoD datasets: PACS [3).

---

PACS
M.CS T-O
Office-Himm Lw0.75
Teraalscogniia 1.1.5
DomainNet
NICO-Animals rw0,67
NiCO-vehiles y-o
L-0.58
NNV
rmt.55
rm022
T=o 50
cm-4
rml 49
T=
H-S22
Tw0.76
Two 50
Tw0 77
tw0.04
Tw0.54
Tw0.79
T=0.80
Light
c-0.80
r-O 60
1-0 85
1-0.40
r-0.73
x-0.80
1-0.22
2000
Figure 3: Comparison of ZooD ranking scores with three cestures-basede ranking methods. The plots illustrate ground truth oft-oo-domain accuracies (r-axis), ranking scores (y-axis), and Kendall's coefficient for 35 PTMs on seven datasets.
VLCS (24). Office- Home 777. TerraIncognita [0], DomainNet 663, and NICO (NICO-Animal &
B3 Each of the datasets has multiple domains. The standard way to conduct the experiment is to choose one domain as test (unseen) domain and use the remaining domains as protocol. The top linear classifier is trained on the training domains only and tested on the test domain. Each domain rotates as the test domain and the average accuracy is reported for each dataset. To get ground-truth performance, DomainBed (29) to fine-tune top linear classifiers for the PTMs on these OoD datasets. We adopt the -ounmann-domain-uut cross-validation setup in Bod with 10 experiments for hyper-parameter selection and run trials. We triple the number of iterations for DomainNet (5000 to 15000) as it large-scale dataset requiring more iterations [17] and decrease the number of experiments for hyper-paramedee selection from 10 5. More details on the experimental setup are in Appendix A.I
training domains, which is named saae--one-doainn
follow
4.1 Comparison with IID Ranking Metrics
IDD ranking methods. divide existing ranking methods into two groups. One group consists of methods that employ PTM's classification layer for ranking. These methods include NCE (76) and
LEEP (58) The other group consists of approaches that only use PTM's s extracted features. These methods include H-Score [8] and LogME [91). Additionally, also use kNN with k=200 [8] as
baseline.
Evaluation metrics. To evaluate PTMs on OoD datasets with ranking methods, we follow leave-one- domain-our validation protocol validation for training domains and take average results as the performance prediction for the held-out test domain. To compute the correlation between ranking scores and ground -ruth performance, we use two metrics. First, to compare the ranking of a transferability metric with accuracy, we employ Kendall's coefficient r (38) Unlike Pearson's 's correlation, 7 measures correlation based on the order of two measures. Consequently, it is better criterion for ranking. Second, to measure the performance of transferability metric for top- rop-model selection, we utilize weighted Kendall's s coefficient T. [78]. The To gives more weight to the ranking of nop-performing models compared with the rest of the models. Therefore, it is better comparative criterion for top model selection.
For ZooD and kNN, we further adopt leave-one-domen dom--dmmin--uu
First, we compare our method with feature- based scoring methods: kNN, H-Score, and LogME. These methods, similar to our method, rank models based on the penultimate layer. We compare ZooD with these methods for the full set of 35 PTMs. We plot ranking scores and ground-truth accuracies 3 For quantitative comparison, we also provide 7 values. It can be seen that ZooD is better correlated with fine-tuning accuracy than other ranking methods on most of the datasets. For example, our method has r of 0.85 compared with LogME's 7 0.77 on Office- Home and a r of 0.40 compared with LogME's s 7 0.04 on TerraIncognita.
Results.
7

---

MACS Xm0008
VLCS TmD 36
Office-homme tm0.90
Terralncogiita rm-0.52
DomainNet tmb.87
MCC0-Animals rmb.90
NICO-ehicle Tm 85
T=0000
T=
x=:0770
r= 12
T=0.08
T= 0..02
T=0.00
T=:0.22
T=00.00
T=00.8
x=:0.4
x=0000
T=:0.89
t=0.90
Group
Group
Gmap
Figure 4: Comparison of ZooD ranking scores with two canisfiiiationllye-bbbsste ranking methods. The plots illustrate ground-truth out-of-domain accuracies (z-axis), ranking scores (y-axis), and Kendall's coefficient for 25 PTMs that have classification layers on seven datasets.
Table 1:
Comparisons: (a) Tue between ZooD and feature-based transferability estimation methods using all of our PTMs. (b) Tu: between ZooD and classification transferability estimation methods. For this comparison, we consider 25 models that have classification heads. (c) Our method v.s. brute-force fine-tuning terms computing cost. For this comparison, we consider all 35 models.
(a) Tw for feature based
(b) Tw for Classification based
(c) Speed up over brute-force
KNN
LogME
Zool)
LEEP
ZeoD
GPU
ZooD
Fise-muning
Speed Up
PACS
0.22
2662.27
9899x
Office-Home
0.78
0.68
0.86
0.36
Office Home
Ottie-Home
Jemaancogniaa
0.40
0.20
0.02
0.46
Taraaloongnita
Termincognita
3920:37
DomainNet
0.89
0.62
0.65
0.76
DomainNet
0.77
0.87
0.72
DomainNet
11:24
17055.33
1516x
NICO Animals
0.73
0.72
0.89
0.90
NIDD-Amimals
0.58
0.92
0.94
NICK--Aniaas
0.32
2914.401
9107
NICO-Vehicter
0.82
0.75
0.90
0.92
Vehiceesc
0.69
0.92
0.96
NICOA Veticles
0.30
27A4.13
9313x
Furthermore, metric is more stable and consistent. Precisely, T of ZooD varies between 0.40
0.85 compared with 0.04 ~ 0.80 for Log.E. -0.08 ~ 0.67 for H-Score, and 0.16 ~ 0.86 for kNN. The consistency of transferability metric across different datasets is critical since the purpose of transferability metric is to estimate performance on a new dataset without having access to ground-trun -trut accuracy. Whenever an estimation metric inherently unstable, it is hard to determine its reliability for a new dataset.
Note that method uses linear model with Gaussian error to approximate the top classifier. This helps us achieve efficient model assessment, especially on small and medium-sizze sized datasets which the bias caused by model approximation is negligible compared with the estimation error due insufficient data. However, on DomainNet, things may be different. The bias caused by model approximation dominants the evaluation performance on large datasets. Therefore, method does not outperform kNN on NomainNet.
Second, we compare our method with classification based methods: NCE and LEEP. For this comparison, we select subset our PTMs that have classification layers. The results are illustrated in Figure It can be seen that ZooD is also more stable and consistent than NCE and LEEP. Moreover, Our method achieves superior performance on the difficult weal-world TerraIncognita dataset. This dataset consists of obscure and blurry images captured by WildCams installed in different territories. NCE has a negative correlation for this dataset. On the other hand, our method, although not perfect, captures the relation in better way. For this challenging dataset, our method has 0.45 compared with 0.12 and -0.32 for LEEP and NCE, respectively.
Third, we compare the weighted Kendall's 's coefficient of our method with other ranking methods. The weighted Kendall's coefficient a better metric to gauge the performance of a metric for top model selection. We also divide these results into two groups: comparison with feature- scoring methods in Table[Tlland comparison with classification--art based scoring methods Table Our method outperforms Ceature-based scoring methods on 6 out of datasets. Similarly, it also outperforms both
8

---

Table 2: Comparison out-of-domain accuracies between ZooD and SOTA OoD methods. The results of MixStyle [93] and SWAD [17] are from SWAD, and other results are from Guirajani and Lopez- -aaz (denoted with t). Our results are average of three trials.
29
Method
PACS
VLCS
Office-Home
TerraInc.
DomainNet
Avg
ERMT
85.5
77.5
66.5
46.1
40.9
63.3
IRM
83.5
78.6
64.3
47.6
33.9
61.6
GroupDRO
84.4
76.7
66.0
43.2
33.3
60.7
I-Mixup
84.6
77.4
68.1
47.9
39.2
63.4
MLDG
84.9
77.2
66.8
47.8
41.2
63.6
MMD
84.7
77.5
66.4
42.2
23.4
58.8
DANNI
83.7
78.6
63,9
46.7
38.3
62.6
CDANNÍ
82.6
77.5
65.7
45.8
38.3
62.0
MTL
84.6
77.2
66.4
45.6
40.6
62.9
SagNet
86.3
77.8
68.1
48.6
40.3
64.2
ARM
85.1
77.6
64.8
45.5
35.5
61.7
VRExt
84.9
78.3
66.4
46.4
33.6
61.9
RSCÍ
85.2
77.1
65.5
46.6
38.9
62.7
MixStyle
85.2
77.9
60.4
44.0
34.0
60.3
SWAD
88.1
79.1
70.6
50.0
46.5
66.9
ZooD
Single
96.0
79.5
84.6
37.3
48.2
69.1
Ensemble
95.5
80.1
85.0
38.2
50.5
69.9
F. Selection
96.3
80.6
85.1
42.3
50.6
71.0
F. Ratio (%)
24.3
24.5
62.5
76.8
99.8
LEEP and NCE on out of datasets. Moreover, our ranking method is more stable as it performs better on challenging datasets. For example, it has Tu of 0.46 12.6~~0.99 0.92 compared with LogME's Tw of 0.02 ~ 0.90 and H-Score's Tw of -0.20 0.75.
In summary, transfcrability estimation of ZooD correlates better with ground-truth accuracy on most of the OoD datasets compared with previous ranking methods. also outperforms most feature- based metrics for model selection in terms of ww Additionally, it is more stable and consistent across datasets, making a better choice for pre-trained model selection.
4.2 SOTA Results with Our Selection Method
We also compare ZooD (model ranking and feature selection) with several recent SOTA OoD methods and demonstrate that achieves substantial performance improvements. We compare previous OoD methods with three versions of method: 1) Single: fine-tune the top-1 model by transferability metric; 2) Ensemble: fine-tune an ensemble of the top-K models; F. Selectioo: fine-tune an ensemble of the top-K K models with feature selection, which is the expected result using ZooD. By fine-tuning, we mean using ERM with DomainBed settings to fine-ture tune a top linear classifier for the PTMs. Their predictive performance and F. Ratio (the percentage of features used in F. Selection) are listed in the last four lines of Table
In all experiment results, except TerraIncognita (discussed in the next paragraph), our method achieves remarkable improvement against ERM and recent SOTA. For Single, we list the improvements over the previous SOTA as follows: + +14% Office-Home, +7.9% on PACS. +1.7% on DomainNet, and +0.4% on VLCS. This result also shows that even without aggregation, using proper pre- vre-trained model improve OoD generalization by large margin. Notice that our method can be regarded complement to other OoD algorithms. After selecting the top-ranked models, different OoD algorithms can be adapted to fine-tune the models.
The performance of Single does not outperform the previous SOTA on TerraIncognita. This because previous methods fine-tune the whole network. In contrast, we only train classifier top of fixed feature extractor. Terralncognita
much more challenging dataset compared with other OoD datasets, as the majority of its images are obscured by the background. Therefore it requires fully fine-tuning. To show the effectiveness of ZooD with fully fine-tuning, we select the top-1 ranked model and fine-tune the whole model. Our resulted model achieves a +2.6% improvement compared with the previous

---

SOTA. One limitation of ZooD when aggregating multiple models is that fine-tuning the whole models difficult due the limitation of GPU memory. However, for OoD tasks, fine-tuning the whole model may not perform better than fine-tuning the top classifier. For example, the results of fine-tuning the full top-ranked models on PACS, VLCSS and Office-Home are 90.6, 79.1 and 83.4, respectively. Empirically, we find if a PTM is suitable for given OoD task, fine-tuning the top classifier has better OoD generalization than fine-tuning the full model.
As shown in Table 2 simple model ensemble (Ensemble) provides fairly minimal improvement because it may introduce invariant but noisy features. To efficiently utilize multiple models, we propose to select informative features in Section 3.2] Here, we compare the performance improvement by F. Selection with Single and Ensemble. ZooD significantly outperforms both candidates while only using a small portion of aggregated features from top-K K models. Even on the most sophisticated DomainNet, ZooD can improve predictive performance by +2.4% compared with Single and +0. 1% compared with Ensemble.
To find the appropriate number K for the model ensemble, we performed an ablation study. We varied the number of K, e.g. K E {3,5, 7]. The performance changes are plotted in 3 We found the performance by aggregating top-3 mod- els strikes the right balance between performance and computational complexity. Hence, K 3 set to the default value.
FACS
VLCS
07.5
12.0
sclack-futures foatures
16-3
7a2.5 Number Models
Number Madeb
In summary, our ranking metric in ZooD is good enough to select a model that can outperform the previous SOTA methods without adding any bells and whistles. Furthermore, feature selection in ZooD can efficiently utilize informative features from top-K models further improve OoD generaliza- tion. In this work, we do not control for the impact of better PTMs. Given zoo of PTMs, our method aims exploit the power of the zoo for OoD generalization. We can further increase the power of the model zoo by adding more PTMs. Based on extensive experimental results on various OoD datasets, we conclude ZooD makes it easy and efficient to exploit large set of PTMs for OoD generalization.
Figure 5: Comparison of selected- feature ensem- ble vs. all-feature ensemble for varying number of top models in the ensemble.
4.3 Computational Efficiency of ZooD
We illustrate the precision and computational efficiency of ZooD by comparing it with brute-force fine-tuning in terms of GPU hours. The results are shown in Table[Tc] ZooD provides a minimum of 1516x speed-up for DomainNet and maximum of 9859x speed-up for PACS. Cumulatively, our method took a total of 13 GPU hours to evaluate all the PTMS on all the datasets compared with 35140 GPU hours (equivalent 4 GPU years) for brute-force fine-tuning. Therefore, ZooD is scalable and practical method for OoD generalization.
5
Conclusion
Machine learning models rely on IID assumption, which is often violated due to constant distribution shifts in real-world applications. In this work, we argue for leveraging large set of PTMs to improve OoD generalization and propose ZooD, paradigm for efficient PTMs ranking and aggregation. Our paradigm avoids the computationally-prriin proooibblyy fine-tuning by ranking PTMs based on quantifying their inter-class discriminability and inter-domain stability, and selecting the most informative features from top-ranked PTMs ensemble. Extensive experiments show ZooD is superior in ranking correlation with the ground- -ruth performance and achieves SOTA results on various OoD benchmarks.
Acknowledgments
Awais's 's work, in part, was also supported by the Institute of Information & Communications Tech- nology Planning & Evaluation (IITP) grant (No. 2021-0-02068, Artificial Intelligence Innovation Hub and No.RS-2022-001 Development (Kyung Hee University)). We gratefully acknowledge the support of MindSpore for this research.
Artificial Intelligence Convergence Innovation Human Resources
10

---

References
[1] Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar, Invariant risk minimization games. In International Conference on Machine Learning, pages 145-155. PMLR, 2020.
[2] Isabela Albuquerque, Nikhil Naik, Junnan Li, Nitish Shirish Keskar, and Richard Socher. Improving nut-of-distribttion generalization via multi-task supe- pretraining. ArXiv, abs/2003. 2020.
[3] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arv: www.:iiw00.0.009000 907.02893, 2019.
[4] Yuki M. Asano, C. Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and represen- tation lcarning. ArXiv, 1 1.03371, 2020.
[5] Haoyue Bai, Rui Sun, Lanqing Hong. Fengwei Zhou, Nanyang Ye, Han-Jia Yc, S-H Gary Chan, and Zhenguo Li. Decaug: Out-of-distributio generalization via decomposed feature representation and semantic augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6705- 6713, 2021.
[6] Haoyue Bai, Fengwei Zhou, Lanqing Hong, Nanyang Yc, S-H Gary Chan, and Zhenguo Li. Nas-ood: Ncural architecture search for out out-o--diitribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8320-8200000 83200,883,,000 2021.
[7] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2 106.08254 2021.
[8] Yajie Bao, Yongni Li, Shao-Lun Huang. Lin Zhang, Lizhong Zheng, Amir Roshan Zamir, and Leonidas J. Guibas. information theoretic approach to transfcrability task transfer learning. 2019 IEEE Interna- tional Conference on Image Processing (ICIP), pages 2309-22000 20119223112222 2019.
[9] Andrei Barbu, David Mayo. Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenen- baum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in Neural Information Processing Systems, 32:9453-948 04633,963,44 2019.
[10] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV,
[11] Christopher M Bishop. Pattern recognition. Machine learning, 128(9), 2006.
[12] David M Blei, Alp Kucukelbir, and Jon McAuliffe. Variational inference: review for statisticians. Journal of the American statistical Association, 112(518):859-87)7 201.
[13] Fabio Carlucci, Antonio D' Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain gencralization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20229223,,111 2019.
[14] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pages 132-149, 2018.
[15] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In Thirty-fourth Conference Neural Information Processing Systems (NeurIPS), 2020.
[16] George Casella, F Javier Girón, M Lina Martínez, and Elias Moreno. Consistency of bayesian procedures for variable selection. The Annals of Statistics, 37(3): 127- 2009.
[17] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. arXiv preprint arXiv: 102. 02.006604 2021.
[18] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. ArXiv, abs/2003 00499,4 2020.
[19] Adrian Corduneanu and Christopher M Bishop. Variational bayesian model selection for mixture distribu- tions. In Artificial intelligence and Statistics, volume 2001, pages 27-34, Morgan Kaufmann Waltham, MA, 2001.
[20] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D Cox, and James J DiCarlo. Simulating primary visual cortex at the front of cans improves robustness to image perturbations. BioRxiv, 2020.

---

[21] Thomas G Dietterich. Ensemble methods in machine learning. In Internatioonal workshop maltiple classifier systems, pages 1-15. Springer. 2000.
[22] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. Advances in Neural Information Processing Systems, 32: 6450-6461, 2019.
[23] Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do sele-super models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5414 5423, 2021.
[24] Chen Fang, Ye Xu, Daniel Rockmore. Unbiased metric learning: On the utilization multiple datasets and web images for softening bias. 2013 JEEE International Conference on Computer Vision, pages 1657-1664, 2013.
[25] Edward I George and Robert McCulloch. Stochastic search variable selection. Markov chain Monte Carlo in practice, 68:203-214, 1995.
[26] Edward I George and Robert McCulloch. Approaches for bayesian variable selection. Statistica sinica, pages 339-33 003,,33,17 1997.
[27] Subhashis Ghosal, Jüri Lember, and Aad Van Der Vaart. Nonparametric bayesian model selection and averaging. Electronic Journal of Statisties, 2:63-89, 2008
[28] Jean- Bann-BBstien Grill, Florian Strub, Florent Altch' e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach supervuud learning. ArXiv, abbs/2000.07733,, 2020.
[29] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
[30] Kaiming He, X. Zhang, Shaoqing Ren, Jian Sun. Deep residual leaming for image recognition. 2016
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[31] Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines. Pattern Recognition, page 107383, 2020.
[32] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[33] Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Xiaodong Song. Pretrained transformers improve out d--istribuution robustness. In ACL, 2020.
[34] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
[35] Mathew Hoffman,, David Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(5). 2013.
[36] Gao Huang, Zhuang Liu, Kilian Q Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261-2261 2017.
[37] Hemant Ishwaran and Sunil Rao. Spike and slab variable selection: frequentist and bayesian strategies. The Annals of Statistics, 33(2):730-773, 2005.
[38] Maurice G Kendall. new measure of rank correlation. Biometrika, 30(112):81-99 1938.
[39] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang. Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: benchmark of tild distribution shifts. In International Conference on Machine Learning, pages (637-5664, PMLR, 2021.
[40] David Krueger, Ethan Caballero, foern-Hentee Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 581 1815-58266 PMLR, 2021.
[41] Kun Kuang, Peng Cui, Susan Athey, Ruoxuan Xiong, and Bo Li. Stable prediction across unknown environments. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1617-1626, 2018
12

---

[42] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive un- certainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
[43] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. 2017 IEEE International Conference on Computer Vision (ICCV), pages 5543-5551, 2017.
[44] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize; Meta-learning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[45] Fan Li and Nancy R Zhang. Bayesian variable selection in structured high- nigh-dimensional covariate spaces with applications in genomics. Journal of the American statistical association, 105(491): 121-)221242121,, 2010.
[46] Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. ArXiv, ab//2055.04666 2021.
[47] Yichen Li and Xingchao Peng. Network architecture search for domain adaptation. arXiv preprint 20Xiv:0008.05006 2020.
[48] Feng Liang, Rui Paulo, German Molina, Merlise A Clyde, and Jim O Berger. Mixtures of g priors for bayesian variable selection. Journal of the American Statistical Association,
[49] Iou-Jen Liu, Jian Peng, and Alexander G Schwing. Knowledge flow: Improve upon your teachers. arXiv preprint arXiv:I904.0555 19888 2019.
[50] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ArXiv, abs/2103. 14030, 2021.
[51] J. C. MacKay. Choice of basis for laplace approximation. Mach. Learn., 33(1):77-86, 1998.
[52] David JC MacKay. Probable networks and plausible predictions a review of practical bayesian methods for supervised neural networks. Network: computation neural systems, 6(3):469 505, 1995.
[53] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ArXiv, abs// 706.06083, 2018.
[54] Ishan Misra and Laurens van der Maaten. Self-supervieed learning of pretext-invarian representations. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6706-6716, 16,
2020. 2020.
[55] Toby J Mitchell and John J Beauchamp. Bayesian variable selection in linear regression. Journal of the american statistical association, 83(404):1023-1332 1988.
[56] Naveen Naidu Narisetty and Xuming He. Bayesian variable selection with shrinking and diffusing priors. The Annals of Statistics, 42(2))789-817, 2014
[S7] Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2. 2011.
[58] Cuong V Nguyen, Tal Hassner, C. Archambeau, and Matthias W. Seeger. Leep: A new mcasure evaluate transferability of learned representations. In ICMI. 2020.
[59] Robert B OaHara and Mikko J Sillanpää. A review of bayesian variable selection methods: what, how and which. Bayesian analysis, 4(1):55 117,
[60] lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, Advances neural information processing systems, 29:4026 202444444,44000 2016.
[61] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin. Natalia Gimelshein, Luca Antiga. Alban Desmaison. Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, perfomperfomance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Gareet, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035, Curran Associates, Inc., 2019.
[62] Nick Pawlowski, Miguel Jaques, and Ben Glocker. Efficient variational bayesian neural network ensembles for outlier detection. arXiv preprint 1w03:::11030077700 2017.
[63] Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang. Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. 2019 JEEE/CVF Internaioonl Conference on Computer Vision (ICCV). pages 1406-14400 1415, 2019.
13

---

[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. /CML, 2021.
[65] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 4070 1951.
[66] Veronika Ročková Ročkovi and Edward I George. Emvs: The em approach to bayesian variable selection. Journal of the American Stutistical Association, 109(506):828-8444 846, 2014.
[67] Olga Russakovsky. Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015.
[68] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry, Do adversarially robust imagenct models transfer better? ArXiv, abs/2007 00849,, 2020.
[69] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang -hichh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 45 10-44520, 2018
[70] Zheyan Shen, Peng Tong Zhang, and Kun Kunag. Stable learning via reweehhniggginnng In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6692-5699, 2020.
[71] Yang Shu, Zhi Kou, Zhangjic Cao. Jianmin Wang, and Mingsheng Long. Zoo-tuning: Adaptive transfer from a zoo of models. In International Conference on Machine Learning, pages 9626-9637. PMLR, 2021.
[72] Christian Szegedy. Wei Liu, Yangqing Jia. Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
[73] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattenn Recognition (CVPR), pages 2818 2826, 2016.
[74] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-61 14. PMLR, 2019.
[75] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl S. Ni, Douglas N. Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. Commun. ACM, 59:64-755 73, 2016.
[76] A. Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1395-1405, 2019.
[77] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5385 5394, 2017.
[78] Sebastiano Vigna. A weighted correlation index for rankings with ties. Proceedings of the 24th international conference on World Wide Web, pages 1166-1176, 2015.
[79] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemniee Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of -tate-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics. URL /twp::://wwwwaa// awwwew
org/ antho T ogy/ 2020. emn p demos. 6
[80] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka. Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020.
[81] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-377337 3742, 2018.
14

---

[82] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transfor- mations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987-5995, 2017.
[83] Xiaofan Xu and Malay Ghosh. Bayesian variable selection and estimation for group lasso. Bayesian Analysis, 10(4):909-955 9960,99993669990 2015.
[84] Ismet Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Kumar Mahajan. Billion-scale semi-supervised learning for image classification. ArXiv, abs/ 90,00054,, 2019.
[85] Yun Yang and Debdeep Pati. Bayesian model selection consistency and oracle inequality with intractable marginal likelihood. Xrviv preprint Xrv: /701. 100::00011111111 00000001100000 2017.
[86] Yun Yang. Martin Wainwright, and Michael Jordan. On the compu bayesian variable selection. The Annals of Statisticx, 446))2297722222, 2016.
of Selih--dmmensinna
187] Haotian Yc, Chuanlong Xic, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards theoretical framework of of-distribution generalization. Xrviv preprint ar (id:::06660444/ 2021.
[88] Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong. Fengwei Zhou, Zhenguo Li, and Jun Zhu. Benchh Quantifying and understanding two dimensions of -fstrr-disttibttion generalization. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 7947-7958, 2022.
[89] Mingyang Yi. Lu Hou, Jiacheng Sun, Lifeng Shang, Xin Jiang. Qun Liu, and Zhi-Ming Ma. Improved ood generalization via adversarial training and pre-training. In JCML, 2021.
[90] Kaichao You. Yong Liu, Jianmin Wang, Michael I Jordan, and Mingsheng Long. Ranking and tuning pre-trained models: A new paradigm of exploiting model hubs. arXiv preprint arXiv:21 10545, 2021.
[91] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In International Conference Machine Learning, pages 12133- 12143. PMLR, 2021.
[92] Yaodong Yu, Heinrich Jiang. Dara Bahri, Hossein Mobahi, Seungyeon Kim, Ankit Singh Rawat, Andreas Veit, and Yi Ma. An empirical study pre-trainn pre-trained vision models on out-of-distribution generalization. In NewrlPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
[93] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. ArXiv, 20422104.0200,,2222
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.2
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Mainly in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URLY? [No] Will be released upon publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Setup Details in Section Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
15

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of
GPUs, internal cluster, cloud provider)? [Yes] See Section
4. If you using existing assets (e.g., code, data, models) eurating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 'r using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "yYTQy2_4Kvb", "reviewer": "Reviewer_z3dR", "summary": "This paper studies the problem of increasing performance on out of domain (OOD) classification, i.e. generalization/robustness on data distributions different from the training dataset.  In particular, this work focuses on the setting where there are several pre-trained models (PTMs) available.  The goal of the paper is then to use these PTMs in some efficient way to realize high OOD generalization.  The proposed approach breaks this into two parts: (1) ranking the PTMs by predicted generalization OOD, and (2) ensembling them.  The authors propose new approaches for both parts, the first using a ranking metric that uses inter-class discriminative capacity and inter-domain stability computed in a leave-one-out x-validation, and (2) using a feature sub-selection method accomplished using an approximate inference EM approach.  The authors rigorously compare and ablate the proposed approach and show strong empirical gains.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- (S1) Interesting and relevant topic: OOD generalization and robustness is always a critical topic, with growing resources and momentum recently; and increasingly there are many pre-trained models publicly accessible which together makes the setting very relevant.\n- (S2) Strong empirical results with thorough methodological ablations: The proposed pipeline is somewhat complex but the component pieces and design choices are thoroughly evaluated overall, with very strong empirical results\n\nWeaknesses:\n- (W1) Minimal detail in main body: There is mention of a formal analysis, and there are several core algorithms that are fully relegated to the appendix; it would seem appropriate and possible to include more in the main body.\n- (W2) Ablation/examination of pre-trained effects: The authors show strong absolute performance against prior work but don't go into detail on how differences in which pre-trained models were used may have influenced; i.e., are the effects of different/better pre-trained models properly controlled for?\n- (W3) The benefits of ensembling over just using the best pre-trained model selected seem fairly minimal", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o1l4247uv68", "reviewer": "Reviewer_SbCP", "summary": "Naive fine-tuning requires high computational cost for choosing the best performing model on the target task, and the best performing model may not perform well on unseen domains. A simple model also cannot fully utilize the knowledge presented in different PTMs. Motivated by such observations, the paper proposed a ranking metric to inspect the generalization ability of the PTMs on unseen data through quantifying the inter-class discriminability and inter-domain stability. They also propose an EM approach to select the informative features and discard the noisy features. Experimental results show that the proposed approach outperforms the previous SOTA methods with high efficiency.", "questions": "In Line 192, do you refer to the wrong equation?\nIn Table 2, Feature selection contributes more to the performance improvement compared to model Ensemble. What if the feature selection approach is directly applied on the single model? \n\nLine 48, there should be a full stop before “Given…’.\n", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The paper is well written and easy to follow.\n2. The main contributions of the paper consist of a ranking metric of inspecting the generalization-ability of PTMs and a feature selection approach.\n3. The paper conducted extensive experiments (35 PTMs and 7 datasets), and the experimental results outperforms the previous best approaches.\n\n\nWeakness\n1. Regarding the experiment results shown in Figure 2, the proposed ZooD approach does not show much better performance when compared to LogME, except on Terralncognita. However, the ZooD requires a linear model with Gaussian error for evaluation. \n2. For the experiment results in Table 2, the experiment result of the Single model outperforms the previous SOTA by a large margin. However, the Single model here indicates a fine-tuning model that is selected from the PTMs.  I am curious why such a baseline is not included in the table. It seems unfair to compare a selected top-1models from a zoo of 35 models with a single model on the 5 datasets. \n3. Regarding the ensemble results in Table 2, do you have the performance comparisons of the Ensemble approach with the LogME approach? \n4. Regarding the computational efficiency, the paper compared the proposed ZooD with brute-force finetuning. When comparing with a simple ensemble, such as LogME, does ZooD still have advantages?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NN3A2qlkqWD", "reviewer": "Reviewer_gvF7", "summary": "This paper proposes a pre-trained model ranking metric and an ensemble method to improve the generalization on target out-of-distribution tasks. The proposed metric evaluates inter-class discrimination and inter-domain stability in a cross-domain validation manner across a set of domains, so that the model would not fixate on domain-specific features. Besides, authors propose a variational EM algorithm to reduce noise of ensembled features. Extensive experiments of 35 PTMs on six datasets show the effectiveness of the method.", "questions": "(1) Why IID ranking baselines perform so bad on TerraIncognita? Is fine-tuning top linear classifiers a good metric on this dataset?\n(2) ZooD uses a large set of PTMs including the newly proposed Transformer-based models, while the baselines (e.g., Gulrajani and Lopez-Paz [18]) use ResNet-50. My main concern is that the comparisons may be unfair due to differences in the performance of different architectures. So, do the different network architectures affect the results?\n", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) The proposed method is technically solid and reasonable.\n(2) Experiments are comprehensive and convincing.\n(3) The proposed method significantly outperforms SOTA baselines on most datasets.\n\nWeaknesses:\n(1) The complexity of computing proposed metric on m domains is O(m^2), which will be inefficient when there are a large number of domains. Although it is still much faster than brute-force finetuning, how does it compare with other non-finetuning methods?\n(2) Authors better put the results of finetuning the whole network in table 2 to improve readability.\n(3) It would be better if authors report the finetuning results of the PTM selected by the IID ranking baselines (H-Score, LogME, LEEP and NCE) to better compare the ranking methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept