SIMPLE: Specialized Model-Sample Matching for Domain Generalization
Authors: Ziyue Li
Kan Ren
XINYANG JIANG
Yifei Shen
Haipeng Zhang
Dongsheng Li
Venue: ICLR 2023 poster (2023)
Keywords: domain generalization
ensemble learning
pretrained model

Abstract: In domain generalization (DG), most existing methods aspire to fine-tune a specific pretrained model through novel DG algorithms. In this paper, we propose an alternative direction, i.e., to efficiently leverage a pool of pretrained models without fine-tuning. Through extensive empirical and theoretical evidence, we demonstrate that (1) pretrained models have possessed generalization to some extent while there is no single best pretrained model across all distribution shifts, and (2) out-of-distribution (OOD) generalization error depends on the fitness between the pretrained model and unseen test distributions. This analysis motivates us to incorporate diverse pretrained models and to dispatch the best matched models for each OOD sample by means of recommendation techniques. To this end, we propose SIMPLE, a specialized model-sample matching method for domain generalization. First, the predictions of pretrained models are adapted to the target domain by a linear label space transformation. A matching network aware of model specialty is then proposed to dynamically recommend proper pretrained models to predict each test sample. The experiments on DomainBed show that our method achieves significant performance improvements (up to 12.2% for individual dataset and 3.9% on average) compared to state-of-the-art (SOTA) methods and further achieves 6.1% gain via enlarging the pretrained model pool. Moreover, our method is highly efficient and achieves more than 1000 times training speedup compared to the conventional DG methods with fine-tuning a pretrained model. Code and supplemental materials are available at https://seqml.github.io/simple.

Published as a conference paper at ICLR 2023
SIMPLE: SPECIALIZED MODEL-SAMPLE MATCHING FOR DOMAIN GENERALIZATION
Ziyue ; Kan Ren", Xinyang Jiang" Yifei Shen?, Haipeng Zhang', Dongsheng Li² Shanghai University, Miicosooft Research litzy06 6 1wwowedd@gamil... com, kan. wwn.reeemiccoooft.com
ABSTRACT
In domain generalization (DG), most existing methods aspire to fine-tune a spe- cific pretrained model through novel DG algorithms. In this paper, we propose an alternative direction, i.c., to efficiently leverage a pool of pretrained models without fine-tuning, Through extensive empirical and theoretical evidence, we demonstrate that (1) pretrained models have possessed generalization to some ex- tent while there is single best pretrained model across all distribution shifts, (OOD) generalization error depends on the fitness be-
and (2) tween the pretrained model and unseen test distributions. This analysis motivates us to incorporate diverse pretrained models and to dispatch the best matched mod- els for each OOD sample by means of recommendation techniques. To this end, we propose SIMPLE, a specialized model-sample matching method for domain generalization. First, the predictions of pretrained models are adapted to the tar- get domain by a linear label space transformation. A matching network aware of model specialty then proposed to dynamically recommend proper pretrained models to predict each test sample. The experiments on DomainBed show that our method achieves significant performance improvements (up to 12.2% for individ- ual dataset and 3.9% average) compared to state-of-the-as (SOTA) methods and further achieves 6.1% gain via enlarging the pretrained model pool. Moreover, our method is highly efficient and achieves more than 1000 training speedup compared to the conventional DG methods with fine-tuning a pretrained model. Code and supplemental materials are available at https://seqml http://ggmmiiiuuu../oo.ooooo github.io/siipll..
INTRODUCTION
Distribution shift is a common problem in real-world applications, which breaks the independent and identically distributional (i.i.d.) assumption of machine learning algorithms (Wang al., 2022) Mismatches between training and test distributions, which are quite common in reality, can largely deteriorate model performance and make machine learning models infeasible for practical applica- tions (González & Abu-Mostafa, 2015). Therefore, enhancing the generalization ability of models has attracted increasing attention CCha et al. 2021 Zhang et al., 2022).
For its practical significance, various methods have been proposed, e.g., domain alignment (aanin 2016 Gong et al., 2019: Arjovsky et al., 2019). meta-learning (Finn et al. 2017 Dou
cct al.]
ct al.
2019; Du et al. 2020), and ensemble learning (Mancini al., 2018, Cha et al. 2021 Arpit et al.
2021). The effectiveness of DG algorithms is generally verified by tine-tuning pre- trained
ResNet(He et al. (016) model with these algorithms (Gulrajani & Lopez- Paz (020). It has demon- strated that these algorithms improve upon empirical risk minimization (ERM) baseline on ResNet- 50 backbone (Arpit al. 2021: Wiles al.) 2021). Meanwhile, recent studies show that neural architectures and pretraining methods have a large impact on the model robustness to distribution shifts (Radford et al. | Wiles et al. 202T). For example, vision transformers are more robust to texture and style shifts compared with ResNet-based models (Zhang et al. 2022), which are instead superior to transformer- based models on dense image classification tasks (Liu et al. 2022). In terms of pretraining, using pretraining datasets other than ImageNet- Ik improves the generalization per- formance in one test domain, yet leads to performance degradation in another (Kim et al., 2022).
It conducted during the internship of Ziyue Li at Microsoft Research. Correspondence Kan Ren.

---

Published as a conference paper at ICLR 2023
These findings are in line with the No Free Lunch (NFL) Theorem (Wolper 1996). which suggests that no single model can always perform better than any other model without having substantive information about the targeted problem. In DG, usually have very limited information about the test domain, so we are more likely to encounter the above challenge.
Inspired by these attempts, in this paper, we conduct fine fine-grained study the relationship be- tween pretrained models and distribution shifts. From both empirical and theoretical evidence, we show that no free lunch in terms of pretraining for domain generalization, i.e., there is single best pretrained model across shifting test distributions. Specifically, 283 pretrained models with different network architectures, pretraining datasets, and learning objectives are compared for their general- ization performance under different distributional shifts. The results reveal that the pretrained mod- els without fine-tuning generalize well to some unseen domains, but none of these models dominate in all unseen distributions. Furthermore, the theoretical analysis indicates that OOD generalization error determined by the fitness between model (varying w.r.t. the network architectures and model weights) and test distribution. For any network architecture with fixed training distributions, such as pretrained models (landola et al. 2014) He et al. 2016 202 it always possible to find a beneficial or detrimental test distribution with small large generalization error.
Motivated by these findings, we propose an alternative DG paradigm that leverages pretrained mod- els with different network architectures and shifting training distributions, upon which match the most suitable pretrained models for each OOD) sample. As shown in Figure 3 our paradigm (specialized model-sample matching for domain generalization. SIMPLE) first adopts a simple la- bel adapter that projects the label space of the pretraining domair the adapter is shared by pretrained models from the same pretraining domain. Then, matching net- work, which is aware of model specialty, selects set of proper pretrained models and aggregates conduct the prediction for each OOD sample. Notably, this promising alternative exhibits significant performance improvements, averaging 3.9% over the existing SOTA results, with gains of up to 12.2% on single datasets, and significant increase in training efficiency.
I to that of unseen domaing where
them together
To summarize, this work has made the following contributions:
We theoretically and empirically analyze the generalization of pretrained models on shifting un- seen test distribution, revealing no free lunch hypothesis exists that motivates our solution of model- sample matching.
In complementary to traditional DG solutions, we propose a novel DG paradigm which directly leverages pretrained models without fine-tuning, and it has significantly improved the DG perfor- mance in the mainstream benchmark upon other strong baselines.
Besides the performance gain, our method is even more efficient since it does not follow the common fine-tuning approach, shedding new light on using pretrained models in DG tasks.
No FREE LUNCH IN PRETRAINING FOR DOMAIN GENERALIZATION
In this section, we investigate if there exists free lunch in pretraining for DG, that is, whether we can find one single best pretrained model that generalizes across all distributional shifts. To this end, first conduct an empirical analysis on the generalization ability of pretrained models over shifting distributions in Section 2.1 followed by theoretical analysis in Section 2.2
2.1 GENERALIZABILITY ANALYSIS OF THE PRETRAINED MODELS
We here analyze the generalization ability possessed by different pretrained models. Note that ex- isting DG methods generally adopt a specific ImageNet--retrainne model (e-g, ResNet-50), which has been shown not sufficient for generalization (Kumar et al. 1021: Kim al. (022). Thus, for a comprehensive analysis, we first incorporate 283 diverse pretrained models composed of diverse combinations of network architectures, pretraining datasets, objectives, and algorithms. Detailed information on all these models and more experimental settings are in Appendix [A.] For the effi- cient adaptation of pretrained models from pretraining domains to unseen domains, we propose to
'The data distribution where the pretrained models have been learned. Source and target domains in DG share the same label space yet differing from that of pretraining domains.
2

---

Published as a conference paper at ICLR 2023
train only label space adapter upon the pretrained models without modifying any pretrained model parameters, which is remarkably lightweight and will be elaborated detailedly in Section 3.2
Takeaway 1: Pretrained models possess decent generalization ability for some OOD samples.
By grafting such a lightweight label space adapter, we find that the generalization performance of the pretrained model is already promising in
cases. As concrete example, given fixed 2014) pretrained on ImageNet, we employ an adapter that converts its prediction of the original ImageNet label space that of OfficeHome (a dataset of DG benchmark) (Venkateswara et al. 2017), with the adapter training on source domains. This combination achieves an average accuracy of 78.3% on target domains, which is 5.9% higher than SOTA as detailed in Section[4.3
DenseNet-121 model (landola et al.)
No dominant pretrained models across unseen domains. Though pretrained mod- els have possessed some generalization ability in some cases, however, their generalization per- formance relates the specific unseen distributions. The left part of Figure shows the relative performance of the pretrained models evaluated in different domains, with label adapter. Each do- main represents a different data distribution. As can seen, pretrained models vary greatly in performance on different test domains, without any single model being dominant in all domains.
Takeaway 2:
Relative performance between pretrained models across different domains and classes
Pertrained model
SqueezeNot v1.1
Mixer
RegNetY-32GF
ConvNeX: (ft1k)
BETT-L
SelecSLS
MobilaNatv3.s
domain
class
Opessum
Bobcat
Coyote
Empty
Rabbit
Squirrel
Figure 1: Classification performance comparison of pretrained models in different domains and different classes the TerraIncognita dataset (Beery et al., 2018). For clarity of presentation, only partial results are shown. The complete results be found in Appendix A.4
Takeaway Pretrained models exhibit more diverse performance at finer-grained levels. We further examine whether performance divergence also exists at finer-grained level, such as class- level. The right part of Figure presents the relative model performance on classes in TerraIncog- nita dataset. Similar to that at domain-level, varying model performance among different classes is also noticeable. Moreover, there exists a more pronounced divergence in model performance at the finer class level, evidenced in detail in Appendix [A.4] It supports the necessity of incorporating pretrained models and being aware of their specialty at a fine-grained level, which motivates us to investigate and exploit the matching metric between models and test samples.
2.2 THEORETICAL ANALYSIS OF No FREE LUNCH FOR DG
This section provides theoretical analysis to support the findings in Section 2.1 To alleviate the generalization difficulties associated with the train-test mismatch, mainstream DG methods seek domain-invann invariant features that can be generalized beyond the training distribution. However, there several issues consider. First, domain-invariant features learned from source domains can still be biased towards source domains and thus have limited performance on unseen domains (Chal al. 2022). Second, domain-spectie
information has also been found to enhance DG, it may be closely related to the sample labels that help generalize in certain unseen domains (Bui et al. 2021.. These observations suggest that existing DG methods may not guarantee whether a model can generalize across distinct unseen target domains, the reason lies in the limitation of our knowledge of unseen domains, which is line with NFL theorem. The analysis of the OOD generalization of kernel methods also points out the relevant insight that shift in test distribution may help hurt generalization (Canatar al. 2021).
Theorem 1. (Informal; OOD Kernel Ceneralization,Canaae Canatar et al. (202l) Given the kernel matrix Kxx,x') with Mercer decomposition K(xxx')= xxz)' ( Suppose the training data is i.id. generated from the distribution p(x), the target function is given by a" ((x), and the training loss is kernel regression with ERM. The generalization error on distribution p(x) given
3

---

Published as a conference paper at ICLR 2023
by gg Eg Egantched kaa 1 - where O,, dz and O' O'= 1-7 Here Equanched is the generalization error when the training distribution and test distribution are matched (i.e., in-distribution error), P is the number of training samples, K are constants depend on A, and 7/ is a constant depending on and 6.
Theorem ! wwill be detailed in Appendix A.5
Remark 1. (Interpretations of Theorem 7 Theorem 1 shows that Egg is in the form Eg Eymatched + v where the second term is due to distribution shift. Note that, if a pretrained model a
is kept frozen serve static feature extractor for the subsequent trainable linear classifier, the pretrained model acts as a kernel and thus Theorem I applies.
Remark (Generalization depends on the fitness between model and test distributions) The eigen- value of matrix O depends on the alignment between the test distribution p(x) and the kernel basis & which determined by the model (network architecture and model weights). The matrix O' may have negative eigenvalues when p(x) and & are matched well and OOD generalizatinn is even bet- ter than i..dd generalization under such circumstances. On the contrury, when & (xx) is fixed, f(x) can be adversarially chosen such that the matrix O' has large positive eigenvalues and thus the network fails to generalization to OOD data.
Remark 3. (No free lunch for single model in DG) major focus of DG is tackling covariate shift, where p(x) can be set arbitrarily as long as ply x) = p(y]z)) Under covariate shift, no single pretrained model will outperform other models for all p(z), implying no free lunch theorem in DG.
The theoretical analysis above raises natural question: should not the focus be more on matching the pretrained model and testing distributions based on their funess? Given that pretrained models with fixed network architectures and training distributions always face beneficial or detrimental test distributions, we suggest incorporating more pretrained models to create diverse network architec- tures and shifting training distributions, to facilitate generalization. Depending pretrained models to the target distributions, is then possible to match OOD samples to appropriate models, thus bypassing the issue of using single model indicated by the NFL theorem.
the fitness of the
3 THE PROPOSED FRAMEWORK FOR DOMAIN GENERALIZATION
In this section, we present a novel framework, namely specialized model-sample matching for do- main generalization (SIMPLE), that reformulates DG as a matching problem, light the analysis in Section 2 We first introduce the overall framework in Section 3.1 Then, we elaborate on the design of specialty-aware model-sample matching and ensemble in Section 3.2 followed by the
learning algorithm in Section 3.3
3.1 THE OVERALL FRAMEWORK
This section provides an overview of the SIMPLE framework, as shown in Figure 3 Following our analysis in Section 2 one single pretrained model is not sufficient accommodate diverse OOD samples. From an intuitive view, it appears that more models need to be incorporated and selected to solve the problem. Analogous to recommending items (models) users (samples) from the vast item set in recommender system, we formulate DG as a model-sample matching problem, with a model pool containing various models and a model dispatcher responsible for assigning these models to OOD samples appropriately.
Preliminaries. DG aims to tackle the shift of data distribution among different domains by trans- ferring knowledge from seen to unseen domains. Unlike domain adaptation, samples from unseen target domain(s) are inaccessible in DG. For a domain, its input and label space can be denoted as X E Rd and y E CRC, respectively. where d is the dimension of input and C' is the number of classes in y; And its samples are observed constructing a dataset D = {x y: W with sample number N. Consider that we have S source domains D, (Di, ....... Ds} and T target domains D. [D....., D that share the same label space but with different joint distributions on X' x y.
Pretrained model pool. As discussed in Section 2.2 the OOD generalization error depends on the fitness between the pretrained model and the test distribution, which unknown DG. Therefore, building pool with diverse pretrained models is crucial for DG. Note that with abundant pretrained
4

---

Published as a conference paper at ICLR 2023
models being released, is easy to construct model pool by simply downloading pretrained models from public repositories [Wightman] 2019), , without further efforts such as retraining. We collect extensive pretrained models to serve for SIMPLE, detailed in Appendix A.33 We denote model
pool with K models as {ʃa [ each of which is parameterized as 0z.
Label space adapter. As the label space of the pretraining domains generally differs from the one shared by source and target domains, label space adapters are required to make them consistent. The adapter is a linear mapping between these two label spaces (pretraining + source/target), which is shared among different pretrained models from the same pretraining domain (e.g., we only use one adapter for all the models trained on ImageNet- 1k). Specifically, given pretrained model fa. we parameterize the adapted model by (=fx=;;))) as 8%2 [ 0x), where & denotes the parameters of the adapter, hy E A: RCC + Rº and C. is the dimension of the label space of the original pretraining dataset of fk- Through the label adapter, the output of pretrained model fs can be transformed and adapted to target domains as Dik = (fx (zz)), without fine-tuning {6x } K -=1===1 k= In this way, can largely reduce the adaptation cost of the pretrained models onto the new domains.
By the above construction, SIMPLE differs from the existing methods in two aspects. First, most existing DG methods strive to fine-tune a specific pretrained model, which cannot generalize well to variety of unseen domains, as analyzed in Section Second, conventional options utilizing the pretrained models are fine-tuning and linear probing, as shown in Figure A[A) and (B), respectively. In detail, fine-tuning the pretrained model is costly and can hurt generalization ability (Wortsman et al. 2022). while linear probing (i.e., replacing the last layer of the pretrained model and retraining that) may achieve better accuracy in OOD scenarios than fine-tuning the whole model (Kumar al., 2021]. However, the linear probing layer is not transferable across models, making the way either costly when large number of models need adapt new target domains. Our label space adaption shown in Figure C.) instead, is remarkably light since it is shared by all the models from the same pretraining domain, which has also been empirically verified Section 4.2.
Fixed Newly added
Prapoptan
Fine-tuned
Dispatcher
0.1
0.6
Adapter
shared
Pretrained
model
Fine- mre-tueed
Model
Model
Pretrained model((s)
(A) Fine-tuning
(B) Linear probing
Model-sample matching (ours)
Figure 2: Different training paradigms in DG.
Model dispatcher. As shown in Section 2 there is no individual model performing the best across different tasks, thus, appropriate models need to be dispatched to address each specific task. We define a model dispatcher 9p. with parameter p. that takes the sample x; as input and determines the weight wa assigned to model fa for the sample v; with ΣK k- Wk 1. Here Wt represents an
estimate of the relative match between the model fú in the model pool and the sample.
Based on the constructed model pool and the model dispatcher, the prediction for each test sam- ple is an ensemble of the predictions by the dispatched models. The final prediction is given by 0: = >x [wy he (fx()))) Finally, we can define population loss as ED(b, = I E/ Ex,-D, [(].. over the given domain D. The objective is to minimize the task-specific D.
loss 1 (e.g.. cross-entropy loss for classification) over both source domains D, and target domains D, by only minimizing the empirical risk &,, (w),)) P)) w.r.t. v/ and The performance on the target domains, then, measures the generalization ability.
3.2 SPECIALTY-AWARF MODEL-SSMMLA MATCHING AND ENSEMBLE
Following the paradigm introduced in Section 3.1 this section elaborates on the model dispatcher, which consists of model-sample matching network and specialty-aware ensemble layer.
5

---

Published as a conference paper at ICLR 2023
Prodiction
Model dispatcher Pp
ensemble weights
(0.)
(1.5)
Specialty-awar
...
ensemble layer
adepted outputs
matching
m
scores
Label space
Paj
adapter
adapter
Model- -smmll matching
adapter
...shared
embediding
Pretrained model pool
learnabie model
embeaidings
[c......
feature
Learsable
extractor
Input Ii;
Figure 3: The SIMPLE framework. Based on pool of fixed pretrained models, a recommender learns the matching of models and samples for model dispatching with the help of a label space adapter for prediction transformation. Note that only pretrained models from the same pretraining domain share the same label space adapter.
Model-sample matching. To capture the fitness of pretrained models towards OOD samples, network is proposed learn the model-sample matching function. We employ a simple recommen- dation algorithm (but not restricted), neural collaborative filtering (NCF) (He et al. 2017), to initiate a proof-of-concept for our idea. Following NCF, the matching scores m, = m,, mak] R of a sample z; and models {fr } are computed on their latent features e; and [ci..... ........ cK]. through non-linear function activated multi-layer perceptrons (MLP) ically, the sample and the model are first embedded and then transformed into a joint latent space. The feature extractor of one pretrained model fko from our model pool, fixed as the sample fea- ture extractor, generates sample embedding e, for the sample z,. For the embedding of each model fi, we introduce a learnable embedding ek. included in p. which is randomly initialized and opti- mized during training. Both of them are processed by two non-linear functions activated MI.P as -MP((MP((e)) and Cj MLP((MP((e.)) to map into the same space for matching scoring.
MLP(c. c). Specif-
Specialty-aware ensemble. After constructing series of ranked models based on the matching metric, the subsequent goal conduct prediction by selecting the most proper models. We argue that individual model prediction may not cover most of the target distribution, thus, instead of utiliz- ing only the top-1 matched pretrained model, we propose to apply specialty-aware model ensemble to derive the final prediction for the specific sample. It is also considered that ensemble has shown improved robustness (Lakshminarayanan et al., 2016) and model specialty in various tasks also plays key role in prediction (Gontijo-Lopes et al. 2001 which will be further elaborated in Sec- et
tion 3.3 Specifically, we normalize matching scores by softmax function Softmax(2); Σ ctk
to highlight the relative competition among the pretrained models. That is, the ensemble weights w; [i....., W/k ........ wx] E R is computed as w, Softmax(m:).
To save the inference time, we further select models with the top-k (i: < K) matching scores, before their inference for the given sample. In this paper, we set k and the sensitivity analysis on will be shown in Section 4.5 The overall inference cost is small as illustrated in Section 4.2
3.3 OBJECTIVE AND LEARNING FOR SPECIALIZED MODEL-SAMPLA MATCHING
Loss for ensemble learning. The classification loss Lens (t,, p) (#r, wt) is optimized for the likelihood of final ensemble output, update both the matching network and the adapter.
Loss for label space adapter learning. To train the general label space adapter hy for all pretrained models, we incorporate the weighted classification losses of adapted predictions of pretrained mod- els to update the shared adapter, which is defined as follows:
Cadapter (v)) Ek_ Wk (Ūsk. vi) Σk= wik -1 (fb((()))))))))))))
Loss for model specialty learning. The model dispatcher generates ensemble weights to aggregate multiple model predictions for each sample, where models vary in their performance significantly over samples indicated in Section 2 Thus, we expect to assign more weights the models with

---

Published as a conference paper at ICLR 2023
Table 1: DomainBed benchmarking. Baseline results are from original papers with the same setup.
Algorithm
PACS
VT.CS
OfficeHome
DomainNer Terralncogniaa
avg.
Model Pool-A
Non-ensemble algorithms
DANN Ganin et al. 2016
84.6+1.1
78.7+0.3
65.440.
48.4
38.4+0.0
63.1
CORAL (Sun & Saenko 2016
86.0
77.7+0.5
68.640.4
46.4
41.8+0.
64.1
MLDG [La ct al. 2018a
84.8
77.1
68.220.1
46.1
41.840.4
63.6
MMD (Li
201
85.010.2
76.710.9
67.71.0.1
49.311.4
39.4
63.6
C-DANN [La al. 21188
82.8
78.3
65.61..5
47.610.8
38.910.1
62.6
FRM (ulrajan
Lopez--Pa
2020
85.7
77.4
67.5
47.210.4
41.2.0.2
63.8
Fish Shi A021
85.5
77.8
68.61..4
45.
42.7.0.2
63.9
LP-FT (Kumar et et 2022
84.6
76.7+1.5
65.0+0.2
47.7
43.0+0.1
63.3
MIRO (Cha et 2022)
83.4
79.0.0.0
70.5.04
50.4
44.3
65.9
Ensemble algorithms
SWAD (Cha et al
2021
79. 1
70.6+u.2
50.0
46.5+0.1
66.9
EoA (Arpıt et al.
2021
88.6
79.1
12.5
52,3
47.4
68.0
MIRO + SWAD CCha et
2022
18.4
79.6102
72.4
52.9
47.0...0
68.1
SIMPLE
88.610.4
79.9
84.6
57.6
49.2
72.0
Model Pool-B
2021 MIRO + SWAD (Cha et al. 2022
EoA (Arpit al.
95.8
81.1
61.1
60.9
96.810.2
81.7
83.310.1
64.3
10.3
60.7
SIMPLE
99.0
82.7
87.7+0.4
59.0=a.6
61.9
78.1
Compare the training and inference cost of SIMPLE against that of SOTA baseline. (L) means higher (lower) is better. The training time represents the overall back bock-propagation time.
Table 2:
Details are in Appendix A.
Pretrained model type
Learmable
Training lime speedup
Inference GFLOPs (L)
Algorithm
params (t)
v.s. SOTA buseline (?)
MIRO SWAD (ResNet-50)
25.6M
3.9
ImageNet-prctrainned
SIMPLE (Model Pool-A)
0.9M
1000
9.6
Diverse
MIRO + SWAD (RegNetY-
SIMPLE (Model Pool-B)
79.7M
0.9M
1000
1x
13.2
9.6
higher sample-level specialties to achieve the best utilization of the pretrained models. We use the likelihood of the ground truth label p(y: 22;; 6%) on the i-th sample produced by model fx as the evaluation metric of its sample-level model specialty. That is, we try to minimize the estimation risk of the estimated model specialty on the ground truth, i.e., w. and p(y x;;; 0g), as
Cspecialty (p) --E (ys z4; 0%) In(wx) + (1 p (yr my; ())) In(1 wx)]
(2)
Especially is used to optimize the model-sample network and the ensemble layer work jointly specialty-aware model dispatcher.
Therefore, the total loss to minimize is L a. (t), p) a Cadpper (b) + Expeciatty( (p), where a,s aa and a. are loss weights. It is worth noting that the only parameters to update are (v, p). each of which is lightweight compared to the pretrained models which remain fixed in method, yet have been fine-tuned in other previous works.
4 EXPERIMENTS
4.1 EVALUATION PROTOCOL
We conduct experiments on DomainBed suite (Gulrajani & Lopez-Paz 2020) which provides like- for-like comparisons between algorithms with a standard evaluation, as detailed in Appendix [A6]]
Datasets. We experiment 5 real-world benchmark datasets including PACS (4 domains, 9,991 samples, classes) (Li et al. 2017). VL.CS (4 domains, 10,729 samples, classes) (Fang et al.) 2013), OfficeHome (4 domains, 15,588 samples, 65 classes) /Venkateswara et al. (2017). TerraIncog- nita (4 domains, 24.778 samples, 10 classes) (Beery et al. 2018), and DomainNet (6 domains, 586,575 samples, 345 classes) (Peng al. 2019).
Baselines. We compare SIMPLE with strong DG baselines including state-of th-----rrt General DG methods incorporate elaborate learning algorithms including ERM (Vapnick) 1998). CORAL (Sun & Saenko 2016), MLDG (Li et al. 2018a). MMD (Li et al. 2018b). DANN (Ganin et al. 2016), C-DANN Li et al. 2018c (1).)) Fish (Shi et al. 2021) L.P-T (Kumar al. 2021). and MIRO (Chal

---

Published as a conference paper at ICLR 2023
Table 3: Results of (1) ablation study and (2) SIMPLE with different sized model pools.
Algorithm
PACS
VLCS
OfficHHome
Terralncognita
DomainNer
avg.
Model PooLA
test-best single model
62.1
66.6
78.3
40.5
28.3
55.2
random ensemble
40.2
48.6
34.7
16.2
12,0
30.3
SIMPLE (small pool)
84.1
79.8
79.9
56.840.
46.3
SIMPLE
88.6
29.9
84.6
$7.6+08
49.2
72.0
Model Pool-B
test- best single model
95.6
82.1
78.3
40.3
52.7
69.8
random ensemble
38.5
46.5
28.2
19.7
12.5
29.1
SIMPLE (small pool)
+0.0
82.2
80.7
56.8
54.7
74.1
SIMPLE
99.0+0.1
82.7
87.7
$9.0
61.9
78.1
al. 2022). Some other works incorporate ensemble learning, including SWAD (Cha et al. 2021
and EoA (Arpit et al., (02T). MIRO has combined SWAD (MIRO + SWAD) into their approach, with the result being the current SOTA for DomainBed. More details can refer to Appendix AA.7 A.7
Model pool composition. We collect 283 pretrained models to compose model pools, described in detail in Appendix A.3] Based on their pretraining domains, we divide them into pure ImageNet- pretrained model pool (Model Pool-A) and one with pretrained models from different datasets (Model Pool-B). We denote SIMPLE using Model Pool-B as SIMPLE to distinguish it from the one using Model Pool-A.
4.2 EVALUATION RESULTS ON DOMAINBED
This section presents the evaluation results of DomainBed suite, on which we compare SIMPLE with general DG algorithms to verify its effectiveness and efficiency. Specifically, we compare algorithms using ImageNet-pretrained models only (e.g.. SIMPLE using Model Pool-A) and using pretrained models from diverse pretraining datasets (e.g., SIMPLE+ using Model Pool-B), resp.
Performance comparison. As shown in Table D SIMPLE achieves an average performance of 72.0% with saageNet-pretraiing by 3.9% and ranks first in all datasets. SIMPLE even outperforms the second-bes, method by 12.2% on OfficeHome dataset. The results show that our proposed paradigm of using only pretrained models without fine-tuning is more effective than the traditional paradigm that requires fine-tuning. Furthermore, by using Model Pool-B that includes models pretrained on other datasets, SIMPLE further improves the average performance by 6.1 1 over SIMPLE and beats other strong baselines.
models, exceeding the current SOTA competitor (MIRO + SWAD)
Training cost comparison. As SIMPLE does not fine-tune these models but use the matching network to dispatch models, the training cost is almost negligible versus general DG methods. As shown in Table 2 the number of learnable parameters of SIMPLE (0.9M) is minor compared with the normal image backbone network used by MIRO (25.6M for ResNet-50 and 79. 7M for RegNetY- 16GF). In addition, training of SIMPLE uses noticeably less time with 1000 times speedup.
Inference cost comparison. Additionally, the inference cost is controlled through seleetively ac- tivating appropriate models with the highest &(< K) matching metric values, for inference. This reduces the inference cost to a large extent, though more than 200 pretrained models have been in- corporated in the model pool. As shown in Table 2 although SIMPLE uses more inference cost than the model using ResNet-50 as the backbone. SIMPLE uses less inference time than the ex- isting SOTA (MIRO SWAD) using RegNetY-16GF (Singh et al. (002)) the backbone. That is, SIMPLE+ obtains new SOTA results with significantly higher training and inference efficiency.
The significant training time advantage of SIMPLE and its surpassing performance indicate that SIMPLE is an effective and efficient paradigm for DG. We illustrate the matching preference of SIMPLE, i.c., which models are typically dispatched in specific domains, in Appendix A.
4.3 ABLATION STUDIES
We verify the effectiveness of SIMPLE design by answering two research questions (RQ) as below.
(RQ1) Does SIMPLE achieve better generalization performance than the best single model in the model pool? To verify, we train an individual adapter on source domains for each pretrained model and show the "cheating" upper bound of single model performance by reporting the best single model performance best single model has outperformed the current SOTA (MIRO + SWAD) on certain datasets (c.g., OfficeHome and VLCS). Nevertheless, SIMPLE beats it in all the comparisons (RQ1). This implies
test set test-best single model in Table 3 As shown in Table T the
8

---

Published as a conference paper at ICLR 2023
that SIMPLE has somehow compensated for the drawbacks brought by NFL of single model in DG, by utilizing different pretrained models and dispatching them selectively to OOD samples.
(RQ2) Is it necessary to assemble pretrained models according to their specialty? The analysis in Section 2.J shows that pretrained models possess certain generalization ability. Thus, a natural approach to simply aggregate pretrained models without considering their relative specialty on different samples, e.g.. randomly select i models for each sample and average their outputs as the final prediction (Lakshminarayanan et al. 2017). The results of the random ensemble are shown in Table illustrating that random ensemble lags behind SIMPLE by large margin. This verifies the necessity to select and ensemble pretrained models based on their specialty over samples (RQ2).
4.4 PRACTICAL TIPS FOR CONSTRUCTING MODEL POOLS
In this section, we investigate the impact of different model pool properties on the generalization performance, to provide useful guidance for the composition and utilization of model pools. The corresponding results are presented in Table 3 Specifically, we focus on the size and diversity of model pools by comparing four different model pools, i.c., Model Pool-A-Small (ImageNet- pretrained, 15 models), Model Pool-A (diverse pretraining datasets, 17 models), and Model Pool-B (diverse, 283 models). The composition of these model pools can be found in Appendix A.3
244 models), Model
Tip 1: Use larger model pool. Based the analysis in Section larger model pool is fa- vored as it increases the probability that the model pool contains models that match well with each OOD sample. This is consistent with the comparison of Model Pool-A-Smal with Model Pool-A, and Model Pool-B-Sman with Model Pool-B. Both types of model pools show significantly better generalization performance as the size of the model pool increases.
Tip 2: Incorporate diverse models. The performance comparison of Model Pool-A(-Smail and Model Pool-B(-Small) indicates that adopting diverse pretrained models from different pretraining domains can enhance DG performance remarkably.
Tip 3: Increasing model diversity matters more than increasing model pool size. On top of Model Pool-A-Small, Model Pool-A incorporates 224 additional pretrained models from Ima- geNet pretraining domain, while Model Pool-B-Smal adds more models that are pretrained on YFCC100M (Radford et al. 2021). However, SIMPLE performs better upon Model Pool-B-Sman suggesting that the diversity model pool may be more important for generalization.
The above tips from empirical observations that, more models and more diversity can improve gen- cralization performance, again suggest that there is no free lunch for DG. Thus, different pretrained models needed to address DG. SIMPLE provides way to realize this goal both effectively and efficiently by (1) adapting pretrained models to unseen domains via label space adaption with a low cost and (2) dispatching fit models from large model pool to handle each OOD sample.
4.5 SENSITIVITY ANALYSIS
We conduct sensitivity analysis for ie and sample feature extractor SRo detailed in Appendix A.9 and the main findings include: (1) There a marginal effect on the generalization performance im- provement brought by increasing k, and SIMPLE can outperform SOTA baseline (MIRO + SWAD) even when & 2; and (2) SIMPLE robust to the selection of feature extractors.
CONCLUSION
Despite recent studies suggesting that network architectures and pretraining practices affect gener- alization ability to a large extent, no work has explored the use of these easy-to-obtain pretrained models to address domain generalization. Our work provides a comprehensive analysis of generaliz- ing pretrained models to unseen domains and reveals that there is no free lunch of pretrained models in DG. Based on that, we propose novel DG paradigm that leverages fixed pretrained models and dispatches them to OOD samples based on their matching metric to the target task. Exten- sive evaluations show that our proposal is a promising alternative for DG, with better generalization performance and significantly higher training efficiency compared to existing DG methods.

---

Published as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907 (III::997.00893,, 2019.
Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Im- proving model selection and boosting performance in domain generalization. arXiv preprint arXiv:2110.10832, 2021.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gen- eralization using meta-regularization. Advances neural information processing systems, 31, 2018.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456 473, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specifee features to en- hance domain generalization. Advances in Neural Information Processing Systems, 34:21189- 21201, 2021.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Out-of-distrriutton generalization in kernel regression. Advances in Neural Information Processing Systems, 34: 12600-12612, 2021.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Infor- mation Processing Systems, 34, 2021.
Junbum Cha, Kyungjac Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. arXiv preprint arXiv:2203.10788 2022.
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path
networks. Advances in neural information processing systems, 30, 2017.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generaliza- tion via model-aggossin learning of semantic features. Advances in Neural Information Processing Systems, 32, 2019.
Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snoek, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. In European Conference on Computer Vision, pp. 200-216. Springer, 2020.
Chen Fang, Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1657-1664, 2013.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Yoav Freund. Boosting weak learning algorithm by majority. Information and computation, 121 (2):256-285, 1995.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 17(1))2096-2030, 2016.
Rui Gong, Wen Li, Yuhua Chen, and Lue Van Gool. Dlow: Domain flow for adaptation and general- ization. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 2477-2486, 2019.
Raphael Gontijo-Lppes, Yann Dauphin, and Ekin D Cubuk. No one representation to rule them all: Overlapping features of training methods, arXiv preprint arXiv: /t..///..2/9.. 2021.
10

---

Published as a conference paper at ICLR 2023
Carlos R González and Yaser S Abu-Mostafa Mismatched training and test distributions can out- perform matched ones. Neural computation, 27(2):365-3877 2015.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint 1xXiv:2007.014344 2020.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions pattern analysis and machine intelligence, 121100993--101, 1990.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv:2111.0637 2021a.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural col- laborative filtering. In Proceedings of the 26th international conference on world wide web, pp. 173-182, 2017.
Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines. Pattern Recognition, 110:107383 2021b.
Jie Hu, Shen, and Gang Sun. Squeeze-and-excitatnn networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Forrest landola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv: 1869, 2014.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv: 1602.0776 2016.
Maximilian Ilse, Jakub M Tomezak, Christos Louizos. and Max Welling. Diva: Domain invariant
variational autoencoders. Medical Imaging with Deep Learning, pp. 322-348. PMLR, 2020.
Rawal Khirodkar, Donghyun Yoo, and Kris Kitani. Domain randomization for scene-specific car detection and pose estimation. In 2019 JEEE Winter Conference Applications of Computer Vision ( (WACV), pp. 1932-1940. IEEE, 2019.
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In European Conference Computer Vision, pp. 158 171. Springer, 2012.
Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jackoo Lec. Selfreg: Self- supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9619-9628, 2021.
Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. A broad study of pre-training for domain generalization and adaptation. arXiv preprint arXiv:2203.. /819, 2022.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning distorts pretrained features and underperforms outtof--isttiibutinn. In International Con- ference on Learning Representations, 2021. 2021.
Ananya Kumar, Aditi Raghunathan, Robbic Jones, Tengyu Ma, and Percy Liang. Fine- tuning distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202. 10054, 2022.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv: 1172,01444 2016.
11

---

Published as a conference paper at ICLR 2023
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pp- 5542-5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5400-5409, 2018b.
Lei Li, Ke Gao, Juan Cao, Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 224- 233, 2021.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018c.
Ziyue Li, Kan Ren, Yifan Yang, Xinyang Jiang, Yuqing Yang, and Dongsheng Li. Towards inference efficient deep ensemble learning. arXiv preprint arXiv:2301. 12378, 2023. 2023.
Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Rong Jin, Xiangyang Ji, and Antoni B Chan. An empirical study on distribution shift robustness from the perspective of pre-training and data augmentation. arXiv preprint arXiv:2205. 12753, 2022.
Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Best sources forward: domain generalization through source-specific nets. In 2018 25th IEEE international conference image processing (ICIP), pp. 1353-1357, 1357. IEEE, 2018
Mohammad Moghimi, Serge Belongie, Mohammad J Saberian, Jian Yang, Nuno Vasconcelos, and Li-Jia Li. Boosted convolutional neural networks. In BMVC, volume 5, pp. 6, 2016.
Li NIu, Wen Li, and Dong Xu. Multi-view domain generalization for visual recognition. In Pro- 2015.
ceedings of the IEEE international conference computer vision, pp.
Xi Peng, Fengchun Qiao, and Long Zhao. Out-of-domain generalization from a single source: An uncertainty quantification approach. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1406-140 1415, 2019.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference roboties and automation (ICRA), pp.
IEEE, 2018.
Fengchun Qiao. Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556- 12565, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2023
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh,, and Sridha Sridharan. Molti-componett image translation for deep domain generalization. In 2019 JEEE Winter Con- ference Applications of Computer Vision (WACV), pp. 579-588. IEEE, 2019.
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. Correlation-aware adversarial domain adaptation and generalization. Pattern Recognition, 100: 107124, 2020.
Robert E. Schapire. The strength of weak learnability. Machine learning, 5(2):197-227) 1990.
Mattia Segu, Alessio Tonioni, and Federico Tombari. Batch normalization embeddings for deep domain generalization. arXiv preprint vrXiv:2011.1267. 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and
Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint arXiv: 1804. 10745, 2018.
Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C Yuen. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10023-10031 2019.
Yuge Shi, Jeffrey Seely, Philip HS Torr, Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.00933, 2021.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 804-814, 2022.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pp. 143-450. Springer, 2016.
Xavier Thomas, Dhruv Mahajan, Alex Pentland, and Abhimanyu Dubey. Adaptive methods for aggregated domain generalization. arXiv preprint arXiv:2112.04700 2021.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 969-977, 2018.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 2014.
Vladimir N Vapnick. Statistical learning theory. Wiley, New York, 1998.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017.
Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu. Visual domain adaptation with manifold embedded distribution alignment. In Proceedings of the 26th ACM international conference Multimedia, pp. 402-410 2018.
Jindong Wang. Yiqiang Chen, Wenjie Feng, Han Yu. Meiyu Huang, and Qiang Yang. Transfer learning with dynamic distribution adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(1):1-25, 2020a.
13

---

Published as a conference paper at ICLR 2023
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: survey on domain generalization. IEEE Transactions Knowledge and Data Engineering, 2022.
Zhen Wang, Qiansheng Wang, Chengguo Lv, Xue Cao, and Guohong Fu. Unseen target stance detection with adversarial domain generalization. In 2020 International Joint Conference
Neural Networks (IJCNN), pp. 1-8. IEEE, 2020b.
Ross Wightman. Pytorch image models. 2019.
http:://gthbb.cmmmm/mo htwp:/huwb..com///igh.t.taaaaaaa
eytorch-image-me
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuffi, Ira Ktena, Taylan Cemgil, et al. A fine-grained analysis distribution shift. arXiv preprint arXiv:2110.11328, 2021.
David H. Wolpert. The lack of a priori distinctions between learning algorithms. Neural Computa- fion, 8(7):1441-3300, 1996.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7977 2022.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Chongzhi Zhang, Mingyuan Zhang, Shanghang Zhang, Daisheng Jin, Qiang Zhou, Zhongang Cai, Haiyu Zhao. Xianglong Liu, and Ziwei Liu. Delving deep into the generalization of vision trans- formers under distribution shifts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7277-7286, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 171..0941, 2017.
Shaofeng Zhang, Meng Liu, and Junchi Yan. The diversified ensemble neural network. Advances in Neural Information Processing Systems, 33, 2020.
Zijun Zhang. Improved adam optimizer for deep neural networks. 2018 IEEE/ACM 26th Inter- national Symposium Quality of Service (IWQoS), pp. 1-2. IEEE, 2018.
Fan Zhou, Zhuqing Jiang, Changjian Shui, Boyu Wang, and Brahim Chaib-draa. Domain general- ization with optimal transport and metric learning. arXiv preprint arXiv:2007.105777 2020.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. IEEE Transactions on Image Processing, 30:8008-8018, 2021.
Tianyi Zhou, Shengjie Wang. and Jeff A Bilmes. Diverse ensemble evolution: Curriculum data- model marriage. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 5920,, 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697-8710, 2018.
A APPENDIX
A.1 RELATED WORK
Mainstream DG research generally addresses distributional shifts through data manipulation (Tobin
et al. 2017; Peng et al., 2018). robust representational learning (Tzeng et al.) 20144] Arjovsky et al.
2019 and exploration of new learning strategies such ensemble learning Cha al. (2021). We
14

---

Published as a conference paper at ICLR 2023
here only review studies that are mainstream in DG and that relate to method. For more comprehensive survey, please refer to Waang et al. (2022).
Data manipulation. DG arises from the lack of sufficient data to feed machine learning models, so they are biased to the training distribution, resulting in an inability to work well on the test dis- tribution (Wang al. 2022). Thus, various methods resort manipulating the training samples to simulate unseen target domains, among which there are two techniques, namely data augmenta- tion (Shankar et al., 2018) and data generation (Li et al. 2021). Data augmentation has been widely used in general machine learning models to enhance their generalization ability by avoiding over- fitting (Shorten& & Khoshgoftaar (019). Inspired by its importance, Tobin et al. (2017) first uses for DG to simulate test distributions, and subsequently, [Peng et al. (2018); Khirodkar et al. (2019); Tremblay et al. (2018) also adopt data augmentation various ways. addition to the common feature augmentation, |eng et al. (2022) further propose to perform label augmentation. In the field of data generation, instead of augmentation, methods generate new samples or domains by means of such as Mixup (Zhang et al., 2017), auto-encoder (Qiao et al. 2020), generative adversarial net- work (Rahman et al. 2219)).
Representation learning. In another promising direction, some DG studies attempt to extract domain-inwe invariaat features that can be generalized to unseen domains Been-David et al. 2006). or separate domain-shared and domain-specifer parts from the features for generalization (Khosla et al. 2012). Motivated by the transferability of domain-invariant features across varied domains and the expectation of their generalization ability to unseen domains, methods try to obtain specific feature space that is invariant to domain labels. et et al. (2)18b) first introduces ad adversarial training in DG, allowing the generator seeks to trick the discriminator about the domain labels of images and thus generate domain-invarian features. Several works (Shao et al. [2019] Rahman et al. 2020; Wang al., 2020b) follow this direction. In the line of extracting domain- invariant features, other works explicitly align feature distribution learned from source domains, with the differences measured by Wasserstein distance (Zhou et al. 2020), maximum mean discrepancy (Wang et al., 2018, 2020a), or mean and variance (Peng et al., 2019). At a meta level, BBlaji et al. (20T8) consider learning regularization function on the classifier to avoid biasing to domain -speiff information. In contrast, for robust representation learning, some work aims separate domain-invariane variables from the
specific ones of the learned features (Niu et al. 2015 lsse et al. 2020 He et al. 20215).
Zhou al. (2018); Li al. (2023) exploit multiple models to produce prediction results and combine the results with various techniques, e.g., boosting [Schapire (1990); [Freund (1995): Moghimi et al. (2016) or mean aggregation /Zhou et al.] (2018); Zhang et al. (2020), aiming at achieving better performance than individual model alone. As well-known technique to improve generalization performance, ensem- ble learning has also been explored in DG. One general idea of using ensemble learning by existing utilize the relationship between unseen domains and source domains. Specifically, in the work of Mancini et al. (2018), they train an individual classifier for each specific source do- main and one additional classifier to predict the probability of each domain that sample belongs to, and aggregate their predictions by weights accordingly. In addition, aadition,,Segu et al.] (2020) proposes to maintain domain-specifee batch normalization layers, which will be weighted for aggregation in inference. /hhou et al. (202T) instead trains only domain-specific classifier heads while letting them share the same feature extractor. SWAD (Cha et al. (022I) and EoA (Arpit et al., 2021 instead use model ensemble and weight ensemble directly to improve the generalization. Though promising in enhancing performance, ensemble learning is criticized for its computational efficiency, which hinders the practical application (Wang et al.) 2022).
Ensemble learning. Ensemble learning methods Hansen & Salamon 1990
DG methods
Relation to existing work. The main difference between the existing DG works and SIMPLE resides in the fact that these efforts seek to find an optimal model that can settle all distributional shifts. Even DG algorithms that use ensemble learning are trying to find an optimal ensemble that has flatter loss landscape and better generalization ability (Cha et al., 202I). To this end, existing works use a specific pretrained model for initialization, as this performs better than training from scratch (Wortsman et al. 2022), and then fine-tune it with training algorithms like data manipulation, robust representation learning, or ensemble learning as discussed above. In contrast, recent studies have shown that such specific pretrained models may not be sufficient to solve distributional shifts, and that pretraining strategies need to be chosen for different types of distributional shifts. To the best of our knowledge, S SIMPLE is the first work to address DG by directly using different pretrained models without fineingning, by reformulating DG model- sample matching problem. Moreover,
15

---

Published as a conference paper at ICLR 2023
SIMPLE does not fine-tune these pretrained models, but uses domain-level label space adaptation to achieve effective adaptation, which has significantly reduced the cost of training and adaptation.
A.2 MORE DISCUSSION ABOUT SIMPLE
Towards a robustness algorithm that can effectively and efficiently leverage extensive pretrained models for DG, there are certain challenges that remain in our paper. Currently, SIMPLE is not feasible for adding new pretrained models into the model pool for direct utilization. Although re- training is lightweight, rapid development of pre-trained models. We leave this problem, i.e., the cold-start problem which has been long studied in recommendation fields, as further work.
need more straightforward approach address this problem due the
A.3 PRETRAINED MODEL POOLS
This section presents the pretrained models used in SIMPLE. With more and more pretrained mod- els being published. it is straightforward to build pretrained model pool consisting of several public pretrained models for direct adaptation to novel domains. In particular, these models are categorized according to their pretraining domain as saaageeet-pretriing model pool (Model Pool-A) and pool (Model Pool-B) containing models pretrained on datasets other than the ImageNet dataset.
Model Pool-A. DG methods commonly use AanageNet-pretraiing
models to initialize the model weights for further fine-tuning(Kin et al.) (02T). To present fair comparisons with those methods, we first construct an
model pool (Model Pool-A). Specifically, we collect 244 models with diverse network architectures and learning objectives from popular 3rd-party reposito- dataset, i.c., ImageNet, will share the same label adapter transforming the vanilla model outputs to the target label space. Therefore, for Model Pool-A, only one label space adapter will be trained, transform the original predictions for 1,000 classes in ImageNet into the predictions of the target class space.
ries Note that, the models pretrained the
Previous studies also found that ImageNet--retraaneed model (e-g, ResNet-50) is not sufficient for generalization [Kumar et al. 2021: Kim et al., (222), suggesting the need for models pretrained on different datasets. Therefore, models that trained on datasets including ImageNet-211 YFCC100M (Radford et al. 2021). In- stagram 3.6B (Singh et al., 2022), as Model Pool-B. total, we add 39 models, whose weights are obtained from the same sources or from their public repositories!!! tains 283 pretrained models with varied network architectures, learning objectives, and pretraining datasets.
Model Pool-B.
further expand the Model Pool-A by incorporating
77 Thus, Model Pool-B con-
To analyze the impact of model pool size on performance, we take a portion of the models from these two model pools and use them to construct Model Pool-A-Small and Model Pool-B-Small. Specifi-
cially, in Model Pool-A-Small, we incorporate the architectures including AlexNet (1) (Krizhevsky
(3) (landola et al.) 2014). haaullPath-Networ---ee ctt al. 2017). NASNeMMobile (1) Zoph et al. 2018). Res/4t-18/34/50 (3) (He et al., 2016). SE-
cet al.
2012). DenseNet-121/1en
(1) (Chen
ResNet-50 (1) Hu al. (2018). SqueezeNet- 1.01.. (2) (landola al. 2016), and MAE-VIT- Base/Large/Huge (3) (He et al. 20) On the other hand, several DG algorithms also use models pretrained on other datasets, such 1G-1B (Arpit al.] (021) and ILSVRCI2 (Thomas et al.,
2021). Therefore, we build Model Pool-B-Small which contains two more CLIP models (Radford) al. 2021) ViT-B/16 and ViT-B/32, which are trained on subset of the YFCC100M dataset of roughly the same size as ImageNet.
A detailed list of pretrained models is given in table with the sources of the models, the dimen- sions of their outputs and FLOPs (floating point operations per second).
htta:://4gthbb..ccmm//om////1/1
16

---

Published as a conference paper at ICLR 2023
A.4 EMPIRICAL EVIDENCE OF NO FREE LUNCH
In this section, we elaborate on the empirical analysis in Section 2 and provide more experimental details and insights for the analysis.
Transfer and performance measurement of pretrained models. As discussed in Section 2 in- stead of fine-tuning or conducting linear probing to adapt those pretrained models be a predictive model for unseen target domains, we only train a label space adapter that learns the mapping function hy. Then we train this shared adapter hy with the empirical loss ED (wb, p) without fine-tuning the pretrained model parameters [ as formally described in Section 3.2, With the trained adapter, we use the likelihood of the ground truth label p(vs I (x4; on the i-th sample produced by each adapted model, which also indicates the confidence of the ground truth label y/ of the model and Eyey ply Iz;;; 0%) L. We utilize this likelihood as the evaluation metric of its sample-level model specialty.
Experimental settings. First, we analyze the specialty distribution of each pretrained model from an aggregation view (i.e, domains and classes, respectively), and we verify if there exists dominant pretrained model that generalizes best across different unseen domains. We calculate domain-level model specialty as summation of the sample-level specialty over all domain samples as L(x.,e)>D log p(u: | zy;; 0%)), on TerraIncognita (Beery et al. 2018) with four domains. To reflect the relative model performance, we perform min-max normalization for model specialty values in the same domain. These complete results are shown in Figure 4 with only partial results put in Section 2 for a clearer presentation. Then, we further examine whether performance divergence also exists the finer-grained class-level. Similar that domain-level, Figure + presents the relative model performance on 10 classes in TerraIncognita. In addition, to clearly compare model specialty differences at the two levels, we present heatmaps of specialty differences (measured by -ullback-Leibler divergence) for domain and class pairs, respectively in Figure
Based on the results of comparing the performance of all pretrained models on different domains and classes, we can draw more empirical insights as follows:
(1) Most importantly, there is no single model that performs best both across domains and classes;
(2) The divergence of the performance distribution is significantly more prominent at the class level than at the domain level, as evidenced by the comparison of the Figure55ja and (b);
(3) As a side finding, models that perform well in the 'Bird' class generally perform well in the 'Bobcat' class, as indicated by their pair-wise Kullback-Leibles divergence values. However, models that perform well in 'Bird' and 'Bobcat' usually perform poorly in 'Dog' and 'Rabbit' classes. This can be viewed as another piece of evidence of NFL. for shifting distribution shifts.
A.5
No FREE LUNCH THEOREM
We here detail the formal version of Theorem with
with related definitions.
Considering a kernel regression task, training samples {ø* y=", are sampled from i.id. and the labels which contain noisy are generated from a target function y ff (x*)) + e where the noise covariance is (e"te") = e2 2pp. Then the model for regression is trained by minimizing the empirical error (ERM loss):
8r = =arg -aaggnnn min E 2
I
+ A(F.)
JEH
u=1
where H is a Reproducing Kernel Hilbert Space (RKHS) associated with positive semi-definite > IR, and (...)x is the Hilbert inner product. The generalization error
kernel K x') R D x R
on the test distribution p(x) ED (f)(x)) f(x)) 2) f(a)" which is random variable whose
value depends on the sampled training dataset. Therefore, the generalization error is averaged over the distribution of all datasets with sample size P:
Eg (I) ((x) (())
Based the problem setting, Canatar al.
2021 proposes the following proposition:
17

---

Published as a conference paper at ICLR 2023
Relative performance between pretrained models across different domains and classes
domain
class
L1D0
Hird
Cat
Dog
Opansur
Faccoon
Bebcat
Ciyots
Empay
Rablit
Squirel
Figure 4: Performance comparison of all pretrained models on different domains and classes of the Terralncognita benchmark. Note that we omit the model names for clarity, and each row represents the performance of one model.
18

---

Published as a conference paper at ICLR 2023
40
Bird
3.5
2.3
5.1
17.5
0.33
1.5
Bobcat
0.43
3.4
2.1
11
3.7
1.3
4.7
150
L338
0.72
4.2
Coyote
4.4
1.7
0.96
1.5
9.5
1.3
1.8
2.5
12.5
1.5
Dog
2.8
1.2
9.6
1.2
1.9
100
4.7
1.5
Opossum
0.75
0.83
1.8
2.4
5.4
1.5
10
18
0.67
Raccoon
5.7
0.95
2.3
0.38
3.3
1.6
1.7
Squirrel
2.9
3.4
16
7.1
2.3
3.3
4.3
8.6
0.61
0
L100
L38
L43
L46
50
Bird
Bobcat
Cat
Coyote
Dog
Emply
Opossum
Rabbit
Raccoon
Squire
(a) Domain-level.
(b) Class-level.
Figure 5: Kullback-Leiber divergence between the performance distribution over all the leveraged
pretrained models in domain-leval (a) and class-level (b), respectively.
Proposition 1. (Restating Proposition I in Canattrr et al. (2021)) Consider the kernel regres- sion problem defined above, where the model is trained on p(z) and tested p(z). Consider the Mercer decomposition of the RKHS kernel (x,xx) (xx) T A where we defined M x M (M possibly infinite) diagonal eigenvalue matrix Apy 7pōpr and the column vector (xx) (01(z), (2(z)... .e.....,6.. 62M (2)), with (c)(
I. Also consider expansion of the target function in the Mercer basis [(x a ((x) with coefficient vector The dataset averaged OOD generalization error is given by:
Fg E..((a)
7 a (PA AI)
la
1
ID
distribution shift error
k= *=A+NT(P+*M y= where K. must be solved self celf-consistenlly, and we defined the M x M overlap matrix
y= P (P PI
= (P ( -
6 pT = d=((t))d O'= 1-7
Here Fig Fllp(()) denotes the generalization ervor when both training and test distributions are matched p(æ) (i.e., in-distribution error) and is given by:
El.p(a)
52 1-7
T T(PA KI) 2a
Further, the expected estimator is:
Puee
(f" (1); Σ
Pnp ap)p(a))
A.6 EVALUATION SETTINGS
Evaluation protocol. We follow DomainBed protocol to conduct our evaluation, for fair com- parisons with baselines. We use the traniig-domain validation set protocol for model selection. Specifically, one domain in dataset is selected as the target domain and the rest source domains, from which 20% of samples are used as the validation set. runs are repeated times using dif- splits. The out-of-domain test performance
ferent random seeds, thus, with different averaged over all domains will be reported for each dataset. addition, we use the standard number of iterations of 5,000 for all datasets, with early-stop based validated accuracy.
Hyperparameter tuning. Here we state the details hyperparameter tuning in our experiments. We separate Adam optimizers (Zhang] 2018) for the ensemble network and label space adapter. Tible the hyperparameters to tune and their search space. For each domain, we sweep through 48 different hyperparameter settings.
19

---

Published as a conference paper at ICLR 2023
Table 4: Hyperparameters we tune for SIMPLE.
Hyperparameter
Search Range
learning rate used for matching network
[0.0001.0.1]
learning rate used for label space adapter
[0.0001.0.1]
loss weight ae
[0.01, 1.0]
loss weight ad
[0.01. 1.0]
loss weight ag
[0.01, 1.0]
weight decay of Adam optimizer
Nee3.le--.1ee-.lees le-7]
batch size
[32, 64, 128, 256]
A.7 BASELINES
Stochastic Weight Averaging Densely (SWAD) ensemble during model training.
(Cha et al. 2021): SWAD performs weight
Ensemble of Average (BoA) (Arpit et al. 202 EoA combines both model ensemble and weight ensemble by taking an ensemble of moving average models from runs. They ex- periment with three different pretrained models as initialization. The first one is pretrained on ImageNet with ResNet-50 and the second is pretrained on both ImageNet and a much larger additional dataset, IG-IB, with more advanced backbone, ResNeXt-50 (Xie al.) 2017). Additionally, with a pretrained RegNetY-16GF (Singh et al.) 2022), EoA achieves its best results. We denote the last one as BoA+ to indicate it uses the additional dataset and to compare with SIMPLE+.
Random ensemble: In contrast to SIMPLE of learning to select models for ensemble, we also compare it with an average ensemble of & models chosen randomly for each sample.
A.8 MORE ABLATION STUDY
Here we introduce an additional baseline that uses the same of ensemble weights for all sam- ples in domain, rather than generating different ensemble weights to aggregate model predictions differently for each OOD sample.
For ensemble- based approaches. the overall prediction is given by d: w (zx).... fx (x,)), where wk the weight for aggregating the prediction of k-th models fu (3:). SIMPLE, the ensemble weights are given by w = MLP(c. C), where c; is the embedding for the i-th sample and C' contains the embeddings for all the models.
Therefore, there is a special case of proposed ensemble approach, where the ensemble weights w R are randomly initialized and optimized through back-propagation. Here in this simplified version, to is shared for all the samples in the dataset and does not incorporate model or sample in- formation. Compared with this simplified version, SIMPLE explicitly leverages sample and model information (encoded in the embedding vectors c; and C') to generate specialized ensemble weights for each sample, which is more fine-grained. With model and sample embedding, SIMPLE enjoys much lower training cost when incorporating
pretrained models that are not in the model pool.
We implement this special case and conduct experiments on OfficeHome dataset to compare with SIMPLE. By performing sufficient hyperparameter tuning, such simple version achieves erage accuracy of 82.3% on OfficeHome, which worse than 87.7% of SIMPLE (and even worse than 84.6% of SIMPLE using the much smaller Model Pool-A) and 83.9% of the existing SOTA. in contrast suggests that, incorporating information about models and the sample to conduct fine- grained model-sample matching in SIMPLE, is necessary and more effective, than using set of weights and optimized by back-propagation.
A.9 SENSITIVITY ANALYSIS
A.9.1 ANALYSIS OF È IN T MODEL. SELECTION
Intuitively, using more models in an ensemble may lead to better performance (Zhang et al. 2020). However, larger ensemble size also means that the ensemble consumes more computation infer-
20

---

Published as a conference paper at ICLR 2023
ence. Thus, in inference, choose to selectively activate l (< K) models with higher ensemble weights. It is necessary to verify the impact of how many models are used for the prediction of each sample the final performance, which can help balance the effectiveness and efficiency of our approach.
Settings, To demonstrate the sensitivity of changes in the parameter (the number models activated in inference) to the final generalization performance, evaluated the performance of SIMPLE+ with different values of k in the top-k selection the OfficeHome dataset. Specifically, evaluate k E [l,.... 10), with limited and average sweep of the hyperparameters save computational time. Therefore, the accuracy may not be optimal.
Averaged accuracy OfficeHome
86.9 86.91 8.002
85.94
85.33
85
84.59
85.34
84
SOTA
82:61
0
5
10
Activated models inference
Figure 6: The impact of & values of top-k selection in inference, on the generalization performance of SIMPLE+ on the OfficeHome dataset.
As illustrated in Figure the highest accuracy of 87.02% is obtained when the number of activated models reaches the highest value we set. This indicates that the generalization performance can benefit from using more models to compose the ensemble for the final prediction. However, another important finding this result is that the marginal effect of increasing the number of models de- creases as the number of activated models increases. For example, when le is increased from 1 to 2. SIMPLE+ gains 1.22% performance gain. In contrast, when shifts from 8 to 10, the perfor- mance gain is only 0.11%. Therefore, it is predictable that since matching networks can provide decent ranking of pretrained models, we can obtain promising generalization performance with lim- ited activation models to save computational cost. Note that when k = 2, our method has obtained a performance that exceeds the existing SOTA (MIRO + SWAD).
A.9.2 ANALYSIS OF FEATURE EXTRACTOR SELECTION
We perform an ablation study verify the robustness of our matching network the selection of feature extractors. Here we provide details for that claim in Section
Settings. In general, ResNet-based models used feature extractors. We compare the perfor- mance of our method here with ResNet-based feature extractor (ResNet-34) and more advanced network structure (EfficientNet-B7 with Noisy Student) see if these changes result in significant performance differences. Specifically, ResNet-34 and EfficientNet-B7. obtain ImageNet top-1 classification accuracy of 75.0% and 86. 86.9%, respectively. This ablation study is performed on the OfficeHome dataset, with results shown in Table 5
Table 5: Performance comparison of SIMPLE+ using ResNet-34 and BfficientNet-B--NS as the feature extractor. domain
C
P
R
avg.
R4
88.5 +8.5
76.7700.
92.8+0.9
92.640.3
87.7
BIficientNetBB-NSS
85.840.8
77.0+0.7
92.240.5
91.940.7
86.7+0.6
21

---

Published as a conference paper at ICLR 2023
As can be seen, using ResNet-34 or EfficientNet-B7-NS results in similar performance, without one dominating in all domains. Therefore, feature extractor selection has little impact on the general- ization performance of SIMPLE, showing its robustness. Meanwhile, it is noted that the number of parameters of EfficientNet-B7-NS is three times higher than that of ResNer-34. Therefore, we choose ResNet-34 because it provide good performance with less computation cost.
A.9.3 ANALYSIS OF LABEL SPACE ADAPTER TRAINING
As detailed in Section 3.3, the label space adapter of SIMPLE is trained by two losses, Less (1b, p) and Catupter. and the training process will be influenced by the ensemble weights wy. In this section, analyze and verify whether such an influence affects the training of adapters.
Note that, we train the same adapter for all the pretrained models from the same pretraining domain, which more efficient (than training different adapters for different pretrained models) and also avoids unexpected training instability. When considering the large model pool with various models from different pretraining domains (e.g., Model Pool-B),
believe the influence is acceptable or even beneficial since: (1) adapters will observe all the training samples, so their training is sufficient; (2) since the number of parameters of adapters relatively small, the training of adapters will not have significant impact on the final performance; (3) although the training of adapters is influenced by the ensemble weights, unbiased because the ensemble weights are optimized according to the final objective.
Furthermore, we conduct experiments to verify whether training label space adapters under the influence of ensemble weights will affect the training of adapters or degrade the performance.
Settings. We compare single model performance with and without the influence of ensemble weights the corresponding label space adapter. For influence on adapters, we separately train individual adapters for each model. We select several models which are assigned with small ensemble weights in SIMPLE. The evaluation is on 'art' domain of the OfficeHome dataset.
Figure shows the performance difference between (1) a model with the adapter trained with and without ensemble weights (blue bars); and (2) our method SIMPLE and the SOTA baseline for reference (orange bar). As can be seen, (1) single model performance does not significantly degrade under the influence of ensemble weights (even less than the difference between our method SIMPLE and the existing SOTA); (2) moreover, for some models in Figure 7 training a shared adapter (i.e., influenced by the ensemble weights) can make a single model perform even better. The possible reason is that training a shared adapter for multiple models avoids overfitting, leading to better generalization.
A. TRAINING AND INFERENCE COST COMPARISON
We compare the training and inference costs of SIMPLE with general DG methods, including SOTA and methods that use ensemble learning. Specifically, for training costs, we evaluate the number of learnable parameters and the overall training time. And for inference, we compute GFLOPs. To fairly compare training costs, we run experiments of ERM, SWAD, and SIMPLE on a single Nvidia Tesla V100 and compare their overall pack-propagation time from the start of training the end (or early-stop). And based on the statistics of ERM and SWAD, we estimate the training time for EoA and MIRO, respectively. The results are shown in T
Training cost comparison. As shown in Table 2 training SIMPLE paradigm uses noticeably less time. SIMPLE takes only 0.1% of the time of ERM on PACS. The significant training time advantage of the method and its surpassing performance suggest that SIMPLE is an effective and efficient paradigm for domain generalization.
Inference cost comparison. Although ensemble methods like EoA and SEDGE achieve better generalization performance at the cost higher inference FLOPs, SEDGE still manages to save a large amount of inference cost compared to the previous best ensemble model (half of the inference FLOPs compared to EoA) This is because SIMPLE only selects models with the highest k(< K) ensemble weights. Therefore, only i of K models are activated for inference per sample, which reduces the inference cost to large extent. In addition, compared to SOTA (MIRO + SWAD), which uses larger network architecture achieve better results than using ResNet-50, SIMPLE
22

---

Published as a conference paper at ICLR 2023
Performance difference
-2
refer- ence
Model index
(Model with separate adapter) minus (Model with shared adapter) (SIMPLE) minus (existing SOTA (MIRO))
Figure 7: The difference of single model performance with and without the influence of ensemble weights, on the OfficeHome dataset.
keeps the inference cost using significantly lower GFLOPs than SOTA uses to obtain the new SOTA results.
Table 6: The comparison training and inference cost. Here the training time implies the overall
pockppropagation time. The training times of ERM and SWAD are derived from the statistics of
runs, and we estimate the training time of EoA and MIRO based on that of ERM. The run for SWAD
on DomainNet failed due to oft-of-memory.
Learnable
Training time (L)
Inference
Algorithm
parameters (1)
PACS
VLCS
OfficeHome
Terra
DomainNet
GFLOPs (J)
ERM
25.6M
3.4h
3.6h
3.3h
3.4h
9.8h
3.9
with ResNet-50
SWAD
25.6M
2.1h
2.4h
2.1h
2.1h
3.9
with ResNet-50
BoA
153.4M
20.4h
21.6h
19.8h
20.4h
58.8h
23.5
with ResNet-50
MIRO + SWAD
25.6M
3,4h
3.6b
3.3h
3.4h
3.9
with ResNet-50
MIRO + SWAD (SOTA)
with RegNetY-16GH
79.7M
10.6h
11.2h
10.3h
10.6h
30.5h
15.2
SIMPLE with Model Pool-A
9.2s
12.0s
16.9s
9.6s
55.9s
9.7
SIMPLE with Model Pool-B
0.9M
8.8s
10.3s
20.9s
9.3s
79.
9.6
A. MATCHING PREFERENCE ANALYSIS
SIMPLE shows excellent generalization performance in the case of dispatching only fixed pre- trained models to predict each OOD sample. Thus, that is, which models are typically dispatched specific domains. This could also provide insights into which types of pretrained models might be more suitable for certain domains.
curious about its matching preference,
Settings, We attempt to analyze the preferences of our approach for network architectures and pretraining datasets. To do so, first measure the importance of the pretrained models and then
23

---

Published as a conference paper at ICLR 2023
perform refined analysis of their architectures and pretraining datasets. Specifically, by taking the model dispatcher trained by SIMPLE, we compute the sum of the ensemble weights assigned to each pretrained model as its importance measure for ranking. Then, classify these pretrained models according to the basic architecture and the pretraining dataset. Treating the ranking of the measure the importance of each
pretrained models the ranking of the associated types, type by calculating the mean reciprocal rank (MRR).
Caltech101
LabelMe
SUIN09
VOC2007
VLCS
Cartoon
Photo
Skosch
PACS
Art
Clipart
Product
Photo
Office-Home
L,100
1,38
1.43
L-46
Terra Incognita
(canera trup location
Clipart
Infographic
Painting
DomainNet
Figure 8: Samples that included in DomainBed, from Table in Gulrajani Lopez- Pazez-Paz Paz 2020
Analysis of network architecture. Figure 13
14
15 T6 and 17 show the raw rankings of the can see that SIMPLE
pretrained models on different datasets. From the ranking information, assigns markedly different pretrained models to samples from different domains. This may imply that these domains, as shown Figure 17, may differ widely from each other and thus need to be handled by varying combinations of pretrained models. We then classify these pretrained models based on their basic architecture, i.e., CNN- ViT-based, and MLPP-baeed, with their MRR values over different datasets and domains shown in Figure 9 It can be seen that ViT-based models are dispatched more frequently than CNN-based models in most domains, with the exception domains. Observations over these domains suggest that CNN-basee models tend to be used more for real images (c.g., photo), while ViT-based models are chosen to handle stylisti or textural variations (e.g., sketch).
Analysis of pretraining datasets. We then analyze on the pretraining datasets, including ImageNet- 1k, ImageNet-211 CLIP (Radford et al. 2021). and SWAG (Singh et al. 2022). used for pretrained models. The MRR results of these different pretraining datasets on five datasets are shown in Fig- 10 with scores grouped by domain. On hand, can be seen that our method shows preference for SWAG, CLIP, and ImageNet-21k over ImageNet-1k dataset. This preference can be supported by recent studies finding that pretraining on ImageNet- 1kk insufficient for gener- alization (Kumar et al.] 2022) and that changing the dataset may improve generalization perfor- mance (Kim al. 2022). D. the other hand, despite this preference, SIMPLE dispatches different models for each domain (e.g. on Terralncognita-LII
neither SWAG nor CLIP is chosen much, but rather ImageNet-pretraing models are used), suggesting that there still no free lunch for DG in the selection of pretraining datasets.
httss::/immggenet.rrg/////
24

---

Published as a conference paper at ICLR 2023
Basic architecture
CNN
MLP
0.6
PACS
VLCS
OfficeHome
Terra
DomainNet
0.5
Mean Reciprocal Rank
0.4
0.2
0.1
0.0
Cartion
Photo
Statch
Caltech101
LabelMe
SUND9
VOC2007
Ciipart
Product
Photo
L100
L38
L43
L46
Clipart
Infographic
Painting
QuickDraw
Photo
Stetch
Figure 9: MRR values for the different network architectures used by the pretrained models.
Basic architecture
SWAG dataset
ImageNect 21k
ImageNet-1k
0.6
PACS
VLCS
OfficeHome
Terra
DomainNet
0.5
Mean Reciprocal Rank
0.4
0.3
0.2
0.1
0.0
Art
Carteon
Photo
Sketch
Caltsch101
LabeiMMe
SUN09
vOC2007
Art
Clipart
Product
Photo
L100
L43
L4c
Clipart
infographic
Painting
QuickDraw
Photo
Stetch
Figure 10: MRR values for the different pretraining datasets used by the pretrained models.
25

---

further improve the performance of the Published as a conference paper at ICLR 2023
A.12 LEVERAGING MATCHING PREFERENCE AS TRANSFERABILIT MEASUREMENT
In this section, we leverage the matching preferences of the learned model dispatcher to optimize the model pool size and study the performance of SIMPLE for different model pool sizes.
Specifically, since the model dispatcher of SIMPLE learns to match the best suitable models for unseen samples at a meta-level, its matching preference over unseen domains (without accessing ground truth labels) can be regarded as measurement of model transferability on these unseen samples. Therefore,
can further optimize the model pool size used for each domain in two-stage manner: (1) first learn with large model pool on source domains to generate matching preferences; and (2) then reconstruct smaller model pool to include models preferred by our dispatcher.
Settings. We experiment this two-manner training on PACS and OfficeHome datasets, using differ- ent reconstructed model pool sizes the second stage. Note that we construct a model pool specific to each domain in dataset, based on aggregated matching preferences for all samples of the unseen domain.
The results of PACS and OfficeHome datasets shown in Figure I1 and respectively. As can be seen, given that SIMPLE can automatically match models more suitable for transferring to unseen domains, with only two models (while the best single model fails to outperform since there No Free Lunch). And when using smaller model pool, SIMPLE OfficeHome dataset.
actually go beyond the existing SOTA approach even using model pool
dataset
99.0
Average sccurscy
98.0
97.5
97.0
96.5
SOTA MIRO)
96.0
best single model original result
12
10
30
100
150
200
250
283
Model pool
Figure 11: The performance of SIMPLE on PACS dataset, with model pools of different sizes.
OfficeHome dataset
88
86
Average accuracy
84
SIMPLE
80
existing SOTA EoA single model
original result of SIMPLE
78
1.
10
30
50
100
150
200
250
Model
Figure 12:
The performance of SIMPLE on OfficeHome dataset, with model pools of different
sizes.
26

---

Published as a conference paper at ICLR 2023
(swag) n14
(swag) Vit h14
(swag) reginaty_ 16gt
(swag) vIr h14
(swag) vit_I16
(swag) vit 126
(swag) vit h14
[swag] vit_
Top-2
(swag)
(swag) vit b16
(swag) wit_
(swag) vit_b16
vit_bi6
(swag)
(clip) VIT-B/16
ftim)) conviext tiny j__2kk
(clip)
lop-4
regnety _2gf
VIT-8/16
(swag) regnety 16gf
(clip) VT-B/32
(timm) swin base patch4 window7 224 in22k
(clip) VIT-8/32
(timm) convnext xlarge 384 In22ft2k
(timm) convnext base In22k
(timm) convnext nano ols
(t/mm) convnext small ln22k
Top.6
(timm))
(timm) base /50 $16 224 jn21k
(tlm)) Jarge patch4 Window .24
mobllenetv3 Jarge__ra 100
(timm) convnext_large.eo _n22k
Top-7
mill in21k
(timm)
(timm)
(timm) convnext base in22ftk
Top-8
swin base patch4 window7 224 20222_22222222_ Jn22k
vit Nt_large_patchhe p_tpppatch22pp2 224
(timm) convnext base_aseeeee.._ee. Jn22k
.jn21k
(timm) base In22
(clip)
(timm) convnext pano bof
(clip) VII-H116
Top
VII-8/16
(Glip)
(timm) beit base wwt__paatsseeee_eeeee ppwch16 224
(timm) gmlp $16_1162200 224
(timm) cait_xxs22 _24
Top-10
B/32
Figure 13: Ranking of the models in the four domains of PACS using the sum of the ensemble weights assigned the models. The four columns from left to right in the figure correspond to the different domains (Art, Cartoon, Photo, Sketch) in the dataset.
27

---

Published as a conference paper at ICLR 2023
(timm) swin swin_large patch4 window 224 In22k
(clip)
(clip)
(swag) regnety 16gt
VII-R/16
VII-H/16
(timm) convnext base fn22k
(swag) vit 126
(swag) regnety_ 16gf
(timm)
Top-2
levit_388 384
(timm)
(swag) vit h14
ftimm) convnext tiny jn22k
(tim)) vit_small_aa [t__smal_patchh2____ _24
wwtmlmmalaaaa.. jn22k
(t/m)
(timm) convnext Xlarge @a2kge_in22k
(clip) VIT-B/32
(timm) convnext convneet_xlarge in22k
Jarge patch4 window7 224 in22k
lop-4
(timm)
(timm)
(swag)
(timm)
vit base patch32 224 in21k
large in22k
b16
xlarge in22k
(timm)
(t/m)) convnext xlarge in22ft2
Top.6
vit base patch32 224 in21k
(swag) regnety 32qf
(cilp) VIT-8/32
(timm)
(timm)
(swag) regnety 16gf
(clip)
Top-7
vit Jargerrgeee..... /50 $32 224 jn21k
vit L_ttinn_____ $16 p8 224 Jn21k
VIT-B/16
(timm)
(timm) resnetv2 50x1 50x1_bitm _iiim .n21k
Top-8
vit_small vit_small_patchh6
6llpppath16pp _244 224 In21k
(MAE) vit wwt_huge_patch1-
(clip) VIT-//32
(timm) large patch16 224 In22k
(timm) large patch4 window7 224
(timm) pptpplllpp2.0 224 In21k
(timm) levit 178s
Top
(bimm) resnetv2 50x1 bitm in21k
(timm) resnetv2 111x1_1liiiii jn21K
(timm) resmlp big 24_224 224
(timm) levit 128
Top-10
Figure 14: Ranking of the models in the four domains of VLCS using the sum of the ensemble weights assigned the models. The four columns from left to right in the figure correspond to the different domains (Caltech101, LabelMe, SUN09, VOC2007) in the dataset.
28

---

Published as a conference paper at ICLR 2023
(timm) convnext xlarge Jn22ft1
(swag)
(swag) vit /16
(swag) vir h14
Top-1
regnety_372
(timm) convnext small 384 jn22ft1k
(timm) convnext large_330 384 jn22nt1
(swag) vit h14
[swag] vit vit_/I6
Top-2
(timm) Swin Jargerrggggoooo patch4 window7 224
(t/m) convnext wwwpiiaargeeaa __84 _n22ft2
(timm)
(swag) vit bl6
Top-3
convnext_sman jn22k
(timm) convnext small jn22ff2k
(timm)
(swag) vit_ [t_h14
(swag) @ggnoty_32gf
Top-4
convnext Baarccxxxxlrgg_ineee
(pretrainedmodels) dpn68
(limm) convnext small jn22ff1k
(swag) regnety_16gt
(Limm) convnext large jn22ff2k
(bmmm) large patch4 window7 774
(timm) _384
(timm) Swin large patch4 window? 224
ftimm) convnext_large in22k
Top-6
convnext xlarge _384 jn22ft1k
(timm) swin base patch4 window 224
(timm) convnext base In22k
(swag)
(timm)
regnety_1oo
convnext xaanrxxtxxaageeeeo in22k
(tim,) swin base patch4 window? in22k
(swag)
(timm)
(timm)
Top-8
vit_bi6
Sonvncxt_xlarre__oo.
convnext convncxi_basc Jn22k
(t/m) basc 384 jn22ft1k
(timm)
(swag)
(timm)
Top-9
dpn68
regnety _3gff
tf_mixnet]
(timm) convnexl Jarge
(tim)) convnexl xilrnee_xxarge in22ft1k
(limm) convnext_ Jarge_ing22
(clip) VIT-8/32
1op-10
jn22ft1k
Figure 15: Ranking of the models four domains of OfficeHome using the sum ensemble weights assigned the models. The four columns from left to right in the figure correspond to the different domains (Art, Clipart, Product, Photo) in the dataset.
29

---

Published as a conference paper at ICLR 2023
(timm)
(timm) convnext pano hof
(MAE) Vir nuge patch14
(MAE) patch14
convnext large_aaaa In22fti
ftimm) convnext xiarge_33gg 384 jn22Rt1
(Swag)
(MAE)
[swag] vit vit_/I6
Top-2
vit Mi_h14
vit Wt_hugc_patchie
(timm) swin basc patch4
(Umm)
(timm)
(swag) h14
window7_224
Convnext_nano___e
convnext_ small
(timm) convnexi xlarge jn22Ri1
(timm) convnext Jarge in22ftk
(clip)
(timm) convexxt base
lop-4
VIT-B/16
(timm) convnext large In22ft1k
(timm) convnext xlarge In22ft1k
(timm) ecaresnet101d pruned
(MAE)
large patch16
(timm) convnext base Jn22fik
(t/m)) convnext xlarge In22ft2
(timm)
(swag) vit f16
Top.6
beit base patch16 224
(timm) twins Sarns_svt__oe bas_
(MAE) vit WM_large_patchhe
(swag) vit h14
(t/mm) twins syt__argeeeeeee
Top-7
(timm) convnext large in22ft1
(timm)
(timm)
(timm) convnext_nane
Top-8
pit jr__b_iistiledd 224
wwins_svt_large
(timm) swin, large patch4 window7 224
(timm)
(timm) In22f11
(timm) base In22
Top
riny_386 Jn22ftk
(umm) swin Jarge patch4 window? 224 In22k
(swag) regnety_3222
(swag) regnety_ 32gf
swin_large
(swag) regnety_16gt
Top-10
Figure 16: Ranking of the models in the four domains of TerraIncognita using the sum of the en- semble weights assigned to the models. The four columns from left to right in the figure correspond to the different domains (1,100, 1.38, L43, 146) in the dataset.
30

---

Published as a conference paper at ICLR 2023
(clip)
(c100)
(ciip) VT B/32
(swag) h14
(swag: vIt
(ctlo) B/T
Top-1
VI7 B/16
B/7
(timm)) conmexxt _asee____eeeeeee 334
(clip)
(clip)
(swag)
(swag) vit_b16
Top-2
VI--8/32
vic h.4
(Ummm] convnext xlarge 384 In22
[mmm) convnext xlarge
(Ummi)
(swag) vit it_n24
Swin, Jarge patch4
(swag)
(swag) vit_126
Top-3
304 In22ftk
window?
vit_i16
224 In22k
(Swag]
Top-4
wit/166
vit
rignnty 16gf
in22n1k
(timn))
(swag) /16
(fimm) convnext base 38-7 Jn22flk
(timm) Swin, Jarge patchd window 224
(him)) convhext xlarge 354 in22ttik
(timm)
Top-5 awin (arge _ porch4
convnext large jn22ft1
windoww? 224
(2mm)) convnext (arge 384 2222
(Ummm) convnext smail 384 1022R11
(Ummm) convnext xtarge 384 in22n1
(0mm) convenxt large 304 in2211
(Swag) vit_b66
(clpp) VIT-8/15
Top-6
(swag)
(b/mm)
(swag)
(clpp)
(imm) small
(timm) base
Top-7
384
VIT-1/32
384
(lmmm)
(Umm) swin (arge patchi windaw? 274
(clip) VIT B/16
(Limm) convnext base in2261k
(vimmm) Via (arge patch1 6, 224
(timm)
Top-8
convnext small In22HfI
convnext base in2261k
Top
[t1mm) convnext xlarge In22ft1
convnext large ln22f1k
regnety_ 32gg
(swag)
(ttmm Jarge
(timm) r26 $32 224
(timm) convnext in22ft1k
in22A1k
(timm) convnext large jn22fl1
(Umm) convicxt small jn22f1k
(Umm) Jargo palch16
(Uimm) awin Dasc potch4 window? 224
(tmm)) convnext tiny jn22nik
(tmm) convnext sma!! 384 In22ft1k
Top-10-
Figure 17: Ranking of the models in the four domains of DomainNet using the sum the ensemble weights assigned to the models. The six columns from left to right in the figure correspond to the different domains (Clipart, Infographic, Painting, QuickDraw, Photo, Sketch) in the dataset.
31

---

Published as a conference paper at ICLR 2023
Table 7:
List of pretrained models from of the following sources: (timm, pretrainedmoodls: clip, MAE or SWAG). The output dimension is the number of classes in the pretraiing domain.
Model id
Source
Model Name
Output Dimension
MFLOPs
pretrainedmodels
squeezenetl squeezenet1_1
1000
785.29
2
pretrainedmodels
1000
335.60
3
timm
resnetv2.152x2.bit.eee
1000
4.10
4
timm
resnex150_32x<>
1000
4062.06
5
timm
resnext50d_32x44
1000
4292.47
6
timm
rexmet_110
1000
379.98
timm
rexnet 130
1000
631.64
8
timm
rexnet 150
1000
835.36
9
timm
rexnet_200
1000
1458. 16
10
timm
selecsls42b
1000
2843.06
11
timm
sclecsls60
1000
3425.65
12
timm
selecsls60b
1000
3461.41
13
timm
semnasnet. 100
1000
297.21
14
timm
seresnet50
1000
3923.46
15
timm
crresnxx16d..22x4e
1000
2605.98
16
timm
seresnext26t_22x6
1000
2575.64
17
timm
seresnext50.322e
1000
4064.47
18
timm
skresmet18
1000
1732.17
19
timm
skresnet34
1000
3496.38
20
timm
skresnext50332x44
1000
4260.47
21
timm
spnasnet_100
1000
318.08
22
timm
ssl_rennet88
1000
1734.79
23
timm
ssl_resnet50
1000
3921.04
24
timm
ssl ssl_resnext50_32xx
1000
4062.06
25
timm
swin base patch4 window7 224
1000
102.33
26
timm
swin large patch4 window7 224
1000
215.38
27
timm
prin_smalllpatchh_.iidd
1000
61.28
28
timm
swin_tiny-patte wwintinyy-pacchh..windoo wwtps//aaaaaaawwwooooooaam 2224 swslrresnet18
1000
40.99
29
timm
1000
1734.79
30
timm
swsl_resnet53
1000
3921.04
31
timm
swsl www.eeeeeett0...000
1000
4062.06
32
timm
tf efficientnet b0
1000
351.21
33
timm
efficientnet bo ap tf aeffiiientnethho_ne tf www.ffiiietntttcc..eeccooo _c_____ceeeeeeee _4c if wweffficceetett.c._buee tf efficientnet.ce tf efficientnet lite0 lite0 tf mixnet_ If mfxiient_m tf_mixnet_ m.mobilenetv33_areee. if mobilenetv3 large 100
1000
351.21
34
timm
1000
351.21
35
timm
1000
154,99
36
timm
1000
154.99
37
timm
1000
1513.29
38
timm
1000
350.60
39
timm
1000
492.45
40
timm
1000
295.36
41
timm
1000
214.28
42
timm
1000
137.04
43
timm
1000
195.96
44
timm
mobilenetv3 large minimal 100
1000
190.21
45
timm
tf www.ilenetts_smmllll.o.
1000
33.62
46
timm
h.t.molleett33ssmallle. If m.mooiieeetv33.c smmllllllll..mmamm.
1000
46.07
47
timm
1000
42.55
48
timm
tnt 1ts.patchh1.2222
1000
33.18
49
timm
tv w=deenenet1211 tv resnet34
1000
2703.08
50
timm
1000
3501.19
51
timm
tv resnet50
1000
3921.04 .04
52
timm
tv L.eesnext50322x44
1000
4062.06
53
timm
twins-pepvt.bar
1000
561.73
54
timm
Lanns
1000
794.99
32

---

Published as a conference paper at ICLR 2023
55
timm
twins-pepyt. penss-ppptssmall twins_svt_bor
1000
317.77
56
timm
1000
534.18
57
timm
twins svt large twins svt small visformer_small
1000
940.45
58
timm
1000
192.95
59
timm
1000
4532.08
60
timm
vit_ viL.base.aatcll... 6224
1000
192.20
61
timm
vit. base paascpp 224 miil
1000
192.17
62
timm
vit_base-parle Satasssppacc32.22222 cait_s24_224
1000
192.09
63
timm
1000
99.52
64
timm
cait. xxs24 224
1000
38.79
65
cait xxs36 224 coat_lite_min
1000
43.87
66
timm
1000
135.35
67
timm
coat_lite_wt smtel coat lite tiny coat mini coat_ting convit_ convit_small convit.tiny
1000
170.24
68
timm
1000
107.48
69
timm
1000
284.68
70
timm
1000
206.37
71
timm
1000
192.18
72
timm
1000
88.16
73
timm
1000
32.85
74
timm
cspresnext50 deit base distilled patch16 224
1000
2942.24
75
timm
1000
192.93
76
timm
deit. base patch16 224
1000
192.20
77
timm
deit_smallddistiledd pptch16_222
1000
76.22
78
timm
seit_small_pattee 224
1000
75.85
79
timm
deit_tiny ddistileedpppteeeeee wpwpchllc.22210p 6_224
1000
33.04
80
timm
deit.tiny-patcie -pt.h
1000
32.86
81
timm
densenet121
1000
2703.08
82
timm
densenet169
1000
3204.45
83
timm
densenet201
1000
4092.84
84
timm
densenetblur12
1000
2931.62
85
timm
dla34
1000
2927.88
86
timm
dla46
1000
556.25
87
timm
dla46x
1000
518.84
88
timm
dla60
1000
4059.07
89
timm
res2nt
1000
3955.43
90
timm
dla60_res2next
1000
3323.87
91
dla60x
1000
3380.01
92
timm
dla60x_c
1000
565.83
93
timm
dm nfnet fl
1000
45.10
94
timm
dpn68 dpn68b
1000
2218.44
95
timm
1000
2218.44
96
timm
eca @ra_nfnet_00
1000
2.73
97
timm
ecaresnet50d
1000
4151.53
98
timm
ccaresnet50d.pruueed
1000
2413.62
99
timm
ecaresnet MaaresnetOld.pprunee
1000
3313.54
100
timm
ccaresnetlight
1000
3916.31
101
timm
efficientnet b0 efficientnet.bl
1000
368.01
102
timm
1000
543.33
103
timm
efficientnet.es
1000
1705.92
104
efficientnet. es es_pruned efficientnet_ lite0 ese_vovnet fbnetc. 100 gernet m
1000
1705.92
105
timm
1000
367.40
106
timm
1000
1271.76
107
timm
1000
367.22
108
timm
1000
2865.4 44
109
timm
gernet_s
1000
709.75
110
timm
ghostnet_100
1000
140.10
111
timm
resnet18 gl ylb
1000
1734.79
112
timm
aluon_resnet:4 vlb
1000
3501.19
113
timm
alloonrrsnette y1b
1000
3921.04
33

---

Published as a conference paper at ICLR 2023
114
timm
gluon .vlc
1000
4151.11
115
timm
gluneetesnet50.ve
1000
4151.45
116
timm
gluon resnext50 32x4d
1000
4062.06
117
timm
gluon seresnext50 32x4d gmixer_24_222
1000
4064.47
118
timm
1000
78.45
119
timm
gmip-s16.222
1000
55.07
120
timm
hardcorenas
1000
212.67
121
timm
hardcorenas.b
1000
238.99
122
timm
hardcorenas.c
1000
256.37
123
timm
hardcorenas d hardcorenas e hardcorenas_f
1000
273.29
124
timm
1000
321.83
125
timm
1000
324.90
126
timm
hrnet_w 18 hrnet. w18 small hrnet.w 188_smml___2 eggacy_seresned legacy sgeacysseresne34
1000
4116.73
127
timm
1000
1538.74
128
timm
1000
2494.39
129
timm
1000
1734.88
130
timm
1000
3501.34
131
timm
legacy_seresnet50
1000
3701.95
132
timm
1000
2375.58
133
timm
Dggacy_seresmext50.2222
1000
4064.47
134
timm
levit 128
1000
55.91
135
timm
levit 128s
1000
54.54
136
timm
levit_192
1000
112.07
137
timm
levit_256
1000
194.78
138
timm
levit_384
1000
426.22
139
timm
mixer.ble bor.lbll2222. 224
1000
166.90
140
timm
mixer_b16 11162222mmiii
1000
166.90
141
timm
mixer 116 224
1000
344.89
142
timm
mixnet I
1000
529.61
143
timm
mixnet_m
1000
323.54
144
timm
mixnet_s
1000
228.17
145
timm
mixnet xl
1000
862.22
146
timm
mnasnet. 100
1000
299.91
147
timm
mobilenetv2_1000
1000
286.90
148
timm
mobilenetv2.1100
1000
410.63
149
timm
mobilenetv2_120d
1000
637.93
150
timm
mobilenetv2_140
1000
555.31
151
timm
mobilenetv._larre__l
1000
205.38
152
timm
mobilenetv3 l 100 miil
1000
205.38
153
timm
mobilenetv3.rv rw
1000
205.38
154
timm
nfnet_10
1000
13.09
155
timm
pit_ 224
1000
209.98
156
timm
pit_b_ boddiiitilledd24
1000
210.96
157
timm
pit.s_224
1000
99.93
158
timm
pit.s.distilled.22.
1000
100.48
159
timm
pit ti 224
1000
39.09
160
timm
pit ti distilled 224
1000
39.34
161
timm
pit_xs_224
1000
61.83
162
timm
pitxxs_distilled 224
1000
62.20
163
timm
regnetx_00) _022
1000
189.83
164
timm
regnetx_00e
1000
379.27
165
timm
regnetx_006
1000
573.07
166
timm
regnetx 008
1000
762.68
167
timm
regnetx 016
1000
1528.6 64
168
timm
regnetx_033
1000
3029.51
169
timm
regnetx_044
1000
3780.90
170
timm
regnety .022
1000
190.29
171
timm
regnety-O0e 0004
1000
383.25
172
timm
regnety _006
1000
573.40
34

---

Published as a conference paper at ICLR 2023
173
timm
regnety-OO0
1000
760.28
174
timm
regnety-016
1000
1537.44
175
timm
regnety 032
1000
3029.41
176
timm
regnety 040
1000
3789.28
177
timm
repvgg_ho
1000
3239.87
178
timm
res2net50_ __8
1000
4010.80
179
timm
res2net50 26w 4s
1000
4082.33
180
timm
res2net50_48w, .2s
1000
3989.85
181
timm
res2next50
1000
4005.80
182
timm
resmip_ 12 224
1000
69.52
183
resmlp 12 distilled 224
1000
69.52
184
timm
resmip_24_222
1000
83.49
185
timm
resmlp_24. 22__distilledddia _224
1000
83,49
186
timm
resmlp 36 .22
1000
97.45
187
timm
resmlp 36.distilled 224
1000
97.45
188
timm
resmlp_bie htpp_bi22.2... 1gg_24_224
1000
233.73
189
timm
resmlp_bij 2bi.244_222 24.224422222222 in22ft
1000
233.73
190
timm
@ssml_big_24_diieee
1000
233.73
191
timm
resnest14d
1000
2636.61
192
timm
resnest26d
1000
3475.13
193
timm
resnest50d 1s4x24d
1000
4222.56
194
timm
resnest50d 4s2x40d
1000
4201.50
195
timm
resnet18
1000
1734.79
196
timm
resnet
1000
1964.95
197
timm
resnet26
1000
2247.86
198
timm
resmet26d
1000
2478.27
199
timm
resnet34
1000
3501.19
200
timm
resmet34d
1000
3731.34
201
timm
resnet50
1000
3921.04
202
timm
resnet50d
1000
4151.45
203
timm
resnetblur50
1000
4914.30
204
timm
lbtpt__btttooe resnetv2.500 bbt distillee
1000
2.05
205
timm
www.larre-aacchle 224
1000
436.37
206
timm
vit vit_large_550 |arge_r50_s32.222... s32.224
1000
387.23
207
timm
vit_small paccllaapphll... 224
1000
75.85
208
timm
vitssmall_patch32.1
1000
75.79
209
vit_small_r26_332.22
1000
57.42
210
timm
vit.tiny-patchh6-
1000
32.86
211
timm
vit tiny r s16 p8 224
1000
42,02
212
pretrainedmodels
alexnet
1000
681.57
213
pretrainedmodels
densenet121 densenet169
1000
2732.91
214
pretrainedmodels
1000
3240.65
215
pretrainedmodels
densenet201
1000
4139.87
216
pretrainedmodels
dpn68
1000
2241.70
217
pretrainedmodels
nasnetamobile
1000
551.83
218
pretrainedmmdell
resnet18
1000
1734.79
219
pretrainedmodels
resnet34
1000
3501.19
220
pretrainedmodels
resnet50
1000
3921,04
221
pretrainedmodels
se se_resnet50
1000
3707.13
222
swag
regnety-16gf.inlk
1000
15223.83
223
swag
regnety-32gf.inl
1000
30849.35
224
timm
beit.base.patch16.22
1000
171.92
225
timm
beit. lait.large./p p 224 convnext base
1000
364.30
226
timm
1000
616.58
227
timm
convnext_base wwwnnextbbase_384iin2221f1
1000
616.58
228
timm
convnext_base in22fftk
1000
616.58
229
timm
convnext large
1000
1205.29
230
timm
mownextt_lagee 384 104_in22ft1k
1000
1205.29
231
timm
convnext_ large in22ft1k
1000
1205.29
35

---

Published as a conference paper at ICLR 2023
232
timm
convnext
1000
2340.25
233
timm
convnext_nano.e Jhn
1000
2340.25
234
timm
convnext ols
1000
2386.90
235
timm
convnext small convnext_small Swwneextsmalll_88__n22222 _384_in22ft1k
1000
392.34
236
timm
1000
392.34
237
timm
convnext_small Inni convnext.ting
1000
392.34
238
timm
1000
307.45
239
timm
convnexi.tiny in222flkk lkk convnext_tiny _hnf convnext tiny in22ftlk convnext xlarge 384 in22ft convnext_xlageeinn22ff vit vit.base.patch16
1000
307.45
240
timm
1000
4261.97
241
timm
1000
307.45
242
timm
334
1000
1980,92
243
timm
1000
1980.92
244
MAE
1000
192.21
245
MAE
vit large patch16 vit huge-
1000
436.38
246
MAE
patch14 huge_patch144
1000
785.64
247
swag
vit
768
164.43
248
swag
vit.16
1024
339.30
249
swag
vit_h14
1280
584.25
250
swag
regnety-16gf
3024
15220.81
251
swag
regnety332gr
3712
30845.63
252
timm
vit base patch16 224 miil in2 lk
11221
199.67
253
timm
mixer b16 224 miil in21k
11221
174.39
254
timm
mobilenetv3 1aapetev_aarre1110000000 100 1000mmil_inn2) wwwpbbaseepptcch_wiii.i 1i1444_winow7..cc inn2k
11221
205.38
255
timm
21841
122.71
256
timm
swin.large.patchie hww.://ww..aacc....ioooo////o//a
21841
245.93
257
timm
beit.base-patchl latsse_pacchh6.2222 16.224..22222221
21841
187.21
258
timm
beit.large.patch16 6__22_in2222
21841
384.68
259
timm
convnext base in22k
21841
636.96
260
timm
convnext large in22k
21841
1235.83
261
timm
convnext_small _n22k
21841
407.62
262
timm
convnext_tiny _in22k
21841
322.74
263
timm
convnext xlarree in22k
21841
2021.65
264
timm
resnetv2 50x bittminn2kk
21843
42.78
265
timm
resnetv2.50x3 2oas333bbm__iiit bitmmin21k
21843
128.30
266
timm
2ssnet22110xx bitm in21k
21843
42.78
267
timm
pesnetv2.152x2_bimm.in
21843
85.54
268
vit_base_patch16 6_224_nn2kk
21843
207.49
269
timm
vit.base-patch3222244 2n242_222222
21843
207.38
270
timm
vit base r50 s16 224 in21k
21843
244.24
271
timm
mixer b16 224 in21k
21843
182.18
272
timm
mixer_l16 216_244 in21
21843
365.27
273
timm
htthhue_ppachh 22224444222200000 22k_i22k
21843
811.09
274
timm
vit.large.pacch 6__22__in211
21843
456.75
275
timm
vit.large.patch32.222 224__n221
21843
456.61
276
timm
vit.large_550.333 wwt55500332224444000 222222222n2
21843
407.61
277
timm
vit small patch16 224 in21k
21843
83.50
278
timm
vit small patch32 224 in21k
21843
83.45
279
timm
vit_small_ (aaall_r6__33222200 (26__332222211111 224_in21k
21843
65.07
280
timm
1kk
21843
36.70
281
timm
vit vit_tiny_r_s/ brayyrrs16_ppp 224 in2
21843
45.86
282
clip
ViT-B-16 ViT-B-32
arbitrary
188.32
283
clip
arbitrary
188.32
36


Reviews:
[{"review_id": "z8OF7B3bhe", "reviewer": "Reviewer_At6g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to leverage a pool of pretrained models without expensive fine-tuning. They empirically demonstrate the proposed method achieves SOTA results on several benchmarks without significant loss in inference speed. \n", "summary_of_the_review": "From the way the paper is written, many aspects are not clear to me, which I stated in the section above. Overall the paper looks quite interesting and the empirical evaluation of the paper looks significant. However, I don’t know how to properly evaluate this paper when there are so many unclear things in it. In particular, the theoretical part and big parts of the paper are very confusing, as well as the model description. I am willing to change my evaluation when my questions are clarified.", "strength_and_weaknesses": "**Strengths:**\n\nThe authors propose a creative way how to use a pool of pretrained models to achieve better generalization on distribution shifts. They demonstrate SOTA results, which significantly outperform previous models. They demonstrate the necessity of parts of their algorithm via an ablation study.\n\n**Weaknesses and comments:**\n\n - The writing is a bit confusing. First, the authors say that they *“adopt a simple la- bel adapter that projects the label space of the pretrained domain to that of the target domain”*, but then they say *“with the adapter training on source domains”*. After reading several times, I understand what they meant in the beginning, but upon first reading, it was very confusing. The authors should stick to the strict terminology that they introduce in the very beginning, e.g. “pre-training domain” (the one that corresponds to ImageNet) and then “source” and “target” domains from a typical domain generalization literature. \n- No free lunch in domain generalization was demonstrated in several works before, e.g. in the DomainBed paper that the authors cite. So Takeaway 2 is an extended version of those results, but with varying architectures. It is even less surprising, given the fact that the authors are not trying to address distribution shift at all there.\n- For me, it is not obvious why Theorem 1 would explicitly generalize to neural networks of fixed size.\n- Also by just tuning the label on the source domain (not the same as image-net pretraining), it is not obvious that the model can even achieve minimal test risk and if all theoretical justifications would be true. An arbitrary model trained under ERM can exhibit quite arbitrary behavior under distribution shift, as known from many previous theoretical results. By only allowing to tune the last layer, the authors put quite a strong constraint on the class of models under which optimal risk can be achieved, therefore such models are not guaranteed to generalize. \n- Remark 3 is also not clear to me. Any causal model would achieve optimal risk under the assumption of covariate shift. \n-  How *“fitness of the pretrained models to the target distributions”* is evaluated?\n- *“Table 1. Baseline results are from original papers with the same setup”*. Does it mean that the method’s performance for benchmarks is demonstrated on “one” model (originally proposed in the corresponding paper). So it is possible that the best architecture which would allow achieving a higher number in this table might have not been considered? My concern is that majority of the papers were focused on demonstrating the effect of the *learning method given a fixed architecture* and that with another architecture, the same learning method would achieve another number. So comparing it to the pool of > 200 models is not entirely fair. \n- Ablation study RQ2: the averaging strategy indeed looks not optimal. But why the “averaging” weights can’t be learned by backpropagation? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "rC5cQ1nUIh", "reviewer": "Reviewer_ySiZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the generalization performance of pretrained models on domain generalization or data distribution shift problems, leading to the ''no free lunch hypothesis'' of pre-trained models in domain generalization settings.  The authors then propose a new DG method that aims to directly leverages pretrained models without fine-tuning. The experimental results on the mainstream benchmark show that the proposed method outperforms other strong baselines. ", "summary_of_the_review": "Overall, I personally think that this submission is below the acceptance rate for an ICLR paper. I would suggest the authors focus more on the theoretical part to make it stronger. ", "strength_and_weaknesses": "[Strengths]: The proposed method is relatively simple and easy to implement. The experimental results are convincing and partly support the main claims of the paper.\n\n[Weakness]: My major concern is about the theoretical contribution of the paper. I did not have time to verify the results in Theorem 1, but even if they are all correct, I could not find the role of pre-trained models in the generalization error in a DG. Furthermore, the idea of (pre-trained) model-sample matching in the proposed method may be impractical since in DG we are not allowed to have access to the samples in the target domain.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FGT71_WO9yI", "reviewer": "Reviewer_qJrT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates performance of different pretrained models on domain generalization and shows that there is no single best pretrained model that generalizes across all distribution shifts. Based on the finds, the authors propose a new method to learn to dispatch the best matched models for each out-of-distribution sample and predict with ensemble of the models. The method does not need to fine-tune the pretrained models and achieves obvious performance improvements.", "summary_of_the_review": "The paper provides an interesting investigation of different pretrained models on domain generalization, which well motivates to learn to dispatch different models for different target samples. The idea is interesting, but there are still some unclear parts in the methods and experiments that need to be clarified as mentioned in the weaknesses. I would consider to raise my score if the authors could well explain or address the weaknesses. ", "strength_and_weaknesses": "[+] Instead of fine-tuning pretrained models on source domains, the paper proposes to learn to dispatch pretrained models by matching the models to each test sample, which is efficient and interesting.\n\n[+] The proposed method achieves significant performance improvement compared with the other state-of-the-arts. The ablation studies also demonstrate the effeciveness of the method.\n\n[-] The method tends to select the best matching pretrained models for domain generalization. However, it is not sure whether method works because it handles the domain shifts or due to the strong capability of the pretrained models. For instance, the CLIP model in Model Pool B is trained on a large dataset, which contains a huge amount of different domains. \n\n[-] Since the method requires to process the inputs by all models in the Model Pool, there should be a big computations and memory usage cost in both training and test stages. While it is shown in Table 2 that the method achieves low FLOPs as shown. Can the authors explain this? The GPU memory usage is also important for reproducing the method.\n\n[-] The method trains a shared label space adapter using Model Pool A. However, it is not clear how to train the adapter with Model Pool B, which contains pretrained models with different sizes of output spaces. If different adapters are used for different models, will the training procedure be influenced by the ensemble weights? For example, some of the adapters will not be trained well since the weights are small.\n\n[-] The method learns model embeddings for the model-sample matching, but there is no information of the models considered in the embeddings. How can we make sure the embeddings related to different models?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cn-U_nJrC5o", "reviewer": "Reviewer_KF3m", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Domain generalization is a challenging problem as target data is not concurrently available along with source data. One approach is to have a set of pre-trained source models and develop a strategy for domain generalization. A paper by Balaji et al., (NeurIPS 2018) suggested the approach of using a set of pre-trained models and set the problem of domain generalization as a regularization problem. This paper has taken a similar approach but assumes the availability of a large number of pre-trained models. The domain generalization problem is then solved y dispatching the pre-trained models to OOD samples based on their matching metric to the target task. Extensive evaluations show that the proposed approach has better generalization performance and significantly higher training efficiency compared to existing DG methods.", "summary_of_the_review": "Domain generalization is harder than domain adaptation. The proposed approach of leveraging a large set of pre-trained models to help with generalization is a good idea. One can see this approach as an extension of nearest neighbor rule, in which the more training data you have, the better the classification performance is. In the proposed approach, each model can be viewed as a training sample. This makes me wonder if some asymptotic performance results can be derived as is done with nearest neighbor rules. Discuss what happens if you assume a very very large number of pre-trained models.", "strength_and_weaknesses": "Strengths:  An approach supported by theoretical analysis and empirical evidence based on extensive evaluations for domain generalization that assumes the availability of a large number of pre-trained source models. The success of domain generalization depends on how close the one or more pre-trained models are in terms of addressing the domain shift. Extensive experimental results and ablation studies are included.\nWeaknesses: This approach  reminds me of a NeurIPS paper by Y. Balaji, S. Sankaranarayanan and R. Chellappa, “MetaReg: Towards Domain Generalization Using Meta-regularization”, Proc. Neural and Information Processing Systems, Montreal, Dec. 2018. There is a brute-force philosophy embedded in the approach. Even more performance gains can be achieved if one can assume the availability of 500 or 1000 or 10,000 models! ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster