OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?
Authors: Liangze Jiang
Damien Teney
Venue: ICML 2025 poster (2025)
Keywords: OOD generalization
distribution shifts
learning to select training algorithms
Primary area: deep_learning->robustness
TL;DR: Choosing the right training algorithm is key to OOD generalization. This paper proposes to learn to predict which candidate algorithm could produce the most robust model given dataset characteristics.

Abstract: Out-of-distribution (OOD) generalization is challenging because distribution shifts come in many forms. Numerous algorithms exist to address specific settings, but *choosing the right training algorithm for the right dataset* without trial and error is difficult. Indeed, real-world applications often involve multiple types and combinations of shifts that are hard to analyze theoretically.

**Method.**  This work explores the possibility of *learning* the selection of a training algorithm for OOD generalization. We propose a proof of concept (OOD-Chameleon) that formulates the selection as a multi-label classification over candidate algorithms, trained on a *dataset of datasets* representing a variety of shifts. We evaluate the ability of OOD-Chameleon to rank algorithms on unseen shifts and datasets based only on dataset characteristics, i.e., without training models first, unlike traditional model selection.

**Findings.**  Extensive experiments show that the learned selector identifies high-performing algorithms across synthetic, vision, and language tasks. Further inspection shows that it learns non-trivial decision rules, which provide new insights into the applicability of existing algorithms. Overall, this new approach opens the possibility of better exploiting and understanding the plethora of existing algorithms for OOD generalization.

OOD-Chameleon Is Algorithm Selection for OOD Generalization Learnable?
Liangze Jiang 12
Damien Teney"
Abstract
(a)
Spurious correlation
Task
Task
Task
Outtof-diitribuutin (OOD) generalization is chal- lenging because distribution shifts come many forms. Numerous algorithms exist address spe- cific settings, but choosing the right training algo- rithm for the right dataset without trial and error is difficult. Indeed, real-world applications often involve multiple types and combinations of shifts that are hard to analyze theoretically.
ERM
GroupDRO
?
ReSample
ERM
Covariate shift
GroupDRO
Mixup
?
Label shift
(b)
Extracting data characteristics
Input
Descriptors training set
Method. This work explores the possibility of learning the selection of a training algorithm for OOD generalization. We propose proof of con- cept (ODDChameleen) that formulates the selec- tion as a multi-label classification over candidate algorithms,, trained on dataset of datasets repre- senting a variety of shifts. We evaluate the ability of OOD-Chameleon rank algorithms on unseen shifts and datasets based only on dataset charac- teristics, i.e. without training models first, unlike traditional model selection.
Distribution shifts
Train
Running 2oo of algorithms
Target
Test
Dataset datasets
Performance on test set
(c)
ReSample
Train robust model on the task
Task
Figure I. Choosing the right training algorithm an often- overlooked key factor in OOD generalization. (a) Different training algorithms perform differently on different distribution shifts. (b) We propose automate the selection of learning algorithm. We train a selector (OOD-Chameloon) on "dataset of datasets" that exemplifies variety of shifts. (c) For novel task/dataset, the learned selector predicts the best algorithm to train robust model.
Findings. Extensive experiments show that the learned selector identifies high- seghhperformin algo- rithms across synthetic, vision. and language tasks. Further inspection shows that learns non-trivial decision rules, which provide new insights into the applicability of existing algo- rithms. Overall, this new approach opens the possibility of better exploiting and understanding the plethora of existing algorithms for OOD gen- cralization. Code: https:/ /ittbbb. com/ Haangae-Jann//ooo-Chaeeee
and test data differ. "OOD" a catch-all term that encom- passes many types of distribution shifts (Wiles et al., 2021; Ye et al., 2022; Nagarajan et al., 2020). In medical imaging for example Beakden-Rayne et al., 2020), a model may process scans from various demographies (covariate shift), pathologies (label shift), and co-occurrences of patient at- tributes (spurious correlations). In many applications, these shifts are combined, making theoretical analysis challeng- ing (Wiles et al., 2021; Yang et al., 2023; Jeon et al., 2025).
1. Introduction
(OOD) generalization refers to a model's ability to remain accurate when the distributions of training
The trade-offs learning algorithms. There exists a multitude of algorithms' designed to improve OOD gen- eralization, from standard ERM (Vapnik, 2000) simple interventions such as Resampling (Japkowicz & Stephen,
EPFL. Switzerland Iddaap Research Institute, Switzerland, Corre- spondence to: Liangze Jiang <liangze.jiang jinggrrsiiann@cp...c.... ch'>.
Proceedings of the 2dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
IIn this work, "algorithm" refers method, or particular element thereof that takes the training data and produces trained model.

---

Is Algorithm Selection for OOD Generalization Learnable?
2002), GroupDRO (Sagawa et al., 2019), and much more complex ones (Liu al., 2023a). However, each algorithm usually targets specific distribution shift settings (Figure la). Indeed, numerous studies by Gulrajani & Lopez-Par Paz (2020) and others (Wiles et al., 2021; Nguyen et al., 2021; Ye et al., 2022; Liang & Zou, 2022; Benoit et al., 2023; Yang et al., 2023) showed that no single intervention surpasses ERM across a range of datasets. The reason that OOD gener- alization is fundamentally underspecified (D' Amour al., 2022; Teney et al., 2022) and different methods make differ- ent assumptions that can each shine in different conditions. This means that choosing the right learning algorithm for a given situation is important to improve OOD general- ization. This possibility has thus been tantalizing:
model thus learns exploit the algorithms' performance in variety of conditions. It can then recommend an appropri- ate algorithm for any new task (Figure Ic).
Results. We evaluate the model on seven applications in synthetic (Sagawa al., 2020), vision (CelebA, MetaShift, OfficeHome, etc.) and language domains (CivilCoomentts MultiNLI). The model is capable of selecting suitable algo- rithms for unseen shifts and datasets, achieving significantly lower test error than any static choice of algorithm. We verify that it achieves this by learning non-trivial, non-linear relations between dataset characteristics and algorithms' performance. Crucially, we examine the learned model to extract its decision rules, which reveal which dataset charac- teristics are important for various algorithms to outperform one another.
"It would be helpful for practitioners to be able to select the best algorithms without comprehensive evaluations and comparisons." " (Wiles et al., 2021)
Contributions. Our findings support a positive answer to the title: OOD generalization can be improved by learning to better use existing algorithms. This helps in dealing with complex types of shifts that are difficult to study theoreti- cally, and provides insights on the applicability of existing algorithms. Our contributions are summarized as follows.
However, identifying the right algorithm without trial and error is challenging due to the intricate interplay between the algorithms'' inductive biases and the complex types of shifts in real-world data. Moreover, the trial-and-error might be impossible because of the lack of OOD evaluation data.
We are the first to identify and formalize the problem of algorithm selection for OOD generalization ($2).
This paper proposes to learn to predict, given a dataset, the best learning algorithm for training robust model. This a priori prediction contrasts with traditional model selection that first requires training many models or relies on restrictive heuristics (e.g. accuracy/agreement on the line or activation coverage, see Garg et al. (2022); Baek et al. (2022); Miller et al. (2021); Teney et al. (2023); Liu et al. (2023b)). Concurrent work by Bell et al. (2024) selects algorithms for spurious correlations only, based on past per- formance on benchmarks most similar to the new task. We similarly rely on past performance, but (i) we do not require any training run for the new tasks, (ii) we address more types of shifts and (iii) we use non-linee learnable predic- tors instead of non-pan non-parametric saarest-neighoo retrieval.
We present solution (OOD-Chameleon) that takes in dataset characteristics and predicts the suitability of can- didate algorithms to train a robust model ($2.2).
We show with extensive experiments that OOD- Chameleon (i) adaptively chooses suitable algorithms (64), (ii) learns non-trivial data/algorithm interactions
($4.5) that transfer to unseen shifts and datasets ($4.3
4.4), (iii) reveals which dataset properties make algo- rithms outperform one another ($4.5).
We release tool for future research ($3) to construct datasets with controlled and potentially mixed shift types, and magnitudes. including spurious correlations, label shifts and covariate shifts.
OOD- Chameleon. We frame the algorithm selection as a multi-label classification over a set of representative al- gorithms.? We train our model to predict the algorithms' suitability given some dataset descriptor (Figure 1b), such as dataset size, degrees of shifts, etc. As training examples, we build dataset of datasets representing diverse types, mag- nitudes, and combinations of shifts, by re-sampling existing datasets with fine-grained annotations, such as CelebA (Liu et al., 2015) or CivilComments (Borkan et al., 2019)). The
2. OOD-Chameleon: Algorithm Selection for OOD Generalization
2.1. Is the Selection of the Best Algorithm Possible?
The no-free-lunch theorem (Wolpert & Macready, 1997) prevents a universal solution to the selection of the best learning algorithm. But distribution shifts appearing in real- world data are not arbitrary² (Wiles et al., 2021; Goldblum et al., 2024). A learning approach could thus be trained for effective algorithm selection on a particular distribution
We consider algorithms with proven efficacy on different types of shifts that are actually used in ML. deployments, and not only in one-off academic papers (see discussion in Section 4): ERM, GroupDRO, Oversample, Undersample, and Logits adjustment.
II shifts were arbitrary, inductive reasoning and machine leaming would not he possible (Wolpert, 1996).
2

---

Is Algorithm Selection for OOD Generalization Learnable?
of distribution shifts. Moreover, measurable properties of dataset can be indicative of an imbalance/shift, and thus of the suitability of algorithms. We therefore propose to train a selector that takes dataset characteristics as input (e.g. dataset size, degree of distribution shifts), and predicts the suitability of various candidate algorithms.
model: A(D = .hg. The OOD performance of A is de- fined the WG error of he on D
We propose OOD- NOD-Chameleon to automate the choice of the best algorithm among M candidates for unseen tasks. This is motivated by the many existing results showing that different algorithms perform differently in different condi- tions (see Section 1). OOD-OOO---hai MOD-Chameleon will learn to choose among candidate algorithms based on their past performance in a variety of conditions. To do so, build meta-dataset D as the training data, which is dataset of datasets rep- resenting a variety of distribution shifts (Section 3). Each jth dataset is associated with an algorithm Am and its WG performance Pym E [0, 1] on the dataset. Formally, the meta-dataset is defined as collection of triplets:
Conjecture 1 (Learnability). There exists a learnable mapping from measurable dataset characteristics (e.g.. size, feature imbalance, shift indicators) algorithm suitability, such thut selector trained on sufficiently diverse and densely sampled distribution of distribution shifts can gen- eralize to unseen, but related, shifts within the support of the training distribution.
This assumption is crucial since there is no guarantee (just like any other learning method) for the selector to generalize beyond the support of the training distribution of distribution shifts. Yet, as we will present, our approach encompasses a wide variety of shift types and magnitudes, which can be densely sampled as training data for the selector.
D = {f(D)), Am, Pjm }
(1)
where f(D))) is a dataset descriptor (Rivolli et al., 2022).
Dataset descriptors. To make the learning tractable, we need a function f Supp(D"r) - R' that summarizes various dataset properties a fixed-length vector. Recent work examined properties relevant to the performance of learning algorithms (Nagarajan al., 2020; Hermann et al., 2023; Yang et al., 2024; Chen et al., 2022; Ye et al., 2022; Wang et al., 2024) but most cannot be measured without first training a model, defeating our purpose. Others (Arango et al., 2023; Öztürk et al., 2022) used trivial properties (c.g. number of classes) that are clearly insufficient in our case.
Distribution shift taxonomy. We consider the common distribution shift taxonomy based on three types (Yang et al., 2023): covariate shifts (CS), label shifts (LS), and spuri- ous correlations (SC). The first two correspond to changes in P(X) and P where X and Y represent inputs and class labels. To formalize spurious features, we view X as containing core features X. always predictive of Y, and possibly other features X. whose presence is also indicated by an attribute A. A spurious correlation means that A and Xa are correlated with Y in the training data, even though X, cannot be relied on to make correct predictions on test data. This implies therefore a shift of X.
We consider two sets of properties for our dataset descrip- tors: (i) distribution shift characteristics and (ii) data complexity characteristics. The former set includes the shift magnitudes defined in Section 2.1 (dge, dis, dea) and the availability of the spurious feature (r). These are pro- vided as ground truth or estimated (see Appendix F). The latter set includes the size of the training set n and the input dimensionality d. We leave for future work the possibility of learning descriptors end-to-end with the algorithm selection.
To measure OOD performance, we use standard worst- group (WG) error on test data, where groups G E Y x A are
combinations (Sagawa et al., 2019). WG performance is also what we will (indirectly) optimize for. The rationale is that it is independent of the test distribution and can thus be addressed by considering only the distribu- tion (imbalances) of training data. WG performance thus promotes general robustness to distribution shifts.
Training an algorithm selector. We can now use D (Eq. 1) to train an algorithm selector (typically a neural net- work) that predicts, given the training set of task, the most suitable algorithm. We can formulate this concept either as regression or a classification. To implement the selector as regression, we train a model ((w,-)) f(D")) x A + R that maps a dataset descriptor f(D"t) and algorithm ident-- fier A to a predicted performance. We could train for the regression objective:
We focus primarily on the most common setting in OOD generalization (Yong et al., 2022) where the training data includes labels of a potentially spurious attribute A. We also evaluate the use of pseudo attributes in Appendix F.
2.2. Algorithm Selection as a Supervised Learning Task
min En LMSE (AASE(((v)) ((w.L
(2)
Our eventual goal is to train robust models on future (yet unknown) tasks. A task is defined by the training and test splits that make up a dataset D = D D = {(x,, y)) U (1 M= Training a model means run- ning a learning algorithm A(-) to obtain a parametrized
"Similar concepts include signal/noise ratio (Yang et al., 2024), magnitude (Wang et al., 2024; Joshi et al., 2023), simplicity (Qiu et al., 2024), spurious/core information ratio (Sagawa et al., 2020). "By abuse of notation, A an algorithm's 's 1-hot identifier here.
3

---

Is Algorithm Selection for OOD Generalization Learnable?
where MSE is the mean square error. The selector can then predict the performance of several algorithms on an unseen task. The top prediction is used to train robust model.
tion, classical results from statistical learning theory apply to the algorithm selector, and (ii) the data-driven view is particularly valuable as it handles complex shifts that are difficult to analyze theoretically. For example, a label shift is effectively addressed with class balancing (Idrissi et al., 2022). It is much less clear, however, how to address a label shift combined with a covariate shift and a mild spurious correlation, for example Garg et al. (2023).
The alternative implementation as a multi-label classifica- tion task is motivated by the fact that neural networks are easier to train for classification than regression (Devroye et al., 2013). A classification also aligns better with the goal of selecting algorithms rather than estimating their absolute performance. We choose a multi-label formulation rather than a single-label/multi-e
because the algorithms do not compete against one another, and multiple algorithmss can sometimes be equally suitable.
In summary, building OOD- Chamelcon proceeds in three steps (see also Figure 1b).
1. Obtaining a collection of datasets with a variety of distribution shifts (see Section 3).
Specifically, we define the training classification labels as follows. For jth dataset in D1, we have M (the number of algorithms) records (f(D))), Am; Pjm J.M We ag- gregate each such set of M records into single train- ing sample (S(D)), YA) where YA E {0, 1 is a one- or multi-hot vector indicating the suitability of the candi- date algorithms. An algorithm is considered suitable if (Pim- - minmP,m) S e for a small threshold e (we use 0.05 for all experiments and perform ablation study in Ap- pendix E.2). This aggregation converts performance num- bers into discrete labels, which acts as sort of "denoising" since algorithms with close performance are deemed sim- ilarly suitable. We use these labels to train a multi-label classifier ((w, :) J(D"r) + (0, 1j for the objective:
2. Assembling the meta-dataset (D)) i.c. pre- computing dataset descriptors and training mod- els with the candidate algorithms to obtain their "ground truth" performance.
3. Training the algorithm selector (Φ) on TD.
3. A Tool to Construct Distribution Shifts
We now describe the construction of the dataset collection, which exemplifies various shift types and magnitudes. The same tool can be used for future research to obtain datasets with controlled, mixed-type shifts. It takes as input a dataset that have fine-grained annotations (i.e. each sample is anno- tated with its class label and one or more attributes), such as CelebA (Liu et al., 2015) CivilComments (Borkan et al., 2019), and outputs a dataset with controlled distribution shifts by resampling the source dataset.
min ED LECE (0(w, (f(w,(DD f(D))) YA)
(3)
where LBCE is the binary cross-entropy. Unless noted, DOD-Chameleon refers to this multi-label classification im- plementation. We use the simplest objective for clarity and proof-of-concept; future work can adjust the objective for de- sirable properties, such as a DRO-like objective (Rahimian & Mehrotra, 2022) or a multi-task objective.
Quantifying the distribution shifts. To have precise con- trol over the degree of shifts, we need an unambiguous way to quantify them. Assuming the test set is balanced (i.e. all groups are equally represented), then how imbalanced/bi- ased the training data is (to test data) directly reflects the degree of shifts. Therefore, we define the following degrees:
Applying the algorithm selector. The trained selector can then predict the suitability of algorithms on an unseen task. When multiple algorithms are predicted as suitable at test- time, the one with the top prediction logit (which corre- sponds to the most confident one) is used to train robust model, see Appendix E.3 for an ablation study on test-time algorithm selection strategies.
Spurious correlation (dac): The ratio of training sam- ples whose class label and attribute agree, i.e. where correct classification of the attribute entails a correct classification of the class.
2.3. Discussion and Summary
Label shift (dha): The imbalanceness of training class label distribution.
OOD-Chameleon can be seen data-driven performance prediction, or a learned priori model selection (i.e. be- fore training any target model). A related line of work is the data-driven selection among pre-trained vision mod- els (Zhang et al., 2023; Achille et al., 2019; Öztürk et al., 2022). In all cases, the intuition to exploit knowledge of past performance in known conditions for new tasks. In our case, (i) due to the supervised classification formula-
Covariate shift (des): The imbalanceness of training attribute distribution.
See Figure 2 for an illustrative example with a 2-way classi- fication of shapes with two color attributes, where - is the set cardinality, and Σ, | = n the size of training set D"r, with |Ge| being the number of samples in each group. These degrees are in [0, 1] by definition and 0.5 means the absence
4

---

Is Algorithm Selection for OOD Generalization Learnable?
Core ( X
it is in the set of "suitable" algorithms (as defined in Sec- tion 2.2). We also report the worst-group error (lower is better) of models trained with the selected algorithms, av- eraged across unseen tasks. The former evaluates the algo- rithm selection itself. The latter evaluates the actual benefits in error reduction by relying on the algorithm selector.
== 1,a =
=1 y=
|9 + Gssl
due-
9i
Σ; 6.1
Spurious (X,)
1911 |
di.
Σ, 19c
dew
|1 + 192l
Gs
9
Candidate algorithms. We select five algorithms, namely ERM (Vapnik, 2000), GroupDRO (Sagawa et al., 2019), oversampling minority groups (Japkowiez & Stephen, 2002), undersampling majority groups, and logits adjust- ment (Menon et al. (2021); Nguyen et al. (2021); Kini et al. (2021), which encourages a relative larger margin for the minority groups. We choose to focus on algorithms (i) with strong proven performance, competitive or superior to more complex ones (Nguyen et al., 2021; Gulrajani & Lopez-Paz. 2020; Idrissi et al., 2022; Yang et al., 2023). (ii) that do not require extensive hyperparameter tuning, (iii) that each ad- dress different types of shifts (Nguyen et al., 2021), (iv) that clearly belong to different families, namely regularization-, reweighting and marggn-basee approaches. Building OOD- Chameleon on other existing algorithms is a direct extension that however requires more computational resources, which we thus leave for future work.
Σ, 19.1
y -1-1,a-
y = 1,a
Figure 2. Illustration of the quantification of distribution shifts. In this example, dec 33/8, di dl==0am 0.5, and des 55/8.
of a shift. In practice, we can different class-attribute pairs available in datasets such as CelebA or CivilComments (see Table 9 for some pairs). Different class-attribute pairs naturally induce different availabilities of spurious features, depending on how the attributes are embedded in the data.
Constructing datasets. Next, given a desired training set size and a triple (des: dis dac) E [0, 1]3 that indicates desired degree of shifts, we could solve linear equations (the ones in Figure 2, plus E, |Gel 12) to obtain the necessary number of samples per group Gi to achieve these proper- ties in the training set. The test set D is balanced, i.e. [Gal = /ite/ /nmm. of groups. We sample the corresponding number of samples from the source dataset and compose them into task with desired properties. We then perform this process many times with different properties to obtain many different datasets. See Appendix B for details and Figure 7 for example datasets.
Algorithm selection baselines and ablations.
Random selection: randomly selecting algorithms.
choosing the single best algorithm accord- ing to its performance on all tasks in the meta-dataset.
Global best:
Naive descriptors: using trivial dataset properties from Öztürk et al. (2022) as input train the selec- tor, instead of our dataset descriptors. This evaluates if our descriptors provide relevant information.
4. Experiments
Oracle selection: upper bound, which uses the best algorithm for each dataset.
We evaluate OOD-Chameleon on seven applications from three domains: synthetic (Sagawa et al., 2020), vision (CelebA (Liu et al., 2015), MetaShift (Liang & Zou, 2022), OfficeHome (Venkateswara et al., 2017), Colored- MNIST (Arjovsky et al., 2020), and language (CivilCom- ments (Borkan et al., 2019), MultiNL (Williams et al., 2017)). See details of the datasets in Appendix A. As we will show, all cases benefit from the adaptive selection. We also verify that it achieves this by learning non-trivial de- cision rules that are transferrable across tasks. Finally, we examine the learned selector to understand when the algo- rithms can outperform one another.
Regression: selector trained from the regression (Eq. 2).
Models. The algorithm selector is implemented as an MLP unless otherwise noted. We evaluate other implemen- tations in Table 5. As for target models, the synthetic exper- iments (Section 4.2) follow Sagawa et al. (2020) and use linear model, which is sufficient to solve the synthetic task. The vision experiments (Section 4.3) use linear probing on a pre-trained ResNet18 (He et al., 2015) or CLIP model (ViT- B/32) (Radford et al., 2021). We show in Appendix E. that MoD-Chameleon is effective for both linear probing and fine-tuning paradigms. The language experiments (Sec- tion 4.4) use linear probing on pre-trained BERT (Devlin et al., 2019) or BERT fine-tuned on CivilComments. In all cases, we train for long enough to ensure convergence with identical hyperparameters across runs. The rationale is that hyperparameter search should not be allowed since OOD validation data cannot be relied on (otherwise it could be simply used as training data to achieve OOD generalization).
4.1. Experimental Setup
For each domain (synthetic, vision, language), create meta-dataset ID then train an algorithm selector. See Appen- dices C-D for details. We then evaluate the generalizability of the algorithm selector on unseen tasks with properties disjoint from DD We report the 0-1 accuracy (higher is bet- ter), which considers an algorithm prediction as correct if
5

---

Is Algorithm Selection for OOD Generalization Learnable?
Difference WG test error (%5
75
85
85.0
(chrll skow) des (degren dy serifsg) day labilaybbty) (ipput dim.)
50
80
Test worst porst-group error
02.5
25
75-
Non-parametrized base (Global Best
22
Test ACCC
-25
Test 0-1
70-
77.5
85
21
75.0
-50
-75
60
72.5
56
70.0
U-0
U-E Methads
G
U-L
0
2000
4000
6000
8000
N/A
Size of mota-dalaset
Removed
descriptor
Figure 3. (Left) Discrepancy in performance among algorithms. Across many tasks, we compare pairs of algorithms (c.g. L-G means comparing Logits adjustment and GroupDRO) and show the distribution of performance differences in worst-group test error. (Middle) The generalizability of the algorithm selector improves with a larger meta-dataset. (Right) We estimate the importance of each piece of the dataset descriptor with leave-nn--descriior--au training of the algorithm selector.
4.2. Synthetic Experiments
Table I. Results on the synthetic tasks. The learned algorithm
selector approaches the performance of the oracle selection.
Data. We consider a binary classification following closely the synthetic example from Sagawa al. (2020). The group distribution is implicitly defined from the defi- nition of inputs as dimension d generated from Gaussian distributions:
Methods
0-1 ACC. (%)
WG crror (%)
Romarks
Oracle selection
100
19.0
Upper bound
Ba] E IR22d with &e and va of
Random selection
62.9+0.s
24.0=0.1
con-parametrrzed
Global best
72.5
22.7 20.1
Non parametrized
Naive descriptors
52.1
23.9
Özüük et al. (2022)
Regression
79.74a.7
20.4
Equation
Je y ~ N (yl, Ta)), Ia Iy o
COD-Chameleon
86.3g0
19.9
Equation
The availability of the spurious features is defined as o //2 We create ~ 7,000 tasks for I'' spanning different duc; dis, des, training sizes n. dimensionality d, availability r, and another ~ 2,000 for evaluation (details in Appendix C).
features, etc. We simulate different availabilities of the spu- rious feature by using different attribute annotations from CelebA, e.g. mouth slightly open as class label and wearing lipstick as spurious attribute (details in Ap- pendix D). With MetaShift, we similarly use fcat, dog} as classes and ( (indoor, outdoor) as attributes to gener- ate 129 tasks. With OfficeHome, we created 100 tasks by randomly sampling pair of domains and pair of classes for each dataset (e.g. classifying (pen, kni fe} in (Art, Cl partt) }). For each task of OfficeHome, the number of samples of each group is naturally imbalanced, creating dif- ferent distribution shifts by design. With Colored-MNNE -NIST, we use the digits as classes and colors as attributes to gen- erate 180 tasks. We build the meta-dataset with 80% the CelebA tasks to train the selector, then evaluate either on the remaining 20% of CelebA tasks, the MetaShift, OfficeHome, Colored-MNISY tasks, respectively.
Results. Table I first confirms, from the gap between random selection and oracle selection, that the ability to choose the right algorithm provides substantial benefits. We show that OOD-Chameleon is accurate at predicting the most suitable algorithm on unseen tasks. Both the 0-1 accuracy and the goost error are substantially better than the baselines.
Additional results in Figure 3 (left) indeed show that dif- ferent algorithms perform differently across tasks/datasets. Figure 3 (middle) shows that OOD-Chamelcon is more accurate with the meta-dataset scaled up. Yet, it already sig- nificantly outperforms the best pon-meraeetrized baseline ("Global best") with relatively small (~~ 200) meta-dataset. In Figure 3 (right), we conduct leave-one-descriptoronn training of OOD- NOD-Chamelcon, excluding the descriptor at a time. The drops in accuracy (compared to the leftmost bar) show that every element provides useful information. Those that matter most are the input dimen- sionality d, then the availability r and degree dec of the spurious correlation.
element from
The availability of spurious features. Unlike with syn- thetic data, there is no straightforward measure of the availability for real-world data. We use a proxy En // En la where ly (resp. 4,) is the average distance be- tween every sample and the center of the cluster of its class (resp. attribute). Distances are measured in the embedding space of the corresponding pre-trained model (Section 4. I). Intuitively, the availability is higher when the cluster w.r.t. the attribute more compact than to the label (see details and discussions in Appendix D.
4.3. Vision Experiments
Data. We generate 720 tasks from CclebA (Liu et al., 2015), again with various data sizes, types of shifts, spurious
6

---

Is Algorithm Selection for OOD Generalization Learnable?
Table Results on vision tasks. The selector trained on a meta-dataset built from CelebA, and then evaluated on separate set of tasks from CelebA (Left) as well as MetaShift (Right). It remains accurate across datasets, meaning that it learns generalizable decision rules.
CelebA
MctaShift
Methods
ResNet18
CLIP (VIT-B/32)
ResNet18
CLIP (VIT-B/32)
0-1 ACC. f
WG error
0-1 ACC. 1
WG eror
0-1 ACC. 1
WG error
0-1 ACC.
WG error
Oracle selection
100
44.9 #0.2
100
36.3 #0.3
100
36.4 #0.3
100
24.5 =0.3
Random selection
28.5 20.5
53.4 10.6
27.0 10.3
45.4 10.2
33.3 10.8
43.1 10.2
34,1 40.3
32.3 =0.1
Global best
35.7 =1.0
51.3 20.5
32.6 20.4
43.4 10.1
39.4 +0.3
42.4 10.2
36.4 40.3
31.7 20.4
Naive descriptors
64,5 20.9
49.7 20.5
66.8 20.6
40.7 20.4
68.8 20.7
40.3 203
66.6 20.3
28.1 20.4
Regression
53.6 #1.1
49.8 20.5
46,7 #0.9
41.2 #0.3
51.2 #0.3
40.9 #0.2
49.2 #0.3
28,3 =0.2
001 Chameelln
75.0
47.7 20.2
78.5
39.1 +0.2
80.6 +0.7
39.0 20.3
76.7 20.5
27.2
Table 3. Comparison with static algorithm selection, i.e. using the same one used on all test tasks. OOD-Chameleon performs much better by adaptively choosing an algorithm for cach task. It approaches the performance of an oracle selection, as well as its distribution of cases where each algorithm is used (see the colored bars where each color represents a different algorithm).
CelebA
MetaShift
Methods
ResNet18
CLIP (VIT-B/32)
ResNet18
CLIP (VIT-B/32)
Alg. Selection
WG emror J.
Alg. Selection
WG error +
Alg. Selection
WG crror
Alg. Selection
WG error J
Oracle Selection
44.9
36.3
36.4 10.3
24.5
ERM
57.8
49.1 =0.2
47.1 10.3
35.6 20.3
GRoupDRO
52.5 +0.4
45.7 =0.2
45.0 +0.3
33.4 +0.3
Logits adjustment
53.1 +0.5
41.4 =0.8
40,5 +0.5
28.0 #0.4
Undersample
49.8 +0.5
41.6 =0.2
40.1 +0.3
28.3 +0.3
Oversample
52.4
45.8
45.1
33.2
00D CoD-Chameleon
47.7 10:2
39.1 =0.2
39.0 10.3
27.2 20.3
Results. In Table 2, observe again that OOD- Chameleon performs well in selecting suitable algorithms for unseen tasks. Most importantly, the selector trained on a meta-dataset constructed from CelebA also performs well on MctaShift (Table 2), OfficeHome and Colored-Me -NISST (the latter two are in Appendix G) This means that learned generalizable relations between dataset properties and algorithms' performance, not idiosyncratic patterns spe- cific to one dataset.
CivilComments (Borkan al., 2019) and 180 from MultiNLI (Williams et al., 2017) (details in Appendix D).
Results. Table 4 and Table 18 in the appendix show that OOD-Chameleon is also effective in the language domain. The results closely align with those on vision tasks and confirm the findings from Section 4.3.
4.5. What Does OOD-Chameleon Learn?
Now we examine the decision rules learned by the selector. We first evaluate its complexity. To do so, we compare in Table 5 several implementations of the algorithm selector and make the following observations.
In Table 3, we compare with single-algorithm baselines, Le. using the same algorithm for all tasks. Our adaptive selection performs much better, confirming the premise that no single algorithm is a solution to all OOD scenarios. Interestingly, the proportions of cases where each algorithm is selected closely resemble those of the oracle (see the colored bars). In Appendix F, we evaluate our model with estimated dataset descriptors, e.g. for cases where attributes are not available at test time. The predictions of the selector remain accurate without further adaptation. In Appendix we show that a selector trained on datasets of smaller data sizes can generalize larger ones; and compare our approach with ensemble, which is much more expensive.
First, linear model performs significantly worse than an MIP. This shows that the MI.P learns non-trivial rules and that there exist non-linear relations between dataset characteristics and the performance of algorithms.
Second, k-NN model is also significantly worse than an MLP. This shows that our model works not only by memorizing a large number of example tasks, which a k-NN also does. On the contrary, accurate predictions on unseen datasets require non-trivial generalization.
4.4. NLP Experiments
Third, a simple decision tree performs almost as well as an MLP (details in Appendix J). Unlike an MLP, a tree intrinsically interpretable, which offers the possibility of discovering new insights about existing algorithms.
Data. The setup is analogous to the vision experi- ments (Section 4.3). We generate 720 tasks from
7

---

Is Algorithm Selection for OOD Generalization Learnable?
Table Results NIP tasks. The selector is trained on a meta-dataset built from CivilComments, then evaluated separate set of
tasks from CivilComments as well as MultiNLI. It remains accurate across datasets, meaning that it learns generalizable decision rules.
CivilComments
MultiNIJ
Methods
BERT
BERT (Finetuned)
BERT
BERT (Finetuned)
0-1 ACC.
error
0-1 ACC. t
WG error
0-1 ACC.
WG emrr
0-1 ACC.
WG error
Oracle selection
100
53.7 203
100
19.4 24.4
100
55.9 2033
100
54.2 20.6
Random selection
14.5 20.7
66.6 #0.4
50.0 10.1
23.5 10.2
19.8 10.3
67.1 #0.2
20,5 #0.5
63.6 =0.7
Global best
29.9 20.8
62.7 20.4
55.9 21.2
22.9 10.3
34.4 10.4
63.6 20.0
27.2 20.0
61.4 20.5
Naive descriptors
68.8 #1.9
57.2 #0.4
81.1 #0.8
21.7 #0.3
64.3 #0.5
59.2 #0.4
68.2 #0.5
57.4 =0.4
Regression
51.3 21.0
57.6 10.4
47.9 20.7
22.6 20.4
57.7 20.8
59.4 10.2
40.8 10.3
60.6 =0.2
OOD-Chameleon
81.9 21.1
55.8 20.4
90.9 11.4
20.7 20.2
79.4 10.0
58.3 20.2
74.4 20.9
56.6 20.2
Attributing algorithm effectiveness to data characteris- tics. To complement the general rules revealed by decision trees, we look at factors affecting local decisions between any two algorithms. We perform leave-one-descriptrr- out training of pairwise algorithm selectors, i.e. which con- sider only two candidate algorithms. We plot the results in Figure 5. For a chosen pair of algorithms, each bar shows a drop in performance relative to the full descriptors (leftmost bar). This drop indicates the importance of specific piece of information for distinguishing the two algorithms. In the case of Oversamplel Saaieee/nndd Undersample for example, the data size (n) and degree of spurious correlation (dec) are the most im- portant. This observation is consistent with the analysis in (Nguyen et al., 2021) that implies that, while undersampling can cope with more distribution shifts than oversampling, it is inferior when the number of samples in the minority group is small (i.e. when n or dhc too small).
Table 5. Alternative implementations of the algorithm selector. The lower performance of linear and k-NN models supports the im- portance of non-trivial, non-lincar relations between dataset char- acteristics and algorithm performance (experiments on CelehA).
ResNet18
CLIP B/32)
Implementution
0-1 ACC.
WG error
0-1 ACC.
WG error
Linear
63.4 #1.0
49.6 #0.3
67.6 #1.8
42.1 +0.3
K-NN
38.6 2077
49.3 2
50.0 11.
41.5 10.4
Decision tree
73.1 20.
48.1 20.3
24.3
39.4 20.5
MLP
75.0 213
47.7 10.2
78.5 20.8
39.1 10.2
Interpreting decision trees. We visualize in Figures 10- 11 the rules learned by selectors implemented as decision trees (see Figure 4 for simplified illustration). Interpreting these trees leads to the following recommendations on the applicability of algorithms. (i) Undersample is preferred when the size of the dataset is not too small and the distri- bution shifts are severe. (ii) When there are indeed only a small number of samples but relatively large shifts, one could resort to Logits adjustment. This aligns with Nguyen et al. (2021) who suggested that Logits adjustment should be more effective than Undersample when the number of unique samples is low for minority groups. (iii) GroupDRO is more useful when there are enough samples and rela- tively large distribution shifts. (iv) For mild shifts, ERM or Oversample are the best options. Overall, such observa- tions provide rich material for future investigations on the applicability of existing learning algorithms.
95
Removed
Test 0-1 ACC. (%)
90
descriptor
85
dsc
d
des
75
70
Oversample/ Undersample
Undersample/ ERM
GroupDRO/ Logit adjust.
Figure 5. Leave-one-deccriptoroouu training of pairwise algo- rithm selectors. The drops in accuracy reveal the importance of various factors in the suitability of one algorithm vs. another.
Logit adjustment
LS
Oversample
Mild
Strong
Undersample
SC
GroupDRO
Small
Large
We also observe that the importance of the different pieces of information varies for other pairs of algorithms (Fig- ure 5 middle/right bars). We can thus identify which char- acteristics of the data are important in practice for specific algorithms to outperform one another. These are rich ob- servations for future work that could confront them with existing theories on the applicability of these algorithms.
ERM
LS
Figure 4. Simplified illustration of learned model selector im- plemented as a decision tree (see Figures 10-11 for full versions).
8

---

Is Algorithm Selection for OOD Generalization Learnable?
5. Related Work
6. Conclusions
OOD generalization is a wide field of study, see e.g. Liu et al. (2021b) for a survey. The conclusion from several studies is that there is no one-fits-all solution to distribution
This paper highlighted the importance of algorithm selection for OOD generalization. We described a data-driven solu- tion to automate this selection, by turning it into a standard supervised multi-label classification problem. Crucially, we demonstrated empirically that training our algorithm selec- tor on collection of comi-synthetic datasets is sufficient to learn decision rules that generalize to unseen shifts and datasets. This is a strong empirical result, since the model could as well have overfitted to idiosyncratic properties of example shifts and datasets. Instead, it learns generalizable relations between dataset properties and algorithms' perfor- mance. We verified that the selection relies on non-trivial patterns, and also ruled out the possibility that it simply memorizes a large number of example cases.
shifts (Gulrajani & Lopez-Paz, 2020; Wiles et al., 2021;
Nguyen et al., 2021; Ye et al., 2022; Liang & Zou, 2022; Yang et al., 2023; Bell et al., 2024). Indeed, standard ERM baseline is generally very effective, but there also exist a variety of training algorithms with demonstrated benefits in specific OOD settings. This motivates our goal of improving the selection among existing algorithms, which we seek to automate with data-driven approach.
Algorithm selection. Model selection (Forster, 2000; Raschka, 2020) and algorithm selection (Rice, 1976; Ker- schke et al., 2019) are integral parts of machine learning workflows. In the context of OOD generalization, recent work (Liu et al., 2023b; Baek et al., 2022; Garg et al., 2022; Miller et al., 2021; Lu et al., 2023) proposes heuristics to predict a models' OOD performance on unlabeled target data. These heuristics have downsides: (i) they can only be estimated after training one or several models, (ii) they re- quire (unlaheled) test data, and (iii) they rely on unverifiable assumptions (Teney et al., 2023). A major difference of our work is to aim for priori selection of the best algorithm, i.c. before training on the target data.
Limitations and future work.
Algorithm baselines: We focused on algorithms known to be reliable, simple but effective in different settings (and often superior to fancier ones, see Idrissi et al. (2022)). Future extension will include more algorithms and their variations, e.g. by including some of their hyperparameters in the search space.
Learnable dataset descriptors: We hand-designed interpretable dataset descriptors, but they depend on attribute annotations and might miss important infor- mation. The next step is to learn descriptors end-to- end with the selector, e.g. with Set Transformers or Dataset2Vec (Lee et al., 2019; Jomaa et al., 2021).
Concurrent work by Bell et al. (2024) identifies a motivation similar to ours. They propose to select among algorithms to deal specifically with spurious correlations, based on the algorithms' past performance on benchmarks similar to the target data. The differences with our work are: (i) we use learning approach rather than a simple similarity; (ii) our learning process relies on semi-synthetic data rather than existing benchmarks, which helps cover a broader set of distribution shifts; (iii) we consider label shifts, covariate shifts and their combinations, not only spurious correlations.
Generalization to other shift types: We considered three types of shifts (spurious correlations, label shifts, covariate shifts) and their combination, which covers most, if not all, shifts studied in the literature. However, generalizing to other shift types might require additional curation. For example, covariate shifts can happen in many forms except for the attribute distribution shifts we studied.
Meta-learning. Our approach learns from a "dataset of datasets" i... a distribution of tasks, which is a principle of meta-learning (Vilalta & Drissi, 2002; Hospedales et al., 2022). Most similarly to us, Achille et al. (2019); Öztürk al. (2022); Arango et al. (2023); Zhang et al. (2023) learn to select among pretrained models for downstream tasks, or for outlier detection Zhao et al. (2021). Contrary to us, these works first require training multiple candidate models and they do not target OOD generalization.
Generalization to broader scenarios: Our tool to quan- tify distribution shifts applies binary classification with binary attributes, which is the default setting in studying spurious correlations. Extension to multiple classes and multiple attributes is valuable. Besides, opti- mizing a combination of multiple relevant performance metrics, instead of a single one, is promising.
Real-world evaluation: A crucial next step is to study how the algorithm selector trained on a semi-synthetic dataset collection generalizes to real-world tasks, such as WILDS (Koh et al., 2021: Sagawa et al., 2022).
AutoMI.. Our approach relates to AutoMI,, which aims to automate ML workflows, often by riil--nn-errro (Hutter et al., 2019; He et al., 2021). Our work differs in that (1) it aims for a priori algorithm selection; (2) it can provide new insights on the applicability of existing algorithms.
Understanding algorithms' applicability: We ex- tracted insights from the trained selector, which could be used to form the basis of future investigations on the ap- plicability and inductive bias of many other algorithms.

---

Is Algorithm Selection for OOD Generalization Learnable?
Acknowledgements
Z., Natarajan, V., Nielson, C.,, Osborne, T. F, Raman,
R., Ramasamy, K., Sayres, R., Schrouff, J., Seneviratne, M., Sequeira, S., Suresh, H., Veitch, V., Vladymyrov, M., Wang, X., Webster, K., Yadlowsky, S., Yun, T., Zhai, X., and Sculley, D. Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research, 2022.
We thank Caglar Gulcehre, Devis Tuia and Thien-Anh Nguyen for the helpful discussions, and the anonymous reviewers for their thoughtful feedback.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Conference of the North American Chapter of the Association for Computational Linguis- ties: Human Language Technologies, Volume I (Long and Short Papers), Association for Computational Linguistics. 2019.
References
Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C.,, Soatto, S., and Perona, Task2Vec: Task Embedding for Meta-Learning, In IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2019.
Devroye, L., Györf,, L., and Lugosi, G. A Probabilistic The- ory of Puttern Recognition. Springer Science & Business Media, 2013.
Arango, S. P., Ferreira, F., Kadra, A., Hutter, F., and Grabocka, J. Quick-Tune: Quickly Learning Which Pre- trained Model to Finetune and How. In International Conference on Learning Representations (ICLR), 2023.
Forster, M. R. Key Concepts in Model Selection: Perfor- mance and Generalizability. Journal of Mathematical Psychology, 2000.
Arjovsky, M., Bottou, L.,, Gulrajani, I., and Lopez-Paz, D. Invariant Risk Minimization. arXiv:1907 0220990,.2....... 2020.
Garg, S., Balakrishnan, S., Lipton, Z. C., Neyshabur, B., and Sedghi, H. Leveraging unlabeled data to predict out- of-distribution performance. In International Conference on Learning Representations (ICLR), 2022.
Back, C., Jiang, Y., Raghunathan, A., and Kolter, J. Z. contnent-on-thhliine Predicting the Performance of Neural Networks under Distribution Shift. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Garg, S., Erickson, N., Sharpnack, J., Smola, A., Balakrish- nan, S., and Lipton, Z. C. RLSbench: Domain Adaptation Under Relaxed Label Shift. arXiv:2302.03000 000200 2023.
Bell, S. J., Bouchacourt, D., and Sagun, L. Reassess- ing the Validity of Spurious Correlations Benchmarks. arXiv:2409.0m 0488, 2024.
Goldblum, M., Finzi, M. A., Rowan K., and Wilson, A. G. Position: The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning. In 41st International Conference on Machine Learning (ICMI.) 2024.
Benoit, H. I.., Jiang, I.., Atanov, A., Kar, O. F., Rigotti, M., and Zamir, A. Unraveling the Key Components of OOD Generalization via Diversification. In International Conference on Learning Representations (ICLR), 2023.
Guirajani, I. and Lopez-Paz, D. In Search of Lost Domain Generalization. In International Conference on Learning Representations (ICLR), 2020.
Borkan, D., Dixon, I.., Sorensen, J., Thain, N., and Vasser- man, L. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion 2019 world wide web conference, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learn- ing for Image Recognition. arXiv: 1:Xiv:5512.03385 2015.
He, X., Zhao, K., and Chu, X. AutoML: A Survey of the State-of-the-Art. arXiv: / 1908. 00709, 2021.
Chen, Y., Xiong, R., Ma, Z.-M., and Lan, Y. When Does Group Invariant Learning Survive Spurious Correlations? In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Hermann, K., Mobahi, H., Fel, T., and Mozer, M. C. On the Foundations of Shortcut Learning. In International Conference on Learning Representations (ICLR), 2023.
D'Aurrr, A., Heller, K., Moldovan, D., Adlam, B., Ali- panahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., Hormozdiari, F., Houlsby, N., Hou, S., Jerfel, G., Karthikesalingam, A., Lucie, M., Ma, Y., McLean, C., Mincu, D., Mitani, A., Montanari, A., Nado,
Hospedales, T., Antoniou, A., Micaclli, P., and Storkey, A. Meta-Learning in Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2022.
10

---

Is Algorithm Selection for OOD Generalization Learnable?
Hutter, F., Kotthoff, L.,, and Vanschoren, J. (eds.). Auto- mated Machine Learning: Methods, Systems, Challenges. Springer Nature, 2019.
Liang, W. and Zou, J. MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Train- ing Conflicts. In International Conference on Learning Representations (ICLR), 2022.
Idrissi, B. Y., Arjovsky, M., Pezeshki, M., and Lopez-Paz, D. Simple data balancing achieves competitive worst- group-accuracy. arXiv:2110. 14503, 2022.
Liu, E. Z., Haghgoo. B., Chen, A. S., Raghumathan, A., Koh, P. W., Sagawa, S., Liang, P., and Finn, C. Just Train Twice: Improving Group Robustness without Training Group Information. In International Conference on Machine Learning (ICML), 2021a.
Japkowicz, N. and Stephen, S. The class imbalance problem: A systematic study. Intelligent Data Analysis, 2002.
Jeon, M., Choi, S., Lee, H., and Yeo, T. An analysis of model robustness across concurrent distribution shifts. Transactions on Machine Learning Research (TMLR), 2025.
Liu, J., Shen, Z., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. Towards arXiv:2 002.008.366244 33624, 2021b.
generalization: A survey.
Jomaa, H. S., Schmidt-Thieme, L., and Grabocka, Dataset2 Learning dataset meta-features. Data Min- ing and Knowledge Discovery, 2021.
Liu, J., Shen, Z., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, Towards Ou--Or-Distribttion Generalization: A Survey. arXiv:2108. 13624, 2023a.
Joshi, S., Yang, Y., Xue, Y., Yang, W., and Mirzasoleiman, B. Towards Mitigating Spurious Correlations in the Wild: A Benchmark and more Realistic Dataset. arXiv:2306. 11/06./199 1157, 2023.
Liu, Y., Tian, C. x., Li, H., Ma, L.., and Wang. S. Neuron Activation Coverage: Rethinking Out-of-distributin De- tection and Generalization. In International Conference on Learning Representations (ICLR), 2023b.
Kerschke, P., Hoos, H. H., Neumann, F., and Trautmann, H. Automated Algorithm Selection: Survey and Perspectives. Evolutionary Computation, 2019.
Liu, Z., Luo, P., Wang, X., and Tang. X. Deep Learning Face Attributes in the Wild. In 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
Kini, G. R., Paraskevas, O., Oymak, S., and Thrampoulidis, C. Label-imbalanced and group-sensitive classification under overparameterization.. In Ranzato, M., Beygelz- imer, A., Dauphin, Y., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Sys- tems (NeurIPS), 2021.
Lu, Y., Qin, Y., Zhai, R., Shen, A., Chen, K., Wang. Z., Kolouri, S., Stepputtis, S.,, Campbell, J., and Sycara, K. P. Characterizing Out-of- Distriibiiiitttion Error via Opti- mal Transport. In Thirty- Seventh Conference on Neural Information Processing Systems, 2023.
Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S. Long-tail learning via logit adjustment. In International Conference on Learning Representations (ICLR), 2021.
Kirichenko, P., Izmailov, P., and Wilson, A. G. Last Layer Re- Re-Training is Sufficient for Robustness to Spurious Cor- relations. In International Conference Learning Rep- resentations (ICLR), 2022.
Miller, J. P., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L. Accuracy on the Line: On the Strong Correlation Be- tween Out-of- cittof-Distributionn and In- Distribution Generaliza- tion. In International Conference on Machine Learning (ICML), 2021.
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M.. Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., Lee, T., David, E., Stavness, I., Guo, W., Earnshaw, B., Haque, I., Beery, S. M., Leskovee, J., Kun- daje, A., Pierson, E.,, Levine, S., Finn, C., and Liang, P. WILDS: A Benchmark of in-the-Wild Distribution Shifts. In International Conference Machine Learning (ICML), 2021.
Nagarajan, V., Andreassen, A., and Neyshabur, B. Un- derstanding the failure modes of out-of-distribure gen- eralization. In International Conference on Learning Representations (ICLR), 2020.
Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. Set Transformer: A Framework for iitettionbasse cromutation-Invaiiann Neural Networks. In International Conference on Machine Learning (ICML), 2019.
Nguyen, T., Nagarajan, V., Sedghi, H., and Neyshabur, Avoiding Spurious Correlations: Bridging Theory and Practice. In NearIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
Lee, Y., Yao, H., and Finn, C. Diversify and Disambiguate: Out-of-Disttriution Robustness via Disagreement. In International Conference on Learning Representations (ICLR), 2022.
Oakden- Rayner, L., Dunmmon, J., Carneiro, G., and Re,
C. Hidden stratification causes clinically meaningful

---

Is Algorithm Selection for OOD Generalization Learnable?
failures in machine learning for medical imaging. In ACM Conference Health, Inference, and Learning. Association for Computing Machinery, 2020.
Sagawa, S., Koh, P. W., Lee, T., Gao, I., Xie, S. M., Shen, K., Kumar, A., Hu, W., Yasunaga, M., Marklund, H., Beery, S., David, E., Stavness, I., Guo, W., Leskovec, J., Saenko, K., Hashimoto, T., Levine, S., Finn, C., and Liang, P. Extending the WILDS Benchmark for Unsuper- vised Adaptation. arXiv:21 12.0220..00000 2022.
Öztürk, E., Ferreira, F., Jomaa, H., Schmidt-Thiee Thieme, L., Grabocka, J., and Hutter, F. Zero-shot AutoML with Pre- trained Models. In International Conference on Machine Learning (ICML), 2022.
Sohoni, N., Dunnmon, J., Angus, G., Gu, A., and Ré, C. No Subclass Left Behind: Fine-Grained Robustness in Coarse- Grained Classification Problems. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Pagliardini, M., Jaggi, M., Fleuret, F., and Karimireddy, S. P. Agree to Disagree: Diversity through Disagreement for Better Transferability. In International Conference on Learning Reprexentations (ICLR), 2022.
Teney, D., Peyrard, M., and Abbasnejad, E. Predicting Is Not Understanding: Recognizing and Addressing Un- derspecification in Machine Learning. In Avidan, S., Brostow, G., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), European Conference on Computer Vision (ECCV). Springer Nature Switzerland, 2022.
Qiu, G., Kuang, D., and Goel, S. Complexity Matters: Fea- ture Learning in the Presence of Spurious Correlations. In Forty-Fire Pirss- International Conference on Machine Learn- ing (ICML), 2024.
Teney, D., Lin, Y., Oh, S. J., and Abbasnejad, E. ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Qiu, S., Potapczynski, A., Izmailov, P., and Wilson, A. G. Simple and Fast Group Robustness by Automatic Fea- ture Reweighting. In 40th International Conference Machine Learning (ICML), 2023.
Vapnik, V. N. The Nature of Statistical Learning Theory. Springer, 2000.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. Krueger, G., and Sutskever, I. Learning Transfer- able Visual Models From Natural Language Supervision. arXiv:2103.00022 2021.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Pan- chanathan, S. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 5018-5027, 2017.
Rahimian, H. and Mehrotra, S. Frameworks and results in distributionally robust optimization. Open Journal of Mathematical Optimization, 3:1-85, July 2022. ISSN 2777-5860. doi: 10.5802/ojmo.15 URL http: dxx doi org. 0 5802 / jmo. 15.
Vilalta, R. and Drissi, Y. A Perspective View and Survey of Meta-Learning. Artificial Intelligence Review, 2002.
Wang, 7.,, Kuang, P., Chu, Z.., Wang, J., and Ren, K. To- wards Real World Debiasing: A Fine-grained Analysis On Spurious Correlation. arXiv:2405.152400 2024.
Raschka, S. Model Evaluation, Model Selection, and Algo- rithm Selection in Machine Learning. 1rXiv:1811.1280 2020.
Wiles, O., Gowal, S., Stimberg, F., Rebuffi, S.-A., Ktena, I., Dvijotham. K. D., and Cemgil, A. T. A Fine-Grained Analysis on Distribution Shift. In International Confer- ence Learning Representations (ICLR), 2021.
Rice, J. R. The Algorithm Selection Problem*. In Rubinoff, M. and Yovits, M. C. (eds.), Advances in Computers. Elsevier, 1976.
Williams, A., Nangia, N., and Bowman, S. R. A broad- coverage challenge corpus for sentence understanding through inference. 17X.::1704.05226, 2017.
Rivolli, A., Garcia, L. P. F., Soares, C., Vanschoren, J., and de Carvalho, A. C. F. Meta-features for meta- learning. Knowledge- Basedd Systems, 2022.
Wolpert, D. and Macready, W. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1997.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Dis- tributionally Robust Neural Networks. In International Conference on Learning Representations (ICLR), 2019.
Wolpert, D. H. The Lack of A Priori Distinctions Between Learning Algorithms. Neural Computation, 1996.
Sagawa, S., Raghunathan, A., Koh, P. W., and Liang, P. An Investigation of Why Overparameterizztion Exacerbates Spurious Correlations. In International Conference on Machine Learning (ICMI.,, 2020. 2020.
Yang, Y., Zhang, H.,, Katabi, D., and Ghassemi, M. Change is Hard: A Closer Look at Subpopulation Shift, In 40th
12

---

Is Algorithm Selection for OOD Generalization Learnable?
International Conference Machine Learning (ICML), 2023.
Yang, Y., Gan, E., Dziugaite G. K., and Mirzasoleiman, B. Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias. In International Conference on Artificial Intelligence and Statistics, 2024.
Ye, N., Li, K., Bai, H., Yu, R., Hong, L.,, Zhou, Li, Z., and Zhu, J. OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization. In IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), 2022.
Yong, L. N., Zhu, S., Tan, L., and Cui, P. ZIN: When and How Learn Invariance Without Environment Partition? In Advances in Neural Information Processing Systems (NeurIPS), 2022.
You, C.. Min, Y., Dai, W., Sekhon, S., Staib, L., and Duncan, Calibrating Multi-modal Representations: Pursuit of Group Robustness without Annotations. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Zhang, Y.-K., Huang. T.-J., Ding, Y.-X., Zhan, D.-C., and Ye. H.-J. Model Spider: Learning Rank Pre- Trained Models Efficiently. Advances in Neural Information Pro- cessing Systems (NeurIPS), 2023.
Zhao, Y., Rossi, R., and Akoglu, L. Automatic Unsuper- vised Outlier Model Selection. Advances in Neural Information Processing Systems (NeurIPS), 2021.
13

---

Is Algorithm Selection for OOD Generalization Learnable?
Appendix
The appendix provides the following additional details and results.
Appendix provides an overview of the used source datasets.
Appendix B details the construction of datasets with desired distribution shifts.
Appendix C describes the data and detailed setup of the controlled experiments of Section 4.2.
Appendix D describes the data and detailed setup of the realistic experiments of Sections 4.3 and 4.4.
Appendix contains various ablation studies (hyperparameters, training paradigms, etc).
Appendix presents additional results on the use of estimated dataset deseriptors as input to the selector.
Appendix presents additional results on vision tasks of Section 4.3.
Appendix H presents additional results language tasks of Section 4.4.
Appendix I presents additional results on efficiency considerations.
Appendix J shows additional details of the algorithm selector implemented by decision free of Section 4.5.
A. Dataset Overview
In our experiments, we create tasks with distribution shifts by subsampling from 6 real-world datasets from computer vision and natural language processing domains. Here we provide details for each of them. Some content in this section is borrowed from (Yang et al., 2023), and we appreciate their clean and concise dataset summary. For visual examples, Table 6 and Table 7.
CelebA. CelebA (Liu et al., 2015) a large-scale face attributes dataset commonly used in facial recognition, attribute prediction, and generative modeling research. It contains over 200,000 celebrity images covering approximately 10,000 identities, each annotated with 40 binary facial attributes (e.g., smiling, eyeglasses, blond hair) and 5 landmark locations (e.g., eyes, nose, mouth corners). The images exhibit substantial variability in pose, lighting, background, and occlusion. The standard task involves predicting facial attributes or generating/cditing facial images using these annotations. The dataset is publicly available for non-commercial research use.
MetaShift. MetaShift (Liang& & Zou, 2022) is general method of creating image datasets from the Visual Genome project. Here, we make use of the pre- pre-processed Cat vs. Dog dataset, where the goal is to distinguish between the two animals. The spurious attribute is the image background, where cats and more likely to be indoors, and dogs are more likely to be outdoors. We use the "unmixed" version generated from the authors' codebase.
OfficeHome. OfficeHome (Venkateswara et al., 2017) is a multi-class image classification dataset commonly used in domain adaptation research. consists of around 15,500 images across 65 object categories (e.g.. keyboard, pen, mug), drawn from four visually distinct domains: Art, Clipart, Product, and Real- World. The standard task to classify the object category across varying domain pairs. The dataset is publicly available for non-commercial research use.
Colored-MNIST.. Colored-MNIST (Arjovsky et al., 2020) is synthetic variant of the MNIST dataset commonly used to study spurious correlations and domain generalization. It is constructed by coloring the grayscale MNIST digits with label-dependent or abbllinddeendden color schemes to introduce a spurious correlation between digit class and color. Multiple environments are created by varying the strength of this correlation, allowing researchers evaluate models' robustness to distribution shifts. The standard task is digit classification under different color-based domain shifts.
CivilComments. CivilComments (Borkan et al., 2019) a binary classification text dataset, where the goal is to predict whether internet comment contains toxic language. The spurious attribute is whether the text contains reference to eight demographic identities (male, female, LGBTQ, Christian, Muslim, other religions, Black, and White).
MultiNLI. MultiLLI (Williams et al., 2017) is a text classification dataset with 3 classes, where the target is the natural language inference relationship between the premise and the hypothesis (neutral, contradiction, or entailment). The spurious attribute is whether negation appears in the text, as negation highly correlated with the contradiction label.
14

---

Is Algorithm Selection for OOD Generalization Learnable?
Table Example imputs for ColSred-MNIST (Arjovsky al., 2020), CelchA (Liu et al., 2015), MetaShift (Liang & Zou, 2022) and OfficeHome (Venkateswara et al., 2017). The table is adapted from (Yang et al., 2023).
Dataset
Examples
2
Colored-MNIST
CelebA
MetaShift
OfficeHome
Table 7. Example inputs for Commennssnnss (Borkan et al., 2019) and MultiNLJ (Williams et al., 2017). The table is horrowed from (Yang
al., 2023).
Dataset
Examples
"Munchins looks like munchins. The man who dont want show his taxes, will tell you everything..."
CivilComments
"The democratic party removed the filibuster steamroll its agenda. Suck it up boys and girls." "so you dont use oil? no gasoline? no plastic? man you ignorant losers are pathetie."
"The analysis proves that there
link between PM and bronchitis." "Postal Service were to reduce delivery frequency."
MultiNLI
"The famous tenements (or lands) began to be built."
B. Construction of the Meta-dataset
In Section 3, we describe framework that allows for constructing datasets with diverse distribution shifts by sampling from synthetic distributions or existing datasets, here we provide more details and examples. There are two use cases in Section 4.2, Section 4.3& respectively, where in both cases we know the distribution of each group (recall that the combinations of different values of attribute a and class form different groups). Specifically, in Section 4.2, we have the group distributions as Gaussian distributions so that we can sample the desired numbers of samples from those distributions, while in Section 4.3 and 4.4 we have a decent amount of samples in each group of the fine-gradn ane-grained annotated real-world datasets used (CelebA, MetaShifts, OfficeHome, Colored-MNIST for vision tasks and CivilComments, MultiNLI for NLP tasks), and we can then also sample the desired numbers of samples from each group.
In Section 3, we define the degrees of distribution shifts as function of the number of samples for each group. Therefore, to obtain a dataset with specific degrees distribution shifts, one only needs to solve the number of samples for each group and sample them from the group distribution. Note that the number of samples can be scaled up or down depending on the expected size of the dataset. All the constraints to be solved are therefore:
9 + |Gal
dis Gi + |Gal
9 + 192
dgc
des
Σ, 9s1
Σ, [9
Σ, |9
Σ |9 n, ( 2 0)
(4)
0 dgc 1, 0 de < 1, 0 < des 1
Solving the constraints gives the feasible solution set of the degrees of distribution shifts, as shown in Figure We see that not any value in the cube can be chosen because of the constraint | 2 for Vi.
15

---

Is Algorithm Selection for OOD Generalization Learnable?
Solution Set of Inequalities Forming Closed Shape
0.
0.8
0.6
1. 0.
0.4
-000
1.0
0.8
0.6
0.2
op.4
0.4
6
dze
1.0
Figure The feasible degrees spurious correlation (dac). covariate shift (dex) and label shift (ds).
Generating datasets with diverse characteristics. We then generate datasets with desired size, distribution shifts, and availability of the spurious feature. More precisely, we vary the following.
The degrees of distribution shifts (as long as they are picked from the feasible solution set in Figure 6).
The size of the dataset.
The group distributions (for simulating different availabilities of the spurious feature). For synthetic experiments, they more controllable in that the group distributions are known and the availability of spurious feature is well-defined (see Section 4.2). For vision tasks and NLP tasks, we respectively use CelebA and CivilComments to create the dataset of datasets. Since they are annotated with various labels, we can use different pairs of them as label Y and spurious attribute A. See Table 9 on the pairs we choose. Each different label-attribute pair results different group distributions, and how "spurious" the spurious attribute
For the synthetic experiments (Section 4.2), we additionally vary the dimensionality of the data points d (See Table 8). See Figure 7 for example tasks generated from CelebA. We summarize the characteristics of the generated datasets in the following sections.
Data size: 1000
Data size: 500 Spurious correlation: 0.82, label shift: 0.41, covariate shift: 0.33
Spurious correlation: 0.12, label shift: 0.26, covariate shift: 0.80
count:
count: 180
count: 134
count: 77
count: 618
count: 78
count: 46
count: 243
Hair Gender
Gender
Figure 7. Example datascts (only training set is shown for each) with different data size and distribution shifts, generated from CelebA The meta-datasct is composed of many more such datascts, each exhibiting different charactcristics.
16

---

Is Algorithm Selection for OOD Generalization Learnable?
C. Controlled Experiments: Datasets and Detailed Setup
We provide details on the experimental setup of the controllable experiments of Section 4.2.
Dataset of datasets. In total, we created 9,240 tasks, see Table 8 for the statisties for these tasks. Specifically, we generate the training sets of the tasks with the combinations of values in Table 8, and for the degrees of distribution shifts, we consider 2 cases: (1)3 3 shifts (spurious correlation, covariate shift and label shift) all present each in different degrees, we uniformly sampled 30 different triple degrees, and (2) there is only 1 shift present, in this case for each shift we consider 9 different values shown in the table (therefore in total 3 9 - 2 = 25 different triple degrees). We generate their test sets with the same number of samples as its training set for each dataset, but keep the number of samples the same for all groups (therefore balanced test sets with all de) 0.5). We randomly split these datasets into 4:1 as the dataset of datasets for training the algorithm selector, and the unseen datasets for evaluation.
Table 8. Statistics for the dataset of datasets in controllable synthetic experiments.
Statistics
Value
Size of training set Input dimensionality
{200, 500, 1000, 2000, 3000, 5000, 10000}
(2, 10, 50, 100)
Availability r == o2/02
{1, 10, 20, 50, 100)
3 shifts
randomly sampled 30 fcasible degrees (Figure 6)
shift
{0.01, 0.05, 0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 0.99)
Algorithm selector. We use 4-layer MLPs to parameterize the algorithm selector, because we found a shallower or simpler model underfits while a deeper model does not provide more improvements.
Tasks. To solve the tasks defined by each dataset in the dataset of datasets, we use Adam optimizer with default hyperparameters, along with L2 regularization. We train for 1000 epochs to ensure convergence on this synthetic example.
D. Datasets and Detailed Setup of the Vision and NLP Experiments
D.1. Availability of the Spurious Features
In the controllable experiments in Section 4.2, we compute the availability of spurious features as o2 / following the spurious-core information ratio defined in Sagawa et al. (2020). The higher the r, the more signal there about the spurious attribute in the spurious features, relative to the signal about the label in the core features.
However, in real-world data (Section 4), it is not straightforward how to generate the dataset of datasets with diverse In Section 4.3, we use different attribute pairs in CelebA (Liu et al., 2015) (one as the label attribute, i.c. the attribute we want to classify, and the other as spurious attribute) as shown in Table 9 (Left), to account for different availability of spurious features. Intuitively, the obviousness of different types of attributes (in terms of size, color, etc) signifies different availability of the spurious feature (caused by the attribute).
Similarly, we use different attribute pairs in CivilComments (Borkan et al., 2019), shown in Table 9 (Right) to ensure diverse availabilities of spurious feature in the created tasks.
Table Attribute pairs in CelebA (Liu et al., 2015) (Left) and CivilComments (Borkan et al., 2019) (Right) that are used to construct different availability of the spurious feature.
CelebA
CivilComments
Label
Spurious attribute
Label
Spurious attribute
Mouth Slightly Open
Wearing Lipstick
Male
Attractive
Smiling
Female
Toxicity
Black Hair
Male
Black
Oval Face
High Cheekhones
White
17

---

Is Algorithm Selection for OOD Generalization Learnable?
By using different attribute pairs, we are able generate tasks with different availabilitiess However, unlike the controllable experiments, here there is no straightforward way of quantifying the availability of spurious feature. Therefore, as mentioned in Section 4.3, we use proxy availability which is the average distances of the samples' embeddings to their labels or attributes clustering center, respectively. Specifically, we first obtain the samples embeddings {2& her from the used backbone (ResNet1 or BERT/BERT (fine-tuned) and then compute the clustering center of each label and attribute as My and paa The availability r is then:
Ey Ly
Ln la
by 2 My
La = |
Ma
(5)
1
y 81 = y/
1/4
1
This definition is intuitively similar to what is defined for availability in controllable experiments, where we had r = /o Intuitively, smaller average distance w.r.t. attributes signify an easier spurious feature and therefore, higher availability. Only with Equation 5, we are able to compute availability for any tasks, and thus enables transferring the algorithm selector trained from CelebA tasks, to unseen tasks from other datasets (i.e. MetaShift (Liang & Zou, 2022), OfficeHome (Venkateswara et al., 2017), ColOTed-MNIST (Arjovsky et al., 2020) and MultiNLI (Williams et al., 2017))
D.2. Detailed Experimental Setup
We provide the detailed experimental setup for the vision experiments in Section 4.3 and NLP experiments in Section 4.4.
Dataset of datasets. In total, we create 720 tasks from CelebA (Liu et al., 2015), see Table 10 for the statistics for these datasets. Specifically, we generate the training sets of the datasets with the combinations of values in Table 10, and for the degrees of distribution shifts, we consider 2 cases: (i) 3 shifts (spurious correlation, covariate shift and label shift) all present each in different degrees, we uniformly sample 30 different triple degrees, and (ii) there is only shift present, in this case for each shift we consider 10 different values (therefore in total 30 different triple degrees). We generate their test with half the number of samples their training set for each dataset, but keep the number of samples the same for all groups (therefore balanced test sets with all d = 0.5). We randomly split the generated CelebA tasks into 4:1 training and test splits, and use the training part for the meta-datase D.
For evaluation, we first evaluate on the unseen tasks from CelebA. In addition, we create 129 datasets from MctaShift (Liang & Zou, 2022) (fcat, dog} and (indoor, outdoor) split) and 180 datasets from Colored-MNIST (Arjovsky et al., 2020) for evaluation, where we follow the same generation procedure. The generated number of tasks for MetaShift and Colored-MNIITT is smaller because: (i) These two datasets only have one attribute annotated (indoor/outdoor and colors, respectively),
can only consider I attribute-label pair and hence one availability of spurious features. (ii) Their number of samples per group for MetaShift is not as sufficient as CelebA, making some of the datasets infeasible (i.e. the Equation 4 as no solution for some dataset characteristics). We also create 100 tasks from OfficeHome for evaluation, from a slightly different procedure, see OfficeHome paragraph in Appendix G.
For NLP experiments, we follow a similar process and generated 720 tasks from CivilComments (Borkan et al., 2019) and generat 180 tasks from MultiNL (Williams et al., 2017).
Algorithm selector. We use 2-layer MLP (with hidden size 100) parameterize the algorithm selector, because we found shallower simpler model underfits while deeper model does not provide more improvements.
Tasks. To solve the tasks defined by each dataset in the dataset of datasets, we either do (1) linear probing on ResNet18 or CLIP (ViT-B/32) or (2) fine-tuning ResNet18. In the first case (all tables except for Table 11), we use Adam optimizer with default hyperparameters, along with L2 regularization. We train for 1000 epochs to ensure convergence. In the second case (Table 11), we use SubpopBench (Yang et al., 2023) and its default hyperparameters to fine-tune ResNet18. We use basic data augmentations (resize, crop, etc).
18

---

Is Algorithm Selection for OOD Generalization Learnable?
Table 10. Statistics for the dataset of datasets in vision and NIP experiments.
Statistics
Value
Size of training set Input dimensionality Availability
{200, 500, 1000) N/A
4 different attribute pairs, see Table 9
shifts
randomly sampled 30 feasible degrees (Figure 6) randomly sampled 10 degrees in (0,1) for each type of shifts
shift
Table 11. Evaluation of training paradigms for tasks. The algorithm selector generalizes well with both linear probing and fine-tuning
Linear probing
Fine-tuning
Methods
0-1 ACC.
WG error +
ACC. t
WG error
Oracle Selection
100
44.9 40.2
100
32.3 =0.4
Regression
53.6 #1.1
49.8 40.5
73.6 #1.8
34.5 =0.3
NOD-Chameleon
75.0 #1.3
47.7 4.02
67.8 41.3
34.8 #0.4
E. Ablation Studies
E.1. Different Training Paradigms
Here we conduct an ablation study of the training paradigms for the tasks. This necessary to check because, solving the tasks by different training paradigms affects the algorithms' OOD performance Pjm and therefore changes the distribution of meta-dataset D = (f(D)) Am, P-m}- In Table 11, verify that the algorithm selector works well with both linear probing and fine-tuning with CelebA tasks. The results indicate that the learned algorithm selector can also accurately select suitable algorithms when the models for the tasks become over- werrparametrized in the case of fine-tuning.
E.2. Algorithm Suitability Threshold
In Section 2.2, we convert the algorithm selection problem into multi-label classification task, because multiple algorithms can be suitable for given dataset/task. This involves defining a small threshold e. Specifically, an algorithm is considered suitable if (P,m min, Pym) S for a small threshold c. We have used 0.05 for all our experiments, and here we provide an ablation study on it, in Table 12. The method proves robust to other choices than the e = 0.05 used throughout the paper. Intuitively, when is too big, the meta-dataset loses discrininative power; too small, it subject to noise.
Table 12. Ablation study on Worst-group error is shown (lower is better).
CelebA (ResNet)
CelebA (CLIP)
etaShift (ResNet)
MetaShift (CLIP)
0.0
48.3
39.8
39.9
27.7
0.025
48.1
39.6
39.7
27.3
0.05
47.7
39.1
39.0
27.2
0.10
48.0
39.4
39.6
27.8
E.3. Algorithm Selection Strategies at Test Time
As discussed in Section 2.2, multiple algorithms can be predicted as suitable for dataset/task at test time, due to our multi-label classification formulation. When this is indeed the case, we always select the algorithm with the highest prediction logits, which corresponds to the one the algorithm selector is the most confident in. An alternative would be to use binary predictions, and if multiple algorithms are predicted to be suitable, randomly select one of them. Here we show that, empirically, the former option (top logits) much better, in Table 13.
19

---

Is Algorithm Selection for OOD Generalization Learnable?
Table 13. Comparison of test-time algorithm selection criteria.
Worst-group error is shown, lower is better.
Selection criterion
CelebA (ResNet)
CelebA (CLIP)
MetaShift (ResNet)
MetaShift (CLIP)
Top logits (used elsewhere)
47.7
39.1
39.0
27.2
Binary predictions
48.5
39.8
39.4
27.4
E.4. Performance Metrics
In our experiments, we mostly use the worst-group (WG) error as the performance metric. The rationale is that it is independent of the test distribution and can thus be addressed by considering only the distribution (imbalances) of training data. WG performance thus promotes general robustness to distribution shifts and has been used in many prior works.
However, our formulation is compatible with other performance metrics. The algorithm selection should work as long as there are useful relations be learned between the data descriptors, the algorithms and the performance metrics. Here investigate alternative metric, the averaged-grow group enror, in Table 14. The conclusions similar to those in Section 4 the algorithm selector can still select suitable algorithms with averaged-grow group error as the performance metric.
Table 14, Training and evaluating the algorithm selector with averaged-group error. Lower is better.
CelebA (ResNet)
CelebA (CLIP)
MetaShift (ResNet)
MetaShift (CLIP)
Oracle selection
33.8
26.4
24.6
14.8
Random selection
36.6
29.5
27.8
18.1
Global best
35.7
28.0
26.9
16.6
Naive descriptors
35.8
27.7
26.7
16.9
Regression
35.5
28.3
27.0
16.4
OOD-Chamelcon
35.1
27.2
26.3
15.9
F. Algorithm Selection with Estimated Dataset Descriptors
Here we study the scenarios where the information compute dataset descriptors, such the attribute for each sample (Liu al., 202 la), cannot be obtained test time. Recent works (Liu et al., 2021a; Kirichenko et al., 2022; Qiu et al., 2023; Lee et al., 2022; Pagliardini et al., 2022) for OOD generalization aim to eliminate the need for attribute annotation by either: (i) infer the attribute annotation and then use them to run algorithms that require attribute annotation, (ii) run ERM on the training set assuming that the ERM learns the spurious feature, and then build invariant classifier on top of the ERM classifier (e.g. model that disagrees with the ERM).
We leverage the above first line of research, i.e. inferring the attribute annotation and use them compute the dataset descriptors the target training Then, the attributes by clustering the embeddings from frozen backbones, following Sohoni et al. (2020); You et al. (2024), The intuition is that different attributes, such as cows in grass or desert, can be considered as 'subclasses' or 'hidden stratifications' and they are observed to be separable in the feature space of the deep models. Hence, for example we can infer which cows belong to which environment, by clustering on their embeddings. particular, the training samples are passed through the backbone of the task (i.e. R or CLIP our case), and get the embeddings. For each semantic class, we cluster the corresponding samples with K-means and assign different attributes to different clusters. This gives each sample its inferred attribute annotation. We can then these inferred attribute annotations compute dataset descriptors, where they are used to compute the degree of spurious correlation dur and covariate shift des. as well as the availability of spurious features r (see Appendix D.1 on how we compute the availability).
OOD-Chameleon to select the suitable algorithms. We infer
With the inferred dataset descriptors, we use the algorithm selector to predict suitable algorithms. In Tab 15, we show that the suitable algorithms are still predictable with estimated dataset descriptors. Interestingly, while having performance drops in most cases, using estimated dataset descriptors boosts the performance on MetaShift with ResNet18.
20

---

Is Algorithm Selection for OOD Generalization Learnable?
Table 15. OOD-Chameleon with inferred attributes (and dataset descriptors) CelchA and MetaShift. 0-1 Accuracy (higher is better) shown.
CelebA
MetaShift
Methods
Attribute
CLIP
CLIP (ViT-B/32)
ResNet18
ResNet18
(ViT-B/32)
Oracle Selection
N/A
100
100
100
100
Regression
53.6 #1.1
46,7 40.9
51.2 #0.3
49.2 #0.3
DhD-Chameleon
75.0 +1.3
78.5 40.8
80.6 #0.7
76.7 #0.5
Regression
52.1 #0.9
50.2 #1.2
53.7 #0.6
47.7 #1.0
OOD-Chameleon
73.2 40.6
72.0 41.1
83.4 #0.8
72.6 +0.5
G. Additional Results on Vision Tasks
Section 4.3, we see that when training the algorithm selector on tasks built from CelebA, the algorithm selector not only able to select suitable algorithms on unseen tasks of CelebA also on unseen tasks of MetaShift. This suggests the learned data/algorithm relation is robust and transferrable across data domains. Here we provide more experiments Colored-MNIST
CoLOred-MINIST.
train the same CelebA meta-dataset and evaluate the algorithm selector on CoISST tasks (Arjovsky et al., 2020). In Colored-MNIST dataset, there are images of 10 digits from 0-9 and the digits are divided into classes (i.e. 0-4 is class 0, 5-9 is class 1). In addition, the two classes of digits are in two different colors (e.g. red and green). When the colors of the digits correlate with the shapes of digits, spurious correlation occurs. We create 180 tasks from ColOred-MNIST, each task exhibits different magnitudes of spurious correlation (SC), label shift (I.S) and covaraite shifts (CS) and the size of datasets span (200, 500, 1000). In Table 16, we see that the algorithm selector proves be effective on ColoTed-MNIST as well.
further support, in particular,
Table 16. Results algorithm selection for unseen ColOTed-MNIST tasks. Algorithm selectors are trained the dataset generated from CelebA and evaluated unseen Colored-MNIST tasks. ResNet18 and CLIP (ViT-B/32) refer to the models used in the tasks.
Golored-MNIST
Methods
ResNet18
CLIP (VIT-B/32)
0-1 ACC. t
WG error 4
0-1 ACC. t
WG error 4
Oracle Selection
100
21.1 #0.3
100
13.3 #0.4
Random Selection
29.3 +1.1
28.1 10.3
39.3 10.5
19.0 10.3
Global Best
50.1 1.077
25.1 10.4
63.3 11.2
16.0 10.2
Regression
79.4 10.5
23.8 20.3
54.2 40.9
16.1 +0.4
DOD-Chameleon
82.7 10.8
23.5 #0.4
75.3 40.7
15.6 10.4
OfficeHome. We additionally verify our approach with OfficeHome (Venkateswara et al., 2017). In OfficeHome, the there exists 65 classes of objects (such as Mug, Pen, Spoon, Knife, and other objects typically found in Office and Home settings.) and each of the classes have samples in 4 differrnt domains: Artistic images, Clip Art, Product images and Real-World images. The samples of each domain and object pair are uneven.
We sampled tasks from OffceHome by randomly sampling pairs of domain and object (e.g. {pen, kni knife) fe) in (Art, C1 clipart}) }), and then collect the corresponding samples from OfficeHome. We created 100 tasks in total, and evaluate the trained algorithm selector from CelebA on these tasks, in Table 17. The results show that NoD-Chameleon is still able to select suitable algorithms, evidenced by the lowest worst- group error across the 100 tasks. This provides strong additional support for the utility of our approach to select suitable algorithms across datasets and diverse types of shifts.
21

---

Is Algorithm Selection for OOD Generalization Learnable?
Table Results on algorithm selection for unseen OfficeHome tasks (worst-group error shown, lower better). Algorithm selectors
are trained on the meta- dataset generated from CelebA and evaluated on unsenn OfficeHome tasks. ResNet18 and CLIP (VIT-B/32) refer the models used in tasks.
Office-Home (ResNet)
Office-Home (CLIP)
Oracle selection
(14.8)
(11.4)
Random selection
19.3
15.1
Global best
18.2
14.4
Naive descriptors
18.8
14.6
Regression
18.5
14.3
OOD-Chamelcon
17.9
13.5
H. Additional Results on NLP Tasks
In Figure 18, we show a comparison with single-algorite baselines, i.c. using the same algorithm for all tasks, on CivilComments and MultiNLI tasks. Similar to vision experiments (Table 3), our adaptive selection performs much better, confirming the premise that single algorithm solution to all OOD scenarios.
Table 18. Comparison with static algorithm selection.
Observations are similar Table
CivilComments
MutlNLI
Methods
BERT
BERT (Finetuned)
BERT
BERT (Finetuned)
Alg. Selection
WG erro 4
Alg. Selection
WG error
Alg. Selection
WG emoor
Alg. Selection
WG
Oracle Selection
53.7 #0.3
19.4 =0.4
55.9 +0.3
54.2 #0.0
ERM
78.2 20.4
28.2 =0.7
76.1 10.6
68.2 20.4
GroupDRO
10.3
24.0 =0.2
62.2 10-4
64.6 20.3
Logits adjustment
77.5
22.9
77.6 +0.5
64.0
Undersample
$7.7 10.5
21.6 =033
39.1 10.2
$7.2 20.3
Oversample
59.7 10.3
24.1 20.2
62.8 10.5
63.5 20.7
OOD-Chameleon
55.8 20.4
20.7 =0.2
$8.3 20.2
56.6 20.2
Additional Results on Efficiency Considerations
The main point of training algorithm selector is to amortize its cost in the long run. Lee. the selector is trained once to generalize to unseen datasets and shifts, as addressed in our experiments.
I.1. Comparison with Ensemble, A Baseline with Much Higher Cost
In the paper, compared baselines with similar costs to solve downstream task. We now additionally compare method with an additional baseline of uniformly ensembling multiple methods' predictions, whose computational requirement is multiple times higher than our method for each new downstream task. Our method performs significantly better than this much more expensive one, as shown in Table 19.
Table 19. Comparison between OOD- Oh-hmhhml and Ensemble.
CelebA (ResNet)
CelebA (CLIP)
MetaShift (ResNet)
MetaShifi (CLIP)
Uniform ensemble
50.8
42.6
41.1
29.6
OOD-Chameleon
47.7
39.1
39.0
27.2
I.2. Generalizing from Small Datasets to Larger Ones
One potential the algorithm selector training on datasets with smaller data sizes and then using it to select suitable algorithms on larger datasets test time, which could further amortize the cost. Here we provide preliminary experiments
22

---

Is Algorithm Selection for OOD Generalization Learnable?
on this front. The algorithm selector is now trained with datasets of size smaller than or equal 1k and used to select algorithms for datasets of size 2k and 3k. As shown in Table 20, that the algorithm selector still accurately predicts suitable algorithms.
Table 20. Performance on CelebA with varying data sizes.
Data size
CelebA (ResNet)
CelebA (CLIP)
up 1000 (i.e. results in Table 2) 2000
75.0
78.5
81.7
83.8
3000
80.4
79.3
This concurs with our hypothesis that, by training on a meta-dataset of datasets with a range of smaller sizes, the model can learn to generalize to larger datasets by identifying patterns of the algorithms' performance w.r.t data sizes, We already showed (in Figure 3-right) that the variability in dataset size during training was critical.
J. Additional Details on the Decision Trees
Here we include more analysis when the algorithm selector is implemented by decision trees, which is omitted in the main paper due to space constraints. In Figure 8 and Figure 9, we show the decision rules implemented by (depth-3) decision tree when directly training on the CelebA meta-dataset. Since the problem is a binary multi-label classification task, the 'value" part is a list of size (5, 2), and the rows correspond to 'ERM', "GroupDRO', 'OverSample', 'Logits adjustment', "UnddrSSmple', respectively. One point that is not shown in the figures is that the final predictions are achieved by selecting the algorithm with the largest logit/probability (as mentioned in Section 2.2) when several of them are favorable. From the important in different cases: when ResNet-18 is used to run the
two figures, we that different pieces of the descriptor algorithms, data size degree of label shifts di, and the availability r are important to inform the algorithm selection; while the degree of spurious correlation dsc. degree of label shifts di, and the availability are important in the case of CLIP.
Decision trees trained to mimic (the best pest-perffrming) MLP algorithm selectors. To understand what has been learned by the MLP- lla-mparaetrrized algorithm selector, we also investigate training a decision tree to directly mimic the function implemented by the MLP. In our experiments, gives a similar performance to the previous version of the decision tree, but has the advantage of understanding the decision rule as a standard multi-class classification (hence the decision rules are more understandable). Specifically, we obtain the predictions of the trained MLP (the predictions are one-hot since we pick the algorithm with the highest predicted logit/probability, mentioned in Section 2.2) the meta-datased and use them to train a decision tree. To make the trees more interpretable, we convert the degree of shift by Id,) 0.5| (where | is the absolute value) since by design the strength of shift is only identified by how far it is from 0.5 (balanced). In Figure 10 and Figure 11, we show the learned trees. We can see clearly how the different data characteristics determine the choice of the best algorithms. See Section 4.5 on more discussions on these two figures.
Overall, these decision rules of the tree can further help practitioners understand the applicability of the existing algorithms.
23

---

Is Algorithm Selection for OOD Generalization Learnable?
Sent Core
222 114
HEA FAR 2021
SEC
THE SET
BRINOS
BRIN CAD
JPT- EAS
CRIN
20 2333
23
12
72
12 241
20
Figure 8. Visualization of algorithm selector implemented as a decision tree, trained on the CelebA meta-dataset with ResNet18 models.
(12) 41.))
POT 2031
2070
DE 180 21
2ATS 3541
1
120
Figure Same as Figure 8, with CLIP (ViT-B/32) architectures.
24

---

Is Algorithm Selection for OOD Generalization Learnable?
d,
enroapy
dbac Undemomple
Tree
2x 5.441
15 750.
arreapy
stttopy
dlas BRM
class Undersample
4, 0..45 aniippy
# 350.0 anzppy
dy 2 0.22
du 0.355
crtmppy
calmyy
class HRM
class Undersomple
class Undemampte
class Undersample
cttragy 2205
entropy " 2,299
carogy 1.484
cntragy B.6
2.051
1.043
1.704
cilllopy 0.9
class IRM
class Uversample
class Legit Aquemmentt
class Underamgll
class - Augiis.Aiisttoo class - Lidersample
class GRogURO
class Undersamplo
Figure 10. Visualization of decision algorithm selector trained mimic the one-hot predictions (obtained by selecting the top logits) of the best-perforning (MLP) algorithm selector obtained from CelebA meta-dataset with ResNet18 models. Since it is trained on one-hot labels, this version of the tree thus has the advantage understanding the decision rule standard multi-class classification. To make the decision rule even more interpretable, we convert degrees of shift with (dg) 0.5| where - is the absolute value. This accounts for the fact that, by design, strength of shift only identified by how far from 0.5 (balanced). See Figure for simplified version of this tree.
2.322
cloo Lagis Adjrodeni
du 5..295
du 5.0.6
extippp 2.204
cellogyy 1.745
class ERM
class Cladessmple
1273 entmay
2.165
enaopp
1.862
Lugils Aljjaximmi
Ciddaaampl
GiuyDRO
сиtopр 2.185
estropy =n 2.26
L.50H
1.562
1.05
1.692
0.75%
1.87
cless ERM
Overssmppe
Undensmple
Oversample
class Undersample
Logits Adusmmen
Uidersmmle
GroupURO
Figure
Same as Figure 10, with CLIP (ViT-B/32) architectures.
25


Reviews:
[{"review_id": "P4pIejunQw", "reviewer": "Reviewer_sRjw", "summary": "This paper tackles the interesting problem of algorithm selection for OOD generalization. In particular, the paper proposes a meta-learning-like algorithm which takes in summary statistics from a dataset as well as a distributionally robust learning algorithm. This tuple is then mapped to the performance of training said algorithm on the dataset described by the summary statistics. For a new dataset, this mapping can be used to predict the likely performance of various distributionally robust learning algorithms without actually training these models. This allows for a flexible algorithm choice across a wide set of shifts and spurious correlations. The effectiveness of the method is demonstrated empirically on synthetic, image, and text datasets. Further, the paper provides general insights on the conditions under which the benchmarked distributionally robust learning algorithms perform well.\n\n### Update after rebuttal\n\nI thank the authors for their rebuttal. I think my concerns have been addressed sufficiently to warrant a weak accept. The connection to DRO still deserves further investigation, though, which is why I don't see myself in a position to champion the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful review. We are glad that you found the studied problem and insights interesting. We address the questions below and propose several improvements to the paper.\n\n---\n**`W1: No guarantee beyond the support of the training distribution:`**\n\nCorrect, this is true of any learning method. The subtle but important innovation is (1) to lift the challenge from OOD generalization across *specific* shifts, to a meta level of generalizing across **distributions of distribution shifts**. And (2) to show that we can automatically **densely sample** semi-synthetic datasets (to train an algorithm selector) **within** a distribution of realistic distribution shifts.\n\nThis is a conceptually novel approach to OOD generalization. We propose to state more clearly our assumptions and limitations to avoid any overclaiming.\n \nPlease also see the response to **Reviewer FvV1** (**Comment 1/2**) regarding theoretical claims.\n\n---\n**`W2: Variety of covariate shifts:`** Thanks for pointing this out, we clarified section 3. In summary, we simulate changes in $P(X)$ by intervening on the domain distribution. Covariate shifts can indeed happen in many ways, but this simple approach is both tractable and generates realistic shifts (e.g. with image data, introducing more from an *indoor* domain in one dataset vs. more from *outdoor* in another dataset).\n\nTo simulate different degrees of covariate shifts, we draw samples with different distributions of attribute or domain labels. The variability covers the full range with a ratio of samples from any domain in $[0,1]$ (thus including severe shifts). \n\nPlease also see the response to **Reviewer 9QSc** (**Q2**) for a general view of section 3.\n\n---\n**`W3: Techniques for estimating covariate/label shifts:`**\n\nWe added to the paper a discussion of the literature devoted to this (e.g. [1]). To clarify, the estimation method we use applies to both label and covariate shifts: all it does is to infer sample-wise domain labels then use them to compute the estimated degrees of shifts. \n\n[1] Unified View of Label Shift Estimation, Garg et al.\n\n---\n**`Q1 (arbitrary shifts):`** We will expand and clarify: if shifts were arbitrary, inductive reasoning and machine learning would not be possible [2,3] and the proposed approach also could not help. We only claim the possibility of training an algorithm selector that is effective on a distribution of distribution shifts, not on *any* possible shift.\n\n[2] The Lack of A Priori Distinctions between Learning Algorithms, Wolpert\n\n[3] Domain Adaptation under Target and Conditional Shift. Zhang et al.\n\n---\n**`Q2:`** Great suggestion - done.\n\n---\n**`Q3 (worst-group error):`** Agreed. We repeated our experiments and now additionally report the **averaged-group error**. The conclusions are similar to those in the paper.\n\n(The experimental setup follows Table 2 for all new results.)\n| |CelebA (ResNet)| CelebA (CLIP) | MetaShift (ResNet) | MetaShift (CLIP) |\n| - | - | - | - | - |\n|Oracle selection|(33.8)|(26.4)|(24.6)|(14.8)|\n|Random selection|36.6|29.5|27.8|18.1|\n|Global best|35.7|28.0|26.9|16.6|\n|Naive descriptors|35.8|27.7|26.7|16.9|\n|Regression|35.5|28.3|27.0|16.4|\n|OOD-Chameleon|**35.1**|**27.2**|**26.3**|**15.9**|\n\n---\n**`Q4 (spurious attributes):`** Our main experiments follow the setup from much of prior work on OOD generalization (domain generalization, spurious correlation, e.g. [4,5], and many more) which indeed assumes access to attribute labels. We also evaluate scenarios without such labels (Appendix E).\n\n[4] In Search of Lost Domain Generalization, Gulrajani et al.\n\n[5] Change is Hard: A Closer Look at Subpopulation Shift, Yang et al.\n\n---\n**`Q5 ($\\epsilon$):`** We performed additional experiments. The method proves quite robust to other choices than the $\\epsilon=0.05$ used in the paper (bold row). Intuitively, when $\\epsilon$ is too big, the meta-dataset loses discriminative power; too small, it is subject to noise. \n\n| $\\epsilon$ | CelebA (ResNet) | CelebA (CLIP) | MetaShift (ResNet) | MetaShift (CLIP)|\n| - | - | - | - | - |\n|0.0  | 48.3 | 39.8 | 39.9 | 27.7 |\n|0.025| 48.1 | 39.6 | 39.7 | 27.3 |\n|0.05 | **47.7**| **39.1**| **39.0**| **27.2**|\n|0.10 | 48.0 | 39.4 | 39.6 | 27.8 |\n\n---\n**`Q6 (comparison of pairs):`** This highlights the discrepancy in performance among methods under different shifts. Without this, there would be nothing to learn. E.g. the comparison of under-/oversampling (blue) shows that many datasets benefit specifically \n from one *or* the other.\n\n---\n**`Q7 (oracle selection):`** Yes.\n\n---\n**`Q8 ($\\mathcal{G}_i$):`** Number of samples in group $i$ (defined L114, illustrated in Figure 2). We will make it clearer.\n\n---\n**`Q9 (ensembling):`** It is indeed possible. We tested two ensemble methods in response to **Reviewer FcV1** (**Comment 3**).\n\n---\nWe incorporated the above clarifications and improvements into the manuscript. Let us know if there are remaining questions or weaknesses that we could address. Thanks!", "claims_and_evidence": "See next section.", "methods_and_evaluation_criteria": "In principle, the method makes sense and should work *under the assumption that the distribution of dataset descriptors is representative of descriptors encountered at deployment time*. I don't think that this assumption is explicitly acknowledged in the paper, though. While it seems like the empirical results show some degree of generalization across datasets and tasks, I am skeptical whether this will necessarily translate in general. Since the learning process described in 2.2 is standard empirical risk minimization and not a distributionally robust algorithm itself, there is no guarantee that this generalization pattern should necessarily hold in general beyond the support of the training distribution. I believe that the paper is over-claiming the generalization capabilities of the proposed method. Also, as discussed in my questions below, it would be worthwhile to report a broader notion of error beyond just the worst group. Evaluating distributional generalization methods solely using worst-group error risks overfitting to small or narrowly defined groups, causing instability and obscuring performance trade-offs with other critical groups.", "theoretical_claims": "The paper does not provide any theoretical claims. It would be great to gain some more theoretical insights from learning theory to understand under what conditions a generalizable algorithm predictor is learnable. Some connections to meta learning could probably be useful here.", "experimental_designs_or_analyses": "In general, I think that the experimental design makes sense and is similar to other works in the meta learning community. My biggest concern here is the lack of clarity in Section 3 (see comments below in strengths and weaknesses).", "relation_to_broader_scientific_literature": "The related work section sufficiently discusses the role this work plays in the context of OOD generalization, algorithm selection, and meta learning.", "essential_references_not_discussed": "Prior work appears to be sufficiently discussed.", "other_strengths_and_weaknesses": "**Strengths**: The question posed by the paper---is it possible to select a distributionally robust learning method based on properties of a dataset---is interesting and relevant to the reliability community. The paper also includes experiments across data modalities (synthetic, image, language). I also find the insights from section 4.5 on the task <---> method fit interesting.\n\n**Weaknesses**: I think Section 3 is currently not very clear. In particular, it is unclear to me how the covariate shifts are generated and what degree of variability can be achieved. While label shifts and spurious correlations are a bit more limited in terms of their simulation flexibility, covariate shifts can happen in many different forms. I don't find this sufficiently discussed in the current draft which makes me question the method's effectiveness in the downstream experimental results. As noted by the authors themselves, the fact that the dataset descriptor is designed with engineered features constitutes a limitation that is likely to affect the proposed method. In particular, key dataset properties are often not directly observable but need to be estimated from the data. While the paper talks about estimating spurious correlations, it does not seem to discuss techniques for estimating the presence and intensity of covariate and label shifts. The effectiveness of the method could be strongly impacted by the quality of this estimation.", "other_comments_or_suggestions": "- The color bars in the tables are a great way to show the algorithm allocation. Big fan of this visualization!\n- Section 3: Should Figure 3 be Figure 2?\n- Lines 156 (right column): I assume you use *binary* cross entropy here? This would be worth making explicit to avoid confusion.", "questions_for_authors": "1) Lines 91-92 (right column): \"But distribution shifts appearing in real-world data are not arbitrary.\" I am not sure what is meant here by \"arbitrary\". Can you clarify?\n2) Line 98 (right column): Can you introduce some foreshadowing examples here already on the dataset characteristics?\n3) Lines 113-116: I agree that worst group error is a key metric to consider but would it be possible to report a broader notion of error beyond just the worst group? Evaluating distributional generalization methods solely using worst-group error risks overfitting to small or narrowly defined groups, causing instability and obscuring performance trade-offs with other critical groups.\n4) Lines 123-124: I am not sure this is true. In most practical applications where shifts occur (and where OOD generalization is therefore important), directly observing labels for spurious attributes is prohibitive. Can you clarify this statement?\n5) What is the method's sensitivity to the suitability hyperparameter $\\epsilon$?\n6) Why do you compare pairs of methods in Figure 3 (left)? I am not sure what this figure is telling me.\n7) Is oracle selection simply done by running all algorithms on all tasks and then picking the best model across all tries?\n8) Line 216: $\\mathcal{G}_i$ has never been introduced before. What is this?\n9) Can you comment on the applicability of ensembling to the problem you raise? Would it be possible to train all methods and then use a weighted average to generate predictions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "69jSHVMPBw", "reviewer": "Reviewer_sPTX", "summary": "This paper introduces an interesting task: choosing the right training algorithm for the right dataset. The authors propose an approach called OOD-CHAMELEON, which predicts whether a given algorithm can generalize to unseen OOD data based on its past performance on the current dataset. I find this work sufficiently novel and engaging, as adapting existing algorithms to different OOD scenarios may be more practical than designing entirely new ones. However, some aspects of the paper lack clarity in detail. If the authors can address my concerns, I would be willing to reconsider my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful feedback. We are glad that you found our work novel and well presented.\n\n---\n\n**`Q1 (algorithm selection at test time):`** We simply pick the method with the *highest logit* (L159 right column), which corresponds to the one the model is the most confident in.\n\nAn alternative would be to use binary predictions, and if multiple algorithms are predicted to be suitable, randomly select one of them. Empirically, the former option (top logits) is much better. See below for new experiments comparing these options (worst-group error is shown, lower is better, and experimental setup is the same as Table 2 in the paper).\n\n| Selection criterion | CelebA (ResNet) | CelebA (CLIP) | MetaShift (ResNet) | MetaShift (CLIP) |\n| -- | -- | -- | -- | -- |\n| Top logits (as in the paper)         | 47.7 | 39.1 | 39.0 | 27.2 |\n| Binary predictions (new experiments) | 48.5 | 39.8 | 39.4 | 27.4 |\n\n---\n**`Q2 (loss function):`** Yes it uses the **binary** cross-entropy, we will make this clearer.\n\n---\n**`Q3 (dataset descriptor):`** Each component in the vector is either a real number (e.g. the degree of distribution shifts or the availability of spurious feature) or an integer (e.g. size of the dataset). We will make the definition clearer. \n\n---\n\n**`Q4:`** The answer is subtle but absolutely critical to understanding the value of our approach.\nThe key innovation is (1) to lift the challenge from OOD generalization across specific shifts, to a meta level of generalizing across **distributions of distribution shifts**. And (2) to show that we can automatically sample semi-synthetic datasets (to train an algorithm selector) after choosing a realistic distribution of distribution shifts.\n\nThis is conceptually *a completely novel approach to OOD generalization*. While it is true (*for any learning method!*) that there is no guarantee of generalizing beyond the support of the training distribution, the training distribution in our approach encompasses a wide variety of shift types, magnitudes and their combinations, which can be densely sampled (point (2) above). We acknowledge that new shift types might require additional curation (we only claim that our approach should work a distribution of distribution shifts, as noted in section 2.1). We propose to state more clearly our assumptions and limitations to avoid any overclaiming.\n\n**OOD generalization on data sizes**: However, we hypothesize that it is possible to generalize to OOD data size -- the algorithm selector has learned to generalize to larger data sizes, potentially by exploring the different data sizes available. In our response to **Reviewer 9QSc** (**W1**), we show that this is indeed the case. This is a case where the algorithm selector generalizes to OOD data.\n\n---\nLet us know if there are remaining questions or weaknesses that we could address. Thanks again!", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "no theory in this paper", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "They try to connect algorithm selection and OOD generalization", "essential_references_not_discussed": "yes", "other_strengths_and_weaknesses": "Strengths :\n\n1. interesting topic, and I think utilize Algorithm Selection to solve OOD problem is novel.\n2. This paper is well-organized and easy to read.\n3. Complete experimental instructions\n\nWeaknesses:\n\nsee Questions", "other_comments_or_suggestions": "see Questions", "questions_for_authors": "1. If the method follows a multi-label classification approach as in Equation (3), how is the most effective algorithm selected during the testing phase? While I understand the authors' claim that \"multiple algorithms can sometimes be equally suitable,\" this reasoning does not seem applicable to the testing stage.\n\n2. Given that the task is multi-label classification, why does Equation (3) use cross-entropy (CE) instead of binary cross-entropy (BCE)?\n\n3. Is the dataset descriptor represented as an indicator vector? Although the authors provide a computational method in the appendix, I still find its definition unclear.\n\n4. The authors claim that OOD-CHAMELEON is a data-driven performance prediction model, which intuitively means leveraging past performance to infer future performance. While I accept this assumption when the datasets used for performance evaluation remain unchanged, how can this assumption still hold in an OOD setting? I am concerned about cases where an algorithm performs poorly on the in-distribution (ID) dataset but is actually well-suited for handling unseen OOD data.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Nqe4U0p3b", "reviewer": "Reviewer_9QSc", "summary": "This paper presents _OOD-Chameleon_, a principled and bottom-up approach for selecting training algorithms in out-of-distribution (OOD) generalization tasks. By framing algorithm selection as a multi-label classification problem, the method leverages a meta-dataset of datasets representing diverse distribution shifts to select algorithms based on test data descriptors. The key contribution is enabling zero-shot algorithm selection by leveraging related tasks/datasets, eliminating the need for grid-searching over algorithmic choices. The method is evaluated across synthetic, vision, and language tasks, demonstrating strong performance in identifying high-performing algorithms for unseen datasets.\n\n**Strengths**\n- **Addresses a timely problem:** No single algorithm consistently outperforms empirical risk minimization (ERM) across diverse OOD settings, making dataset-aware and automated algorithm selection valuable.\n- **Strong empirical evaluation:** The model achieves near-oracle performance across multiple domains (synthetic, vision, NLP) and generalizes to new datasets, outperforming multiple representative baselines.\n- **Interesting analysis:** The learned selector captures non-trivial dataset-algorithm relationships and provides analysis on when different interventions are effective. \n- **Clean and simple framework:** The multi-label classification approach is intuitive and extensible, supported by thorough ablation studies over model class (linear, k-NN, decision trees) and cost function (multi-label vs. regression).\n\n**Weaknesses**\n- **High upfront cost:** Constructing the meta-dataset requires significant computational resources, as it involves training models across many datasets and configurations. It is unclear how well this would scale to larger-scale settings, or whether a predictor trained on a metadataset of small datasets would generalize to a larger dataset at test time\n- **Generalization to unseen shifts:** This (supervised learning) method should perform well when test datasets resemble those in the meta-dataset, but its ability to generalize under significant distribution shifts is unclear. Careful curation of the meta-dataset and dataset descriptors feature engineering is required, which in turn requires some domain knowledge about the test datasets. The paper should explicitly discuss this.\n- **Lack of evaluation on larger-scale algorithmic selection tasks:** While the framework is general, experiments primarily focus on semi-synthetic datasets (e.g., CelebA, MultiNLI) and worst-group accuracy tasks. Evaluating the method on large-scale OOD problems—such as language model pretraining or ImageNet-scale classification—would better demonstrate its broader applicability. Even analyzing how basic hyperparameters (e.g., optimizer settings, scheduler choices, vocabulary size) and pretraining dataset composition affect final model performance would be interesting.\n- **No compute-matched baseline:** The study lacks a direct comparison against random grid search using equivalent compute resources (i.e., the cost of constructing the meta-dataset). Additionally, potential synergies with search-based approaches are unexplored—for instance, using OOD-Chameleon’s predicted algorithm selection to warm-start a grid search over the test dataset.\n\n**Questions & Suggestions:**\n- How does the method handle algorithms that require different levels of metadata? Could this bias the predictor toward metadata-rich approaches (e.g., GroupDRO vs. ERM)?\n- The section on quantifying distribution shifts (Section 3) is difficult to follow. The heuristics for measuring shifts should be explained more clearly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback. We are glad that you found our work timely, valuable, and appreciated its clarity and insights.\n\n---\n\n**`W1 (Would a predictor trained on a meta-dataset of small datasets generalize?):`** We performed additional experiments, reported below. The algorithm selector is now trained with datasets of size <= 1k and used to select algorithms for datasets of size 2k and 3k. We see that the algorithm selector still accurately predicts suitable algorithms.\n\nThis concurs with our hypothesis that, by training on meta-dataset of datasets with *a range of smaller sizes*, the model can learn to generalize to larger datasets by identifying patterns of the algorithms' performance w.r.t data sizes. We already showed (Figure 3-right) that the variability in dataset size during training was critical.\n\n| Data size | CelebA (ResNet) | CelebA (CLIP) |\n| -- | -| -|\n| up to 1000 (i.e. Table 2) |  75.0    | 78.5     |\n| 2000     | 81.7     | 83.8     |\n| 3000     | 80.4     | 79.3     |\n\n(Numbers are algorithm-selection accuracy; higher is better)\n\n---\n\n**`W2 (Generalization to unseen shifts):`** Because of space limits, we refer to the response to **Reviewer sPTX** (**Q4**). We are adding an extended discussion of this important point to the paper.\n\n---\n\n**`W3 (large-scale evaluation):`** Agreed, all of these points are important considerations that deserve an entire dedicated publication. Automating OOD algorithm selection is a completely new take on OOD generalization, and the point of this study (as stated in the title) is to investigate the *feasibility* of such a radically different approach.\n\nWe concur that these are all very exciting extensions, including automating the extraction of useful dataset characteristics, and we are actively working on it.\n\n---\n\n**`W4 (compute-matched baseline):`** It is unclear what would be the amount of compute to match: we clarified in the paper that the whole point of training an algorithm selector is **to amortize its cost in the long run**. I.e. the selector is trained once to generalize to unseen datasets and shifts, as addressed in our experiments.\n\n**Comparison with ensemble, a baseline with much higher cost.** In the paper, we compared baselines with similar costs to solve a downstream task. To address the proposed suggestion, we now additionally compare our method with an additional baseline of ensembling multiple methods predictions, whose compute requirement is multiple times higher than our method for each new downstream task. For space reasons, these new results are reported in the response to **Reviewer FvV1** (**Comment 3**). Our method performs significantly better than this much more expensive one.\n\nRegarding your comment on 'synergies', we additionally explored the combination of our method with ensemble-based methods -- using the predicted algorithm selection to adjust ensemble weights. We found that this strategy further improves in certain cases, albeit with a higher cost. Please again refer to the response to **Reviewer FvV1** (**Comment 3**) for more details.\n\n---\n\n**`Q1 (different levels of meta-data):`** The results indiate that the predictor does **not** have such as bias. For example, in Figure 3-left, the comparison between ERM/Undersample (orange) shows that ERM outperforms Undersample in many cases even though the latter uses meta-data (i.e. spurious attribute labels). Prior work [1,2,3] already pointed out that ERM can outperform complex methods that use rich meta-data, which is part of the underlying motivation for our work.\n\n[1] OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization, Ye et al.\n\n[2] In Search of Lost Domain Generalization, Gulrajani et al.\n\n[3] MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts, Liang et al.\n\n---\n\n**`Q2 (measuring distribution shifts):`** We reworked this section, which should now be much clearer. In summary, for each sample in the training data of a dataset, we have class label and attribute (or domain) label annotated. Then:\n- The bias of class label distribution indicates the degree of label shifts.\n- The bias of attribute (or domain) label distribution indicates the degree of covariate shifts.\n- The correlation of class labels and attribute (or domain) labels indicates the degree of spurious correlation.\n- Why is it sufficient to use only training data to measure the distribution shift? Because we use an unbiased test set, then how biased a distribution is (to an unbiased one) directly reflects the degree of shifts.\n\n---\n\nLet us know if there are remaining questions or weaknesses that we could address. Thanks again!", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "A new dataset-aware zero-shot method for algorithm selection", "essential_references_not_discussed": "None that I know of.", "other_strengths_and_weaknesses": "Please see my review", "other_comments_or_suggestions": "Please see my review", "questions_for_authors": "Please see my review", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lETcTwRo4D", "reviewer": "Reviewer_FvV1", "summary": "The paper introduces OOD-Chameleon, a framework for automatically selecting training algorithms to improve out-of-distribution (OOD) generalization. Instead of trial-and-error, it learns to predict the best algorithm based on dataset characteristics. The authors create a meta-dataset by re-sampling datasets with various distribution shifts and train a model to classify which algorithms (e.g., ERM, GroupDRO, oversampling) perform best. Experiments in vision and language tasks show that OOD-Chameleon outperforms static algorithm selection and learns interpretable decision rules, demonstrating that algorithm selection for OOD generalization is learnable and effective.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the thoughtful review. We are happy that you found our work novel and the analysis insightful.\n\n---\n\n**`Comment 1 (Theoretical analysis to explain the success of OOD-Chameleon):`** The bottom line is that our meta-learning-like approach turns algorithm selection into a *supervised* problem where, on the theoretical side, classical results from statistical learning theory apply. In particular, we use either an MLP or a decision tree to parametrize the algorithm selector. The approximated VC dimensions of these classes are well defined and can be used for bounding generalization errors of the selector. \n\n---\n\n**`Comment 2 (Theoretical analysis from an OOD framework):`** This is a valid question. Identifying generalization bounds when multiple types of shifts co-exist is an open problem. Most theoretical work focuses on a given type of distribution shift [1,2,3]. **The difficulty of theoretically handling complex mixed types of shifts is a core motivation** for our empirical approach (see L40, L66, L185). A theoretical treatment would be highly valuable but besides the point of this paper and deserves a full separate work.\n\n[1] Understanding the Failure Modes of Out-of-Distribution Generalization. Nagarajan et al.  \n\n[2] A Unified View of Label Shift Estimation. Garg et al.  \n\n[3] A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization. Xu et al.  \n\n---\n\n**`Comment 3 (additional algorithms/baselines):`** Our approach is **not** tied to specific algorithms. The ones used in the current implementation are justified L245-255: (1) they belong to different categories, (2) can address different types of shifts, (3) were proven to be effective in seminal works. Most importantly, no method is **strictly stronger** than these [4,5], and we show that *better algorithm selection with these 'simple' methods is sufficient to improve OOD generalization*.\n\n[4] Simple data balancing achieves competitive worst-group-accuracy, Idrissi et al.\n\n[5] Change is Hard: A Closer Look at Subpopulation Shift, Yang et al.\n\n**Algorithm selection baselines:** There are no existing competing methods; our evaluation includes various single-algorithm selection baselines. We propose an additional experiment with a new strong **ensemble** baseline: every algorithm is used to train a model whose predictions are ensembled (**row 1**). Our approach (**row 2**) significantly outperforms this ensemble. The ensemble also requires training and evaluating as many models as algorithms vs. only one for our method.\n\n\n| (Worst-group error, lower is better) |CelebA (ResNet)|CelebA (CLIP)| MetaShift (ResNet) | MetaShift (CLIP) |\n|-|-|-|-|-|\n|Uniform ensemble (new)|50.8|42.6|41.1|29.6|\n|OOD-Chameleon (from the paper)|**47.7**|39.1|**39.0**|27.2|\n|+ Weighted ensemble (new)|48.3|**38.9**|39.2|**26.8**|\n\nAs suggested by **Reviewer sRjw**, we also investigated combining OOD-Chameleon with an ensemble (**row 3**): predictions are ensembled after weighting by the predicted score of the algorithm. This further improves OOD-Chameleon in certain cases. It confirms the quality of the predicted suitability of algorithms, but removes the benefit of only having to train a single model.\n\n---\n**`Comment 4 (meta-dataset diversity):`** This is true for any supervised model where the training data coverage will inevitably affect the performance (also similarly, the pretraining data coverage for a pre-trained model affects its capability). So we rather believe this is a feature instead of a limitation, since this allows for building a more powerful predictor with meta-dataset scaled up. This is confirmed in Figure 3-middle. Another point to note in that figure is that, only a *relatively small size* of meta-dataset (~200) is sufficient to outperform the best non-parametrized baseline by a large margin.\n\n---\n**`Q1`**: please see the response to **Reviewer sPTX** (**Q4**). \n\n---\n**`Q2 (Office-Home):`** We performed additional experiments as requested with Office-Home. We created 100 datasets by randomly sampling a pair of domains and a pair of classes for each dataset (e.g. classifying pen/knife in Art/Clipart). The number of samples is naturally imbalanced, creating mixed types of distribution shifts by design. We use the algorithm selector from the paper trained with the \"CelebA\" meta-dataset, which we use to predict suitable algorithms for these variants of Office-Home. The (worst group error) results are below. They provide strong additional support for the utility of our approach to select suitable algorithms across datasets and diverse types of shifts. \n\n\n|| Office-Home (ResNet) | Office-Home (CLIP) |\n| - | - | - |\n| Oracle selection | (14.8) | (11.4) |\n| Random selection | 19.3 | 15.1 |\n| Global best | 18.2 | 14.4 |\n| Naive descriptors | 18.8 | 14.6 |\n| Regression | 18.5 | 14.3 |\n| OOD-Chameleon | **17.9** | **13.5** |\n---\nLet us know if there are remaining questions or weaknesses that could be addressed. Thanks!", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. This study proposes a data-driven algorithm selection method, rather than relying on traditional trial-and-error strategies. The idea of automatically selecting training algorithms is novel in the field of OOD generalization.\n2. The paper provides some interpretability through decision tree analysis, examining how different algorithms perform under various dataset characteristics, which enhances the credibility of the study.\n\nWeaknesses:\nSee comments.", "other_comments_or_suggestions": "Comments:\n1. Lack of Theoretical Guarantees – The paper primarily relies on empirical experiments but lacks a clear theoretical analysis to explain why OOD-Chameleon can successfully predict the optimal algorithm.\n2. Although decision tree analysis provides some insights, the study does not deeply discuss the problem from the perspective of generalization bounds or an OOD theoretical framework.\n3. The paper compares simple baselines (ERM, GroupDRO, Oversampling), but it may lack stronger relevant methods as additional baselines.\n4. The method relies on a meta-dataset, but if the meta-dataset lacks diversity, the performance may degrade on new tasks.", "questions_for_authors": "Questions:\n1. If the shift types in the training data and test data are completely different, can the selector still work effectively?\n2. Test the performance of this method in more complex OOD scenarios, such as the Office-Home dataset.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the work merits acceptance. The paper addresses the timely issue of selecting OOD generalization methods where there is not a consistent method across different types of shifts, and shifts may be unknown in practice. Overall, the reviewers found their proposed meta-learning method using dataset properties to predict which OOD generalization will most likely result in good OOD performance. Ironically, there are some concerns about the OOD generalizability of the method and suggestions that perhaps some OOD generalization method should be applied to the OOD method selection problem, this concern does not outweigh the strenghts of the paper. Consequently, I recommend accepting the paper.

*Strengths.* The question posed by the paper is interesting and relevant to the reliability community. The experiments span synthetics, image, and language datasets. The paper provides some insights into what the meta-learner learns, which is, in turn, insightful about the connection between dataset properties and OOD generalization methods. The reviewers also found the paper to be well-organized, easy to read, and clear. The reviewers also found the simple, intuitive, and extensible nature of the framework to be a notable strength.

*Weaknesses.* The selector learned may itself suffer from OOD-generalization. While the paper is empirical, there is room for more (informal) connections between the dataset properties and the theoretical underpinnings of the method. The paper only evaluates a limited number of OOD generalization algorithms. The inclusion of more state-of-the-art algorithms would substantially improve the paper. Specifically, another limitation of the paper is that it does not deal with mixtures of shift types. The algorithms to select from are also closely tied to one shift type (e.g., covariate shift and label shift). Perhaps the selection is much more complicated and less generalizable when more sophisticated algorithms that may address many shift types are included.

The author provided additional experiments, albeit limited, to show that learning on smaller-scale datasets still generalizes to larger-scale datasets---though only within the same orders of magnitude. They also provided additional error metrics and other empirical results requested by the reviewers. Their clarification of their contributions was generally convincing to the reviewers.

The strengths of this paper outweigh its weaknesses; this provides a potentially useful strategy for addressing the distribution shift. Therefore, I recommend accepting the paper.
Decision: Accept (poster)