ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets
Authors: Damien Teney
LIN Yong
Seong Joon Oh
Ehsan Abbasnejad
Venue: Submitted to ICLR 2023 (2023)
Keywords: OOD generalization
underspecification

Abstract: Several studies have empirically compared in-distribution (ID) and out-of-distribution (OOD) performance of various models. They report frequent positive correlations on benchmarks in computer vision and NLP. Surprisingly, they never observe inverse correlations suggesting necessary trade-offs. This matters to determine whether ID performance can serve as a proxy for OOD generalization. This paper shows that inverse correlations between ID and OOD performance do happen in real-world benchmarks. They could be missed in past studies because of a biased selection of models. We show an example on the WILDS-Camelyon17 dataset, using models from multiple training epochs and random seeds. Our observations are particularly striking with models trained with a regularizer that diversifies the solutions to the ERM objective. We nuance recommendations and conclusions made in past studies. (1) High OOD performance may sometimes require trading off ID performance.(2) Focusing on ID performance alone may not lead to optimal OOD performance: it can lead to diminishing and eventually negative returns in OOD performance. (3) Our example reminds that empirical studies only chart regimes achievable with existing methods: care is warranted in deriving prescriptive recommendations.

Under review as a conference paper at ICLR 2023
ID AND OOD PERFORMANCE ARE SOMETIMES INVERSELY CORRELATED ON REAL-WORLD DATASETS
Anonymous authors Paper under double-bling review
ABSTRACT
Context. Several studies have empirically compared in-distribution (ID) and out- of-distribution (OOD) performance of various models. They report frequent pos- itive correlations on benchmarks in computer vision and NLP. Surprisingly, they never observe inverse correlations suggesting necessary trade-offs. This matters determine whether ID performance can serve as a proxy for OOD generalization.
Findings. This paper shows that inverse correlations between ID and OOD perfor- mance do happen in real-world benchmarks. They could be missed in past studies because of biased selection of models. We show example the WILDS- Camelyon17 dataset, using models from multiple training epochs and random seeds. Our observations are particularly striking with models trained with regu- larizer that diversifies the solutions to the ERM objective (Teney et al. 2022a).
Implications. We nuance recommendations and conclusions made in past studies.
High OOD performance may sometimes require trading off ID performance.
Focusing on ID) performance alone may not lead to optimal OOD performance: can lead diminishing and eventually negative returns in OOD performance.
Our example reminds that empirical studies only chart regimes achievable with existing methods: care is warranted in deriving prescriptive recommendations.
INTRODUCTION
Past observations. This paper complements existing studies that empirically compare in- distribution (ID) and out-of-distributtor (OOD) performance of deep learning models (Andreassen et al. 2021 Djolonga et al., 2021 Miller et al. 2021 Mania Sra 2020 Miller et al., 2020 Taori al.
2020: Wenzel al. 2002).. It has long been known that models applied OOD data suffer a drop in performance, e.g. in classification accuracy. The above studies show that, despite this gap. ID and OOD performance are often positively correlated perrelateddJcroos.. across models on benchmarks in computer vision (Miller et al.) 202I) and NLP (Miller et al. 2020).
Past explanations. Frequent positive correlations are surprising because nothing forbids opposite, inverse ones. Indeed, ID and OOD data contain different associations between labels and features. One could imagine e.g. that an image background is associated with class C₁ ID and class C2 OOD. The more model relies on the presence of this background, the better its ID performance but the worse its OOD performance, resulting in an inverse correlation. Never observing inverse correlations has been explained with the possibility that real-world benchmarks might contain
We use "OOD" to refer to test data conforming to covariate shifts (Shimodaira 2000) w.r.t. training data. We use "correlation" refer both linear and non-lincar relationships.

---

performance Under review as a conference paper at ICLR 2023
Toy representations, one point = one model Inverse correlation
Figure 1: Past studies suggest that positive correlations be- tween ID/ / OOD performance are ubiquitous. This paper shows, with a counterexample, that inverse correlations are possible and can be accidentally overlooked. The possible need for an ID/OOD trade-off thus not merely theoretical and should be envisioned, e.g. preventing blind reliance on ID performance for model selection.
Positive correlation
only mild distribution shifts (Mania & Sra 2020). We will show that such observations can also be an artefact of study design.
A recent large-scale study. Wenzel al. 2022) show that not all datasets display clear positive correlation. The authors observe other patterns that sometimes reveal underspecification (D Amoor al. 2020; Teney al., 2022b; Lee et al. 2022). or severe shifts that prevent any training / test
transfer. Surprisingly, they never observe inverse correlations:
betwen accuracy and robustness, where more accurate "We did not observe any trade-off between models would overfit to spurious features that do not generalize." (Wenzel et al., 2022)
On the contrary, we do observe such cases and showcase it on dataset from the above study.
Explaining inverse correlations. We name the underlying cause a misspecification, by extension of underspeciffcation which was previously used to explain why models with similar performance can vary in OOD performance (D' Amour et al., 2020; Teney et al. 2022b Lee et al., 2022). In cases of misspecification, the standard ERM objective (empinical risk minimization), which drives ID performance, conflicts with the goal of OOD performance. ID and OOD metrics can then vary independently and inversely to one another. In Section [ we present a minimal theoretical example that illustrate how an inverse correlation pattern originates from the presence of both robust and spurious features in the data. Section 6 we show that different patterns of occur with different magnitudes of distribution shifts.
Summary of contributions.
An empirical examination of ID vs. OOD performance on the WILD-Camelyone dataset (Koh ct al., 2021) that shows an inverse correlation pattern conflicting with past evidence (Section 3)
An explanation and empirical verification that past studies could miss such patterns because of biased sampling of models (Section
A theoretical analysis showing when inverse correlations patterns can occur (Sections 5
A revision of conclusions and recommendations made in past studies (Section
PREVIOUSLY-OBSENVE PATTERNS OF ID vs. OOD PERFORMANCE
Past studies conclude that ID and OOD performance tend to vary jointly across models on many real-world datasets (Djolonga et al. 2021 Miller al. 2021: Taori al., 2020). Millet al. report an almost-systematicc linear correlation between probit-scaled ID and OOD accuracies. Mania & Sra 2020) explain this trend with the fact that real-world benchmarks contain only mild distribution shifts Andreassen et al. 2) find that pretrained models perform "above the linear trend" in the early stages of fine-tuning. Their OOD accuracy rises more quickly than their accuracy early on, even though the final accuracies agree with linear trend.
Most recently, the large-scale study of Wenzel al. (2022) is more nuanced: they observe linear trend only on some datasets. Their setup consists in fine-tuning an ImageNet-pretrainee model on chosen dataset and evaluating it on matching ID and OOD test sets. They repeat the procedure with variety of datasets, architectures, and implementation options such data augmentations.
The scatter plots of OOD accuracy in Wenzel et al. (202)) show four typical patterns (Figure
"The "linear trend" is really linear: applies to probit-scaled accuracies non-linear transform). Mania & Sra (2020) explain the linear trend with (1) certain data points having similar probabilities of occur- ring in ID and OOD data, and (2) the probability being low that model classifies some points correctly that higher-accuracy model classifies incorrecty..
2

---

Under review as a conference paper at ICLR 2023
0.8
INCREASING LINE
VERTICAL LINE
HORIZONTAL LINE (Comain iniggrppoow-uuiiie
NO CLEAR TREAD
(Domanliet cipattwa-deetch)
(PACS kertch-oe-phttt
(Terrninsagnits Laot-
0.4
000
0.0
0.00
0.25
0.50
0.75
1.00
0.30
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
acruanyy
borurary
sccurity
accuney
Figure 2: Typical patterns observed in Wenzel et al. (2022) (reproduced with permission).
1. Increasing line (positive correlation): mild distribution shift. ID and OOD accuracies are positively correlated. Focusing classical generalization and ID performance brings concurrent OOD improvements.
2. Vertical line: underspecification (D' AAmour et al. 2020 Lee ct al. 2022 Teney et al. 2022b)
Different models obtain a similar high ID performance but different OOD performance. The objective of high IDD performance does not sufficiently constrains the learning. Typically, multiple features in the data k. biased or spurious features) can be used to obtain high ID performance, but all of them are equally reliable OOD data. To improve OOD performance, additional task-specific information is necessary, e.g. additional supervision or inductive biases (custom architectures, regularizers, etc.).
3. Horizontal line, low OOD accuracy: severe distribution shift. No model performs well OOD. severe shift prevents any transfer between training and OOD test data. The task needs to be significantly more constrained e.g. with task-specific inductive biases.
No clear trend: underspecification. Models show a variety of ID and OOD accuracies. The difference with (2) is the wider variety along the ID axis, e.g. because a difficult learning task ERM objective.
yields solutions of lower ID accuracy from local minima
The authors note the absence of decreasing patterns, which are however possible in theory.
Decreasing line (inverse correlation): misspecification. The highest accuracy ID and OOD are achieved by different models. Optima of the ERM objective, which are expected to be optima in ID performance, do not correspond to optima in OOD performance. This implies a trade-off: higher OOD performance possible at the cost of lower ID) performance.
When does an inverse correlation occur between ID and OOD performance?
Intuitively, it can occur when there a pattern in the data that predictive in one distribution and misleading in the other. For example, object classes Ci and C2 are respectively associated with image backgrounds B₁ and B, in ID data, and respectively B, in B2 (swapped) in OOD data. Relying on the background can improve performance either distribution but not both simulta- neously. While such severe shifts might be rare, the next section presents an actual example.
NEW OBSERVATIONS: INVERSELY CORRELATED ID OOD PERFORMANCE
We use the WILDS- DILL---Cmllyyn 7 dataset described by Koh et al. 2022 in a manner similar to Wen- zel et al. 2022). These authors evaluate different architectures, assuming different inductive biases will produce models covering a range ID OOD accuracies. For simplicity, we rely instead on different random seeds since Amourr al. 2020) showed this be sufficient to cover variety of ID / /OOD accuracies on this dataset. To increase this variety even further without manually picking architectures, also train models with the diversity-inducing method of Teney et al. (2022a).
Experimental details. We use DenseNet-121 models pretrained by the authors of the dataset using 10 different seeds. For each of these 10 models, re-train the last linear layer from random initialization for to 10 epochs. while keeping the other layers frozen. These are referred to as "ERM models". We perform this re-training with 10 different seeds which gives 10x ERM models (10 pretraining seeds x 10 re-training seeds x 10 numbers of epochs). In addition, we repeat this re-training of the last layer with the diversity-idducing method of Teney et al. (2022a) (details in the box below). These are referred to as "diverse models".. Each run of the methods produces 24 different models, giving a total of 10 24 such models (10's as above 24).
3

---

Under review as a conference paper at ICLR 2023
Background: learning diverse solutions to learning task.
A range of methods exist to identify multiple neural networks of similar high IDD performance but that differ in other desirable properties such as OOD performance, interpretability, adversarial robustness, etc. These methods are relevant in cases of underspecification (D D'Amour et al. 2020 i.e. when the standard ERM objective does not constrain the solution space to a unique one.
Recent methods consist of optimizing multiple models while encouraging diversity in their fea- ture space (Heljakka et al. 2022; Yashima et al.] (022),, prediction space (Paglirrdini et al.) 2022
Lee al. 2022). gradient space (Ross al. 20188 2020) Teney al. 2022a b).
This study uses the method of Teney et al. (2022a) which encourages gradient diversity. The method trains many copies of the same model in parallel in case, linear classifer top of a frozen DenseNet backbone (see Figure
Training image
Figure 3: Method used to train a di- verse of models. Each training image goes through a frozen pre- trained DenseNet to produce features h=f(x). We train set of linear classi- fiers {9. }= on these features. diver- sity loss minimizes the pairwise simi- larity between their input gradients.
FiO
Feature extractor (frozen DenseNet)
Fs
diverse models (linear classifiers)
5
+ (
9
Pairwise similarity of inpot gradients
Labels
Ediveriity
The models optimized by standard SGD minimize the sum of standard classification loss (cross-entropy) with diversity loss that encourages diversity across models. Using À a weight hyperparameter, the complete loss is L Celassification + Ediversity. The second term encourages each copy to rely on different features by minimizing the mutual alignment of input gradients:
diiversity Ex TrrETiaining ΣR Ej-i+1 Vn gh))
()) with h = f()).
(1)
These pairwise dot products quantify the mutual aligmment of the gradients. Intuitively, mini- mizing this loss makes each model locally sensitive along different directions in its input space. Assuming that g produces a vector of logits (as many as there are classes), Vn g()) refers to the gradient of the largest logit w.r.t. the classifier's input h. We use n=24 copies and weight A=10 that were selected for giving a wide range of ID accuracies. See Teney et al. (2022a) for details about the method. Variations also described by |ooss al. (2020) and Teney al. (2022b).
Aucusss (5% (ai-00)
Recunss (1% tol-oa))
Accunty (%% tenvvos))
8
8
20
88
88
8
55
10
75
60
50 100
100
70
75
33
35
90
95
100
Accuracy (%4 val-id)
Accuracy 0%) val-ad)
Acceniccy (%) vall A4)
Accunaty (%) val
Figure 4: Our new observations show that higher OOD accuracy can sometimes be traded for lower ID accuracy. Each panel corresponds to different pretraining seed. Each dot represents linear classifier on frozen features, re-trained with a different seed and/or number epochs. They are re-trained with standard ERM (red dots @) diversity-iduucin method (gray dots *). The latter set includes models with higher OOD/ / lower ID accuracies. See Appendix A for additional plots.
Results with ERM models. Figure 4 we plot the ID vs. OOD accuracy of ERM models as red dots (+). Each panel corresponds to a different pretraining seed. The variation across panels (note the different Y-axis limits) shows that OOD performance varies across pre-traiming seeds even though the ID accuracy is similar, noted by Koh et al. (202I). Our observations visible within each panel. The dots (models) in any panel differ in their re-training seed and/or number of epochs. The seeds induce little variation, but the number of epochs produce patterns of decreasing
4

---

Under review as a conference paper at ICLR 2023
trend (negative correlation). Despite the narrow ID variation (X axis), careful inspection confirms that the pattern appears in nearly all cases (see Appendix [ zoomed-in plots).
Results with diverse models. We plot models trained with the -inducing method (Teney ctt al. 2022a) as gray dots (o). These models cover a wider range of accuracies and form patterns that extend those of ERM models. The decreasing trend is now obvious. This trend is also clearly juxtaposed with a rising trend where ID/ OOD performance are positively correlated. This suggests point of highest OOD performance after which the model overfits to ID data. Appendix [A] shows similar results with other pretraining seeds. The patterns are not always clearly discernible because large regions of the performance landscape are not covered, despite the diversity-inducing method. We further discuss this issue next.
4
WHY PAST STUDIES MISSED NEGATIVE CORRELATIONS: A BIASED SAMPLING OF MODELS
We identified several factors explaining the discrepancy between our observations and past studies.
ERM models alone do not always form clear patterns (red dots in Figure 4. In our observations, the models trained with a diversity-inducinn method (gray dots were key in making the suspected patterns more obvious, because they cover a wider range of accuracies.
The ID/OOD trade-off varies during training, as noted by Andreassen et al. (202T). This vari- ation across training epochs is responsible for much of the newly observed patterns. However, models of different architectures pretraining seeds are not always comparable with one another because of shifts in their overall performance (see e.g. different Y-axis limits across panels in Figure D. Therefore the performance across epochs should be analyzed individually per model.
The "inverse correlation" patterns are not equally visible with all pretraining seeds. In some cases, a careful examination of zoomed-in plots is necessary, see Appendix [A] This is reminder that stochastic factors in deep learning can have large effects and that empirical studies should randomize them as much as possible.
To demonstrate these points, we plot our data (same in Figure while keeping only the ERM models trained for 10 epochs and including all pretraining seeds on the same panel. F that these small changes reproduce the vertical line observed by Wenzel al. (2022). which com- pletely misses the inverse correlations patterns visible in Figure
Aacuray (ts tox-cod) ood)
Figure 5: We plot again the ERM models of Figure 4 (red dots but only include models trained for fixed number of epochs and combine all pretraining seeds in the same plot. This repro- duces the vertical line from (Wenzel et al., (022). which com- pletely misses the patterns of inverse correlation.
Under- sampled
regime
Accuracy valid)
A general explanation is that past studies undersample regions of the ID/OOD performance space. They usually consider variety of architectures an attempt to sample this space. However, different architectures do not necessarily behave very differently from one another (see the box below). We lack methods to reliably identify models of high OOD performance, but the diversity- inducing method that use yields models spanning wide range of the performance spectrum.
Why isn't it sufficient to evaluate variety of architectures?
Different architectures do not necessarily induce radically different behaviour: Even CNNs and vision transformers were shown have similar failure modes (Pinto al.] 2022). Distinct ar- chitectures can share similar inductive biases that are e.g. due to SGD, such as the simplicity bias (Scimeca al. 2002 Shah al. 2020) or neural anisotropies (Ortiz-Jimenez et al., 2021).
Therefore, independently trained models are not necessarily diverse despite the variety of archi- tectures. ID/ OOD performance may only vary along similar directions across models.
5

---

Under review as a conference paper at ICLR 2023
5 THEORETICAL ANALYSIS OF A LINEAR CASE
In this section, we present minimal case that shows a trade-off between ID and OOD performance and aids understanding the cause of such pattern. Let y E R be a target variable to be predicted by model, and a the features used as input to the model. These features are concatenation of invariant and spurious features (defined implicitly below): a: = Finv T with Zinv E Rd
and sup RRd Following Arjovsky et al. (2019); Rosenfeld et al. (2020); Zhou al. (2022) we
consider the simple data-ge sitta-ggneaaingg process defined by the following structural equations:
yf = 7 zinv + Ciuv
Esp y'1 + α Cspu
(2)
where e E feID, GOOD} } is an environment index referring ables éinv and Expn represent symmetric independent random noise with zero-mean, sub-Gaussian tail probabilities, and Var (finv) > 0, Var(<spu,s) 0, Vi [1, dspu) The vector Thu deter- mines the relation between the target variable and the invariant features and is identical across en- vironments. In contrast, the vector a affects the spurious features and varies among environments. Therefore the invariant features are similarly predictive in ID and OOD data while the spurious ones not.
OOD data. The random vari-
To study the relationship between ID and OOD performance a hypothetical predictive model, we assume that this model relies subset of the features æ. This subset is identified by binary mask T E (0,1) 10,1]dur+dapo Suppose have already selected diw invariant features and dapu spurious features, such that +dpuu) = d = || where the subscript d denotes the number of selected features.The The features selected by & are [2inv,1, ... Tinv,dine' Tspu,1, ... "spu,dggu! Let E denote either the in- and out- -i--a---omain expectation as E and gOOD_ We use 3 to denote the optimal parameter of the linear regression for certain domain, i.c., B3 = El [xxx)]]] E
Then the MSE loss of the fitted linear regressor is Ely $²(x) B.)². Further, let (X] Xg
denote the eigenvalues of TE| [B ( (x)()) and [og. vg w the corresponding eigenvectors.
Given a feature mask d we now examine how the ID and OOD losses of the model vary when including an additional spurious feature feature into & (see Appendix B for a proof).
Theorem 1. Including additional sparious feature leads the following change in loss L:
(11( ( = [ 22(x) T Ely
D241
T
3ID 12
d+1
< 0
LOOD ((1+)) LOOD Q1 + Q2 Q3
with Q1, Q2 Q3 defined as follows:
Q1 IOODly - 1 2x(x)
T G00D - Ely d (x)
T 300012
d
Q2
Σ
[ + (z)r) i dy. (2 (
1
2
t=1
ID,
\
OOD,d
+ (>))) T ia
doD (2 1
A ID,d+1
0OD
o 2 ((a )2 (a d+1 (20))3
d+2
23 = -0 (OODD c
> 0.
d+1
(A (2 x 00D.å+1
d+1
Further, if the new feature is sufficiently unstable in the test domain, Lee. if 241 )2 (a d+1 )22 is sufficiently large such that:
(2 ID, )2 2 2000 00D.
(( )2 (a d+1
0OD 1
d+1
1Q1 Q21
(BOOD (x))) T v 100 00D,å+12
then have Qa> > 1Q: + Q21 and therefore
LOOD ( LOOD ( > 0.

---

Under review as a conference paper at ICLR 2023
Because Espu is zero-mean symmetric random noise, the sign of Q does not the matter in the results. Theorem [ shows that adding a spurious feature to those used by the model can affect its ID and OOD losses in opposite directions, implying trade-off between ID and OOD accuracy. In other words, this minimal case shows that a simple model without/with an extra (spurious) feature can exhibit an inverse correlation between its ID and OOD performance.
ORDERING ID OOOD PATTERNS ACCORDING TO SHIFT MAGNITUDE
The above analysis shows that inverse correlation patterns are essentially due to the presence of spu- rious features, i.e. features whose predictive relation with the target in ID data becomes misleading OOD. Occurrences of spurious features increase with the magnitude of the distribution shift. There- fore, the possible patterns in ID/ OOD performance presented in Section can be ordered according to the magnitdee of the distribution shift they are likely occur with (see Figure
Posilive transfer
Underspecifieation
Misspecification
transfer
Distribution shift
Mild
(Too!) Severe
Typical pattern
Positive correlation
Vertical line/ho clear trend
Negative correlation
Low horizontal line
(toy representation)
B1 Cecoaa
Valid approaches for improving OOD performance
Simply Focus on improving ID performance
Task relevatt inductive biases e.g. arch., regularizers. angmentation with relevant transformations. Non-iid, training data e.g. multiple training domains.
Open question
Example datasets
ImageNet ImapeNet
PACS sketch
WILDS
DomainNet infograph quickdraw
photograph
Camelvon17
Recht al. 2019
Li al. 2017
Koh et al. 2021
(Koh al. 2021
Figure 6: Various patterns of ID vs. OOD performance occur at different levels of distribution shift.
With the smallest distribution shifts (leftmost case in Figure 6). for example training on ImageNet and testing on its replication ImageNet v2 (Recht et al. 2019). ID validation performance closely correlates with OOD test performance. This OOD setting is the easiest because one can focus on improving classical generalization and reap concurrent improvements OOD.
With a larger distribution shift, more features are likely to be spurious, which is likely to break the ID/ FOD correlation. The task of improving OOD performance is likely to be under- mis- specified, i.e. there is not enough information to determine which features a model should rely on to perform well OOD. Valid approaches include modifying the objective function, injecting task- specific information with custom architectures (e.g. building-in invariance to rotations as in Teney & Hebert (216))) well-chosen data augmentations, or inhomogencous training data such as multiple training environments (Li et al. 2017) and counterfactual examples (Teney et al. 2020a).
With extreme distribution shifts, most predictive features are overwhelmingly spurious and is very difficult to learn any one relevant in OOD data (rightmost case in Figure 6).
The proposed ordering of patterns is rather informal and could be further developed following the two axes of diversity shifts and correlation shifts proposed by Ye et al.] (2022) (see also Wiles et al. (02T)) ) More recently, Wang & Veitch (002)) showed that the suitability of various methods for OOD generalization depends on particularities of the underlying causal structure of the task - which must therefore be known to select a suitable method. Identifying which ID/ OOD patterns occur with particular causal structures might serve tool to understand the type of OOD situation
one is facing and identify a suitable method.
7

---

Under review as a conference paper at ICLR 2023
7 REVISITING RECOMMENDATIONS MADE IN PAST STUDIES
We have established that observations in past studies were incomplete. We now bring nuance to some recommendations and conclusions made in these studies.
Focusing on a single metric.
"We see the following potential prescriptive outcomes (...) correlation between OOD and ID performance simplify model development since can focus single metric." (Miller
et al. 2021)
We demonstrated that inverse correlations are possibility, hence there exist scenarios where an metric would be misleading. In general, relying on a single metric during model development is ill-advised (Teney et al., (020b). Even more so here since it cannot capture trade-offs along multiple axes. A model with a suboptimal better OOD generalization. Our recommendation is to track multiple metrics e.g. performance on multiple distributions or qualitative interpretable predictions on representative test points.
performance may have learned features that enable
Improving ID performance for OOD robustness.
"If practitioners want make the model more robust OOD data, the main focus should be to improve the ID classification error. (...) We speculate that the risk of overfitting large pretrained models the downstream set minimal, and it seems to be not a good strategy to, e.g., reduce the capacity of the model in the hope of better OOD generalization." (Wenzel et al. 2022)
This recommendation assumes the persistence of positive correlation. On the opposite, we saw that positive correlation precede regime of inverse correlation (Figure [4] left panels). If the goal is to improve OOD, focusing on ID performance is blind alley since this goal requires to increase ID performance at times, and reduce it at other times.
Future achievable OOD performance.
As obvious as it is, feels necessary to point out that empirical studies only chart regimes achiev- able with existing methods. Observations have limited predictive power, hence more care seems warranted when deriving prescriptive recommendations from empirical evidence.
Evidently, the best possible performance on Camelyon17 is limited to the Pareto front ob- served in our experiments. For example, the current state of the art on this dataset (Robey) et al., 2021, WILDS Leaderboard) injects additional task-relevant knowledge to bypass the un- der/misspectiiiatiion of the ERM objective, and exceeds both highest ID and OOD perfor- mance. The important message remains that given hypothesis class (DenseNet architecture in our case) admits parametrizations whose ID and OOD performance do not necessarily correlate.
Possible invalidation of existing studies.
The possibility of inverse correlations may invalidate studies that implicitly assume a positive one. For example, Angarano al. (2022) evaluate the OOD robustness of backbone computer vision architectures. They find that modern architectures surpass domain generalization (DG) methods.
However, they discard any model but those with the highest ID performance, a.k.a. "training domain validation" in Gulrajani & Lopez-Paz (2021). This means that any model with a high OOD performance but non-optimal ID ignored. They also train every model for fixed, large number of epochs (30). This may additionally prevent from finding models with high OOD performance since robustness progressively lost during fine-tuning (Andreassen et al. 2021).
By design, this study is therefore incapable of finding OOD benefits of any architec- ture htperpaammeeerrs/////////
method that requires trading off some ID performance. Most impor- tantly, once the implicit assumption of positive correlation is enacted by throwing away models with non non-maximal ID performance, there is no more opportunity to demonstrate its validity.
8

---

Under review as a conference paper at ICLR 2023
8 RELATED WORK
Inverse correlations can be found in the literature, although always highlighted by their authors.
An close examination of Gokhale al. (2022 Table 2) reveals clear inverse pattern with three benchmarks for natural language inference (NLI). The task is known for biases and shortcuts in the training data, and the OOD test sets in these benchmarks correspond to severe distribution shifts. Our proposed explanation (right end of the spectum in Figure (6) therefore aligns with these observations. Experiments on question answering from the same authors use data with milder distribution shifts. Correspondingly, they show instead a positive correlation.
Kaplun et al. 2022) Figure 7) find that the CIFAR dataset contains subset CCIFAR-1O-NeR on which the performance of visual classifiers is inversely correlated with their ID) performance.
Xie et al. (2020) Section 5.3) discusses cases of inverse correlation on the CelebA dataset with their In-N-Out method albeit within overall positive trend.
McCoy et al. (2019) show that BERT models trained with different random seeds vary widely on their performance on the HANS benchmark for NLI, while their ID) performance on the MNLI dataset similar.
Naganuma et al. (2022) published an extensive evaluation of OOD benchmarks after the initial release of this paper. They consider much wider range of hyperparameters than previous studies. As expected from our claims, they observe broader types of relations between ID/OOD perfor- mance than the previously reported "linear trend".
Work on adversarial examples has examined the trade-off between standard and adversarially- robust accuracy |Raghunathan et al.] (2020); Yang et al. (2020); /Zhng al. (201)). This agrees with our explanation since adversarial inputs correspond to extreme distribution shifts.
Work on transfer learning has occasionally shown cases of negative transfer, a related phe- nomenon where improving performance in one domain hurts in others.
DISCUSSION
This paper showed that inverse correlations between ID OOD performance are possible not only theoretically, but also happen real-world data. We do not know how frequent this situation is. Although we examined single counterexample|] we also showed that past studies may have systematically overlooked such cases. This suffices to show that one camnot know a priori where a given task falls the spectrum of Figure 6 It is thus ill-advised to blindly make the assumption of a positive correlation, which was suggested in the past.
Can avoid inverse correlations with larger training set? Scaling alone without data curation seems unlikely to prevent inverse correlations. Fang et al. (2022) examined a more general question and determined that the impressive robustness of the large lisioo-and-langguge model CLIP is deter- mined by the distribution stem from biases the training distribution (e.g. a class C₁ appearing more frequently with image background B3 than any other). And biases in a distribution do not vanish with more i.i.d. samples. Indeed, more data can cover more of the support of the distribution, but this coverage will remain uneven, i.e. biased. The problem can become one of "subpopulation shift" (Santurkar et al.) 2020) rather than distribution shift, but it remains similarly challenging.
its training data rather than its quantity. Similarly, inverse correlations
Training full networks with diversity standard ERM models and even more strikingly - with linear classifiers trained with diversity- inducing method (Teney et al. (022a)). To the best of our knowledge, this method has not been applied to deep models because of its computational expense. would be interesting to confirm our observations on networks trained entirely with this or other diversity -induuiit-nnninnggggg methods (Section 3)
method. We showed inverse correlations with
Qualitative differences along the Pareto frontier. We focused on quantitative performance. Inter- pretability methods could also be used to examine whether models of various ID OOD trade-offs rely on different features and generalization strategies, as done in NLP by Juneja et al. (2022).
Recent work by Eastwood et al. (2222) also recognizes issues of model selection for OOD general- ization. They get around selection based on either ID or OOD validation performance with a new
WWe only ran our experiments on Camelyon17. The dataset was not picked post hoc hecause of unusual results.

---

Under review as a conference paper at ICLR 2023
domain generalization method (Quantile Risk Minimization) with tunable trade-off. Other excit- ing advances by Wang & Veitch (202)) examine existing approaches to improve OOD performance with their suitability to different distribution shifts and underlying causal structures.
REFERENCES
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of-distribution robustness throughout fine-tuning. arXiv preprint arXiv:2 15831, 2021.
Simone Angarano, Mauro Martini, Francesco Salvetti, Vittorio Mazzia, and Marcello Chiaberge. Back-to-bonen bone:: Rediscovering the role of backbones in domain generalization. arXiv preprint arXiv:2209.01 121, 2022.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.00 02893, 2019.
Alexander D' A'mour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beu- tel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Under- specification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03395. 2020.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021.
Cian Eastwood, Alexander Robey, Shashank Singh, Julius von Kügelgenn, Hamed Hassani, George Pappas, and Bernhard Schölkopf. Probable domain generalization via quantile risk minimization. arXiv preprint arXiv:2207.0994/ 2022.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (CLIP). arXiv preprint arXiv:2205.01399, 2022.
Tejas Gokhale, Swaroop Mishra, Man Luo, Bhavdeep Singh Sachdeva, and Chitta Baral. Gen- eralized but not robust? comparing the effects of data modification methods on out-of-domain generalization and adversarial robustness. arXiv preprint arXiv:2203.07653, 2022.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In Proc. Int. Conf.
Learn. Representattinss 2021.
Ari Heljakka, Martin Trapp, Juho Kannala, and Arno Solin. Representational multiplicity should be exposed, not eliminated. arXiv preprint arXiv:2206. 200.02206...890, 2022.
Jeevesh Juneja, Rachit Bansal, Kyunghyun JoooJ reveals generalization strategies. arXiv preprint arXiv::205. /20000/..2111 /2411, 2022.
João Sedoc, and Naomi Saphra. Linear connectivity
Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: A pointwise framework of learning. arXiv preprint arXiv:2202.0990 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. WILDS: A benchmark of wn-the-wild distribution shifts. In Proc. Int. Conf. Mach. Learn., 2021.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from under- specified data. arXiv preprint arXiv:2202.0344 08 2022.
Da L.,, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 5542-5550, 2017.
Horia Mania and Suvrit Sra. Why do classifier accuracies show linear trends under distribution shift? arXiv preprint arXiv:2012. 15483, 2020.
10

---

Under review as a conference paper at ICLR 2023
R Thomas McCoy, Junghyun Min, and Tal Linzen. Berts of feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv preprint arXiv:1911.02969, 2019.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Proc. Int. Conf. Mach. Learn., 2020.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy the line: the strong correlation between dustr---iitributiin and in-distribution generalization. In Proc. Int. Conf. Mach. Learn., 2021.
Hiroki Naganuma, Kartik Ahuja, Joannis Mitliagkas, Shiro Takagi, Tetsuya Motokawa, Rio Yokota, Kohta Ishikawa, and Ikuro Sato. Empirical study on optimizer selection for out-of-distribution generalization. NeurlPS 2022 Workshop DistShift, 2022.
Guillermo Ortiz-Jimenez, Itamar Franco Salazar- Reque, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. A neural anisotropic view of underspecification in deep learning. In Proc. Int. Conf. Learn. Representations, 2021.
Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree dis- agree: Diversity through disagreement for better transferability. arXiv preprint arXiv:220..04414 2022.
Francesco Pinto, Philip HS Torr, and Puneet K Dokania. An impartial take to the CNN trans- former robustness contest. arXiv preprint vrXiv:2207.11347, 2022.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint arXiv:2002.10716, 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Proc. Int. Conf. Mach. Learn., 2019.
Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Proc. Advances in Neural Inf. Process. Syst., 2021.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. arXiv preprint arXiv:2010.0576,, 2020.
Andrew Ross, Weiwei Pan, Leo Celi, and Finale Doshi-Velez. Ensembles of locally independent prediction models. In Proc. Conf. AAAI, 2020.
Andrew Slavin Ross, Weiwei Pan, and Finale Velez. Learning qualitatively diverse and inter- pretable rules for classification. arXiv preprint 1rXiv:1806.087. 16, 2018.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. BREEDS: Benchmarks for subpopula- tion shift. arXiv preprint 20Xi::2008.04855 2020.
Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shortcut will dnns choose? study from the parameter-space perspective. In Proc. Inf. Conf. Learn. Representations, 2022.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. arXiv preprini TTt::2006.077100 2020.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 2000.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Proc. Ad- vances in Neural Inf. Process. Syst., 2020.
Damien Teney and Martial Hebert. Learning to extract motion from videos in convolutional neural networks. In Asian Conference on Computer Vision, pp. 412-428, Springer, 2016.
11

---

Under review as a conference paper at ICLR 2023
Damien Teney, Ehsan Abbasnedjad, and Anton den Hengel. Learning what makes difference from counterfactual examples and gradient supervision. arXiv preprint 20X1:20004000334, 2020a.
Damien Teney, Kushal Kafle, Robik Shrestha, Ehsan Abbasnejad, Christopher Kanan, and Anton van den Hengel. On the value of out-of-distribution testing: An example of Goodhart's law. In Proc. Advances in Neural Inf. Process. Syst., 2020b.
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplicity bias: Training diverse set of models discovers solutions with superior OOD generalization. In
Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2022a.
Damien Teney, Maxime Peyrard, and Ehsan Abbasnejad. Predicting is not understanding: Recog- nizing and addressing underspecification in machine learning. arXiv preprint arXiv:2207.02598, 2022b.
Zihao Wang and Victor Veitch. unified causal view of domain invariant representation learning. arXiv preprint arXiv:2208. 10t.:22288.09887, 2022.
Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel, Max Horn, Do- minik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, al. Assaying out- of-distribution generalization in transfer learning. arXiv preprint 27X19:200..09239, 2022.
WILDS Leaderboard. http:////aaaaaaaaaccaaaaaaaaaaaa
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuffi, Ira Ktena, Taylan Cemgil, et al. A fine-grained analysis on distribution shift. arXiv preprint arXiv:2 10.1 2021.
Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In- n-out: Pre-training and self-training using auxiliary information for out-of-distributio robustness. arXiv preprint arXiv:2012.04550 2020.
Yao- Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ Salakhutdinov, and Kamalika Chaud- huri. A closer look at accuracy vs. robustness. Proc. Advances in Neural Inf. Process. Syst. 2020.
Shingo Yashima. Teppei Suzuki, Kohta Ishikawa, Ikuro Sato, and Rei Kawakami. Feature space particle inference for neural network ensembles. arXiv preprint arXiv:2206.0004 2022.
Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-bench: Quantifying and understanding two dimensions out-of-distribbtioo generalization. In Proc. IEEE Conf. Comp. Vis. Patr. Recogn., 2022.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Proc. Int. Conf. Mach. Learn., 2019.
Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In Proc. Int. Conf. Mach. Learn., 2022.
12

---

Under review as a conference paper at ICLR 2023
A
ADDITIONAL RESULTS
AGUURO 7% 900-00
3
7%
15
60
80
8
60
10
80
6
18
12
180
15
85
06
Amenay
Aroway 18, 10)
Far Terewer
mwry
AOUPO RAR--OE
Pac
1%
mwr 0% nol 4)
Figure 4 we show that higher OOD accuracy can sometimes traded off for lower ID accuracy. Each panel shows results from different pretrained model (i.e. pretrained with a different random seed). Each dot represents a linear classifier re-trained on features from this pretrained model with standard ERM (red dots with diversity-inducing method (Teney al., 2022a) (gray dots o). The latter set includes models with higher OOD/ / lower ID accuracies.
Figure 7:
ACCIBOO 7% 18-000
400.00
Accano)
(6-40)
(94-)
0 01
ACDBOC 25 (0-00)
8
2
2
Accuno
Figure 8: Same as in Figure 7 but using val-ood (instead of test-ood) as the OOD evaluation
set.
Scmo 15 (thoc)
1%
67
67 Acany 13,
Accuncy (S willi
Accanct (%. mlke)
Accuncy veldd]
Acc may 1% saled)
witent
Asumccy
Accurco (ioa
Accrnoi 2% mala)
val-d)
Figure 9: Same as in Figure zoomed-in on ERM models (red dots
13

---

Under review as a conference paper at ICLR 2023
B PROOF OF THEOREM 1
Let aa == Φb a (mzz Tinv,l, .. Z inv,dmy" Dspu,,, ... 2 spp be the d features already selected
and z4+1 4 a+1 (mz) the features after adding new spurious feature spun,dgnnt1 to z
Let (X Xg ,..... x denote eigenvalues of Elar] arg and [of og...., vgj corresponding eigenvectors.
Assumption 1. The projection [aac wf] each basis corresponding to feature is non zero, i.e.
EE" [ar
d d >> O, Vee (eID, eOOD), iE(d).
This ensures that coefficients of a feature can not be always 0, otherwise we can simply remove it.
Proof. Let IID and gooD denote the solution of linear regression in the ID and OOD domains, i.c.
3 arg min E (y 2 1)22
8
dd BOOD = arg min E FOOD (y x 3)²
(4)
8
Now let us compare the OOD loss after we include x spsn,dguu+11 practice, we can only obtain 3ID and then apply it on both the ID and OOD domains, which elicits the following errors:
LID (2)) =E (y - x )D Goon((2) (.)) - (y x T TTD)
(5)
= =000 (y 2) BTD) -ED-EOO FOOD (y - 2 300D EOOD (y o 300D)
(6)
sf
is well known that the residual of the linear fitting y by æy on the ID domain
CID (
E [y- xz 2
x
E'y 22(x) 11
(7)
Similarly, we have
C E
(v
T
3+1 8
(8)
Since a spuuddppas does not lies in the space spaned by æ. so the space spanned by d'41 is strietly larger than x. Togther with Assumption I we have
LID( LID
E
x
EID ly
x T
d41 d+1
d+1"
12 0,
(9)
and also
5
-
52
i+1 BOOD
ly 2x
100D
E
BOOD
ly
3 gOOD 0.
(10)
d+1
By the proof in Appendix B.6.3 (above Eq. 29) in Zhou et al. (2022). we have
d
ct
Σ
OOD
[x T d OOD
To
(11)
X IID_á
ooD
By Eq. (20) in Zhou et al. (202)), we have
X IID \ OoD = (a!!D)2 2
(12)
So we have:
cd+1 dd+
(ROOD
p T oa oonddj3 (o (1o0D.))
2
1D,d
ooD,d
2
OOOD
T/ D 2 (200,,411)
x
1 (oD
A ID,å+1
101,d+112 2 ((a 1D )2
2+1 (aa2t
(a (0 22
EE OOD
v)
T v
(13)
d+1
00D,d+1
d+1
From Eq. 11 and 13 we have the desired result.
14


Reviews:
[{"review_id": "lxpNY5YyNRq", "reviewer": "Reviewer_94SR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper highlights that an inverse correlation can exist between the in-distribution (ID) and out-of-distribution (OOD) performance of a model. Empirically, the paper supports the claim by training different linear heads with ERM and regularized ERM in [1]. The paper also provides a theoretical study for linear models on the negative impact of spurious features. \n\n\n[1] Teney et al., Evading the simplicity bias: Training a diverse set of models discovers solutions with superior OOD generalization. CVPR 2022", "summary_of_the_review": "The paper provides an interesting empirical evaluation on a specific phenomenon on the impact of spurious correlation and OOD generalization. However, the paper provides limited theoretical insights and lack sufficient analysis. The empirical study is limited to ERM on a single dataset. ", "strength_and_weaknesses": "Strengths:\n- Investigation on the impact of spurious features and the correlation between ID and OOD performance is an important task. \n- The paper provides extensive background and review of related works.\n\nWeaknesses: \n- Organization can be largely improved. For example, Section 1, 2, and 7 are mostly devoted to review of literature and comparison with previous works. A more concise organization would be better. The background section in Section 3 can be moved to Appendix which diverts the attention of readers.  \n- Insufficient novelty due to lack of in-depth analysis. It is well expected that the presence of spurious features and reliance on it can negatively impact the OOD performance. The analysis only applies to linear models, which is insufficient to support the empirical results.\n- Lack of empirical evidence. The paper only uses ERM and regularized ERM as training objectives. No domain generalization methods are investigated, which leads to an open question whether the trend still hold with methods that are specifically designed for improving domain generalization.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vhym-f7_Pj", "reviewer": "Reviewer_nE6K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "An emerging observation is the strong empirical correlation between IID and OOD robustness. This paper presents a counterexample, showing that IID-OOD performance can be inversely correlated on the WILDS-Camelyon17 dataset. The paper also presents a series of arguments to counter specific claims presented in past work, bringing nuance to the discussion of IID-OOD correlation.", "summary_of_the_review": "The evaluation is quite limited, and the paper is more of a position piece. Overall, I do not find the argument extremely compelling.", "strength_and_weaknesses": "Strengths:\n* I like the claim that when trying to study IID-OOD correlations, the space of models that people have considered in past work is actually quite small. And, by training with diversity-promoting regularizers, one can uncover more of the pareto frontier.\n\nWeaknesses:\n* I found the paper's central claim to be a bit unsurprising. Concretely, most past work in this area, e.g., the \"Accuracy on the line\" paper (Miller et al. 2022), claim there is a strong correlation between IID and OOD performance for many datasets and distributions. However, this is not a \"for all\" claim: the paper is not arguing that every dataset and shift has this same correlation. Instead, they are merely pointing out that this correlation is surprisingly strong and evident in many existing benchmarks.\n* For that reason, I found the paper's main result, that one can find a *single* dataset where the correlation does not hold, to be rather unsurprising. Especially given that the dataset they study, WILDS-Camelyon17, is already a dataset for which Miller et al. 2021 says the IID-OOD correlation is rather weak. \n* Moreover, there are other cases in the literature where the IID-OOD corelation is weak or breaks down. For example, the NLP spurious correlations body of work points out a lot of cases where similar models generalize in widely different ways (e.g., https://arxiv.org/abs/1911.02969). The CLIP paper presents another case where finetuned models can have different IID-OOD tradeoffs (improving IID can hurt OOD), and there are numerous follow-up work's that try to mitigate this trade-off. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JsVvdy5Ybg", "reviewer": "Reviewer_HGmQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work discovers a concrete example on the WILDS-Camelyon17 dataset that ID performance is negatively correlated to OOD performance. A theoretical example based on a simple linear model is also presented to show the trade-off between ID and OOD performance. The authors explain why past studies missed such a negative correlation and bring recommendations to the OOD generalization studies. ", "summary_of_the_review": "See Strength and Weakness.", "strength_and_weaknesses": "Strength:\n1. This work verifies the trade-off between ID and OOD performance happening in a real-world dataset.\n2. The authors present detailed discussions about the experimental details and the comparison to the past studies.\n3. The authors provide a theoretical analysis of a linear model.\n\nWeakness:\n(I list my questions in this section.)\n\n1. The authors do not clearly introduce the OOD generalization task and evaluation. \n    - **Q1.** Is it a generalization problem from the training dataset to a specific test dataset? or from the training dataset to a set of multiple test datasets? \n    - **Q2.** Do you consider the average accuracy or the worst-case (e.g. worst-domain) accuracy?\n\n2. The authors do not discuss the assumptions or conditions of the OOD generalization problem. \n    - **Q3.** For example, if there is no additional information about the unseen test datasets and only the training dataset is accessible, what are the reasons to reject a model with good ID performance? Is this a subjective guess on test data, rather than a scientific model selection scheme? \n    - **Q4.** Do the negative correlations between ID and OOD performance imply that the OOD generalization task is unlearnable?\n\n3. This work focuses on a specific case of OOD generalization from the perspective of models and algorithms, but ignores the weakness of the common-used datasets, e.g. the domain generalization benchmarks. \n    - **Q5.** We consider a multidomain dataset, e.g. PACS. According to the negative correlation between ID and OOD accuracy, is it possible that there exists a fifth domain such that the current SOTA domain generalization method performs worse than ERM? \n    - **Q6.** Can we conclude that current domain generalization benchmarks lack enough domains to evaluate OOD performance?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "07-L7s_WCJZ", "reviewer": "Reviewer_yZ1A", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies patterns in the performance of ML models on in-distribution (ID) and out-of-distribution (OOD) data. The central claim of this work is that ID performance can be inversely correlated with OOD performance in practice, which has not been explicitly observed before in deep ML models. This work then provides theoretical justification in an ERM model for why this might occur, and discusses the implications these results have on the broader research field. \n", "summary_of_the_review": "See above", "strength_and_weaknesses": "I appreciate the thorough exposition on prior works and past observations in the paper’s introduction. Experimentally, I think the use of diversity-inducing methods to enlarge the sampling space of models is clever and well motivated. The recommendation and discussion sections are also particularly informative, highlighting possible over-simplifications in conclusions drawn by prior works. \n\nHowever, I am leaning towards recommending rejection of this work in its current state. There are three important weaknesses in this paper that limits its contribution towards our collective knowledge about OOD generalization.\n\nThe first weakness is that the empirical observations made in this work are entirely based on experiments conducted on the Camelyon17-Wilds dataset. This begs one to question whether the claims made in this paper have any transferability to wider data domains, such as natural images, chest x-rays, satellite images, and NLP datasets. The Camelyon17-Wilds dataset defines “in” and “out” of distribution based on the hospital of origin of the slide image. This is comparatively a narrow scope of ‘in’ distribution, where the distributional shift is almost entirely accounted for by a change in the staining protocol. Other types of distributional shifts, such as those that include both subpopulation shifts and feature changes, might induce different generalization behaviors. \n\nThe second weakness lies with the model training protocol. In particular, this work studies the ID and OOD performance of models that arise from training the last layer of a network (a linear model) while varying the initial seed and number of epochs. A danger with this design choice is that observations and theories established for linear models are rarely true for deep neural networks (e.g. the “double descent” generalization curve of over-parameterized models, and the fragility of influence estimation in the deep learning context), and unconverged models behave differently from fully converged models. As such, the observed negative correlation between ID performance and OOD performance could be the direct result of using gradient descent with linear models. For instance, if one considers a case where some spurious features are weakly and positively correlated with an invariant feature in the training domain, then the predictive power of these spurious features to the label is necessarily smaller than the invariant features. One can then apply standard arguments (e.g. https://www.cs.toronto.edu/~rgrosse/courses/csc2541_2021/slides/lec01.pdf) to show that the implicit regularization effect of the gradient descent algorithm causes the invariant features to be learnt early on, and the spurious (and weakly predictive) features are only learnt later on in the optimization process. If this were true, one would predict that by regularizing the number of epochs in the optimizer, one can achieve better generalization on test domains where the predictive direction of such spurious features are flipped. This seems to be indeed the case as the authors observe that “this variation across training epochs is responsible for much of the newly observed patterns”. Modern deep networks, with both explicit and implicit regularizations, may not require such early-stopping to “unlearn” spurious features, and as such do not behave in the same fashion as the models studied in the paper. The authors argue that diverse architectures are not sufficient when studying these trends, but this doesn’t absolve the authors of the responsibility of investigating architectures beyond linear models if they wish to make any general claim about neural network training. \n\nThe third weakness lies in the lack of quantitative evaluations for hyperparameters and choice of metrics. From the plots in Appendix A, one observes that the OOD accuracies vary significantly between different pretraining seeds. Furthermore, not every plot exhibits the negative correlation behavior. A careful investigation of the characteristics of the pretrained feature extractor and visualizing other metrics (such as those used in Wenzel 2022) could hope to elucidate factors that can explain these variations.\n\nRegarding questions, I think most of my concerns can be addressed experimentally through additional datasets and models. I will raise my score accordingly. \n\nLastly, while not directly affecting my score, I think the presentation of Theorem 1 could be made more concise by replacing several repeated expressions with appropriate symbols. The author might also consider modifying the analysis in replacing the “feature selection” model with a direction of steepest descent argument as I linked above to reduce the amount of simplifications required in the analysis. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject