ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets
Authors: Damien Teney
LIN Yong
Seong Joon Oh
Ehsan Abbasnejad
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Generalisation; machine learning
TL;DR: ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets

Abstract: Several studies have compared the in-distribution (ID) and out-of-distribution (OOD) performance of models in computer vision and NLP. They report a frequent positive correlation and some surprisingly never even observe an inverse correlation indicative of a necessary trade-off. The possibility of inverse patterns is important to determine whether ID performance can serve as a proxy for OOD generalization capabilities.

This paper shows that inverse correlations between ID and OOD performance do happen with multiple real-world datasets, not only in artificial worst-case settings. We explain theoretically how these cases arise and how past studies missed them because of improper methodologies that examined a biased selection of models.

Our observations lead to recommendations that contradict those found in much of the current literature.
- High OOD performance sometimes requires trading off ID performance.
- Focusing on ID performance alone may not lead to optimal OOD performance. It may produce diminishing (eventually negative) returns in OOD performance.
- In these cases, studies on OOD generalization that use ID performance for model selection (a common recommended practice) will necessarily miss the best-performing models, making these studies blind to a whole range of phenomena.

ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets
Damien Teney
Yong Lin
Idiap Research Institute, Switzerland damien. twweeeeedaa.....mmm
Hong Kong Uni. of Science and Technology ylindf@comna ust.hk hk
Seong Joon Oh
Ehsan Abbasnejad
University of Tübingen, Germany coallaoh@gmail.com
University of Adelaide, Australia ehsan abbasne ja@adeladddeddd edu au
Abstract
Context, Several studies have compared the in-distribution (ID) and out-of- distribution (OOD) performance of models in computer vision and NLP. They report frequent positive correlation, but surprisingly, almost never an inverse correlation that would be indicative of a necessary trade-off. Such inverse patterns are possible theoretically, and their occurrence in practice is important to determine whether ID performance can serve proxy for OOD generalization.
Findings. This paper shows that inverse correlations between ID and OOD perfor- mance do happen with multiple real-world datasets, not only in artificial worst-case settings. We explain theoretically how these cases arise and how past studies missed them because of improper methodologies that examined a biased selection of models.
Implications. Our observations lead to recommendations that contradict those found in much of the current literature.
High OOD performance sometimes requires trading off ID performance.
Focusing on ID performance alone may lead to optimal OOD performance. It may produce diminishing (eventually negative) returns in OOD performance.
In these cases, studies on OOD generalization that use ID performance for model selection (a common recommended practice) will necessarily miss the best- performing models, making these studies blind whole range of phenomena.
Introduction
Past observations. This paper complements existing studies that empirically compare in-distribution (ID) and out-oo-diisrriuttior]
performance deep learning models 2 6
25
22
24 42 50).
It has long been known that models applied to OOD data suffer drop in performance, c.g. in classification accuracy. The above studies show that, despite this gap. ID and OOD performance are often positively correlated" across models on benchmarks in computer vision 25 and NLP 24!
Past explanations. Frequent positive correlations are surprising because inverse correlations are similarly possible theoretically. Indeed, ID and OOD data contain different associations between labels and features. One could imagine e.g. that an image background is associated with class C ID and class C2 OOD. The more a model relies the presence of this background, the better its
We use "OOD" to refer test data conforming to covariate shifts 41 w.r.t the training data.
We use "correlation" refer both linear and non-linear relationships.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Toy representations (one point model)
Figure I: Several past studies suggest that positive cor- relations between ID/ DOO performance are ubiquitous. This paper shows empirically and theoretically why verse correlations are also possible and can be acciden- tally overlooked. The possibility of ID /OOD trade-offss goes counter the common practice of model selection based on ID performance, which is recommended in many benchmarks for OOD generalization.
Positive correlation
Inverse correlation
COD Ausency
ID performance but the worse its OOD performance, resulting in an inverse correlation. The fact that inverse correlations were almost never observed in practice was explained with the fact that real-world benchmarks only contain mild distribution shifts [22]. However, we will show in Section 4 that the reason may also be in flawed experimental design.
recent study [50] shows various correlation patterns across datasets, but never inverse ones:
"We did not observe any trade-off between accuracy and robustness, where more accurate models would overfit to spurious features that do not generalize." " (50)
This does not match our past general experience, so we set out to expose in this paper actual cases of inverse correlations, using popular datasets used for OOD research. Our results show inverse correlations across models trained with varying numbers of epochs and random seeds. These patterns even more striking when models are trained with regularizer that identifies a diverse set of solutions to the ERM objective [66].
Explaining inverse correlations. We name the underlying cause a "misspecificattion" an extension of the "underspecification" used previously to explain why models with similar ID performance can vary in OOD performance [5] [6 47). In cases of misspecification, the standard ERM objective (empirical risk minimization) aligns with a maximization of the ID performance but is in conflict with the OOD performance. ID and OOD metrics then vary inversely one another. In Appendix C we present a minimal theoretical example to illustrate how inverse correlations originate from the presence of both robust and spurious features in the data. In Section[7] show that different patterns of ID/ performance occur with different magnitudes of distribution shifts.
Summary of contributions.
An empirical examination of ID vs. OOD performance on several real-world datasets showing inverse correlation patterns that conflict with past evidence (Section 3).
An explanation and an empirical verification of why past studies missed such patterns (Section 4).
A theoretical analysis showing how inverse correlation patterns can occur (Appendix C
A revision of conclusions and recommendations made in past studies (Section
2 Previously-observed patterns of ID performance
Multiple studies conclude that ID and OOD performance vary jointly across models on many real- world datasets (6 25 42). Miller et al. (25) report an almost-systemalt
linear correlation between probit-scaled ID and OOD accuracies. Mania and Sra 22 explain this trend with the fact that many benchmarks contain only mild S Andreassen et al. (2) find that pretrained models perform "above the linear trend' in early stages of fine-tuning. The OOD accuracy then rises more quickly than the ID accuracy early on, though the final accuracies agree with a linear trend across models.
Most recently, the large-scale study of Wenzel al. [50] more nuanced: they observe linear trend only on some datasets. Their setup consists in fine-tuning an ImageNet-pretrainee model on a chosen dataset and evaluating it on ID and OOD splits w.r.t. this dataset. They repeat the procedure with variety of datasets, architectures, and other implementation choices such as data augmentations. The scatter plots of ID/ /ODD accuracy in [50) show four typical patterns (Figure
'The "linear trend" is not really linear: it applies to probit-scaled accuracies non-lineat transform). [22] explains linear trends with (1) data points having similar probabilities of occurring ID and OOD, and (2) a low probability that model correctly classifies points that higher-accurage model misclassifies.

---

BOASIN [Dana siatbsrsitt)))
VIRTICN INE (PACE wendeveppoc
HORPOORTA [Davaslic Hapapwireddee
(Tumstogg (oofersaad)
04
0.30
0.35
0.90
0.5
1:30
500
6.76
6.60
100 0.00
0.3
0.90
0.25
1.30
0.25
6.62
2.34
Figure 2: Typical patterns observed in 50 (reproduced with permission).
1. Increasing line (positive correlation): mild distribution shift. ID and OOD accuracies are posi- tively correlated. Focusing on classical (ID) generalization brings concurrent OOD improvements.
Vertical line: underspecification 16 Different models obtain similar high perfor- mance but different OOD performance. The objective of high ID performance does not sufficiently constrains the learning. Typically, multiple features in the data (a.k.a. biased or spurious features) can be used to obtain high ID performance, but not all of them are equally reliable on OOD data. To improve OOD performance, additional task-specific information is necessary, e.g. additional supervision inductive biases (custom architectures, regularizers, etc.).
3. Horizontal line, low OOD accuracy: severe distribution shift.
No model performs well OOD, severe shift prevents any transfer between training and OOD test data. The task needs to be significantly more constrained e.g. with task-specific inductive biases.
No clear trend: underspecificatioo. Models show a variety of ID and OOD accuracies. The difference with (2) is the wider variety along the ID axis, e.g. because difficult learning task yields solutions of lower ID accuracy from local minima the ERM objective.
The authors note the absence of decreasing patterns, which are however possible in theory.
5. Decreasing line (inverse correlation): misspecification. The highest accuracy ID and OOD are achieved by different models. Optima of the ERM objective, which are expected to be optima in ID performance, do not correspond to optima in OOD performance. This implies trade-off: higher OOD performance is possible at the cost of lower ID performance.
When does an inverse correlation occur between ID and OOD performance?
Intuitively, it can when pattern in the data predictive in distribution and misleading in the other. For example, object classes C1 and C2 are respectively associated with image backgrounds B, and B3 in ID data, and respectively B₂ in B, (swapped) in OOD data. Relying on the background can improve performance on either distribution but not both simultaneously. While such severe shifts might be rare, the next section presents actual example.
New observations: inversely-correeated
performance
This section is an in-depth examination using the WILDS- Camelyon17 dataset [15] We include experiments on other datasets in Section and Appendix Here, we use Camelyon17 7 manner similar to Wenzel et al. [50]. These authors evaluated different architectures and assumed that their different inductive biases can produce models that cover a range ID/OOD accuracies. In contrast and for simplicity, we rely instead on different random seeds since
S showed that this is sufficient to
80
Acanne US tol)
Acaneer (aal-ad))
Acauty ooddooo))
65
60
8
8
8
8
85
98
10
60
60
80
85
90
150
70
80
25
90
95
100
1%,
(%)
Figure 3: Our new observations show that higher OOD accuracy can sometimes traded for lower II) accuracy. Each panel corresponds to different pretraining seed. Each dot represents linear classifier on frozen features, re-trained with a different seed and/or number of epochs. They are re-trained with standard ERM (red dots @) diversity-inducing method (gray dots #). The latter set includes models with higher OOD/lower ID accuracies. See Appendix A for additional plots.

---

cover a variety of ID/ / /OOD accuracies on this dataset. We also want to minimize the experimenter s bias. Therefore, to further increase variety without the manual arbitrary selection of architectures of we also train models with the general-purpose, diversity-inducinn method of [46
50,
Background: learning diverse solutions.
A range of methods have been proposed to train multiple networks to similar ID performance while differing in other properties such as OOD generalization. These "diversification" methods are relevant in cases of underspecification [ when the standard ERM objective does not constrain the solution space unique one. Recent methods train multiple models (in parallel or sequentially) while encouraging diversity in feature space
[1] [57], prediction space [28] [6], or gradient
space [36 35
47
We use method of [46) that encourages gradient diversity.
Training image
Feature extractor (frozen DenseNet)
Set of diverse models (lincar classifiers)
(36)
Pairwise similarity of inpul gradients
I Lahels
Figure 4: Method used to train diverse set of models. Each training image goes through a frozen pretrained DenseNet produce features h=f( We train a set of linear classifiers {g: features. diversity loss minimizes the pairwise similarity between their input gradients.
[9:] Hir1
these
The method trains many copies of the same model in parallel - in our case, a linear classifier on top of frozen DenseNet backbone (see Figure 4). The models are optimized by standard SGD to minimize the sum of a standard classification loss (cross-entropy) with diversity loss that encourages diversity across models. Using À weight hyperparameter, the complete loss is L= Lelassification + À diiversity- The second term encourages each copy to rely on different features by minimizing the mutual alignment of input gradients:
Ldivensty EEET D 2j Vn (
with h = f(x).
(1)
These pairwise dot products quantify the mutual alignment of the gradients. Intuitively, minimiz- ing (1) makes each model locally sensitive along different directions its input space.
Assuming that g produces a vector of logits (as many as there are classes), Vn g(-) refers to the gradient of the largest logit w.r.t. the classifier's ' input h. We use n=24 copies and weight X=10 that were selected for giving a wide range of ID accuracies. See (46) for details about the method.
Experimental details. We use 10 DenseNet- 121 models pretrained by the authors of the dataset with different seeds (15). For each, re-train the last linear layer from random initialization for I to 10 epochs, keeping other layers frozen. These are referred to as ERM models. We perform this re-training with 10 different seeds which gives 1 ERM models (10 pretraining seeds x 10 re- training seeds x 10 numbers of epochs). In addition, we repeat this re-training of the last layer with method of [46] (details in the box below). These referred to as diverse the models. Each run of the methods produces 24 different models, giving a total of 10 24 such models.
Results with ERM models. In Figure 3 we plot the ID vs. OOD accuracy of ERM models as red dots (+). Each panel corresponds to a different pretraining seed. The variation across panels (note the different Y-axis limits) shows that OOD performance varies across pre-training seeds even though the
ID accuracy is similar, as noted by [15]. Our new observations are visible within cach panel. The dots (models) in any panel differ in their re-training seed and/or number of epochs. The seeds induce little variation, but the number of epochs produce patterns of decreasing trend (negative correlation). Despite the narrow ID variation (X axis), careful inspection confirms that the pattern appears in nearly all cases (see Appendix A for zoomed-in plots).
Results with diverse models. We plot models trained with the diversity method [6) as gray dots (o). These models cover a wider range of accuracies and form patterns that extend those of ERM models. The decreasing trend is now obvious. This trend is clearly juxtaposed with rising trend of positive ID/ OOD correlation. This suggests a point of highest OOD performance after which the model overfits to ID data. Appendix A shows similar results with other pretraining seeds. The patterns

---

not always clearly discernible because large regions of the performance landscape are not covered, despite the enversity-inducle method. We further discuss this issue next.
Past studies missed negative correlations due to biased sampling of models
identified several factors explaining the discrepancy between our observations and past studies.
ERM models alone do not always form clear patterns (red dots in Figure B In our observations, the models trained with a diversity-inducing method (gray dots @) were key in making the suspected patterns more obvious, because they cover wider range of accuracies.
The ID/OOD trade-off varies during training, as noted by (2.. This variation across training epochs responsible for much of the newly observed patterns. However, models of different architectures or pretraining seeds are not always comparable with one another because of shifts in their overall performance (see c.g. different Y-axis limits across panels in Figure 3). Therefore the performance across epochs should be analyzed individually per model.
The "inverse correlation" patterns are not equally visible with all pretraining seeds. In some cases, a careful examination of zoomed-in plots is necessary, see Appendix A This is reminder that stochastic factors in deep learning can have large effects and that empirical studies should randomize them as much as possible.
To demonstrate these points, we plot our data (same in Figure while keeping only the ERM models trained for 10 epochs and including all pretraining seeds on the same panel. Figure that these small changes reproduce the vertical line observed by Wenzel et al. [50), which completely misses the inverse correlations patterns visible in Figure 3
Figure 5: We plot again the ERM models of Figure (red dots but only include models trained for fixed number of epochs and combine all pretraining seeds in the same plot. This reproduces vertical line from (50), which completely misses the patterns of inverse correlation.
Under- sampled regime
Accuraor
Accuracy 1%, valdd)
general explanation is that past studies undersample regions of the IDOOOD performance space. They usually consider variety of architectures in attempt to sample this space. However, different architectures do not necessarily behave very differently from one another (see the box below). We lack methods to reliably identify models of high OOD performance, but the diversity-indueche method that we use yields models spanning a wide range of the performance spectrum.
Why isn't it sufficient to evaluate variety of architectures?
Different architectures do not necessarily induce radically different behaviour [39] Even CNNs
and vision transformers have similar failure modes [29]. Distinct architectures share similar
inductive biases due e.g. to SGD, the simplicity bias (39 40 or neural anisotropies [27].
5 Occurrences in other datasets
In addition to Camelyon17, we performed experiments five additional datasets used in OOD research and found inverse correlations on four out of five. We present two of them below and the others in Appendix B
In these experiments, we train standard architectures with well-known methods: standard ERM, simple class balancing [12), mixup (00 selective mixup [56], and post hoc adjustment for label shift [19] (we did not use the diversification method from Section (3). We repeat every experiment
with 10 seeds and plot the ID/ OOD accuracy from every epoch in the following figures.
5

---

5.1 WildTime-arXiv
Data. The WildTime-arXiv [55] dataset contains text abstracts from arXiv preprints. The task is to predict each paper's category among 172 classes. The ID and OOD splits are made of data from different time periods.
Methods. We fine-tune a standard BERT-tiny model with new linear head, using any of these
well-known methods: standard ERM, simple class balancing [12]. mixup [60], selective mixup [56). and post hoc adjustment for label shift T9 (we did not use the diversification method from Section 3 We repeat every experiment with 10 seeds and record the ID and OOD accuracy at every training epoch. We then plot each of these points in Figure and highlight the epoch of highest ID or OOD accuracy per run (method/seed combination).
Results, As discussed in Section 5.1 there is a clear trade-off both within methods (i.e. across seeds and epochs) and across methods.
49
Resampling
OOD accuracy (%)
[uniform clases)
Mixup and mixup
44
Adjusting
lobel shift
Epochs of best 00D accuracy
accuracy (%)
Figure 6: Results on WildTime-ad arxvv.
5.2 Waterbirds
Data. The waterbirds dataset 37 is synthetic dataset widely used for evaluating OOD generaliza- tion. The task to classify images of birds into 2 classes. The image backgrounds are also of two types, and the correlation between birds and background is reversed across the training and test splits. The standard metric is the worst-group accuracy, where each group is any of the combinations of bird/background.
Methods. We follow the same procedure as described above. We experiment two classes of architectures: ResNet-50 models pretrained on ImageNet and fine-tuned waterbirds, and linear classifiers trained of features from the same frozen (non-fine fine-tuned) ResNet-50.
Results. We observe in Figure patterns of inverse correlations in both cases.
(5%
Asemmzo drou8 Moorte OOD
000 (vers acorray
78
Roummping
(anform graass)
79
best ID sccuracy
Selective mioes
(eto dowalyy Likel USA)
Fpachs bedt atcurity
Epacis socuracy
73
Epoche bet 300 accuracy
32
67
77
83
89
95
ID accuracy (%)
ID accuracy (%)
Figure 7: Results on waterbirds with linear probing (left) and fine-tuned ResNer-50 models.
5.3 Occurrences in the existing literature
A literature review reveals other cases across range of topics, many of which were not particularly highlighted by their authors and required close examination.
A close examination of 9 Table 2] reveals a clear inverse pattern on three benchmarks for natural language inference (NLI). This task known for biases and shorteuts the training data, and the OOD test sets in these benchmarks correspond to severe distribution shifts. Our

---

proposed explanation (right end of the spectum in Figure 8) therefore aligns with these observations. Experiments on question answering from the same authors use data with milder distribution shifts. Correspondingly, they show instead positive correlation.
Kaplun et al. [14] Figure 7] find that the CIFAR dataset contains a subset (CIFAR-IO-Neg) on which the performance of visual classifiers is inversely correlated with their ID performance.
Xie et al. [53] Section 5.3] discuss cases of inverse correlation the CelebA dataset with their In-N-Out method - albeit within an overall positive trend.
McCoy et al. [23] show that BERT models trained with different seeds vary widely in performance the HANS benchmark for NLI while their ID performance MNLI is similar.
Naganuma et al. (26) performed an extensive evaluation on OOD benchmarks after the initial release of this paper. They consider a wider range of hyperparameters than existing works, and as expected from our claims, they observe broader range of ID/ /ODD relations than the "linear trend".
Liang et al. [8 focused on datasets with subpopulation shifts and analyzed the relation between performance across subgroups. They also find a linear trend to be an inaccurate description. They observe non-linear relations with a transition point between models showing a negative correlation and others showing positive one.
In their paper on Model recycling, Ramé et al. 33 include plots of ID/OOD performance (Appendix A) that are nothing like linear correlations. Instead, bell-shaped curves unambiguously indicate necessary trade-off between ID and OOD performance.
Work on adversarial examples has examined the trade-off between standard and adversarially- robust accuracy 30, 54 [5]. This agrees with our explanations (Figure X since adversarial inputs
correspond to extreme distribution shifts.
The literature on transfer learning has previously shown occasional cases of negative transfer, a
related phenomenon where improving performance in one domain hurts in others (20 49).
Theoretical analysis
The theoretical possibility of an inverse ID/OOD correlation is quite obvious. In Appendix C), we show below theoretically how to construct toy example where ID and OOD negatively correlated. We then explain how ID and OOD metrics can diverge as training progresses, or as one trains a model on data containing more and more spurious features.
However, the theoretical analysis is not central to our argument. The more important open question whether these cases happen merely in pathological situations, or also in real-world data. Prior work strongly argued for the former, but we show that the latter actually correct. The key contribution of this paper is therefore in the observations actual data, rather than in theoretical argument.
7 Ordering OOD patterns according to shift magnitude
The above analysis shows that inverse correlation patterns are essentially due to the presence of spurious features, i.e. features whose predictive relation with the target in ID data becomes mislead- ing OOD. Occurrences of spurious features increase with the magnitude of the distribution shift. Therefore, the possible patterns in ID/ OOD performance presented in Section 2 can be ordered according to the magnitude of the distribution shift they are likely to occur with (see Figure B.
With the smallest distribution shifts (leftmost case in Figure 8). for example training on ImageNet and testing on its replication ImageNet v2 (32), ID validation performance closely correlates with OOD test performance. This OOD setting is the easiest because one can focus on improving classical generalization and reap concurrent improvements OOD.
With a larger distribution shift, more features are likely to be spurious, which is likely to break the ID/ OOD correlation. The task of improving OOD performance is likely to be under- misspecified, ie. there not enough information to determine which features a model should rely on.
Valid approaches include modifying the training objective, injecting task-specific information (e.g. building-in invariance to rotations as 43) ). well-chosen data augmentations, or inhomogeneous training data such multiple training environments [7 or counterfactuals [4 [
7

---

Positive transfer
Underspecifiatioo
Misspccification
No transfer
Distribution shift
Mild
(Too?) Severe
Typical pattern (toy representation)
Positive correlation
Vertical linelno clear trend
Negative corretation
Low horizontal line
CO0
000
E Acusp
Accustoo
Ada oay
1D Agi
Valid approaches for improving OOD performance
Simply focus improving ID performance
Task relevant inductivve biases e arch., regularizers. Data augmentation with tusk-relevant transfornatoonss. Non-L.d.. training data e.g. multiple training domains.
Open question
Example datasels
ImageNet
PACS sketch photograph
WILDS Camelyon17
DomsinNet inligraph quickdraw
ImageNet v2 32
13
Figure 8: Overview of ID vs. OOD patterns occurring different levels of distribution shift.
With extreme distribution shifts, most predictive features are overwhelmingly spurious and it is very difficult to learn any one relevant in OOD data (rightmost case in Figure 8)
The proposed ordering patterns is rather informal and could be further developed following the two axes of diversity shifts and correlation shifts proposed by (58] (see also (52)) More recently, 48 showed that the suitability of various methods for OOD generalization depends on particularities of the underlying causal structure of the task which must therefore be known to select suitable method. Identifying which ID/OOD patterns occur with particular causal structures might serve as a tool to understand the type of OOD situation one is facing and identify a suitable method.
8 Revisiting advice from past studies
We have established that observations in past studies were incomplete. We now bring nuance to some recommendations and conclusions made in these studies.
Focusing on a single metric.
"We see the following potential prescriptive outcomes (...) correlation between OOD and ID performance can simplify model development since we can focus on a single metric."
25
We demonstrated that inverse correlations do occur, hence there exist scenarios where ID per-
formance is misleading. Relying on single metric for model development is ill-advised 45
especially it cannot capture necessary trade-offs. We recommend tracking multiple metries e.g. performance multiple distributions interpretable predictions representative test points.
Improving ID performance for OOD robustness.
"If practitioners want make the model more robust on OOD data, the main focus should be to improve the ID classification error. (...) We speculate that the risk of overfitting large pretrained models to the downstream test set minimal, and it seems to be not good strategy to, e.g., reduce the capacity of the model in the hope of better OOD generalization. " (50)
This recommendation assumes the persistence of positive correlation. On the opposite, we saw that a positive correlation can precede regime of inverse correlation (Figure 3 left panels). If the goal is to improve OOD, focusing on ID performance is blind alley since this goal requires to increase ID performance times and reduce it at others.
Future achievable OOD performance.
As obvious as it is, feels necessary to point out that empirical studies only chart regimes achievable with existing methods. Observations have limited predictive power, hence more care seems warranted when deriving prescriptive recommendations from empirical evidence.
The best possible performance e.g. on Camelyon17 is obviously not limited to the Pareto front of our experiments. The state of the art on this dataset [3 51 injects additional task knowledge to bypass the under/misspecifiatiion of ERM, and exceeds both our highest ID and OOD performance.
8

---

The important message remains that a given hypothesis class (DenseNet architecture in our case) admits parametrizations on which ID and OOD metrics do not necessarily correlate.
Possible invalidation of existing studies.
The possibility of inverse correlations may invalidate studies that implicitly assume positive one. For example, Angarano et al. [3 evaluate the OOD robustness of computer vision backbones. They find modern architectures surpass domain generalization methods. However, they discard any model with submaximal ID performance by performing "training domain validation" as in [0]. Any model with high OOD performance but suboptimal ID ignored. They also train every model for fixed, large number of epochs. And this may additionally prevent from finding models with high OOD performance since robustness often progressively lost during fine-tuning (2).
By design, this study 3 is incapable of finding OOD benefits of architectures or methods that require trading off some performance. Most importantly, once the assumption of positive correlation is enacted by throwing away models with submaximal ID performance, there is no more opportunity to demonstrate its validity.
An even more recent example of this fallacy is found in 21 The authors construct new OOD benchmark, train and tune baselines for maximum ID performance, then observe:
"We find no [positive inverse] correlation between in- and out-of-distributtion performance. All methods consistently achieve 98-99% ID test performance." " 21
environment
But the authors did not give the chance to make any other observation. As in Figure the ID
criterion means that we only get to observe a thin vertical slice of the ID vs. OOD plot
9 Discussion
This paper highlighted that inverse correlations between ID/OOD performance are possible, not only theoretically but also with real-world data. It is difficult to estimate how frequent this situation also showed that past studies may have systematically overlooked such cases. This suffices to show that cannot know priori where task falls on the spectrum of Figure g It is clearly ill-advised to blindly assume a positive correlation.
is. Although we examined a single counterexample,
Can we avoid inverse correlations with more training data? Scaling alone without data curation seems unlikely to prevent inverse correlations. [ examined more general question and determined that the impressive robustness of the large vision-and-langune -add--nnd-gggee model CLIP is determined by the distribution of its training data rather than its quantity. Similarly, inverse correlations stem from biases in the training distribution (e.g. class C₁ appearing more frequently with image background B, than any other). And biases distribution do not vanish with more i.i.d. samples. Indeed, more data can cover more of the support of the distribution, but this coverage will remain uneven, ic. biased. The problem can become one "subpopulation shift" 38 rather than distribution shift, but it remains similarly challenging.
Training full networks with a diversity-inducing method. We showed inverse correlations with standard ERM models and with linear classifiers trained with the best of our knowledge, this diversity method has not been applied deep models because of its computational expense. It would be interesting to confirm our observations on networks trained entirely with diversity-inducci methods.
method [46). To
Qualitative differences along the Pareto frontier. Besides quantitative performance, interpretability methods could examine whether various ID/ / /ODD trade-off models rely on different features and generalization strategies as done in NLP in 13
Model selection for OOD generalization has recently seen promising advances. get around selection based on either ID or OOD validation data with a tunable trade-off in their Quantile Risk Minimization method. And [48] examined existing approaches OOD generalization from their suitability to various distribution shifts and causal structures.
"Statements in [21] are f'actually correct but misleading. They are equivalent to "we did not detect X" but leave it to the reader to figure out "we designed our experiments such that there is way of detecting X".

---

References
[1] Ehsan Abbasnejad. Damien Teney, Amin Parvanch, Javen Shi, and Anton van den Hengel. Counterfactual vision and language learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[2] Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of-distribution robustness throughout fine-tuning, arXiv preprint arXiv:2106.15831/ 2021.
[3] Simone Angarano, Mauro Martini, Francesco Salvetti, Vittorio Mazzia, and Marcello Chiaberge. Back-to-bones: Rediscovering the role backbones in domain generalization. arXiv preprint arXiv:2209.0112,, 2022.
[4] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz, Invariant risk mini- mization. arXiv preprint arXiv: arXiv:/907.000 /te.//9977.999. 2019.
[5] Alexander D' D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew Hoffman, al. Under- specification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03395. 2020.
[6] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transfcrability of convolutional neural networks. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2021.
[7] Cian Eastwood, Alexander Robey, Shashank Singh, Julius von Kügelgen, Hamed Hassani, George Pappas, and Bernhard Schölkopf. Probable domain generalization via quantile risk minimization. arXiv preprint arXiv:2207.09944 2022.
[8] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). arXiv preprint 2rX1v:2205.01399, 2022.
[9] Tejas Gokhale, Swaroop Mishra, Man Luo, Bhavdeep Singh Sachdeva, and Chitta Baral. Generalized but not robust? comparing the effects of data modification methods out-of- domain generalization and adversarial robustness. arXiv preprint arXiv:2203.07653, 2022.
[10] Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In Proc. Int. Conf. Learn. Representations, 2021.
[11] Ari Heljakka, Martin Trapp, Juho Kannala, and Arno Solin. Representational multiplicity should be exposed, not eliminated. arXiv preprint trXiv:2206.08890 2022.
[12] Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive cesst-rrou--cccurre Reasoning, 2022.
Conference on Causal Learning and
[13] Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoe, and Naomi Saphra. Linear connectivity reveals generalization strategies. arXiv preprint arXiv:2205. 2022.
[14] Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: pointwise framework of learning. arXiv preprint arXiv:2202.09931, 2022.
[15] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. WILDS: A benchmark of in-the-wild distribution shifts. In Proc. Int. Conf. Mach. Learn., 2021.
[16] Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspecified data. arXiv preprint arXiv:2202.03418, 2022.
[17] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Deeper, broader and artier domain generalization. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 5542-5550, 2017.
10

---

[18] Weixin Liang, Yining Mao, Yongchan Kwon, Xinyu Yang, and James Zou. On the nonlinear correlation of ml performance across data subpopulations. In JCML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
[19] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In Proc. Ini. Conf. Mach. Learn., 2018.
[20] Shengchao Liu, Yingyu Liang, and Anthony Gitter. Loss-balanceds task weighting reduce negative transfer in multi-tase task learning. In Proc. Conf. AAAI, 2019.
[21] Aengus Lynch, Gbêtondji JS Dovonon, Jean Kaddour, and Ricardo Silva. Spawrious: A benchmark for fine control of spurious correlation biases. arXiv preprint arXiv:2303.05477 2023.
[22] Horia Mania and Suvrit Sra. Why do classifier accuracies show linear trends under distribution shift? arXiv preprint arXiv:2012.15483, 2020.
[23] R Thomas McCoy, Junghyun Min, and Tal Linzen. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv preprint arXiv; 1911
10911.2220 1.02969, 2019.
[24] John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Proc. Int. Conf. Mach. Learn., 2020.
[25] John P Miller, Rohan Taori, Aditi Raghumathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between -istiiibbiitinoonnnnn and in-distribution generalization. In Proc. Int. Conf. Mach. Learn., 2021.
[26] Hiroki Naganuma, Kartik Ahuja, Ioannis Mitliagkas, Shiro Takagi, Tetsuya Motokawa, Rio Yokota, Kohta Ishikawa, and Ikuro Sato. Empirical study on optimizer selection for out-of- distribution generalization. NewrIPS 2022 Workshop DistShift, 2022.
[27] Guillermo Ortiz-Jimenez, Itamar Franco Salazar-Reque, Apostolos Modas, Seyed-Mohsen MoosaviiDezzooll and Pascal Frossard. A neural anisotropic view of underspecification in deep learning. In Proc. Int. Conf. Learn. Representations, 2021.
[28] Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Pranceth Karimireddy. Agree to disagree: Diversity through disagreement for better transferability. arXiv preprint arXiv:2202.04414, 2022.
[29] Francesco Pinto, Philip HS Torr, and Puneet K Dokania. An impartial take to the CNN vs transformer robustmess contest. arXiv preprint arXiv:2207.11344 2022.
[30] Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Un- derstanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint a0Xiv:2002.10716, 2020.
[31] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthicu Cord, Léon Bottou, and David Lopez-Paz. Recycling diverse models for diitrr-distribution generalization. arXiv preprint arXiv:2212. 10445, 2022.
[32] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Proc. Int. Conf. Mach. Learn., 2019.
[33] Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Proc. Advances in Neural Inf. Process. Syst., 2021.
[34] Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimiza- tion. arXiv preprint arXiv:2010..5761,, 2020.
[35] Andrew Ross, Weiwei Pan, Leo Celi, and Finale Doshi- Velez. Ensembles of locally independent prediction models. In Proc. Conf. AAAI, 2020.

---

[36] Andrew Slavin Ross, Weiwei Pan, and Finale Vessi.Velez.. Learning qualitatively diverse and interpretable rules for classification. arXiv preprint arXiv:1806.087/6 2018.
[37] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint (11iv:991..0877 2019.
[38] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. BREEDS: Benchmarks for sub- population shift. arXiv preprint arXiv:2008. 204809990 2020.
[39] Luca Scimeca, Scong Joon Oh. Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shorteut cues will dans choose? study from the parameter-space perspective. In Proc. Int. Conf. Learn. Representations, 2022.
[40] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias neural networks. arXiv preprint arXiv:2006. 07710, 2020.
[41] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 2000.
[42] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Proc. Advances in Neural Inf. Process. Syst., 2020.
[43] Damien Teney and Martial Hebert.
Learning to extract motion from videos in convolutional neural networks. In Asian Conference on Computer Vision, pages 412-428. Springer, 2016.
[44] Damien Teney, Ehsan Abbasnedjad, and Anton van den Hengel. Learning what makes differ- ence from counterfactual examples and gradient supervision. arXiv preprint arXiv:2004.09034 2020.
[45] Damien Teney, Kushal Kafle, Robik Shrestha, Ehsan Abbasnejad, Christopher Kanan, and Anton van den Hengel. On the value of out-of-distribution testing: An example of Goodhart's law. In Proc. Advances in Neural Inf. Process. Syst., 2020.
[46] Damien Teney, Ehsan Abhasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior OOD generalization. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn,, 2022.
[47] Damien Teney, Maxime Peyrard, and Ehsan Abbasnejad. Predicting is not understanding: Recog- nizing and addressing underspecification in machine learning. arXiv preprint arXiv:2207.02598. 2022.
[48] Zihao Wang and Victor Veitch. A unified causal view of domain invariant representation learning. arXiv preprint arXiv:2208.00 06987, 2022.
[49] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. 2019.
[50] Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabrier Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of-distribution generalization in transfer learning. arXiv preprint arXiv:2207.09233: 2022.
[51] WILDS Leaderboard. https:://
swansorddorrroo adu/leaderboard)
[52] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuffi Ira Ktena, Taylan Cemgil, et al. A fine-grained analysis on distribution shift. arXiv preprint arXiv:2110.11322 2021.
[53] Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In-n-out: Pre-training and self-training using auxiliary information for -ut-of-distribution robustness. arXiv preprint arXiv:2012.045500, 2020.
[54] Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaudhuri. closer look at accuracy vs. robustness. Proc. Advances in Neural Inf. Process. Syst., 2020.
12

---

[55] Huaxiu Yao, Caroline Choi, Bochuan Cao. Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. Wild-time: benchmark of in-the-wild distribution shift over time. Proc. Advances Neural Inf. Process. Syst., 2022.
[56] Huaxiu Yao, Yu Wang. Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving Learn., 2022.
robustness via selective augmentation. In Proc. Int. Conf. Mach.
[57] Shingo Yashima, Teppei Suzuki, Kohta Ishikawa, Ikuro Sato, and Rei Kawakami. Feature space particle inference for neural network ensembles. arXiv preprint arXiv:2206.000 00944, 2022.
[58] Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-benchs Quantifying and understanding two dimensions of -ittt--distrbbution generalization. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2022.
[59] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. Proc. Int. Conf. Mach. Learn. 2019.
[60] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.0944 2017.
[61] Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In Proc. Int. Conf. Mach. Learn., 2022.
13

---

Appendix
A Additional results Camelyon17
Aumsay (5 vd.40
Ausmay (5, (0)
Aarmary 1% (4)
mar 1% na 4)
Figure 9: As in Figure we show that higher OOD accuracy can be sometimes be traded off for lower ID accuracy. Each panel shows results from different pretrained model (i.e. pretrained with different random seed). Each dot represents linear classifier re-trained on features from this pretrained model with standard ERM (red dots or with a diversity-induce method [46] (gray dots e). The latter set includes models with higher OOD/lower ID accuracies.
Aount whet
Acunte stott
Acunte whott
Adune stotf
Figure 10: Same as in Figure using val-ood instead of test-ood the OOD evaluation set.
14

---

Acunt val000
65
67
67
Acorncy 1% vel
Accentt (%. mkke)
Accunay 0% vel-d]
Acc 10y 7% valed)
Jcanay (0
Reasnn sioont
Aosia 0% w1oo))
Aousd (NN
04
Asuncy (9--1
Accurcy (h.
Accrncy valid)
val-d)
Figure 11: Same as in Figure 10 zoomed-in on ERM models (red dots
15

---

B Results on other datasets
B.1
CivilComments
Data. The CivilComments dataset 15 is another widely-used dataset in OOD research. It contains text comments from internet forums be classified toxic not. Fach example is labeled with topical attribute (e.g. Christian, male, LGBT, etc.) that spuriously associated with ground truth labels in the training data. The target metric is again worst-group accuracy, where group is any label/attribute combination.
We experiment two classes of architectures: pretrained BERT-tiny fine-tuned on CivilComments, and linear classifiers trained of features from the same frozen BERT-tiny models (a.k.a. linear probing).
Methods. We follow the same procedure described above.
Results. We observe Figures 12 13 different patterns with the two classes of architectures. With linear probing, the ID vs. OOD trade-off is minimal, and the model of highest ID) performance within a run as well as across methods very similar the model of highest OOD performance. With fine-tuning however, the trade-off is more pronounced. The ID and OOD performance usually peak then diminish at different epochs during the fine-tuning. This agrees with previous reports [2] of OOD robustness being progressively lost during fine-tuning.
72
2ao02-11 (0000 aoo
Epochs of best ID (sverage-group) accuracy
(60 Ao acouraa acok
Epochs best OOD (werst group) accuracy
T2
75
78
ID (average group) accuracy (%)
Figure 12: Results on CivilComments with fine-tuned BERT.
71
OOD (worst-gr group)
66
Epochs (averagn-group) accuracy
accuracy (%)
61
Epochs OOD (worst-group) group)
51
46
ID (average groopgrrppppp accuracy (%)
Figure 13: Results CivilComments with linear probing frozen BERT embeddings.
B.2
MadTTmme-MMMI--Raddaae
Data. The WildTime-MIMEN MMIMIC-Readmission [5 dataset contains hospital records (sequences of codes representing diagnoses and treatments) to be classified into two classes, corresponding to the readmission of the patient within a short time. ID and OOD splits contain records from different time periods.
Methods. We follow the same procedure as described above. We train a standard bag-of-embeddiing architecture, which associate each diagnosis/reeamment with a learned embedding, then summed and fed linear classifier. train this model with standard ERM, and with resampling to balance the classes in the training data [12. which is standard approach for imbalanced datasets. We also
16

---

train models with a "mild balancing", where classes are sampled according to distribution half-way between the original one the training data, and uniform (50%-50%) one.
Results. In Figure 14 we observe that ID and OOD performance are mostly positively correlated across methods. The best models obtained with uniform balancing of classes, in which case model selection based on OOD performance could give a small advantage, but it is marginal compared to the improvement over the ERM baseline, which can be clearly detected on both the ID and OOD performance.
59
0OD AUROC (%)
Baseline
Resampling
(mild balancing of classes)
55
Resampling
(perfectly uniform claoses)
Epochs best OOD
51
51
ID AUROC (%)
Figure 14: Results on WidTTie-MIMII-Reedaee
B.3
WildTime- Yearbook
Data. The WildTime-Yearbook [55 dataset contains yearbook portraits. Each image is be classified as male or female, and the ID and OOD splits contain images from different time periods.
Methods, We follow the same procedure described above. We train the simple CNN architecture described in (55). report in bothh the "average-group" accuracy (over the entire OOD test set) and the "worst-group" accuracy (where group is any 5-year period within the OOD test period.
The patterns are slightly different in the two cases but similar conclusions can be drawn from both. There is mostly-positiv correlation. but the highest accuracies (upper-right quadrant), some small trade-off exists. This suggests that fine-grained differences exist that are useful for either ID or OOD generalization, but not both. Although differences are small, this "pointy end" of the spectrum is where the state-of-the-art models compete, hence the relevance of this observation.
Results.
OOD Average-gee group accuracy (%)
66
OOD Worst-gn group acquracy Aoaaa 186)
72
Epochs of best ID accuracy
Epochs of best ID accuracy
82
68
Epochs of
Epochs accuracy
78
0OD
64
accuracy
74
60
97
98
99
100
97
98
99
100
ID accuracy (%)
ID accuracy (%)
Figure 15: Results on WildTime- HidTime--Yerbbok (left: average-group accuracy, right: worst-group accu- racy).
17

---

C Theoretical analysis
We present minimal case of trade-off between ID and OOD performance to aid understanding its cause and show it can occur even in a simple linear setting. Let E IR be the target variable predicted by the model and x the features used as input. Without loss of generality, we consider regression setting for mathematical convenience, though the results are valid for classification as well. The purpose of this analysis is to investigate theoretically how variations in certain input features correspond to changes in risk (i.e. expected loss) on OOD data. We consider the input features to be a concatenation of invariant and spurious elements: x [Finv ; Tspu] with Tinv E Room 61], we consider the simple gattrgennra process defined by the following structural equations: y" := 7 iin + Civv. and F = y'1* + a o Expu where E {eID, COOD) is an environment index referring to ID or OOD data, 1s is vector of ones and o denotes the element-wise product. The random variables Cluv and Cupu represent symmetric independent random noise with zero-mean, nub-Gaussian tail probabilities such that Var (Gmv) > O, Var(<spu.)) > 0, Vi [1, dapul. The vector Y E Ra determines the linear relation between the target variable and the invariant features, which is identical across environments. In contrast, the vector a acts on the spurious features and is anvironment-specifi. Invariant features are similarly predictive on ID and OOD data while spurious ones are not.
and sppu E R Following
Let us now study the relationship between ID and OOD performance of predictive model that relies on subset & of the features æ (not that the subset could be spurious or invariant). This subset is represented with a binary mask $ E [0,1] dur +oape, Suppose we have selected divv invariant features and dapu spurious features, with (dinv + depu) = d 11 [alla denote the selected features by Φ with [Biuv,1, ... Finv.dune: Fspu,,, ...: dppu To investigate the changes in model performance, we consider simple linear regression model. This enables us to theoretically measure the sensitivity of predictions when additional spurious features are added without complications from non-linearities. We denote with TD and ROOD the in- and out- of-domain expectation, and with 33 the optimal parameters of the linear regression for certain domain, i.e. 33- = T8 ())))))))))))))(())) T [x)) TE th (ax) v)- The MSE loss of the fitted linear regressor is then Ely - 2 (x(()) 3a]2. Given the feature mask L... we denote with LID ( and LOOD () the risk of the model on the in and out-of-domain distribution, respectively. Theorem T below (see complete form including assumptions and proof in Appendix [D examines how the ID and OOD risks of the model vary as additional spurious features included into o a to obtain 4 a-1-
l11 the number of selected features. We
Theorem 1. Under Assumption I and sufficient changes in a, including an additional spurious feature leads to:
LID CnD ( E 1 ( ED yy- i (2) LOOD ( (41) LOOD [oo (x(x) T 30OD12 gooD by 1 4+1 (x) T d+1
12 <
1+1
3000
> 0.
Theorem 1 shows that adding a spurious feature those used by the model can affect its ID and OOD losses in opposite directions, implying a trade-off between ID and OOD accuracy. In other words, this minimal case shows that simple model without/with extra (spurious) feature can exhibit an inverse correlation between its ID and OOD performance.
D Proof of Theorem I
Theorem 1. Including an additional spurious feature leads to the following change the risks:
(1 - (1)) = Ey - 1 (xx) 3 - EE'Dy 2 (x)
BDD 12 < 0
d41
LOOD 1 (1+1) (+g) = BOOD y 1 (x) BOOD12 FOOD y 4 (z) 10OD >
LOOD ( - ( Q + Q2 + Q3
18

---

with Q1, Q2. Qs defined as:
Q = E FOOD
ly
(
( Bo Ely + (a) 88
Q2 Σ (B (a)y)] in d)..d) (1 (D
2
pa gOOD
[B ooD ( ) (1 10DD
1O0D,
((a ((a 241 )2
(a )3)
Q: (E 24
0. d+1
d+1
(A 00D,d44 2+1
Further, if the new feature is sufficiently unstable in the test domain, i.e. if ((a 2+1 )2 - (a is sufficiently large such that:
(2 xo0nd41
(41)
(a (a'+1
)1
V (BoOD(D(2))) To u2 2o
1Q: 021
then have Q3 |Q + Q2 and therefore LOOD ) LOOD(Ha) 0.
Let x = 1 (.(mx)) Fiav,1, .... dnv 1s ... spu be the d features already selected,
2d+1 = 4 4 (mz) the features after adding new spurious feature sss to 2 j.
(X]. Xg. -.. xg) the eigenvalues of E[ar] mrg) and [uf, og og] the corresponding eigenvectors.
Assumption 1. The projection
of EE [ wi] on each basis corresponding to feature non zero, i.e.
E
a m > O, (eE cOOD). iE [d].
This ensures that coefficients of feature not be always 0, otherwise can simply remove it.
Proof. Let 3ID and good denote the solution of linear regression in the ID and OOD domains, i.e.,
3 arg min E (y 2: 22]
GOOD = arg min E OOD (y x: 3)2
(3)
g
Now let us compare the OOD loss after we include x s'sp,, practice, we can only obtain 3ID and then apply it on both the ID and OOD domains, which elicits the following errors:
=E (y - x 1 3DD) LID CID(2)) (2))
(4)
LOOD(Hg) E000 (y (DD
=E0OD (y 2 - FOOD (y - x] 3000
FOOD (y d
300n
(5)
It is well known that the residual of the linear fitting y by xa on the ID domain
LiD( = TD y xz (x [
'B 'B [aaa]]
E E t T ppp
(6)
Similarly, we have
LiD( (+1)
EDD
3+1
3
(7)
Since Sspu.d.p-. does not lies in the space spaned by æ; so the space spanned by 0 is strictly larger than æ. Together with Assumption 1, we have
LID(#T)) LID (+1) E.D xz
TgDD
B (V
x. T
3ID 0,
(8)
84
d+
19

---

and also
<2
ed+1 = BOOD ly 2 T @ - -EODD yy
d+1 d41--11 201
> 0.
(9)
By the proof in Appendix B.6.3 (above Eq. 29) in 61 we have
d
c Σ
(EOOD)
I uy OOD 03
FOD,d )
(10)
A P.Dia
By Eq. (20) in 161, we have
X AIDdd 1 foon, = (a! - (a: )
(11)
So have:
d
1+1
[ (LOOD)
[2 T To ) (2, (200
1
51
ID,à
X OOD
1=1
(OOOD
81 100 (1 (O0dd a 1+1
1 ID
00D
2 2 ((a)))) d+1 )² (a
(00)2))
O0OD
v d+1
d+1
(12)
DD 1/2 000,4+1
d+1
d+1
From Eq. 10 and 12 we have the desired result.
20


Reviews:
[{"review_id": "CWRRGOFiXH", "reviewer": "Reviewer_p2Jz", "summary": "The paper examines the previously proposed idea that in-distribution (ID) and out-of-distribution (OOD) performance of models are positively correlated. Through various experiments, the authors demonstrate that ID and OOD performance can in fact be inversely correlated. They also discuss how the previous work have missed this observation due to selection bias, and criticize the conclusions made by them. Moreover, a theoretical analysis on the linear case supplements the empirical findings regarding inverse correlation of ID and OOD performance.", "questions": "- To confidently recommend acceptance of this paper, I need to see the relevance of the theoretical analysis (section 6) to the experiments. In the previous section, I elaborated my recommendations to make this connection happen, and I am asking the authors to address my concern.\n\n- Is there a solid conclusion/suggestion made by the paper apart from the presence of inverse correlations between ID and OOD performance? Apparently, the paper suggests that contrary to the conclusions made by previous work, ID performance can be irrelevant to OOD performance, however, the authors do not provide an alternative guideline for practitioners to make claims about OOD performance. Such discussion would be very helpful for the audience.", "limitations": "Mentioned in the previous parts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Disclaimer: The paper is merely experimental as it is evident from its title. Before reading this report, I have not seen such activity in the OOD generalization literature; Note that benchmarks such as DomainBed [1] are fundamentally different as they do not provide suggestions for OOD generalization based on mere experiments. To understand OOD generalization, I believe the theoretical analysis must be the foundation, and ideally the results need to be backed with experiments. However, similar work have been approved in the literature; for example Wenzel et al. [2] which has motivated the empirical study of ID and OOD performance was published in NeurIPS 2022 despite making the false claims that are elaborated in this work. Therefore, I see the relevance and necessity of a rebuttal such as this work to correct the mistakes made by the previous work, and complement the existing literature. Only for this reason, I value the efforts made by the authors, and I recommend accepting the paper.\n\n- The paper addresses important shortcomings of the previous work in methods and evaluation. \n\n- The paper is well-written, and easy to understand for the audience who might be unfamiliar with the literature. The Reviewers’ FAQ in the appendix is very helpful.\n\n- The empirical evaluation is comprehensive.\n\n[1]  I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” 2020, arXiv:2007.01434.\n\n[2] Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of-distribution generalization in transfer learning. arXiv preprint arXiv:2207.09239, 2022.\n", "weaknesses": "- The premise of section 6 is to explain the inverse correlation witnessed by the experiments on the real-world datasets. To this end, the authors chose a linear model, and proved (Theorem 1) that adding a spurious feature decreases the OOD performance while it boosts the ID performance. It is unclear how this result corresponds to learning diverse models (section 3). In other words, how do the models learned in the experiments are related to adding a spurious feature to the feature set of a regression model? Such correspondence is necessary for section 6 to be part of a cohesive report. \n\n- The work is missing experiments on synthetic data to connect its theory and the real-world experiments. It would have been ideal if the authors used the same experimental method to sample from ID/OOD performance of models fitted to the data generated by the linear model described in section 6. Ideally, the plots in Figure 7 must be reproducible with the toy model and synthetic data for different instantiations of the linear model. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and suggestions.\n\n> I need to see the relevance of the theoretical analysis (section 6) to the experiments. In the previous section, I elaborated my recommendations to make this connection happen, and I am asking the authors to address my concern.\n\nThe theoretical possibility of an inverse ID/OOD correlation is quite direct as noted by reviewer zdij (\"*It is very easy to construct a toy example where ID and OOD are negatively correlated.*\"). Our theoretical analysis (section 6) shows how to construct such a toy example, in a (hopefully) didactic way. It shows how ID and OOD metrics of a model can diverge, as training progresses, or as one considers data with more and more spurious features.\n\n**The important open question was whether such cases are merely pathological or do appear in real-world data.** Prior work strongly argued for the former, but we show that the latter is actually correct. The key strength of this paper is therefore in the observations on actual data [*] and *not in a theoretical argument*. We propose to make this point much clearer in the text.\n\n[*] Simulations on synthetic data would only distract from this main message.\n\n> the authors do not provide an alternative guideline for practitioners to make claims about OOD performance\n\nThe takeaway is precisely that simplistic guidelines do not apply universally. Sec.9 does include actionnable takeaways (\"*Can we avoid inverse correlations*\") and promising research directions (\"*Model selection for OOD generalization*\")."}, {"review_id": "RvL2mIv1S3", "reviewer": "Reviewer_Q2JA", "summary": "Through extensive experiments, the authors showed that ID and OOD performances can inversely be correlated. The key ingredient is to track the performances throughout the training dynamics, instead of looking only at the end point.\n", "questions": "Has the authors conducted an experiment for two or more OOD data performances, for a single set of models trained on an ID data, where the OOD data can arguably be ordered in a way that becomes gradually more OOD to the ID training data? For example, one can train on CIFAR 10, and track its OOD performances on some selected OOD classes from CIFAR 100, and do the same on SVHN as a further away OOD dataset.\n\nI'm interested in the interplay between training dynamics and the more OOD axis.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "By tracking the training process, instead of the final point, the authors gave a more comprehensive view on the matter. This methodology should be more widely adopted, in my opinion.", "weaknesses": "While Theorem 1 demonstrates this diverse performance phenomenon can happen even to simple linear models, how NN does so was not analyzed. I'd like to see the authors analyze a shallow neural net. There could be broader and deeper theoretical insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and suggestions.\n\n> Q1. Theorem 1 demonstrates this diverse performance phenomenon can happen even to simple linear models, how NN does so was not analyzed. I'd like to see the authors analyze a shallow neural net.\n\nThe claims of this paper are much more fundamental: they are not restricted nor dependent of particularities of neural networks. The connection with training dynamics of NNs [1] would be interesting for future work. A relevant analysis of a shallow neural net appeared recently [2].\n\n[1] Pitfalls of Simplicity Bias in Neural Networks, Shah et al. 2020\\\n[2] Simplicity Bias in 1-Hidden Layer Neural Networks, Morwani et al. 2023\n\n> Has the authors conducted an experiment for two or more OOD data performances, for a single set of models trained on an ID data, where the OOD data can arguably be ordered in a way that becomes gradually more OOD to the ID training data?\n\nThis would be interesting indeed. The challenge is to obtain multiple test sets where the distribution shift can be quantified. The suggested ones would be problematic because of the different classes."}, {"review_id": "0bY0Tepead", "reviewer": "Reviewer_yjiA", "summary": "The paper presents an empirical examination of ID vs. OOD performance on several real-world datasets showing inverse correlation patterns can indeed arise in many datasets and experimental settings. This conflicts with past papers mainly observing a positive dependence structure. The paper further discusses why past studies missed such patterns and theoretically analyzes how inverse correlation patterns can arise.", "questions": "- Why was the diversification method not used in the second set of experiments?", "limitations": "- Although limitations of the approach are not explicitly discussed, current shortcomings and future work are discussed in section 8 and 9.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper highlights and important OOD generalization misconception, namely that increased model performance on in-distribution data is enough to achieve good OOD generalization. As the paper points out, it should be self-evident that this should not hold in general. Still, many works suggest that improving ID generalization helps OOD generalization. Model diversity seems to be a key factor that previous studies failed to account for. This is ensured via a training dynamics argument by examining model performance during a training run. I find this treatment convincing.\n- The provided experimental evidence shows that inverse correlations are possible under starker distribution shifts and that previous results on positive correlations were mainly obtained by a restrictive experimental setting. It makes sense that an inverse trend should arise at some point under misspecification.\n- The paper points out specific cases where the positive ID-OOD generalization insight has been observed but also where a negative relationship has been observed. This paper still seems to add value as in contextualizes all of these occurrences while explaining why a positive dependence structure might have appeared.\n- I also like that the paper contains a theoretical section showing that an inverse correlation is expected under a spurious feature. At the same time, I am not sure it adds that much value as it mainly relies on a formalization of the effect of spurious correlations which have been previously theoretically analyzed in the domain generalization literature.", "weaknesses": "- Although the paper does provide meaningful value in analyzing the ID-OOD tradeoff, the paper sometimes mentions this idea as a new insight. I am not sure this is necessarily true. I would make sure not to oversell the \"discovery of inverse ID-OOD tradeoffs\" but not lose focus on explaining how previous papers could have overlooked this possibility (which the paper already does for the most part but not everywhere).\n- I found point 4 in the description of Figure 2 confusing. The difference in the ID axis as compared with panel 2 does not appear to be large. 2 seems to show a more consistent variance while 4 seems to show a growing variance. Would it be appropriate to refer to theses as homoscedastic / heteroscedastic noise models, respectively?\n- Last point in 3.1 (lines 182-183) is missing references; it would be good to back up this claim.\n\n### Post-rebuttal\n\nI have read the author's rebuttal and found that my concerns were adequately addressed. I maintain my positive score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and suggestions.\n\n> I would make sure not to oversell the \"discovery of inverse ID-OOD tradeoffs\" but not lose focus on explaining how previous papers could have overlooked this possibility (which the paper already does for the most part but not everywhere).\n\nThis is a very valuable suggestion. We have edite the related statements throughout the paper to make this point unambiguous.\n\n> I found point 4 in the description of Figure 2 confusing. The difference in the ID axis as compared with panel 2 does not appear to be large. 2 seems to show a more consistent variance while 4 seems to show a growing variance. Would it be appropriate to refer to theses as homoscedastic / heteroscedastic noise models, respectively?\n\nThe comment is valid for these particular examples (reproduced from [48]) but the \"growing variance\" in panel 4 is not a key characteristic (the data points could be all over the place). We added a note in the caption to make this clear.\n\n\n> L182-183 is missing references\n\nWe added two supporting references about negative transfer in transfer learning:\\\n[1] Characterizing and Avoiding Negative Transfer, Wang et al. CVPR 2019\\\n[2] Loss-Balanced Task Weighting to Reduce Negative Transfer in Multi-Task Learning, Liu et al. AAAI 2019\n\n\n> Why was the diversification method not used in the second set of experiments?\n\nBecause reviewers of an earlier version of the paper complained that the first set of experiments relied on the diversification method. We added this new set of experiments to show that diversity could be induced from simple, standard variations in hyperparameters and training methods."}, {"review_id": "CuiFHssgyg", "reviewer": "Reviewer_3Cei", "summary": "The manuscript observe a phenomenon that ID and OOD performance can be inversely correlated because spurious correlations under the partially informative invariant feature (PIIF) assumption. The authors analyze the reasons of this missing phenomenon in previous studies, and conduct dataset-wise and shift-wise experiments. The paper provide a theoretical analysis and an OOD experiment guidance.", "questions": "I suggest extending section 6 and reorganizing the paper based on it.", "limitations": "No limitations, broader impacts, and licenses included.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The introduction of the ID/OOD inverse correlation is reasonable and easy-to-follow.\n- The experiments are comprehensive.\n- Theoretical analysis is equipped to provide insights.\n- (Update) The empirical results coherently reflect previous theoretical findings, which is worth spreading in the OOD community.", "weaknesses": "~~The most obvious weakness of this paper is that the analyses are mostly experiment-centric, which leads to farraginous explanations and insights. This organization makes readers hard to follow the logics and insights of this paper. The authors may strive to highlight the inverse correlation is \"surprising\" instead of trying to explain this phenomenon more natural and intuitive.~~\n\nThe overall idea can be clearer and more insightful if the authors use causality as the backbone. For example, according to section 6, the observed phenomenon is **natural** with the structural causal model analysis under the partially informative invariant feature (PIIF) assumption.\n\nOther weaknesses:\n- Line 193: The structural causal equation definition is not formal. Use \":=\" instead of \"=\".\n- Line 102: Link is not valid.\n\n~~Although I lean to accept, I'm not sure if this version of paper is enough beneficial for the research community.~~\n\nThe authors should revise their paper according to their proposal in the discussion period.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and suggestions.\n\n> The most obvious weakness of this paper is that the analyses are mostly experiment-centric\n\nOn the contrary, this is actually the key strength of the paper. The theoretical possibility of an inverse ID/OOD correlation is almost trivial (especially from a causal point of view, as rightly noted). The key strength of this paper is to show that this pattern is not only a worst-case pathological scenario, but does appear in multiple real-world datasets.\n\nThe suggestion to emphasize the theoretical argument is completely at odds with reviewer zdij, who noted: \"*It is very easy to construct a toy example where ID and OOD are negatively correlated.*\", even recommending to remove Sec.6 and emphasize the paper \"*as a large-scale empirical study that shows that the correlation between ID and OOD on real datasets can be anything*\"."}, {"review_id": "MLMDBAdFA5", "reviewer": "Reviewer_zdij", "summary": "This work performs an empirical study on the correlation between ID and OOD performances of models in vision and NLP. While some previous work claimed that there is usually a positive correlation between ID and OOD, this work argues that there are real datasets on which ID and OOD are negatively correlated, so ID should not be used as a proxy for OOD. The study was originally mainly done on Camelyon17, and this work also includes other datasets in Section 5, as well as similar observations in other papers. Section 6 provides a toy example where there is a trade-off between ID and OOD. Section 7 shows that the positive correlation occurs mostly in cases where the shift is mild, and with a larger shift the correlation could turn into a negative one. Section 8 points out several fallacies in previous work, including suggesting using the ID performance as a single metric, incorrect evaluation methodologies, etc.", "questions": "- In Section 7, it is demonstrated that when the distribution shift is mild, the correlation between ID and OOD is usually positive. Are there any real datasets where the shift is mild but there is still a negative correlation between ID and OOD? In many real applications, it is safe to assume that the shift from train to test is mild. And if under this assumption ID and OOD is usually positively correlated, then one could argue that ID could still be used as a proxy for these applications. Note that I am not asking whether it is *possible* for there to be a negative correlation - It is easy to construct such a toy example. I am asking whether the correlation is always observed to be positive on real datasets with mild shift.\n\n**Post rebuttal note:** I have read the rebuttal and had a discussion with the authors. I prefer to keep my score and vote for accepting this submission.", "limitations": "Limitations are discussed in Appendix A.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- As someone very familiar with this area, I am well aware of the \"accuracy-on-the-line\" folklore that ID and OOD are positively correlated, and it has been reported in a lot of previous work. So to me, this submission is a breath of fresh air. A lot of work on OOD draws conclusions based on pure empirical observations. This is understandable because the OOD problem is an ill-posed problem by nature. However, the caveat is that lots of follow-up work takes these conclusions for granted, proposes new methods based on them, and claims to achieve \"state of the art\" while their evaluation is usually flawed. Such an unscientific methodology has haunted the OOD community for quite a long time, and I believe that this work conveys a strong message that this needs to be changed.\n- This work is very clean and well written. The main argument of this work is by no means novel. In fact, in Section 5.1 the authors list previous papers that made similar arguments. Nevertheless, to my knowledge, this work is the first to convey this important message in such a clean way, and I believe that it can influence the perception and methodology of many researchers in the community.", "weaknesses": "- Section 6 is not very useful, and in my opinion, could be replaced with more experiments. It is very easy to construct a toy example where ID and OOD are negatively correlated. For example, if ID is a binary classification task, and OOD simply flips the labels, then of course they are negatively correlated. The reason why this \"accuracy-on-the-line\" has been there for so long is not because it is theoretically correct, but because it has been observed in many reports by different people. Everyone sufficiently familiar with OOD knows that there is no theoretical correlation between ID and OOD. I think this paper should be posed as a large-scale empirical study that shows that the correlation between ID and OOD on real datasets can be anything, so Section 6 is unnecessary.\n- Section 5 is too short and the experimental results are not well presented. I suggest the authors remove Section 6 and put more experimental results in Section 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review. We are grateful for your appreciation of the importance of this paper for the OOD community.\n\n> Q1. Section 6 is not very useful\n\nWe agree that the empirical results can stand on their own. But we are also fearful of readers dismissing the claims as \"hand-wavy\" unless a toy case is described in symbolic notation, as we do in Sec.6. We propose to move Sec.6 to the appendix, and instead expand Sec.5 to clarify the experimental details + include additional results currently in Appendix C.\n\n> Q2. In Section 7, it is demonstrated that when the distribution shift is mild, the correlation between ID and OOD is usually positive. Are there any real datasets where the shift is mild but there is still a negative correlation between ID and OOD?\n\nVery good question. We do not have such an example. We will however clarify in Sec.7 that this claim can only be guaranteed \"*in the limit*\" (i.i.d. setting)."}]

Author rebuttal:

Meta-review: All reviewers recognized that the paper makes a valuable empirical counterpoint to major narratives in the literature on robustness and the relationship between ID and OOD generalization on real-world distribution shifts. The literature review and experiments are comprehensive, and the paper's new results are well-contextualized with results that were previously reported.

Because the paper brings new empirical insights to a highly topical question, I am recommending it for a spotlight.
Decision: Accept (spotlight)