Spurious Feature Diversification Improves Out-of-distribution Generalization
Authors: LIN Yong
Lu Tan
Yifan HAO
Ho Nam Wong
Hanze Dong
WEIZHONG ZHANG
Yujiu Yang
Tong Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Out-of-distribution generalization
Primary area: general machine learning (i.e., none of the above)

Abstract: Generalization to out-of-distribution (OOD) data is a critical challenge in machine learning. Ensemble-based methods, like weight space ensembles that interpolate model parameters, have been shown to achieve superior OOD performance. However, the underlying mechanism for their effectiveness remains unclear. 

In this study, we closely examine WiSE-FT, a popular weight space ensemble method that interpolates between a pre-trained and a fine-tuned model. We observe an unexpected ``FalseFalseTrue" phenomenon, in which WiSE-FT successfully corrects many cases where each individual model makes incorrect predictions, which contributes significantly to its OOD effectiveness. To gain further insights, we conduct theoretical analysis in a multi-class setting with a large number of spurious features. Our analysis predicts the above phenomenon and it further shows that ensemble-based models reduce prediction errors in the OOD settings by utilizing a more diverse set of spurious features. Contrary to the conventional wisdom that focuses on learning invariant features for better OOD performance, our findings suggest that incorporating a large number of diverse spurious features weakens their individual contributions, leading to improved overall OOD generalization performance. Additionally, our findings provide the first explanation for the mysterious phenomenon of weight space ensembles outperforming output space ensembles in OOD. Empirically we demonstrate the effectiveness of utilizing diverse spurious features on a MultiColorMNIST dataset, and our experimental results are consistent with the theoretical analysis. 

Building upon the new theoretical insights into the efficacy of ensemble methods, we further identify an issue of WiSE-FT caused by the overconfidence of fine-tuned models in OOD situations. This overconfidence magnifies the fine-tuned model's incorrect prediction, leading to deteriorated OOD ensemble performance. To remedy this problem, we propose a novel method called BAlaNced averaGing (BANG) to mitigate the overconfidence problem, which significantly enhances the OOD performance of WiSE-FT.

Published as a conference paper at ICLR 2024
SPURIOUS FEATURE DIVERSIFICATION IMPROVES OUT-OF-DISTRIBUTION GENERALIZATION
Yong Lin*t
Lu Tan*
Yifan Hao*
Ho Nam Wong!
Hanze Dong!
Weizhong Zhang:
Yujiu Yang
Tong Zhang1
1
The Hong Kong University of Science and Technology, Tsinghua University, Uniiersity of Illinois Urbana-Chann Chanaa-gaapaign..
i Fudan University
ABSTRACT
(OOD) data is critical challenge in machine learning. massebble-based methods, like weight space ensembles that interpolate model parameters, have been shown to achieve superior OOD performance. How- ever, the underlying mechanism for their effectiveness remains unclear.
Generalization to
In this study, closely examine WiSE-FT, popular weight space ensemble method that interpolates between a pre-trained and a fine-tuned model. We ob- serve an unexpected "FalseFalseTTrue" phenomenon, in which WiSE-FT success- fully corrects many cases where each individual model makes incorrect predic- tions, which contributes significantly to its OOD effectivvness. To gain further in- sights, we conduct theoretical analysis in multi-class setting with large number of spurious features. Our analysis predicts the above phenomenon and it further shows that ensemble-based models reduce prediction errors in the OOD settings by utilizing more diverse set of spurious features. Contrary to the conventional wisdom that focuses on learning invariant features for better OOD performance, our findings suggest that incorporating a large number of diverse spurious fea- tures weakens their individual contributions, leading to improved overall OOD generalization performance. Additionally, our findings provide the first explana- tion for the mysterious phenomenon of weight space ensembles outperforming output space ensembles in OOD. Empirically we demonstrate the effectiveness utilizing diverse spurious features MultiColorMNIST dataset, and our exper- imental results are consistent with the theoretical analysis.
Building upon the new theoretical insights into the efficacy of ensemble methods, we further identify an issue of WiSE-FT caused by the overconfidence of fine- tuned models in OOD situations. This overconfidence magnifies the fine-tuned model's incorrect prediction, leading to deteriorated OOD ensemble performance. To remedy this problem, propose novel method called BAlaNced averaGing (BANG) to mitigate the overconfidence problem, which significantly enhances the OOD performance of WiSE-FT.
INTRODUCTION
Machine learning has seen significant advancements recently. However, the assumption that test- ing samples follow the same distribution as training samples, known the Identically Independent Distributed (IID) assumption, can be violated real-world applications. When a machine learn- ing model encounters novel testing samples that it hasn't seen during training, it faces the out-of- distribution (OOD) generalization problem.
Ensemble-based models (ESM) have achieved significant success in addressing OOD problems in recent years. Specifically, denote the input as x and the model as fo with parameter 8. Given two models fø and fg. existing ESM works typically consider the output space ensemble (OSE)
"Equal contribution. Corresponding L inccylinnf@ccommeuuuthhh

---

Published as a conference paper at ICLR 2024
Takknng
Labdy
Spakes Foster by
Spakes Festera Au
(x,))
(x.
f(
[0.4, 0,4,0.6,10 0] X
(LC.o))
[0.4, 0, 0.6] X
Dag
(C)
(
Ceed DVA. 1I
1/2f( - 1/2 f(
[0044 0.3, 0.3]
Figure I: Illustration of Falseral phenomenon. Consider to classify camels, cows, and dogs. The invariant feature v the shape of the animal. There are spurious features, i... 1) the background 2,,1, e.g., camels are always on the sand, cows are on grass and dogs are on the floor. 2) the fur of the animals 2, e.g., camels have brown fur, cows have dotted fur and dogs are all in black in the training dataset. Suppose we fit two models, f and f. on the training dataset independently. Assume that f uses the invariant feature V, and 5,,1, and f uses I, and E,,2. f and f both correctly predict the label of a sample from the training distribution. Consider an OOD testing sample of a dog with brown fur on the grass. f puts a large logit for the cow class since the background (grass) is spuriously correlated with cows, i.e., [( FF,1) ) = [0.4, 0.6, 0j. f puts a large logit for the camel class since the texture(brown fur) is spuriously correlated with camels, i.e., f 2222) = (0.4, 0, 0.6]. Both f and f make mistakes on this sample. However, the average of them can make correct prediction, i.e., 1 1/2 20.11) 1/2f(z,,, (2,22) 0..4,..3, 0.3).
which outputs fg(x:) (ar) and the weight space ensemble (WSE) which outputs f(a+ā) WSE is also called weight averaging in literature. Wortsman et (2022); Wortsman et al.) Rame et al. (222) show that ESM can significantly improve the OOD performance and WSE outperforms
OSE Many works, e.g.. Cha et al.] (2021); |Raee al. (2022): : Arpit et al. (2022); [ame et al.
Wortsman et al.] Tian et al. (223); Kumar et al. (2022). adopt WSE to repeatedly improve the SOTA performance on many OOD benchmarks such as DomainBed (Gulrajani & Lopez-Paz, 2020)
and ImageNet variants (Wortsman et al. 2022). See Appendix B.I for more related works.
Consider two types of features for OOD: (1) invariant features that consistently predict the label across distributions, and (2) spurious features that have unstable correlations with the label. Exist- ing OOD theories (Arjovsky et al.) 2019; Rosenteld et al., 2020 Wald et al. 2022] Ahuja et al.) 2020 Zhou al. 2022b) show that an ERM-trained model relying on spurious features can fail in worst-case ESM, which combines multiple ERM-trained models, may still heavily depend on such features and potentially fail in worst-case scenarios well. There have been some previous attempts to explain the effectiveness of model ensemble, but they do not offer satisfactory explanations on the overall OOD improvement of ESM. Furthermore, the difference between weight and output space ensemble remains under-explored (a thorough discussion on related works in Appendix B.2p.
An intriguing phenomenon. To understand the benefits of ESM, we examine the WiSE-FT (Worts- man et al., 2022), which interpolates between a pre-trained and fine-tuned model. When evaluating OOD datasets, we divided them into four groups based on the correctness of predictions made by the individual models. Surprisingly, we found a "FalseFalse True" phenomenon: WiSE-FT can correct predictions on samples where both individual models make incorrect predictions. Further, we show that two individual models learn different feature sets, and WiSE-FT utilizes more diverse features. Based on these observations, we then motivate our theory by a toy example (shown in Figure []. Suppose we have two models, and f. for a 3-class classification task. For a sample from the first class, if produces logits of (0.4, 0.6, 0), and produces logits of (0.4, O, 0.6). The ensemble model's prediction would be (0.4, 0.3, 0.3). This phenomenon can happen when f and f learn different sub- sets of spurious features, represented as and Š, respectively. Recall that the spurious correlations change in OOD. In the example, f generates high logit (0.6) for the second class influenced by S, while produces high logit (0.6) for the third class influenced by S (details in Section
A new perspective on OOD generalization. In Section we extend a popular theoretical setting (Rosenfeld et al.) 2020 Wald et al.. 2022) to a 3-class classification with multiple spurious features. Our theoretical results predicts the aforementioned phenomenon. We show that ESM incorporates more diverse spurious features, which weakens the contributions of individual spurious feature and
2

---

Published as a conference paper at ICLR 2024
further leads to improved overall OOD performance. We also shed light the difference between the weight and output space ensemble. Recall that there has been a significant effort in OOD com- munity to learn invariant features and discard spurious features (Arjovsky et al. 2019). However, these approaches have not shown satisfactory performance when applied to real-world datasets (Gul- rajani & Lopez-Pazi (220)) which may be due the fact that invariant learning requires numerous domains (Rosenfeld et al. (020)) strong regularization (Zhou et al. 2022b). and faces additional dif- ficulties induced by non-linearity (Rosenfeld et al., 2020), overparameterization (Lin et al. 2022a), and optimization challenges (Chen et al., (023c). In contrast, our findings offer a new perspective that spurious features diversification actually improves OOD performance, which can be easily implemented as shown in ensemble-base based models and has achieved remarkable empirical success. To further verify our findings, we introduce MultiColorMNIST in Section 3.4. novel variant of CM- NIST (Arjovsky et al.] (019), with multiple spurious features. Through empirical analysis, we show that individual models trained on MultiColorMNIST utilize different spurious features, and their en- semble achieves superior OOD performance by leveraging this diversity. Notably, while several methods promote feature diversity to enhance empirical performance, none of them have explored the spurious features diversification from a perspective similar to ours (details in Appendix [B2).
An improved method. Our theoretical results indicate that the scaling of f and f should be similar to maintain the improvement of the model ensemble. If f is much more confident than f. resulting in larger scaling for f, the ensemble model can become biased towards f. Unfortunately, the scaling issue arises in WiSE-FI, which combines a pre-trained model and a fine-tuned model in the weight space. Empirical evidence shows that the pre-trained model is well calibrated, whereas the fine-tuned model is highly over-confident on OOD datasets, indicating a larger scaling compared to the pre- trained model. Based on these findings, we propose BAlaNced averaGing (BANG), which combines the pre-trained model with model fine-tuned by over-confidence preventing methods like Label Smoothing and MixUp. We demonstrate that BANG improves vanilla WiSE-FT by approximately 1.9pp in average OOD performance across five ImageNet variants.
To summarize, the following are the main contributions of the paper:
By examining WiSE-FT, popular method of ensemble-based models (EBM) that com- bines the pre-trained and fine-tuned model in the weight space, we discover an unexpected 'FalseFalseTrue phenomenon that WiSE-FT can correct large fraction of OOD samples on which both individual models make wrong predictions. We further show that two indi- vidual models use different sets of features and WiSE-FT utilizes more diverse features.
Through theoretical analysis on a multi-class classification problem with multiple spurious features, we provide natural explanation for the observed phenomenon and show EBM can improve OOD performance through spurious features diversification. Additionally, our findings provide the first-ever explanation for the mysterious phenomenon of weight space ensembles outperforming output space ensembles in OOD scenarios.
Contrary to the traditional belief that emphasizes the exclusive learning of invariant features for OOD, our findings suggest that incorporating diverse spurious features weakens their individual contributions, leading improved overall OOD generalization performance. Through experiments on our MultiColorMNIST dataset, which contains multiple spurious features, we provide concrete evidence for the effectiveness of diverse spurious features.
Based on our theoretical and empirical findings, we show that WiSE-FT can suffer from the over-confidence problem of the fine-tuned model, which skews the ensemble and dete- riorates the OOD performance. We further propose a novel method BANG to remedy this problem, and it significantly improves the OOD performance.
2
UNDERSTANDING ENSEMBLE-BASED MODELS VIA EXAMINING WISE-FT
The FalseFalseTruu phenomenon. In this section, we closely examine WISE-FT WWorssman et al., 2022) to obtain intuition on why EBM can improve OOD performance. Specifically, (Wortsman
al. 2022) ensemble pre-trained CLIP and the model fine-tuned on ImageNet in the weight space.
In Appendix C.I we divide each dataset (ImageNet as ID dataset and five ImageNet variants [
'They are ImageNet-V2 V2
(Recht et al. 2019 ImageNet-R (Hendrycks et al. 102 ImageNet-
A (Hendrycks et al. 2021b). ImageNet Sketch (Wang al.] (2019) and ObjectNet (Barbu et al. 2019). We
refer them IN-V2, IN-R, IN-A, IN-S, and ObjNeet for short. More details in Appendix
3

---

Published as a conference paper at ICLR 2024
Ratio of FilseFo sellus Overa Improsummnn:
5.37
WOST rectoral Freeztt orggn
4.25
3,8
.06
Deplte
Figure 2: (Left) FalseFalse ratio; (Right) GradCAM feature visualization.
as OOD datasets) into 8 groups by whether the pre-trained, fine-tuned and averaged models make correct predictions. We surprisingly find that WiSE-FT can correct substantial part of samples on which both the pre-trained and fine-tuned models make mistakes. Specifically, we calculate the number of "FalscFalleeTrue" samples, i.e., samples on which WiSE-FT is correct while both the pre- trained and fine-tuned models are incorrect. We then calculate the mulseFalseTrue ratio by dividing FalseFalse number over the dataset size. Figure L[efft) shows FalseFalseTrre ratio on each OOD dataset and compares it with "overall improvement", which is the accuracy improvement of WiSE-FT over the best of pre-trained and fine-tuned model. We can see that there are substantial parts of FalscFalseTrue samples in each dataset. Refer Appendix C.I for more details. It is interesting that the FalseFalseTre ratio even higher than the overall improvement in IN-R and IN-A, we provide in-depth analysis and explanation in Appendix
and E.6
Illustration on when FalseFalse True occurs. In this part, we try to understand the FalseFalse True phenomenon. We first consider the output space ensemble to be similar to the weight space ensem- ble in this part and will present an analysis of their difference in Section 3 Suppose we want to distinguish from camels, cows, and dogs. There one invariant feature x, (the shape of the animal) and two spurious features (the background 1,,1 and the fur of the animal 22.2). Camels are typically found on sand, cows on grass, and dogs on the floor. Camels have brown fur, cows have dotted fur, and dogs are all black in the training dataset. See Fig. f illustration. Suppose we fit two different models, f and f on the training dataset. Further assume f uses the feature By and 2,,1, and f uses Ty and x,.2 Both f and f correctly predict samples from the training distribution. Whereas, for a sample from the testing distribution, e.g., a dog with brown fur (æx.2) the grass (æs.1): f puts large logit for the cow class since the background, grass, spuriously correlated with cow, i... f(xx, x.,1) = [0.4, 0.6, 0]: f puts large logit for the camel class since the texture, brown fur, is spuriously correlated with camel, i.e., f(x,; x2,2) [0.4, O, 0.6]. Both I and f make different mistakes under distributional shifts due to using different spurious features. However, the ensemble of them can make a correct prediction, i.e., 1/2f1 (I,, x2.1) + (22 22,2) [0.4, 0.3, 0.3).
Feature visualization. The reasoning above assumes that individual models utilize different fea- tures. GradCam (Selvaraju et al. 2016) visualization of the features used by the pre-trained (zero- shot), fine-tuned, and WISE-FT in Figure Riihht confirms this assumption. The visualization shows that the pre-trained and fine-tuned models rely different features, while WiSE-FT utilizes more diverse features. Additionally, (Allen-Zhu & Li, (020) provides empirical evidence support- ing the use of diverse features by different DNNs with the same architecture trained on the same datasets (with different initialization). They also provide formal thcoretical proof for 2-layer DNNs. We include some of (Allen-Zhr & Li 2020) 's empirical results in Appendix C.2) Additionally, there is more evidence suggesting that DNNs favor sparse feature representations and discard redundant
features (Papyan et al. 2020 Andriushchenko et al. 2023).
3 ANALYSIS ON SPURIOUS FEATURE DIVERSIFICATION
3.1 THEORETICAL SETTINGS
Notation. For simplicity of presentation, consider 3-class classification problem, i.e., y (e), e2. ea). where e; denotes the 3-dimensional unit vector with ith element equaling 1, e.g.,
E2 " In Appendix F.2 we extend the setting to K-class classification. a(k) means
For simplicity of illustration, assume that ff and f learn the same invariant feature. However, this is not necessary for FBM outperform both individual models, demonstrated in Section 3
4

---

Published as a conference paper at ICLR 2024
Dog
Dog
Deg
Dog
Caw
Camet
O
0
0
O
0
0
2a000
With Probabilly
1- 2p/3
pra
p/3
Mad
(81 Q. in distribttion
Fctst column of Q2 O00 distrbbtton
Figure 3: (a) Mag E Rd represents spurious feature, e.g., the background. Each column of p. is an attribute of the spurious feature, e.g., (1),(1), (s,;(2) and Psg are the floor, grass, and sand, respectively. (b) Qz.j E {0. 1|3x3 represents the relationship between labels and spurious features. In the ID distribution, Qu. equals I, indicating that each spurious feature is perfectly correlated with the corresponding class. (c) In the OOD distribution, spurious correlation can fail, e.g., Q. (1) equals e2 with probability p/3, indicating the background of the dog is the grass.
the ltth element of vector a, A(k) means the kth column of matrix A. We use I to represent K x K identity matrix, e.g., I3 [ei, e2, e3]. We omit the subscript of I when confusion arises.
Suppose we have d, invariant features { Id. and de spurious features { where Tu.i Fay E Rd and the whole feature x E Rdx(ds+do) is the concatenation of them, i.c. a = Concat( ( U {
[mo,,,.... Ln,d, 1..1,...., Fssd, Consider that each model f is composed of a featurizer E 10,1) and a classifier w E Rdx3 4 first [=), 02, ,2 my and + [, then σ = 21 + 222. Then the classifier ur E RRdx3 is fit based on the features selected by + as w arg min (e 2) arg min ((...)) [L(v' (x4), y)], where l is the cross- entropy loss function and Dia is the ID distribution. (Remark: Refer to Appendix D.I for detailed discussions on the setting.)
selects feature by æb. For example, suppose
Following (Rosenfeld et al. 2020 Wald et al. 2022). we consider that each Ee, and æet are gen- erated from the label y with the latent invariant features fuo and spurious features Me where Muji Maj E Rdx3 The full data generation process is:
Definition (Data Generation Process). The whole data generation process is as follows: y (ei, C2, e3} x = Concat {I 1 U (xr..
dd 1/=1)
(o y) =N Qu,y, where Qu,i, Qu.j {0, 133x3, Further, Qu = I3 = [er, e2, es) always hold. In the ID distribution Did Qu, I3: and in 00D Dood: the kth column of Q ie., (kkj is as follows for k 1,2,3:
o² La) Po Pe 18) - (No, O Vi,j. (1)
ex,, with probability 1 1-p Unifler, C2, ez). with probubility p.
(k.(k) =
The intuition of the data generation process. We consider the example in Figure Figure
the intuition of ps.j and Q..j- Suppose the spurious feature μa. is the background in Figure D
Here Ps.j = [ps., (1), ps,; Pss [3)] E Rdx3 and each column Pa (k) for k = 1, represents a specific attribute that is associated with class & in the training set. In other words, (,, (2), and ps,; represent attributes of background, namely, floor, grass, and sand, which are correlated with dog, cow, and camel, respectively. Consider dog image (i.e., y el [1. 0, 0, ). We have us. Qyly=ei
y=e, = us. Qa, and further
(a) In the ID distribution Did: (1) e, and Maj Qaoilu=er e = M (1). Then Vaj = N (M( af), indicating that in Dia the background of the dog (i.e., y = E1) is the floor (i.e., Ms. (1)).
(b) In the OOD distribution D Dood. (1,(1) = er with probability - and Q.y(1) ~ Unif(er, e2, Ea} with probability p. Then we have the following:
Ma. Vuyyly = (1), with probability 1-p (Unif((ss()) pa. (2), (3)}, with probability
"Specifically, Qyly=e; = Q[], Qc.(1), where Qa,(1) (1) is the first column of Q
5

---

Published as a conference paper at ICLR 2024
indicating that in the OOD distribution the background of the dog (i.e., y e1) is the floor (i.e., (1.((1) with probability 1-P P and is randomly drawn from floor, grass, and sand (i.e.. Ma,;(1,, Ms.((2), (2), and (3))(3) with p. In other words, p is the probability that spurious correlation no-longer holds and a larger indicates larger distributional shift.
Remark. Our data generation process extends the setting of (Wald et al. 2022 Rosenfeld et al.) 2020) to a 3-class classification problem with multiple features. This extension aligns with the intuition behind popular multi-class datasets used in empirical studies on OOD generalization, such as FulIColorMNIST,, ColoredObject, and CifarMNIST (Zhang et al.] 2021 Lin et al. [022a] /Zou jet al. 2022b a Ahmed et al. 2021). Take ColoredObject for example, correlations between classes and background colors exist in the training dataset but fail with certain probability in OOD,
Definition 2 (Individual models). Denote the whole invariant feature set as V < and
spurious feature set S == { -
Consider f (क) un) and f ( vb). Suppose & learns
V V and S c S, and & learns V V and S C S. Denote [V] Rly |S| n. [ - |S| ng In DI Rovo and IS no Specifically, we have x = Ex,ey EV F. + Ex,es Lz
arg min ugRuxa (v, b), and xΦ = Ex. Fy + Ex.es Lz v = arg min Ria "t,(0,1) .). Definition (Output space ensemble (OSE)). Given the two individual models defined Definition the prediction of the the output space ensemble is fose(x)) = 2w0 (2x)) + i u (xxb))
The predicted class of the sample (xt y) is the class with the maximum logit. Specifically, denote the logit as i = [ax). The predicted class is k = arg max i(h) where i(h) of the hth dimension of the logit L. The model makes correct prediction if I(eg y) holds where I is the indicator function. The accuracy is A(S) = E [ = y)]. We denote the OOD accuracy as
(f) = Eq. [Em.y = y) Qu]] where we use Q, as a short hand for Q..1..... Qs,d,- We discuss the metric in Appendix D.4 We defer the analysis of ID accuracy Appendix D.5 since consider infinite samples and the ID accuracy of all considered models are all close to
Assumption (Small Noise). Denote ní, and n', as the the maximum number of invariant features and spurious features that model can learn, respectively. We need the overall noise to be small to satisfy F (ort, (o())tny)) 2 1 1--E in which F is the cumulative distribution function of standard Gaussian random variable, and K refers to the class number (here we analyze the case K = 3).
Remark. Since we impose random noise on each feature, c.g., To,s My, + z where z (0, o² Ia) where Ia d-dimensional identity matrix and d >> d, - dg. natural to assume the overall noise is controlled, e.g., we have S 10-6 when K = 10, = 1/100, n', + +n', = 20. Assumption (Orthogonal features (Wald et al. [2022 Allen /hu Li 2020 (1) [1 = and | 5k)]l2 1 for i I.... du, (i, k) + (i', k)), k K' = 1,2,3, V., vy (Mo " undd, Ms,1. }.
de, K 1, 2, (2) v. (A) 1 by (K) for any
3.2 THEORETICAL RESULTS
We first show the intuition the simple Example[ then extend to the general setting in Def. 3 Example (Illustrative examples). Consider that there are totally 4 invariant features (Mo,)) and 6 spurious features {x.,} 1j=1. and two individual models (w. 5b) and (w, b) learn non- overlapped features as x = Σ Do,i + E5=1,2 Øs..j, and x = Ei=s, Duis Σj Dz.j.
Proposition 1 (Illustrative examples). Consider Example | suppose Assumption ① and 2 hhold, and there are infinite ID and OOD samples. Omitting small terms containing e, we have Aood
Aood( = -1p.. and Aood(fose) = 1 81
2p" 17p"
729
We can see that OSE improves OOD by Aood (fose) max Aood(f), (o) 1/81p3.
Intuition of the proof (Full proof in Appendix F.I. Let's consider the samples of first class = (1. + 1 Ms, Q..j (1) + where z ~ (0 Vo0 Ia). By Lemma 5] we have wkk) Ed I M (k) + Σ3 (ka(k) ()) for each class
e = 01, 0). Model (w. )) has 260 coly=ee
y=e, = Dim Mr, Qr.i (1)
k 1,2.3. Omitting the small noise term, the predicted logit for class k is un(k) T(xx)) w=

---

Published as a conference paper at ICLR 2024
Ei-1Me,(A) (M,,)Qu,((1)) Σÿ=1 (k)(k) (Ho.)Qa,, The model will mistakenly predict E2 on the samples with true label el when 20 x y=e, < ((2) w y=ee. This will hap- pen when the three events (1) for detailed discussion). Each event occurs with a probability of p/3, resulting
ez}} simultaneously happen in OOD (see Appendix D.7
combination probability of p² /27. This means that with probability of p²/27, we encounter an OOD sce- nario where the model f = (w,, b) incorrectly predicts almost all samples from the first class e the second class e2. This failure occurs because all three spurious features happen to have values that are spuriously correlated with E₂ in the training dataset. three spurious features dominate the prediction of e2, overshadowing the two invariant features that predict the true label e1. For the OSE model, have T (ik) (x0) + w(() (x+))y==,,
other words, the
(1)). The model will mistakenly predict e2 on the samples with true label e₁ when at least five of the six events {Q+,(1) eg ز simul- tancously happen in OOD (see Appendix [D6 for details), whose probability much less than that of f. Intuitively, the failure probability of the averaged model is smaller as it utilizes more spurious features, which are less likely make the same mistakes.
E' (k)) Qu.; Ej=1 (k) (..,)
Proposition 2 (General Results for OSE). Consider Definition Omitting small
7 3
Assumption
hold, and infinite ID and OOD samples. we have Aood(f)
constants involving C, and
((1--pp tae Fp.
Aood(f) Fp
(1-p)R, Vn,
Vn,
Aood(fose) Fp (1-plln.tn.)(0tttt, Vn, +20
Here (x(x)) cumulative density function (CDF)
parameterized by p as defined in Appendix F2
which is monotonically increasing with x as shown in Figure Aa). Suppose two individuals learns the same number of features with no-overlap, i.e., ñp
Poofaa
fh Ru. n. n. na and Nuo n1 O. we have Aood( = Fp (V26) and (o)
1 & RiD Windee000e
spurious F7
66)
(o) = (1) where / = (1 - p) + Vnn."
Figure 4: (a) Illustration of of F(r); (b)
indicating that sose is better than f since F(-) monotonically increasing.
Aood - (f) in Example
Example 2. Consider p = 0.9 and nwo individual models learn none overlapped, i.e., Dan = n0 fixing n. n. 20, and vary ñ. 0,1,.., 5 and it, 0, 1.., 20.
Figure b) illustrates (Joee) Aood (f) on Example 2 fose achieves better OOD perfor- mance than f in most cases. One exception that if f is much weaker than f. e.g. f learns invariant features but learns invariant features, the ensemble model fose is inferior than ʃ.
(Jose)
3.3 THE DIFFERENCE BETWEEN THE OUTPUT AND WEIGHT SPACE ENSEMBLE
It is an open problem on the difference between output space ensemble (OSE) and WSE (referred as OSE-WSE difference). Furthermore, the mysterious phenomenon of weight space ensembles outperforming output space ensembles in OOD scenarios has puzzled researchers (Wortsman al.,
2022 Wortsman et al. Rame et al. 2022). We shed light on this by our bilinear theoretical model 20:
Definition 4 (Weight space ensemble (WSE)) Given the two individual models defined in Definition the prediction of the WSE fwse(x)) = (w + T) (2( + &)).
In Appendix D.2 we show that the OSE-WSE difference in a 2-layer DNN is closely connected
with the OSE-W WSE difference captured by models in Definition 3
Proposition (General Results for WSE), Consider Definition
3
Assumption and in-
finite ID and OOD samples. Omitting small constants involving c, we have Aood
Fg (1 (1=p)( +2n.o) Vn, tn, 114gg
Comparing Proposition 2 and we can see that the only difference between (ood(fwse) and (ooe( is the number of overlapped invariant and spurious features learned by individual

---

Published as a conference paper at ICLR 2024
models, i.e., nuo and Roo- Specifically, when T and o selects overlapped features, fwse and fose makes the same prediction since œ - w and wh 1 w by Assumption 2 and further
(wo + +=)² (2=( + ()) x w xh + i xb. When there is overlapped features: (a) for WSE,
the coefficient of overlapped features is amplified by 2 in + क, and further amplified twice in aD + wt This results in coefficient of the overlapped feature becoming 4 in (w + " =( x). (b) for OSE, i.e., w x + i Φ& the coefficient of the overlapped feature 2. See Appendix D.7.1 for detailed discussion. In Appendix D.7.2 we provide conditions when swse outperforms ʃose. in addition with simulation results and supportive experiments. Our findings provide the first-ever explanation for the mysterious phenomenon of weight space ensembles outperforming output space ensembles in OOD.
3.4 EXPERIMENTAL VERIFICATION ON MuLTICOLORMNIST
Previous efforts in OOD community have focused on learning invariant features and discarding spu- rious features (Arjovsky et al. 2019). However, these approaches have not performed well on real- world datasets (Rosenfeld et al. 2020). This could be due to the requirements of invariant learning, such as the need for numerous domains (Rosenfeld et al.] 2020), strong regularization (Zhou al.) 2022b), and the challenges posed by non-linearity, overparameterrizaton, and optimization (Rosen- feld et al. 2020 Lin et al. 2022a Chen et al., 2023c). In contrast, our findings show that learning diverse spurious features also help with OOD generalization. This approach, shown in ensemble- based models, is easily implementable and has shown remarkable empirical success.
To further verify our findings, contruct MultiColorMMNIST. 10-class variant of CMNIST (Ar jovsky et al. 2019) with 32 spurious features, following Definition [ As shown Figure 5 each sample in MultiColorMNIST consists of 32 color patches, each serving train two neural networks, denoted as fo, and fe., with the same architecture but different initial- izations on MultiColorMNNSST. The results in Table show that the OSE model (fo, (w) fo, (x)) improve OOD performance over individual models CTo (æ) and Se, (x))). In Appendix C.3 (1) we show that each individual model learn subset of spurious features MultiColorMNISTT and OSE utilizes more diverse spurious features (2) we construct SingleColorMNIST with only one spurious feature and show OSE yields little performance gain since both individual models learn the same
spurious feature. We
spurious feature (similar to the results in Rame et al. (2022)
0.70
0.75
0.80
0.85
0.90
model model 2
71.05=11.04 71.77:40.94
60.07:11.04
48.57:1.92
36.9310.70
26.01:10.45
60.75+00.91
49.26-10883
37.74:20.66
26.63-10.42
model ensemble
78.64:10.73
67.61110.80
55.25:10.75
42.34:10.64
29.28:10.40
Table 1: OOD performance of (output space) model ensemble on Multi- Figure 5: A sample from ColorMNIST. The spurious correlation and - - in the training and MultiColorMNIST
testing set, respectively. A larger indicates larger distributional shift
4 BALANCED AVERAGING (BANG)
Our previous results show that EBM boost the OOD performance. An implicit requirement that the scaling of the two models should be roughly the same. If the two models have different scalings, e.g., one model is much more confident than the other, the EBM improvement is weakened.
Proposition 4 (Imbalanced scaling weakens WSE). Consider the Example
! Definition 4
Assumption 7 Consider WSE of two imbalanced models, f (w. +) and fx (Aw XT), where À Aood Ix>vs - -Aooe(Jwse) \==1
Specifically, swse(()) 0.25( + x ① Xτ). We have -34/729p².
See Appendix F3 for proofs and Appendix D.8 for an illustration of the over-confidence charac- terized by X. When À = 1, indicating similar confidence levels between f and fx. the WSE is balanced. However, when À > V5 and fx is significantly more confident than f. fwse becomes biased towards fx. resulting in a performance drop of over 34/729p3. Here we set À = v5 for illus- tration purposes and similar results can be similarly obtained for other À > 1. Unfortunately, we find
8

---

Published as a conference paper at ICLR 2024
Model Model Averaging
IN
IN-V2
ObjectNet
Ang
Zero shot Worisman et 2023
No
68.3
61.9
77.6
49.8
48.2
53,0
58.1
Fine- tuning Wortsman a 2i7e
No
81.3
70.9
65.6
36.7
46.3
49.6
53.8
Fine- (LS)
No
82.0
72.5
63.3
38.3
46.5
51.1
54.3
Fine Mixxu)
No
83.0
72.7
66.4
43.7
48.8
52.4
56.8
suning (Mixup LS)
No
82.9
65.8
48.5
WISE-FT Wortsman et al. 2022
Yes
81.7
72.8
78.7
52.2
53.9
57.3
63.0
BANGTLSS
Yes
82.1
73.3
78.2
55.2
53.7
58.9
63.9
BANG (Mixup)
Yes
81.5
73.0
79.5
57.9
54.5
58.7
64.7
BANG (Mixup LS)
Yes
81,6
73.1
79.7
56.8
58.9
64.9
Table 2: Results of fine-tuning CLIP VIT-B/16 on ImageNet. LS is short for Label Smoothing. The performance of the baseline methods are from the Table 8 of (Wortsman et al. (022).
WiSE-FT, which is the WSE of the pre-trained model (PM) and fine-tuned model (FM), suffers from the imbalanced confidence issue. Specifically, we compare the PM and FM on their confidence and accuracy. The confidence is defined as the largest probability that a model assigns to a class (details in Appendix E..).. Figure shows that the fine-tuned model is highly over-confident, especially on OOD datasets, c.g., ImageNetA have only 0.37 accuracy while the average confidence is over 0.7. Such overconfidence magnifies the FM's incorrect prediction, leading to deteriorate OOD ensemble performance (details in Appendix (.6).
direct fix to the issue of over-confidence is to tune the temperature of the softmax of the fine-tuned model (Kumar et al.) (022). However, this method can not be directly applied to WiSE-FT since WiSE-FT ensemble model weights instead of the outputs. Moreover, the tem- perature scaling tuned on the ID dataset (Kumar al.) 2022) fails to calibrate the fine-tuned model on OOD datasets, where over-confidenes is more severe (results of (Kumar et al., 2022) in Appendix E.6). Therefore, bel smoothing or Mixup during fine-tuning to prevent overconfidence and then average the pre- trained model with such fine-tuned model. (1) Label smoothing replaces the label of the true class (e.g. 1) with a positive value (e.g., 0.8), while distributing the smoothing parameter (e.g., 0.2) evenly among the other classes (Müller et al.) (019)). (2) Mixup (Zhang al.) (207)) gen- erates new samples during fine-tuning by linearly mixing pairs of training data and their labels.
propose BAlaNced averaGing (BANG), which adopt la-
The compatis on zuto shot and Statunel modal
We conduct experiments with CLIP (aT-oo et al. [021]. We impose Mixup or Label Smoothing dur- ing fine-tuning the pre-trained CLIP on ImageNet (IN), and test OOD performance on IN-V2, IN-R, IN-A, IN- S and ObjectNet. Following Wortsman et al.) (2022) BANG averages the pre-trained CLIP model and the model finetuned with LS and MixUp (details in Ap- pendix E.4]. The results in Table show that BANG ef- fectively improve the performance over WiSE-FT. Specif- ically, BANG(LS+MMxupp
53.
07
04
03
2.4
3.8
01a
where both IS and MixUp are adopted, achieves 1.9% higher average OOD aceu- racy than WiSE-FT. Further experimental results in the appendix show that Mixup and Label Smoothing can ef- fectively alleviate the over-confidence of the fine-tuned model on both ID and OOD datasets.
Figure 6: Comparison of confidence and accuracy between zero-shot and finetuned model. In the figure, v refers to IN, for IN-A, O for IN-R, for IN- S. O for IN and X for ObjNet.
Since Mixup and LS also improve the performance of the fine-tuned model, so curious reader would wonder whether the improvement of BANG comes from better calibration or just due to the improvement in the fine-tuned model. We conduct further investigation in Appendix [E.6 to confirm the contribution of better calibration: (1) Dividing the weight of the vanilla fine-tuned model by multiple scalars significantly enhances the performance of weight averaging, which nearly matches the performance of BANG. (2) BANG can correct sub- stantially more samples that is mis-classified by the fine-tuned model. We also show that BANG's effectiveness can not be explained by other data augmentation methods in Appendix E.5
ACKNOWLEDGEMENT
We are grateful for the insightful discussion with Yongqiang Chen, Damien Teney, Alexandra Rame, Pang Wei Koh, Mitchell Wortsman, Difan Zou, and Hao Wang. Thank you for your valuable inputs.

---

Published as a conference paper at ICLR 2024
REFERENCES
Faruk Ahmed, Yoshua Bengio, Harm Van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. In International Conference Learning Representations, 2021.
Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney. Empirical or invariant risk minimization? a sample complexity perspective. arXiv preprint arXiv:2010. 16412, 2020.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning, arXiv preprint arXiv:2012.09816, 2020.
Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learn- ing. pp. 903-925. PMLR, 2023.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.0289) 2019.
Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improv- ing model selection and boosting performance domain generalization. Advances in Neural Information Processing Systems, 35:8265-8277 2022.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Eric Bauer and Ron Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine learning, 36:105 1999.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- Food-101-ma 1od-10nmminin discriminative compo- nents with random forests. In Computer Pision-ECCY 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 461 Springer, 2014.
Leo Breiman. Bagging predictors. Machine learning, 24:123 140, 1996.
Rich Caruana, Alexandru Niculescu-Mizit, Geoff Crew, and Alex Ksikes. Ensemble selection from libraries of models. In Proceedings of the twenty-first international conference on Machine learn- ing. pp. 18, 2004.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Infor- mation Processing Systems, 34:22405-2224 22418, 2021.
Yimeng Chen, Tianyang Hu, Fengwei Zhou, Zhenguo Li, and Zhi-Ming Ma. Explore and exploit the diverse knowledge in model zoo for domain generalization. In International Conference on Machine Learning, pp. 4623-4640. PMLR, 2023a.
Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Towards under- standing feature learning in out-of-disttibution generalization. arXiv preprint arXiv:2304. 11327, 2023b.
Yongqiang Chen, Kaiwen Zhou, Yatao Bian, Binghui Xic. Bingzhe Wu, Yonggang Zhang, MA KAILI, Han Yang, Peilin Zhao, Bo Han, et al. Pareto invariant risk minimization: To- wards mitigating the optimization dilemma in duit----ist generalization. In The Eleventh International Conference on Learning Representations, 2023c.
Xu Chu, Yujie Jin, Wenwu Zhu, Yasha Wang, Xin Wang, Shanghang Zhang, and Hong Mei. Dna: Domain generalization with diversified neural averaging. In International Conference on Machine Learning, pp. 4010-4034. PMLR, 2022.
Mircea Cimpoi. Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEÉE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
10

---

Published as a conference paper at ICLR 2024
MMPieeTrain Contributors. Openmmlab's pre-training toolbox and benchmark. 2023.
https://
github. com/ wwweeenmmmaabmmpppceee
Yihe Deng, Yu Yang, Baharan Mirzasoleiman, and Quanquan Gu. Robust learning with progressive data expansion against spurious correlation. arXiv preprint arXiv:2306. 04949, 2023.
Thomas G Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems: First International Workshop, MCS 2000 Cagliari, Ital,, June 21-23, 2000 Proceedings 1, pp. 1-15. Springer, 2000.
Thomas G Dietterich et al. Ensemble learning. The handbook of brain theory and neural networks, 2(1):110-125, 2002.
Qishi Dong, Awais Muhammad, Fengwei Zhou, Chuanlong Xie, Tianyang Hu, Yongxin Yang, Sung- Ho Bac, and Zhenguo Li. Zood: Exploiting model zoo for out-of-distribttion generalization. Advances in Neural Information Processing Systems, 35:31583-31599 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19299, 2020.
Zhili Feng, Anna Bair, and J Zico Kolter. Leveraging multiple descriptive features for robust few- shot image learning. arXiv preprint arXiv:2307.04317, 2023.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259-3269. PMLR, 2020.
Yoav Freund and Robert E Schapire. A decision-theoretie generalization of on-line learning and an application to boosting. Journal of computer and system sciences, (5(1):113 199-199,,13,, 1997.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversart training of neural net- works. The journal of machine learning research, 17(1):2096-23333 2016.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 2(11):665-66 166,6666633, 2020.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distributiin generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021a.
Dan Hendrycks. Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262, 2021b.
Saachi Jain, Dimitris Tsipras, and Aleksander Madry. Combining diverse feature priors. In Interna- tional Conference on Machine Learning, pp. 9802-9832. PMLR, 2022.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.0200 2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554 561, 2013.
Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles mit- igate accuracy tradeoffs under distribution shift. Uncertainty in Artificial Intelligence, pp- 1041-1051. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2024
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference computer vision (ECCV), pp. 624-639, 2018.
Weixin Liang, Yining Mao, Yongchan Kwon, Xinyu Yang, and James Zou. Accuracy the curve: On the nonlinear correlation of ml performance between data subpopulations. 2023.
Yong Lin, Hanze Dong. Hao Wang, and Tong Zhang. Bayesian invariant risk minimization. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16021-
16000, 2022a.
Yong Lin, Shengyu Zhu, Lu Tan, and Peng Cui. Zin: When and how to learn invariance without environment partition? Advances in Neural Information Processing Systems, 35:24529-24542, 2022b.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distributtin and in-distribution generalization. In International Conference Machine Learning, pp. 7721-7735, PMLR, 2021.
Rafael Müller, Simon Korblithh, and Geoffrey E Hinton. When does label smoothing help? Ad- vances in neural information processing systems, 32, 2019.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? Advances in neural information processing systems, 33:512-523, 2020.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24665 2020.
Seo Yeon Park and Cornelia Caragea. On the calibration of pre-trained language models using mixup guided by area under the margin and saliency. arXiv preprint arXiv:2203.07553 2022.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identificaiion and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947-1012, 2016.
Aahlad Puli, Lily H Zhang, Eric K Oermann, and Rajesh Ranganath. tion in the presence of nuisance-induced spurious correlations. arXiv preprint arXiv:2107.00520 2021.
generaliza-
Shikai Qiu, Andres Potapezynski, Pavel Izmailov, and Andrew Gordon Wilson. Simple and fast group robustness by automatic feature reweighting, arXiv preprint arXiv:2306.11074 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Leon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distrriution generalization.
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Recycling diverse models for out-of-distribution generalization. arXiv preprint arXiv:2212.10445. 2022.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Galli- nari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization, arXiv preprint arXiv:2205.098 09739.. 2022.
12

---

Published as a conference paper at ICLR 2024
Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Pazz Pazz Model ratatouille: Recycling diverse models for out-of--istributie generalization. 2023.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761, 2020.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202.06856, 2022.
Robert E Schapire. The strength of weak learnability. Machine learning, 5:197-227, 1990.
Robert E Schapire. The boosting approach to machine learning: An overview. Nonlinear estimation and classification, pp. 149-171, 2003.
Robert E Schapire. Explaining adaboost. In Empirical Inference: Festschrift in Honor of Vladimir N. Vapnik, pp. 37-52. Springer, 2013.
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450 2016.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pp. 443-450. Springer, 2016.
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton Van den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16761- 16772, 2022.
Junjiao Tian, Xiaoliang Dai, Chih-Yao Ma, Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning, arXiv preprint arXiv:2303. 2000 2023.
Yoav Wald, Gal Yona, Uri Shalit, and Yair Carmon. Malign overfitting: Interpolation and invari- ance are fundamentally at odds. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference Machine Learning, pp. 23965-23999 PMLR.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7971, 2022.
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key to -fiss---iittittttionn generalization? International Conference on Machine PMLR, 2021.
Learning. pp.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
Jianyu Zhang and Léon Bottou. Learning useful representations for shifting tasks and distributions. In International Conference on Machine Learning, pp. 40830-40850. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2024
Jianyu Zhang, David Lopez-Pas Paz, and Léon Bottou. Rich feature construction for the optimization- generalization dilemma. In International Conference on Machine Learning, pp. PMLR. 2022.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6))1452-1466 2017.
Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang. Model agnostic sample reweighting for out-of-distributio learning. In International Conference Ma- chine Learning, pp. 27203-27221. PMLR, 2022a.
Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In International Conference Machine Learning, pp. 27222-27244. PMLR, 2022b.
14

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
2
Understanding Ensemble-based Models via Examining WiSE-FT
3
Analysis on Spurious Feature Diversification
3.1
Theoretical Settings
Theoretical Results
3.3
The difference between the output and weight space ensemble
3.4
Experimental Verification on MuliColorMNNST
BAlaNced averaGing (BANG)
Social Impact
17
B Related Works
17
B. A review on the existing methods
17
B.2 On difference with existing works
18
Supportive Empirical Results for the Theory
20
Talsefalsseruee Phenomenon
20
Deep neural networks learn different features
21
Experiments on MultiColorMNIST.
C.3.1
Increasing the Number of Ensemble
24
Simulation
25
D
Discussions, illustrations, and supportive results for the theoretical parts.
26
D.I
Discussion on the theoretical models
26
D.2
Comparison on our model with 2-layer DNN
26
D.3 Illustration of the transformation matrix Q
28
D.4
On the pessimism of worst-case theoretical analysis for OOD
28
D.5
ID performance
28
D.6
Intuition of OOD Performance Improvement OSE
28
D.7
The Difference Between WSE and OSE in OOD
30
D.7.1
Explaining the difference between WSE and OSE
30
D.7.2
The theoretical condition of WSE outperforming OSE
31
D.7.3 Empirical Verilication
32
D.8
Illustrating the over-contidence.
33
More experimental details and results on BANG
33
F.
Details ImageNet Variants
33
15

---

Published as a conference paper at ICLR 2024
E.2
Details of Places365, StanfordCars, DTD and Food101 (PSDF)
34
Details on calculating the confidence
34
B,4
Experimental Details
34
More Results on BANG and Discussions
36
E.6
WiSE-FT benefits significantly from better calibration
37
Proofs
39
Proof of Proposition
39
................................
Proof of Proposition
F2.1
Proof for Single Model
42
F222
Proof for Weight Space Ensemble
44
F2..
Proof for Output Space Ensemble
46
F2.44
Case Study for
48
F2.55
Close Form of (p(.)
49
F22.6
Close Form of Gl Day Dsy Doo Do
49
Proof of Proposition
50
4
Proof of Proposition
51
F.5
Auxiliary Lemmas
53
Illustrating the Theory of WiSE-FT
55
H
Musttrtinn the Effectiveness of BANG Through the Lens "Accuracy the Curve"
56
16

---

Published as a conference paper at ICLR 2024
A SOCIAL IMPACT
We investigate how to enable machine learning models to generalize in OOD scenarios, which makes machine learning models more reliable in real-world applications.
B RELATED WORKS
B.1 A REVIEW ON THE EXISTING METHODS
cutrof-distribution generalization Machine learning models are based on the L.D. (indepen- dently and identically distribution) assumption. Whereas, the I.I., assumption can be easily vio- lated since the model can easily encounter novel testing samples that are from distributions different with the training distribution. This is also known as the out-of-distribution generalization (OOD) problem. Existing works find that the model performance deteriorates dramatically under distribu- tional shift. This is especially the case when the model rely on spurious features that are unstable in new domain (Geirhos et al. 2020 Arjovsky et al.) 2019 Deng et al.) 2023). OOD problem has attracted great attention in recent years and there are a rich line of works in this direction, such
as Invariant Risk Minimization (IRM) (Arjovsky al. 2019 Lin al. 2022b), model averaging
[Wortsman et al. (022 Ramé et al.) 2022 Cha et al. [022]) I. feature alignment methods (Ganin et al. 2018) and so on.
2016 Sun & Saenko 2016 Li al.
Among them, Invariant Risk Minimization (IRM) has gained significant attention from the re- searchers (Arjovsky et al. 2019) and inspires a great line of works. Recall that there are two kinds of features for OOD generalization: invariant features that can stably predict the labels, and spu- rious features whose correlation with the labels is unstable. IRM tries to build robust models by extracting only invariant features. IRM has strong theoretical guarantees in linear system and clear connection with causal theory. Nevertheless, IRM methods face challenges in dealing with large scale real-world datasets, it has been repeatedly observed that IRM cannot outperform ERM on various datasets. Some works have provided explanation for this, e.g., (Rosenfeld et al. 2020) shows that IRM needs very great number of domains, (Rosenfeld et al. 2020) shows IRM lacks theretical guarantees on non- non-linear models, Lin et al. (2022b) shows it is difficult to learn invariance without domain partition and (Lin et al.] 2022a Chen et al. (023c) show the difficulty of optimizing IRM objects on deep neural networks. In contrast, model averaging is exceptionally powerful and achieve SOTA performance in a lot of benchmark with various models and architecture (Cha et al. 2021
Wortsman et al. 2022 Wortsman et al. Rame et al. 2022 Chu et al. 2002: Arpit et al. 2022
Output and weight space ensemble The ensemble of multiple models is powerful idea that often leads to stronger predictive performance (Caruana et al. 2004) [Detterich] 2000) Bauer & Kohavil 1999; |reiman (996). "ypically, conventional ensemble methods aggregate the outputs of models, as known as the output space ensemnle. The recent application usually average the parameters of models which is generated from the same pre-training model by finetuning (Wortsman ct al.) 2022; Cha et al. 202T), also known as the weight space ensemble. While averaging two models trained from scratch by different initialization often yields poor results (close to random guessing). (Neyshabur et al. 2020) finds that fine-tuning two models from the same pre-trained initialization results in two different models that were connected via linear path in weight-space, along which the performance remains high. This is also known as linear mode connectivity (Frankle) al. (2020). A notable difference between ensemble in ID and OOD study is that the improvement of ensemble in OOD is much more significant than that in IID. (Wortsman et al.] 2022) shows that an ensemble the finetuned model with the pre-trained model improve near Ipp in ImageNet (the ID domain) and over 6-8pp on the variants of ImageNet (the OOD Domain). Actually, model averaging is still among strongest methods for OOD generalization. It still remains mysterious on why averaging methods are so effective for OOD.
Theory of Out-of cittof-ddittiibuionn generalization. Existing theory mostly focus on the worst case metric on analyzing the OOD performance (Wald et al.) 2022 Arjovsky et al. 2019 Rosenfeld et al. 2020 Puli al. 1021 Zhou et al., 2022b). The worst case metric requires model to be robust at any OOD testing distribution. Typically, a model that only uses invariant features can minimize the worst case metric. However, we discuss above, invariance learning is hard in
17

---

Published as a conference paper at ICLR 2024
practice and performs ineffectively real world datasets (Gulrajani & Lopez- Paz 2020 Rosenfeld et al. 2020; Lin et al., 2022b). The worst case metric based theory can not explain the success of model averaging methods. To be specific, the averaging of two models can spurious features that learnt by each individual model due to its pessimism as described in Appendix (D.4] In contrast, our theoretical results characterize the probability of the model failure due to distributional shift, which can successfully explain the experimental results.
B.2 ON OUR DIFFERENCE WITH EXISTING WORKS
Difference with existing works on learning diverse features. There have been some works that promote feature diversity to enhance empirical performance OOD generalization by weight aver-
age (Chu et al. 2022 Rame et al., 2022; 2023). feature concatenation (hhagg & BBttoo 2023
boosted rich feature learning (Zhang et al.) 2022 Chen et al., 2023b; Jain al. [0022] Teney al.;
2022
Feng et al. (023), , and utilizing model zoo (Dong et al. 2022 Chen et al. (023a). Teney et al.
2022 train a set of diverse models and select the best one among them for OOD. Feng et al. (2023)
proposes to use an ensemble of prompts which contains diverse descriptions of class to perform
classification via CLIP. Jain et al.] (2022) train two models with different feature priors and then ensemble the predictions of these models. However, existing explanations either do not distinguish
the invariant or spurious features (Chu et al. 2022 Rame et al. 2022] 2023 Zhang & Bottou 2023) Dong et al., 2022 Chen et al. 2023a), foeus only on learning the potentially missing invariant features (Chen et al., 2023b Feng et al. (202)) In fact, according to existing invariance learning perspective Arjovsky et al. (019)) arguing that models relying on spurious features are prone to failure in OOD scenarios, these methods that learn diverse features while also incorporating spu- rious features may not be able generalize effectively under distributional shift. In contrast, our spurious feature diversification viewpoint provides a explanation by characterizing why and when incorporating more diverse spurious feature diversification can improve OOD performance.
Difference with the existing theoretical results on ensemble and boosting in IID settings. There are existing explanations for the effectiveness of model ensemble in the IID setting, which mainly from the perspective of variance due over-fitting the label noise in finite samples cases (Dietterich et al. 2002). Specifically, model ensemble can have smaller variance in prediction com- pared with each single model. Whereas, we consider infinite sample case, where the variance of the model due to fitting label noise is zero. So model ensemble can not bring significant IID improve- ment in this case. However, the model trained on infinite samples can still fail due to distributional shift (Arjovsky et al. 2019). This is because the model utilizes the spurious features, which are also considered as a kind of bias (Wald et al. 2022). Our results show that model ensemble can reduce the risk of model failure and lead to better expected performance under distributional shift by spurious feature diversification. In other words, model ensemble reduces the probability of the model failure due the bias. This is new result in the OOD problem as shown in Proposition and 2 Notably, Allen-Zhu & Li (2020) also considers ensemble in the IID setting, however, their theory can not explain the OOD performance improvement of ESM models on the data in Defini- explain the FalseFalseTrree phenomenon, or explain the difference of weight and output space ensemble.
tion I
Another related area to this work is boosting. Boosting can benefit by training multiple models, where each model corrects the mistakes made by the previous ones and each model would possibly utilize on different subsets of features. While previous studies on boosting mainly focused on ID scenarios (Schapire) 1990 |Freund & Schapire 1997, Schapire| 2013] 2003), we show that in the context of OOD, the improvement in performance due to using diverse features can be even more significant. This is because different irrelevant features can cause different errors when the distri- bution changes, and diversifying the features helps reduce the impact of each individual feature (as shown in Figure 1). By utilizing a diverse set of models, boosting allows us to take advantage of a wider range of features and effectively deal with the challenges posed by OOD situations.
Difference with existing explanations on the OOD performance of ensemble- based methods (EBM). There are some previous attempts that try to explain the effectiveness of EBM for OOD. Cha et al. (2021) shows that the loss landscape changes under distributional shift and model averag- ing can lead to flatter minima. However, as discussed in Rame et al. (2022), the upper bound of et al. 2021) is uncontrolled and their analysis based on flat minima fails explain many experi-
18

---

Published as a conference paper at ICLR 2024
mental results. Rame et al. (2022) decomposes the OOD loss into the bias, variance and covariance terms. They show that the variance term can benefit from EBM. Different from the results of Rame et al. (2022) that only tackles with the variance term, our results provide concise characterization on the overall OOD performance. Further, |Raee et al. (2022)) 's results can not differentiate between the weight and output space ensemble.
19

---

Published as a conference paper at ICLR 2024
C SUPPORTIVE EMPIRICAL RESULTS FOR THE THEORY
C.I. FALSEFALSETRUE PHENOMENON
In this subsection, take deeper look WiSE-FT (Wortsman et al. 2022 popular model averaging method that averages the weights of the pre-trained and fine-tuned model. (Wortsman al., 2022) obtains the fine-tuned model by fine-tuning the pre-trained CLIP model on ImageNet. They have shown that the averaging of the pre-trained and the fine-tuned model can outweigh both of them on ImageNet (ID dataset) as well as five OOD datasets (ImageNetV2, ImageNetA, ImageNetR, ImageNetSketch and ObjectNet). We denote the pre- treitra model as PM, fine-tuned model as FM, and averaged model as AM.
To understand why model averaging is effective, we divide each dataset into eight groups of samples according to whether the PM, FM and AM make correct predictions, respectively. We further use T/F to denote whether a model makes correct predictions, i.e., T for True and F for False. For example, we use PM(T)-FMED)AMCT) to denote the group of samples on which the predictions of PM. FM and AM are correct, wrong, and correct, respectively. A simple explanation for the improvement of the averaging of two models is that when one model makes a mistake and the other one is correct, the correct model can rectify the mistakes made by the other model. So we evaluate the performance on the group of data where one model makes a wrong prediction while the other model makes correct prediction, i.e., the group containing PM(T)-FM(() and PM(F)-FM(T). We refer to this group of data as TF+FT for short in the following discussion. We also look into another subset TT+FF which contains PM(T)-FM(() and PM(F)-FM(F).
Given a subset G of a dataset D, we use CorrectNum(G: f) to denote the number samples in G that are correctly predicted by model f. e.g.. CorrectNum (FF+TT; (TFrr PM) stands for the num-
ber of samples that are correctly classified by the pre-trained model PM. We propose the metric ImproveContri(9) which estimates how much AM performs better than PM and FM on the group g and how much the improvement on G contributes to the overall accuracy improvement on the whole dataset D:
ImproveContri()) CorrectNum(G; AM) - max Maaxx(CorecNumm(:PPM) CorrectNuG;: FM)
(2)
[D]
For Example, suppose D> contains 1,000 samples and its subset G contains 200 samples. PM.
FM and AM correctly predict 120, 118, 130 samples in G. i.e., CorrectNum((? (G: PM) = 120,
CorrectNum((;: FM) = 118, CorrectNum(G: PM) 130. AM outperform PM and FM by mak- ing 10 more correct predictions on G. further these 10 samples contribute to 1,000 X 100% = 1.0%
10
accuracy improvement on the whole dataset. Note that ImproveContr((2) denotes the accuracy improvement of model averaging on the dataset D, which is also denoted as ImproveContriAALL) in the following discussion. The results of ImproveConti(TT-FFF) ImproveContr((TFFFFT and ImproveContri(LLL) are illustrated in Figure a
We surprisingly find that ImproveContri([] () is significant on TT+FF in all the datasets, which means the averaged model AM can exceed PM and FM on the groups where PM and FM are both right or wrong. Recall that the subset TT+ contains four groups PM(T)-FM(T)-AMCT) AM(F), (M(F))FM(F)-FM(F)) and PM(F))FM(P))PM(C)) We further plot the ratio of the sample size in PM(T)-FM(T)-AN find that PM(T)-FM(T)-MMIT) is nearly the same (about 0.5% ) in all datasets. The group PM(F)- FM(T)-AM(T) is much larger than (M(T)-FM(T)-AM(F), especially in OOD datasets. It indicates that AM can make correct predictions on many samples where the both PM and FM make wrong predictions when distributional shift occurs! Interestingly, we find ImproveContri(TF + FT) is negative on some datasets, e.g, IN-R and IN-S. In Section 4 and Appendix E.6 we find that this is because the fine-tuned model is highly over-confident and the fine-tuned model dominate WISE-FT even when it make mistakes.
PM(T)-FM(T>-
and FM(F))TM(E)))M(TT over [g] in Figure 7bb), respectively. We
In Figure Figure[ of Section 2, we present the results of ImproveContri(TTTFF) to repre- sent the samples where both individual models make incorrect predictions, but the averaged model makes correct predictions. (mproveConti(TT-FF) is calculated as the group ratio of PM())-FM(F) AM(T) subtracted by PM(T)-FM(T)-AMM
Remark:
We use ImproveConti((TTFFF) instead of PM(F)- (M(T)-AM(T) because we believe that there is a certain proportion of samples in PM(F)-FM(F) AM(T) where the averaged model corrects mistakes due the randomness introduced by the non-
20

---

Published as a conference paper at ICLR 2024
5.37
PMTT FMMT)-MIF)
ContteCon
Poncrtapathi
2.72.6
edtviN
2.17
1,07
1,87
1.A
2:11 2.28
-12
0.42
0.41
0.61
1N-22
16-8
IN-A
IN-5
IN IN-42
IN-R
IIAA Ih-5
Figure 7: A closer look at WiSE-FT, which averages the pre-trained CLIP and the model obtained by fine-tuning CLIP on ImageNet. Here ImageNet is regarded as ID domain and the other 5 ImageNet variants are OOD domains, i.e. IN-V2 (ImageNetV2). IN-R (Inag(mmeeNeRR) IN-A(ImageNetA), IN-S (ImageNetSketch), and Obj (ObjectNet). (Left) ImproveContri(9 is defined in Eqn. equation which estimates how the AM (averaged model) performs better than the PM (pre-trained) and FM (fine-tuned model) on the group G and how much the improvement on G contributes to the overall accuracy improvement on the whole dataset D. (Right) The ratio of sample size PM(T)-FM(T)- over the same size of the whole dataset. Here PM(T)-FM(T)-
AM(F) and PM(F)-FM(T))AMCC AM(F) denotes the group where the AM make wrong predictions and the PM and FM models make correct predictions; while both PM and FM make wrong predictions. Putting these two figures together, we can see the AM can correct many samples on which PM and FM make wrong predictions in OOD.
(M(F))FF((F-AAM))
denotes the group where AM make correct predictions
linearity of deep neural networks (DNNs) during weight averaging. To approximate such random- ness, the size of PM(T)-FM(T)-AM(F) This adjustment helps account for more accurate approximation of the sample ratios where the averaged model corrects the samples due to its utiliza- tion of more diverse spurious features.
C.2 DEEP NEURAL NETWORKS LEARN DIFFERENT FEATURES
In Section 2 have shown that the pre-trained and fine-tuned uses different features and the aver- aged model can utilize more diverse features. Actually, (Allen-Zhu & Li, 2020) provides empirical evidence (e.g., Figure 3 and 4 in (Allen-Zhu & Li 2020)) supporting the of diverse features by different deep neural networks with same architecture, even when trained on the same datasets (with different initialization). We add their empirical observations in Figure for easy of reference.
C.3 EXPERIMENTS ON MUNTISTI.ORMNIST
MultiColorMNIST.. We extend the CMNIST (Arjovsky et al. 2019) MultiColorMINIST, which constructed following Definition T MultiooorMMNIST contains 1O classes with 32 spurious fea- tures. Each image has 42 42 x pixels. There are patches in each image and each patch can take one of 10 colors. Figure ?? illustrates two samples from MultiColorMNISTT, Specifically, the label of the sample is generated from the shape of the digit and each color patch perfected correlated with the label. Let C; denote ith color patch for i 1, .... 32, Each C; takes one of the color which is perfectly correlated with y. For example, the 1st patch, i.e., C1 always takes 'white' on samples with label 5; the 2nd patch, i.e., C2, always takes 'yellow' on samples with label 5. Each C, independently generated from the label and we have C, Cyly for + j. See Figure TO for detailed illustration of the data generation process which follows the theoretical Definition In the OOD testing distribution, the spurious correlation can fail with probability p. For example, samples with label 5 randomly pick any color with probability p OOD The data generation process is analogous to the theoretical setting in Definition where each patch is a spurious feature and each color is attribute that the spurious feature take.
SingleCol We also introduce SingIeCOIozMNIIST for better comparision. SingleCol- orMNIST has 10 classes and each image has 42 x 42 3 pixels, which is the same with MultiCol-
21

---

Published as a conference paper at ICLR 2024
ResNet-34 34 learns three features (views) of car: (1) wheel (2) front (3)
ResNet-34 34 learns three features (views) of horse: (3) tail (2) legs (3) head
Figure 3: Visualization of the channels in layec-29 of ResNet-3d trained on CIFAR-1O, picture from (4).
Pigure 'Te ladependently trained BesNet-3e 34 modela (and their ensemble) detert ressorinng, suggosting this stricture. The mumerrccal experimets in Figuru 9 alsu suggees, the existence of multi views.
tirough dificecnt
views, Incependent trained neural networks
Figure 8: Figures taken from (Allen-Zhu Li 2020p which show that different DNN (with the same architecture) learns different features even trained on the same dataset.
orMNIST. However, SingleColorMNIST only contains spurious features. In other words, the 32 patches each image are same. The spurious correlation defined similarly with MultiCol- orMNIST. Figure D illustrates Figusregillustrttes two samples from SingleColorMNIST..
Experimental Details. We use the following configuration for both SingleColorMNIST and Mul- iiColoMNIST.. We use an 2 layer MLP with 64 hidden units to perform classification. We adopt Adam with learning rate 10-3 and batch size 100. train for 5000 steps and report the perfor- mance at the last step. We train two individual models f and f with different random initialization on MultiColorMNIST. We also evaluate the ensemble the two models, i.c., fose (x)) = f(x) + f(x). Each experiment is repeated for n. 20 random seeds.
Results. We vary p in MultiColorMNIST and compare the performance the ensemble model with each individual model. is the probability that spurious correlation no-longer holds in testing environment. A larger indicates larger distributional shift. The results of MultiColorMNIST are summarized in Table 3 We can that model ensemble consistently improve the OOD) perfor- mance. Figure I visualizes how much each model relies on each patch. Specifically, Figure [I shows how much the model changes its prediction when we replace a patch with black color. We can see each individual models uses different feature sets and model ensemble uses more diverse features. Table 4 shows the results of SingleColorMNIST. We can see that model ensemble can not improve the OOD performance in SingleColorMNIIY (since there is only one spurious feature in SingleColorMNISTT and model ensemble can not utilize more diverse spurious features).
Comparing Table 3 and Table we can see that the performance of individual model in MultiCol- orMNIST higher than that SingleColorMNIST when the p the same. This is because the individual model already learns multiple spurious features (even though it is only small subset of the whole feature set as shown in Figure [T]. This is also consistent with our theoretical results that diverse spurious features leads to better OOD performance.
Remark, Recall weight space ensemble (WSE) needs to be conducted between the pre-trained and fine-tuned models or different fine-tuned models starting from the same pre-trained model (Worts- man et al., 2022; Frankle et al. 2020). Since we have suitable pre-trained model for the synthetic dataset, leave the investigation of WSE MuIIIColorMNIST to future work.
22

---

Published as a conference paper at ICLR 2024
Figure 9: Two samples from SingleColorMNISST. SingleColorMINIST has 10 classes and each sam- ple contains spurious feature.
Digit ehope
Label
Panch
Parch
Patch
Figure 10: The data generation process of MultiColorMNIST (follows Definition 1
(:Model faaturo heatmap
(b]Modal 2 feature heatmop (c) Ensemble model feature
heatmma
Figure 11: Visualization of the features uses by each model and model ensemble.
p
model_1
model_2
modeL_ensemble
0.10
100.00=0.00
100.00+1..00
100.00=10.00
0.20
99.99.00.00
99.09
100.00
0.30
99.91100001
99.89-10.04
99.99-10.01
0.40
99.161:0.11
99.1910.13
99.75:10.03
0.50
95.84
96.0610.35
98.13110.14
0.60
87.15+0..74
07.56+00.69
92.31-10.41
0.70
7..05+1.04
71.77+00.4
78.6440.73
0.75
60.07+11004
60.75+0.91
67.61+10.00
0.80
48.5740.92
49.26:10.83
55.25.10.75
0.85
36.9310.70
37,74
42.34.1.64
0.90
26.0110.45
26.63+0.42
20.28
Table Results on MultiColorMNIST
23

---

Published as a conference paper at ICLR 2024
p
model_1
model_2
model_ensemble
0.1
91.0440.03
91.07=0.05
91.04:10.44
0.2
82.34+00.2
82.39=0.07
82.34+0.04
0.3
72.93+0008
72.99=0.10
72.93+00.09
0,4
64.08+10.99
64.21=0.19
64.1010.11
0.5
54.89-1.112
54.99=0.18
54.88
0.6
45.91=0.16
46.09=0.31
45.92=0.19
0.7
37.39=0.15
37.55=0.27
37.39=0.17
0.8
27.86+0.19
28.06
27.87+0.22
0.9
19.28
19.50-0.34
19.29+0.20
Table 4: Results on SingleColorMNNIST
C.3.1 INCREASING THE NUMBER OF ENSEMBLE
we show that the ensemble of two models improves significantly over each indi- vidual model MultiColorMNIST. In this part, are going to show that if increasing the number of models in the ensemble can even increases more significantly.
In Table 1 and 3
Specifically, in Table we show the results of different model number in the ensemble. When the ensemble number is I, it means that we consider a single model (in other words, not performing model ensemble). ensemble number 16, it indicates that we independently train 16 models with different initialization and use the ensemble of these 16 models to make predictions. We can see that increasing the ensemble number can signficantly boost the OOD performance. For example, when p 0.8, the OOD performance of single model (ensemble number equals 1) is 49.33%. The ensemble of two models achieves 55, .2%% OOD accuracy. The ensemble of 16 models can increases the OOD accuracy to 64.85%! This also gives us a hint the effectiveness of model soup, which averages multiple checkpoints trained with different
P Ensemble Number
0.70
0.75
0.80
0.85
0.90
71.66:12.06
60.68:2.22
49.3342.02
37.74
26.744
2
78.8
08.3440.89
55.96:1..77
42.91=0.64
29.89+0.63
4
14.3
1..0
6..04+1.32
47.92
32.74::0.55
8..64
7..73
63.52+1.61
49.15++1.23
33.67++0.93
16
86.76+00.55
77.31+0.87
64.85+1.099
50.63+0069
34.47++0.40
Table 5: Experiments on MultiColorMNIST. larger indicates larger distributional shift.
On the other hand, the dataset only contains a single spurious feature, e.g., the SingleColorMNIST, find that increasing ensemble number does not help the OOD performance. These results included T
0.70
0.75
0.80
0.85
0.90
Ensemble Number
37.40 0.11
32.64
0.11
27.90
+ 0.14
23.32 + 0.14
19.32 + 0.14
37.32 0.03
32.54
0.05
27.78 + 0.04
23.20
+ 0.04
19.18 + 0.05
37.35 0.09
32.57 0.12
27.81 0.13
23.22 0.11
19.23 = 0.12
8
37.30 + 0.01
32.50
0.01
27.74
0.02
23.16
19.16 = 0.00
16
37.35 0.08
32.56
1.56 0.09
27.82 - 0.12
23.22 0.110
19.24 = 0.11
Table 6: Experiments on SingleColorNNIST. A larger p indicates larger distributional shift.
24

---

Published as a conference paper at ICLR 2024
C.4 SIMULATION
In this section, we take some simulations to investigate the performance of theoretical forecasting results of OOD accuracy. Following the data generation process Definition here consider four examples:
example 1-1: n2 = 2, = 3 in model 1; n, = =2, n. = 3 in model 2; overlapped feature 0.01; distribution shift probability 0.9.
number 1o 11 0; noise variance
2. example 1-2: n, 2, n, 3 in model 1; A, = 2, nt = 3 in model 2; overlapped feature 1; noise variance o 0.01; distribution shift probability p 0.9.
number 1l so
3. example 2-1: n. 5, f, 20 model 1; f. 4, n, 20 in model 2; overlapped feature
number Tu00 ao = 0: noise variance о = 0.01; distribution shift probability
0.9.
4. example 2-2: n. = 5, n. = 20 in model I; n. = 5, ng 20 in model 2; overlapped feature
number 1.. 4, so
1; noise variance 0.01; distribution shift probability p 0.9.
In each example, we take 1000 simulations to report the mean OOD accuracy in Table 7 To be precise, the training data size 20000 and the data size 10000 in each simulation. Then com-
Model
Model
Model Average
Model Ensemble
Simulation Results Theoretical Results
0,866
0.866
0.974
0.974
Example 1-1
0.865
0.865
0.973
0.973
Simulation Results
0.866
0.861
0.943
0.940
Example 1-2
Theoretical Results
0.865
0.865
0.948
0.946
Simulation Results
0.940
0.894
0.978
0.978
Example 2-1
Theoretical Results
0.941
0.910
0.980
0.980
Simulation Results
0.943
0.939 0.943
0.999
0.989
Example 2-2
Theoretical Results
0.943
0.992
0.983
Table 7: Simulation for the OOD accuracy in different models
paring the results of theoretical results and simulation results, it is safely to say that our theoretical analysis, as well proper approximations, could take an effective estimation for OOD accuracy.
25

---

Published as a conference paper at ICLR 2024
D DISCUSSIONS, ILLUSTRATIONS, AND SUPPORTIVE RESULTS FOR THE THEORETICAL PARTS.
D.1 DISCUSSION ON THE THEORETICAL MODELS
Our theoretical models in Section 3 is designed to mimic the modern deep learning architectures such as Vision Transforms (ViT) (Dosovitskiy et al.) 2020). Figure 12 provides a comparison between our theoretical models and Vision Transformers.
Similar to VIT, we process images as patches, where each patch corresponds to a specific feature denoted as Patch,. Each Patch, is represented by high-dimensional vectors æ; I in the embed- ding space. Consequently, the whole feature obtained by concatenating the embeddings of each patch, resulting in = [r], .2...... Assuming a total of d, features (d, d, + d, in Section 3 we have a E Iddxde Notably, (Allen-Zhu & Li, 2020) also uses a similar theoretical data model that concatenates the patches to analyze the convolutional neural networks, e.g., Figure 5 in (Allen-Zhu & Li 2020).
To simplify the model, we utilize two-layer structure consisting of binary feature mask O as the feature encoder and a linear classifier w. analogous to ViT which uses the transformer feature encoder with MLP classifier. This two-layer simplification approach has been widely employed in OOD literature (Arjovsky et al., 2019; Rosenfeld et al.] [2020] Zhou et al., 2022b; Peters et al.) 2016 Lin et al., 2022b). The difference between theoretical model and ViT is that ViT process the features sequentially while we select the feature at once.
The binary feature mask & is represented as (0, 1)dt, For instance, if have three features, i.e., = [21, 42, zz), and & = [1, 1, 0j, the learned feature would be x T Ij + z2. Considering 3-class classification task, the linear classifier w R2x3 takes the learned feature T & input and produces a 3-dimensional vector whose elements represent the logits of the three classes. The classifier is optimized to minimize the in nn-distribution (ID) loss based on the learned feature. Therefore, we have:
w = arg min Rid(u, b), Rexxx
where Rea(v, 3) represents the loss of (v, Φ) in the ID distribution.
D.2 COMPARISON ON OUR MODEL WITH A 2-LAYER DNN
In this paper, we consider the model w æ&, where w R and & E (0, (0,1]d.-t pjottd are paramters. Here the input E (dd((++dd)) and see App D.1 for detailed discussion. We then compare our model with a general 2-layer DNN to see why it can capture the difference between weight space ensemble (WSE) and output space ensemble (OSE) in DNN.
Consider a general 2-layer DNN parameterized by (Wa E Rdixda, W, E RdixK) with ReLU activation 8(-) and output fann(X) = W,, 8(W X) for X E Rd. Here we use uppercase X to avoid confusion with our previous since they have slightly different dimensions (App D.I. Since WSE is conducted on the models that is close to a pre-trained model (Wortsman et al.). e.g., (Wao. Woo),
consider fann AW, is small and trainable. By Taylor expansion, we have
where AW, and (Wio + AW)) 5(W + AW.) Wherr
fdnn (X) Wi5s(W..T X) + AW.. T X) Wea6' (W.. XAAW. X) + AW65 XAW. X) +
(a)Fixed Term
(b)Linear Term
(c)Bilinear Term
Where 5'(Y' is 06(Y).
By Further, we incorporate the fact that the second order derivative 8*8()) is aay
zero almost everywhere for ReLU activation function (except at Y 0). Then is the error term induced by the non-linearity of ReLU activation function (while W X has some zero elements). To be precise, as here we just focus on fine-tuning regime and W is not sparse in general situations, it is safely to say that & is small. WSE and OSE are exactly the same for the (a) fixed term and (b) linear term. We will show that WSE and OSE differs on the (c)bilinear term, which is captured by model in Definition B
26

---

Published as a conference paper at ICLR 2024
Predictions (ar17
Closs A/BCC
Cleis A/BrG
Linear Classifer
Linwar Head
MLP
Leamed Fezture
Festure selector (0, 0.
Transformer Encoder
Feature enteding space (Rt F21 Pini 8
Pasch Podicc
00
Projection to High Ciner sionel Embedding Seace
Linear Projection of Flattened Patches
Figure 12: Comparison of our theoretical models in Section with Vision Transformers Dosovit- skiy et al. 2020 Some parts of the figures are adopted from Dosovitskiy et al. 2020).
Consider two models, fann and fann. both close to the pre-trained model. Specifically,
Jann (X) (W1o + AW,) 5((W + AWW) X); +XW))
fanm (X)= (Wio + AW)) 5(( 00 + +WW)) Tx):
Then the output space ensemble of Idna (X) and (X) is
Idnn, 0.5 (Wos + AW,) ((( + AW.) X) + (Woo + AW, T(((Wo AT)) xx) Win XX 0.5(AW + AWw) T( X) + Woo WW (O..AW + x
(a)Fixed Term
(b)Linear Term
+ 0.5 AW, X) x) + AWS (W ( XAW X)
(c)Bilinear Term
fdnn,
-WWo + 0.5(AW, AW.) 5((Wao + 0.3(AW, AW.)) Xx)
WTWSWW.. x + 0.5(AWnn + AWr)) 5(W T X) + Wind' (W X)).5AA. + AW.) x)+
(a)Fixed Term
(b)Linear Term
+ 0.25 AW, + () + AW,)
Bi)linea Term
Comparing Jáun,ose with Jdnn,,wee
that the difference of them lies in the bilinear term:
fdnn,wse fdnn,ose (0.25 + (W.) + AW.) Xx)
0.5 (W. x) + (W
x))
(3)
We can see the bilinear term difference has a clear analogy with our models in Definition Specifically, according to our Definition of OSE and WSE in Definition 3 we have
fwse fose 0.25( + w)) ax ① )))
0.5( T + xΦ))
27

---

Published as a conference paper at ICLR 2024
Comparing equation and equation 4 that is analogous to AW, and @ is analogous to A a. equation 3 and equation differ by a scaling 5' (WWT. which is a fixed matrix independent of the trainable parameter (AW, AW))
D.3 ILLUSTRATION OF THE TRANSFORMATION MATRIX Q
Consider the 3-class classification problem. In the ID distribution, we have,
[1,0, 07
Qs = [ei, e2, e3] I3
0, 1,
(0, 0,
This indicates that each spurious feature is perfectly correlated with the invariant feature, as illus- trated in Figure 13 (left). For instance, Q=, Is implies that the background of the dog, crow, and camel are floor, grass, and sand, respectively.
In the OOD distribution, Q.. is longer equal to I, indicating that the correlation between ani- mals and the background may fail with a certain probability. Figure 13 (right) illustrates 2x, which represents the first column of Q..j Q..j (1) take the value e2 with probability of p/3, indicating that the background of the dog is grass in this case. Similarly, Qe,((1) (1) can take the value eg with probability of p/3, indicating that the background of the dog sand with probability of p/3.
Label
Dog
Dog
Dog
Caw
Cameel
0
Backgrand
O
0
0
1
Q1)
With Probabillty
1 2pr3
px3
p/3
Maa
(9) Q.,, ID distribution
[c] First column of Q..y OOD distribution
Figure 13: Illustrations of the matrix Qaj--
D.4 ON THE PESSIMISM OF WORST-CASE THEORETICAL ANALYSIS FOR OOD
D.5 ID PERFORMANCE
Recall that in Section the OOD accuracy is defined by
y) y) Aood()) = Eq. [ 1
The ID accuracy (f) is defined similarly by fixing [Q......., Q.d,1 ,I...., I]. According
Lemma 3 we know that the ID accuracy all models involved in Definition 2 Example are larger than
D.6 INTUITION OF OOD PERFORMANCE IMPROVEMENT OF OSE
We use Example D to show the main intuition of the output space ensemble (OSE). In Example I two individual models learn nve-overlapped feature, so model ensemble and averaging are the same. According to the proof in Appendix F.T. consider the samples from the first class, i.e., y e,, the
28

---

6p Published as a conference paper at ICLR 2024
predicted logit of the each class is
3 (He, + y= Σ (ea(1))
w T
x
Σ
Mas(1) (Mo,)ee((1),
t=1
j=1
3
2x)) x y Σ uu (2) (1,:200111)) + Σ
Uss (2, (us,)Qu
j=1
w(3)² T x y=e: Σ (3) (c. + Σ (a)(3) (Ms, Qe.,
where we omit the noise term whose impact the accuracy is less than according to Lemmal 3 Further, let μl denote any Peùi and plai and Q denote its corresponding transformation matrix. For example, μ Ma.j and Q Qs.j- Suppose Q(1) Ckg have
1, if k2, p(ki) (IQ(1)) (k(k)) (mek,) u(ki) p(k2)
0, otherwise.
For the invariant features, (1, e₁ always hold and for spurious features, Qs,((1) takes ei with probability 1 1 -. and takes e2 or e3 with 5. respectively. So the predicted logit the each class is simply
3 Σ
m(1) T xΦ y=
=2 +
==1
w(2) xΦ y=
=0 Σ
[1Q,,((1= =e2).
j=1
w(3) x y=ei =0 Σ
(1) =s):
Let us consider the probability of (2) x y=e, > w(1 x y= i.e., the model (Ø, w) mis- takenly predicts the second the class e2 even if the true class e1. This will happen when (1) = ==) holds simultaneously, whose probability would be 27- spurious features takes the value in OOD that is correlated with the second class ez, overwhelming the two invariant features correlated with E1.
g Intuitively,
As for the averaged model, we have
w(1) x y=e.
=4 + Σ
(1) e1).
5=1
mx)) T अτ y= =0 Σ
((Q e2).
==1
T(3)Tzo x y=e: =0 Σ (1) =s).
We will have (2) T y=e, > w(1 x x
y=e, if either of the following occurs
(1) -2) holds simultaneously, whose probability would be p
f22
Five of (I[QQ ..j (1)) takes ea and the remaining one takes es. i.e., 1 (QQ,()))
e2) = and Σ T(Q (1) = ea) = Such probability
725"
The total probability is then 72g intuition.
See Figure 14 for a visualization of the main
104
104
29

---

Published as a conference paper at ICLR 2024
Q(1) =
Q(1)= =
Q(1) 6q
Q.aill) Qua(1) Qall)) Q.a(1) Q.,(1)
Individual Model
Failure Probabiliy=pp/77
( (7)
(L) (1, Qualli Quall) Q.a0) Q.al)) 4.a00 Qa(()) QuslI) Q.a0)
Avearged model
Failure Probabiity=pp/729
Figure 14: Comparison of the failure probability individual model and averaged model. With probability pd /27,, the individual model (L) w)) will encounter OOD distribution where it mistak- enly predicting the second class 02 on the samples from the first class et For the averaged model, such probability would be roughly about p /729. Refer to Appendix D.6 for detailed explanation.
D.7 THE DIFFERENCE BETWEEN WSE AND OSE IN OOD
D.7.1 EXPLAINING THE DIFFERENCE BETWEEN WSE AND OSE
We use the following Example 3 to show the main intuition of the difference between model aver- aging and ensemble.
Example Two individual models learn overlapped features 2x,2 and Øs,3 as
x Øe,1 + -u,2 + Ls, + x3,2 + Is,3, x = Iú,2 + Lu,3 I + F.,1 + 2,,5+
Proposition 5. Consider the Example suppose Assumption [hold, and there infinite ID and OOD samples, the averaged and ensemble models are defined as Definition 3 Omitting small
terms containing E, have Aood(f) Avoa(f) 1 bpp and Aood(fose) = 8I
1p" 8p" and
243
A = 4p
pº
SI
27
Full Proof in Appendix and (k).. By Lemma
F.4
In Example B two individual models learn overlapped feature, x.,i
for l = 1,2,34 3, we have
3
w (k:) Σ (k)) + Σ (k))
j=l
3
w(k) Σ (k Σ (k)(),,
5=3
So have
in kk) + (k:) = Σ (k) 2k), +
Σ
(k.((k) + (k))
1=13
1=1,2,455
For samples from the first class, we also have
x x( 9=
g=e. = Σ nv Qu. Qv.
Σ
ps (1,0 2
1=1.3
1=1,2,455
where 2; ~ NN(0 2I)), Vi. We then have
(10(k) + (kk)) x= + 6
Σ (k) QI, (k) (H,,2Qu,211)
Σ (k)( (Hu (1)( + (k),((K) (Ho3Q..s(1)) +
(5)
1=12
30

---

Published as a conference paper at ICLR 2024
enellsppod ineatda fastrr
ove spore marrent Rell
ovortapped spurious fattre n..
Ovoriaoped spuncus feature
(8)
(b)
Figure 15: (a) (fws))) (f) on Example (b) (fooe)) on Example
As for model ensemble, we have
w(b)) T w + zikk) (ak) x
(k) (Mo, Qu)(())
Σ (a)(k)
5=1,2,3
(ma(k) (H,,
Σ (k)(k) (Han
(6)
223
/=3,1,5
(k) (po.Qu 2k),2(()) (Hr,
(14,) Qv.2(1))
1=13
(ag(k)
(us,) (k), (Mo.3Q..31) + +
1=1,5
Comparing equation 5 and equation 6 we can see that
For model averaging. the overlapped features μ,,2 and Ps.3 (corresponding 0a,2 and 22,3) have coefficients amplified by 2 in $ + and further amplified twice in w + w. This results in coefficients of the overlapped feature becoming 4 in (w + w) x< + + For model ensemble, i... w w + i Td, the coefficients the overlapped feature are 2.
D.7.2 THE THEORETICAL CONDITION OF WSE OUTPERFORMING OSE Recall Proposition 2 that we have
(m (1-p)(m,
(1-
nl 21000 + (fy nl + 21
oodd = Fy
Vn, + nl + 14
(f -.,)) + (f. + m..) Aood(fose) Fp Vn, + n. + 27
A direct consequence of Proposition 2 is as follows, which illustrates when model averaging can be more effective than model ensemble:
Proposition 6. Consider the models in Definition 2 suppose Assumption [and 2 hold, there are infinite ID and OOD samples. Suppose the number of features that & and &b learn are the same,
ie., ny = My = Nuy The = ngg = nly and denote Ps = Tooo Ner Po nvo/ nny. Omitting 3(1-p)ns and
small constants involving we have Aood(fwse) > Aood(fose) when po A Awoe( S Aood when 3(1-p)nn
10
As shown in Appendix [.7.I the coefficient of overlapped feature in model averaging is 4, and The coefficient of overlapped feature in model ensemble is 2. If more & and & learns more overlapped invariant features, the model averaging would put more weight on the invariant features, leading to better OOD performance.
In Figure 4 (c) and (d), we illustrate Aood (fwse) - Aood(?) (f) and Aood(fwse - Aood (fose) on the following example:
31

---

Published as a conference paper at ICLR 2024
Example 4. Consider both models learn the same number of features, i.e., fixing n. n. 10 and R, = ng = 20, vary nuuo = 0..,...; and To = 0, 0,1,.... 5.
We can see that fwse achieves larger OOD improvement over fose when two individual models learns more overlapped invariant features (e.g., larger (h and less overlapped spurious features (c.g., smaller 7l In Appendix(D72.. why this can happen easily in real-world datasets, provide some primary experimental results.
we provide conditions when fwse outperforms fose. discuss
Why does easily happen in OOD on many real-world applications? Recall that there are totally dy invariant features and d, spurious features. It is a common believe that spurious features are high-dimensional and invariant features are low-dimensional, i.c. da >> d, (Arjovsky et al.) 2019 Rosenfeld et al. 2020). Since the spurious features are high dimensional and (Allen-Zhu & Li 2020 Zhang & Bottou 2023) indicate that different models can learn different (limited size) subsets of features, the overlap ratio of spurious feature p, relatively low. On the other hand, there are a small number of invariant features and recent studies (Rosenfeld et al. [022] Qiu et al.) [023]
Kirichenko et al. 2022) show that models always learn some invariant features for the fine-tuned task during ERM fine-tuning regardless of the presence of spurious features, so conjecture that the overlapped ratio of invariant feature Pe is relatively higher.
However, we recognize that our discussion regarding the overlap ratio of invariant spurious features being larger than spurious features is not supported by rigorous proof, but rather it remains a conjec- ture. Further research in this area is necessary to provide conclusive evidence and establish solid foundation for this claim. In the next part, we will conduct experiments to provide some initial support for this conjecture.
D.7.3 EMPIRICAL VERIFICATION
It is very difficult to directly empirically verified Proposition 6
because
For real-world datasets, it is hard to identify whether and how much a model relies on invariant or spurious features. Verifying Proposition 6 needs estimate how much two models relies on the same feature.
For synthetic datasets, such as CMNIST (Arjovsky et al. 2019), there is no feasible pre- trained models available. On the other hand, weight space ensemble needs to be conducted on models close to pre-trained models.
In this part, we design primary experiment to get around the above obstacles. Consider the ensem- ble of two models: pre-trained CLIP (J) and the CLIP fine-tuned on ImageNet (f).
First, we use ImageNet variants (ImageNet- V2, ImageNet-Sketch, ImageNet-A, ImageNet-R, OB- jectNett for OOD performance evaluation. Recall that ImageNet variants share the same invariant features with ImageNet. Also recent studies (Rosenfeld et al. [022]] Qiu et al., [2033] Kirichenko al. 2022) show that ERM fine-tuned models always learn some invariant features for the fine- tuned task regardless of the presence of spurious features. So ff learns the invariant features for ImageNet variants. At the same time, the pre-trained CLIP f can stably perform zero-shot clas- sification on ImageNet and its variants, indicating that also learns good invariant features for ImageNet variants. According to the previous discussion, f and have some overlapped invariant features for ImageNet variants, leading to better weight space ensemble than output space ensemble on ImageNet variants (shown in Figure Le(fe))).
We then evaluate the OSE and WSE on three other distinct datasets, i.e., Places365, StanfordCars, DTD and Food101 (refer as PSDF datasets). These tasks have different label space with ImageNets, and contains different invariant features with ImageNet. Then this case, the model fine-tuned on the ImageNet learns little invariant for PSDF datasets. So overlap invariant features used by the pre-trained model f and fine-tuned f are rather limited, indicating Pe is close to zero. Then according to Proposition WSE would be no better than OSE, This is consistent with the results in Figure right).
32

---

Published as a conference paper at ICLR 2024
0.44
043
646-
Tea-s
Irageten AcE
Figure 16: Comparison of model ensemble and averaging. Left) OOD performance on ImageNet variants, Right) OOD performance on PSDF (Places365, StanfordCars, DTD and Food101).
D.8
ILLUSTRATING THE OVER-CONFIDENCH,
In Section 4 use À to characterize the over-confidence of fx (Aw, AΦ). Specifically, we have
fx(x) X² xΦ
Denote q w T ax which is 3-dimensional vector for 3-class classification problem. Consider an example, i.e.. = [2, 1, 1]. Recall that ((k) is the k-th element of for k 1, 3.. The predicted probability for the first class when À
is
exp(g(1))
(2)
Probability of class 1 = exp(g(I)) exp (2)) exp({(3)) exp(2) + exp(1) + exp(1) 0.576.
When the predicted class of fx would the same for > and = 1. Whereas, when À > 1, the predicted probability for the largest class would be amplified, e.g., when v5
exp(X²q(1)
Probability of class exp(APq(1)) + exp (2) cxp(A2q(3))
exp(2X2)
exp(2X + exp(12) + exp()
-099
So we can see that larger won't 't change the predicted class, but would make fa more confident.
E MORE EXPERIMENTAL DETAILS AND RESULTS ON BANG
E. DETAILS IMAGENET VARIANTS
Details for ImageNet variants:
ImageNer-V22NN recreated version of the ImageNet test set, but with different set of data distribution.
ImageNet-R((N-R) Renditions of 200 ImageNet classes resulting in 30,000 images.
ImageNet Sketch(IN-S): Sketch style images of the same categories as ImageNet, with total of 50000 images.
ObjectNet(ON): Objects in this dataset are captured in cluttered and natural environments at unusual poses.
ImageNet-A((N-AA) This dataset consists naturally occurring images that are misclassi- fied by ResNet-50 model for 200 ImageNet classes.
33

---

Published as a conference paper at ICLR 2024
ImageNet (Deng et al.)
ImageNet-A (Hendrycks et al)
ImageNet-A R (Hendrycks et all)
ImageNerV2 (Recht al.)
ObjectNet (Barbu al.)
ImageNet Sketch (Wang al.)
Figure 17: Datasets on ImageNet and variants. For each dataset, we pick samples of the class lemon and show illustrative images from each dataset. The dataset descriptions similar to that of Wortsman al. (022)..
E.2 DETAILS OF PLACES365, STANFORDCARS, DTD AND FOOD101 (PSDF)
Placex365 (Zhou al. (201))) A scene recognition dataset. In this paper, the validation set from the Places365-standard, which is composed of 36,000 validation images from 365 scene classes.
StanfordCars (Krause et al. (2013): This dataset contains 196 classes of cars. Classes are typically the level of Make, Model, Year, 2012 Tesla Model S or 2012 BMW M3 coupe. In this paper, we evaluate models on the test set, comprising 8,041 images.
Describable Textures Dataset (DTD) (Cimpoi et al. (2014)): DTD is texture database, organized according to list of 47 categories inspired from human perception such as banded, dotted and gauzy. In the paper, we use the test set with 40 images per class.
Food101 (Bossard al. (2014)) This dataset consists of 101 food categories. In the paper, we use the test set with 250 test images for each class.
E.3 DETAILS CALCULATING THE CONFIDENCE
Consider K-class classification problem. Denote the lth element of the output Prob,, indicating the probability the model assigns the (th class. We have Σ Prob; 1. The confidence is defined as as:
Confidence max ((Probi
E.4 EXPERIMENTAL DETAILS
We use the CLIP model ViT-B/1 Radford ImageNet. We use the AdamW optimizer with the default PyTorch Adam hyperparameters and choose 512 as batch size. We use learning rate of 3 x 10-3 gradient clipping at global norm and fine-tune for total of 10 epochs. The settings mentioned above are the same with Wortsman et al. (2022). For our method BANG, we try four smoothing for LS (label smoothing): 0.05, 0.10, 0.15 and 0.20. We adopt 0.10 in our reported results in Table 2 Further results in Table show We do not tume the hyper-parameters of Mixup Zhang et al. (207). We the default hyperparamter AMMreeTrinCCnttiiee
Raadood et al. 2021 ). fine-tune the pre-trained model on
that BANG is relatively insensitive to the
(202)).
34

---

Published as a conference paper at ICLR 2024
Caliration [Kumar. model
The rinetuned
27-
25
3.4
04
33-
brstio Sarmer 20023
3.2
04
0.8 Accray
5.8
(b) The vanilla tuned model calibrated on in
(a) The vanilla fine-tuned model
dataset Kumar (022).
Fine-tuning
Mixp tuning
3.9
0.7
2.7
3.5
2.4
Finstanin
67
06
32
25
Accaray
(c) The model fine-tuned with label smoothing
(d) The model fine-tuned with Mixup
Zero shot and Koup with LS Rne tuning model
36-
3.-
62
0.3
0.1
35
06
65
Accumey
(c) The model fine-tuned with both Mixup and LS
Figure 18: Comparison of confidence and accuracy between zero-shot and the model finetuned with different methods. In the figure, the ID dataset is the ImageNet dataset, which is represented by V. the five OOD datasets are: for ImageNetA, D for ImageNetR, + for ImageNetSketch, for ImageNetV2 and x for ObjectNet.
35

---

Published as a conference paper at ICLR 2024
Methods
Model Averaging
ObjectNet
Aug
Zero-sbot Wortsmon cl
2022
No
68.3
61.9
77.6
49.8
48.2
53.0
58.1
Fine-tuning Wortsman et al. 2022
No
81.3
70.9
65.6
36.7
16.3
19.6
53.8
Flip
No
81,3
70.5
63.1
36.8
44.6
51.4
53.3
Rotate
81.4
65.2
Color
81.4
46.7
50.4
Mixup
No
83.0
72.7
66.1
43.7
18.8
52.4
56.8
Flip
Yes
81.8
72.7
78.2
52.9
53.6
58.4
63.1
Rotale
Yes
81.7
72.8
78.8
52.7
53.7
57.3
63.1
Color
63.4
Mixup
81.5
73.0
79.5
57.9
54.5
58.7
64.7
Table 8: Results of fine-tuning CLIP VIT-B/16 with flip, color, and rotation data augmentation ImageNet.
Puiide
0
53 56 35
(a) With rotate augmentation
(b) With flip augmentation
(c) With color augmentation
Figure 19: Comparison of confidence and accuracy between zero-shot and the model finetuned with different data augmentation. In the figure, v refers to ImageNet dataset, o for ImageNetA, D for ImageNetR, + for ImageNetSketch, & for ImageNetV2 and x for ObjectNet.
E.5
MORE RESULTS ON BANG AND DISCUSSIONS
Mixup and label smoothing can alleviate the over-confidence of the fine-tuned model Compare Figure T8e
18d I8e with Figure Sa we can see that imposing Mixup and LS during fine-tuning can alleviate the corriconiieence of the fine-tuned model on both ID (ImageNet, denoted by v in the figure) and OOD datasets, which is consistent with existing results Park Caragea (002)).
Comparison with Calibrated Ensemble Kumar et al. (022). Kumar et al. (2022) calibrates the fine-tuned model on the ID) dataset by searching for a temperature T of the softmax. Figure T8b shows that the confidence of the calibrated fine-tuned model approximately equals its accuracy on the ID dataset (ImageNet). However, such model is still highly over-confidence in OOD datasets, e.g., the confidence is over 0.6 while the accuracy is lower than 0.4 on ImageNetA (denoted by o), which is consistent with the findings in Ovadia et al. (019)) and also the discussions in the Section 4.2 of Ovadia et al. (2019). So the scaling issue shown in Proposition (itill exists in OOD datasets. Notably, Calibrated Ensemble itself Kumar et al. (2022) can not be directly applied on model aver- aging: Model averaging merges the parameters of each layer. However, calibrated Ensemble only tunes the temperature of the softmax, which does not affect the lower layers, indicating that the layers other than the output layer can still suffer from scaling issues. We try direct adaptation of (Kumar et al. 2022) to WiSE-FT: divide the weights in the last layer w by scalar (temperature) and then perform weight averaging. This also does not yields satisfactory results (Appendix (E.6) and the reason is discussed above.
Comparison between Mixup with other data augmentations We also compare Mixup with other data augmentations. We fine-tune the CLIP on ImageNet with flip, rotate, and color augmentation, respectively. We then performance weight averaging on these fine-tuned model with the pre-trained model as Wortsman et al.) (2022) docs. Table g shows that flip, rotate, and color augmentation can not enhance the performance of model averaging. Figure 19 also shows that these augmentation methods not alleviate the over-confidence of the fine-tuned model.
BANG is relatively insensitive to ayper-parammters. Table shows the performance of BANG with different ppperpparaetter of label smoothing. BANG is relatively insensitive to such hyper- parameters, e.g., the average OOD performance of BANG(Mi))p+LS) all remains at about 64.9% for the four parae-parameters.
36

---

Published as a conference paper at ICLR 2024
Methods
Midel Averaging
IN(ImageNet)
IN-V2
IN-R
IN-A
IN-Skeich
ObjectNel
Avg OOD
Zero-shot Wortsman
2022
68.3
77.6
48.2
53.0
58.1
Fine-tuning Worrsman ot
4022
No
81.3
70.9
65.6
36.7
46.3
49.6
51.8
Fine (unngit-SCUUS)
No
82.0
71.5
62.8
37.7
45.5
50.6
53.6
Fine Sane-tuiin((LS(O..11
82.0
72.3
61.3
38.3
46.5
51.1
54.3
Fine-tuiin((LS(00155
No
82.1
72.1
63.3
38.0
46.6
50.7
$4.1
Fine-tuningg(
72.1
4622
50.5
53.7
Fine-tuning(Miixup)
No
83.0
72.7
66.4
43.7
48.8
52.4
56.8
Fine-tuning Mixup LS(0.05))
73.2
48.5
52.3
56.7
Fine tuningt Mixxup (.8(0.10)
No
82.7
73.0
66.4
43.3
48.6
52.4
56.8
Fine Mixin
56.6
Tue--uning(MMxxpp--L LS(0.200)
No
82.9
73.2
66.4
44.6
48.5
52.4
57.0
Wontsmnn et
63.0
BANGLLS0
Yes
82.2
73.0
78_1
54.7
53.8
58.3
63.6
BANGGLSS(DDDD
BANGKLS(0155)
Yes
82.0
73.2
78.1
$5.0
53.4
58.9
63.7
BANG)LS(0,20)
63.4
BANG(Mixui)
Yes
81.5
73.0
79.5
$7.9
54.5
53.7
64.7
BANG/Mixup- 1.5(0.05))
Yes
81.6
73.1
79.7
58.2
54.8
58.9
64.9
BANGIMixup LSS(..0))
Yes
81.5
73.0
79.8
$7.9
54.8
59.0
64.9
BANG/Mixup- 1.5(0.15))
Yes
81.7
72.9
79.6
57.7
54.6
59.1
64.8
BANG(Mixup LS(0.20))
Yes
81.6
73.1
79.9
$7.8
54.8
59.0
64.9
Table Results of BANG with CLIP-B/16. We show different hyper-haramaaaen parameters of label smoothing. Mixup use the default
of MMPreTrain IMPreTrainContriburoo Contributos) 2023)
WiSE-FT
Exp 1
Exp 2
BANG
63.0%
63.0%
64.1
64.9%
Table 10: WiSE-FT can benefit significantly from better calibration by scaling the fine-tuned model.
(Exp ()x(Exp 2) are described in Appendix E.6
E.6 WISE-FT BENEFTT SIGNIFICANTLY FROM BETTER CALIBRATION
In Section we theoretically show that model WSE can suffer from the imbalance issue where two individual models have different scaling. This can happen if one model is much more confident than the other. Unfortunately, we observe that the popular method, WiSE-FT suffers from this issue. Specifically. WiSE-FT averages the pre-trained model with the fine-tuned model. In Section 4 we show that the fine-tuned model is high-overconfident compared with the pre-trained model. We pro- pose BANG, which averages the pre-trained model with the model fine-tuned with Label Smoothing (LS) or MixUp. Since and MixUp can also improve the fine-tuned performance, we conduct following experiment to isolate the effect of better calibration from better fine-tuned performance.
Scale the fine-tuned model during weight space ensemble. A straightforward method to alleviate over-confidence is to tune the temperature of the softmax of the fine-tuned model (Kumar et al. 2022). However, this method can not be directly applied to WiSE-FT since WiSE-FT averages model weights instead of ensemble the outputs. We first apply direct adaptation of (Kumar et al.) 2022) to WiSE-FT: divide the weights in the last layer w by scalar (temperature), which is equivalent to softmax tempering. However, recall the model averaging (w + w) x + .) also suffer from the imbalance issue of Φ Specifically, Proposition 4 shows that the averaged feature can be biased towards + if the scaling of & is larger than क. So merely adjusting the weight of the classifier w cannot alleviate this bias. The experiment result (Exp 1) in Table I0 also shows that merely re- scaling the classifier can hardly improve WiSE-FT. In practice, we use a transformer (VIT-B/16) with 12 block layers and linear layer. We obtain the averaged model (0. w) as follows
(Exp 1) Re-scale the classifier of FM (fine-tuned, 0, w)) model during averaging, i.e., 0 0.5 + 0) and a (1 a) + ax.
(Exp 2) Re-scale whole network of FM as, (1 aa) at and i (1 a) aw.
We search for the best among 0.2-0.5 (with interval 0.1) for each ood dataset. The results in Table [O]shows can merely scaling the fine-tuned model to alleviate its over-confidence can signifi- cantly improvement the performance of WiSE-FT.
37

---

Published as a conference paper at ICLR 2024
Marghn of model
0.50
0.50
PMCTI-FMITI
FMIF)-FMIF)
0.00
PM(T)-FMIT)
0.00
PM(F)-FMIF)
-0.50
-0.50
PMITT-FMITT PMITI-FMIF)
-0.75
(M(T)-FM(() PMIF)-FM(FI
Margins of zero-shot model
Margins of zero shot model
Figure 20: (Left) Margins of WiSE-FT, where the tine-tuned model is obtained through vanilla fine- tuning. (Right) Margins BANG, where the fine-tuned model is obtained through fine-tuning with MiXUP+LSS
N1/N2
WiSE-FT BANG
10.6%
13.4%
The ratio (versus the entire dataset) of samples where model averaging can correct the prediction in the PM(T)-FM(F) group. N1 denote the number of samples where model averaging can correct the prediction in the PM(T)-FM(() group and N₂ denote the total sample size in the dataset.
Table 11:
BANG can correct more samples which the fine-tuned model make mistakes. In this part, We denote the pre-trained model as PM, fine-tuned model as FM, and averaged model as AM. We divide each dataset into four groups of samples according to whether the PM and FM make correct predictions, respectively. We further use T/F to denote whether a model makes correct predictions, i... T for True and for False. For example, on which the predictions of PM and FM AM are correct and wrong, respectively. We visualize the average margin of fine-tuned and pre-trained models on four groups, i.e., PM(T)-FM(()), PM(T)- FM(F), PM(F)-FM(T), and PM(F)-FM(F). The margin is the difference between the probability assigned the correct class and the maximum probability among the wrong classes, i.e.,
use PM(T)-FM(F) denote the group of samples
Margin Prob; max Prob(k)
kil
where / is the it the true class, k = K, and Prob(k) is the probability that assign to the class k. Averaging models with negative and positive margins can potentially correct mistakes. Fig- 20 (Left) and (Right) visualize the the margins of pre-trained and fine-tuned models each group each datasets for Wise-ft and BANG. In Wise-ft, the fine-tuned model exhibits signifi- cantly negative margins the PM(T)-FM(F) group. Specificallyy Margin Margin(FM) on the group PM(T)-FM(() is negative for WiSE-FT on 20gure indicating dominance of fine- tuned models WiSE-FT even fine-tuned make mistakes. This also explains that why in some datasets, e.g.. IN-R and IN-A, ImproveContri(TFFF] is negative as shown in Figure 7 How- ever, BANG, Margin PM) + Margin(FM) on the group PM(T)-FM(F) is positive on average as shown in Figure 20 Right), suggesting that BANG is capable of correcting more mistakes within the (M(T)-FM(() group. Table [I shows that ratio (versus the entire dataset) of samples where model averaging correct the prediction in the PM(T)-FM(F group. Specifically, let N1 denote the number of samples where WiSE-FT can correct the prediction in the PM(T)-FM(() group and N₂ denote the total sample size in the dataset, Table 11 compares the N₂ /N₂ (averaged over OOD datasets) of WiSE-FT and BANG. Table [I shows BANG can correct substantially more mistakes made by the fine-tuned model.
38

---

Published as a conference paper at ICLR 2024
F PROOFS
F.1 PROOF OF PROPOSITION 1
Proof. (a) Two individual models. Recall that in the 3-class classification problem, w
(1)1), (2), w()) E Rdx3 We first solve the ip on the infinite ID samples. Lemma 5 for
k 1,2,3, 3, have
3
(k) = Σ (o,
Σ Mas Mo. Qo) = Σ (k) j=1
(k),
2=1
g=1
where the last inequality is because: Qu = always hold and Q.. = I in the ID distribution. Then (k) Eb and a (k) en (recall that Q(k) the kth column of the 3 x 3 matrix Q). Then for each (kQ(k) pek = pu(k) and (a(k)) is the kthh column of the d x 3 matrix μ.. Similarly, we have
4
i (k))
Σ (k)
Σ ps,s()) (k))
We first look at the model (1,, + and consider the OOD accuracy of the samples from first class k 1. For each sample from the first class in OOD, we have
x
Σ pr Qu,(
Eus Σ z
where 2j ~ NNO, 0² 12), Vi. The model (10 क) makes correct prediction on the samples from
e1 if the following holds
w( (1) T x y=e. > w2) xblly=e and w(1) x lybb > voyy=e
So for each OOD sample, we have
w(1)
T xΦ) y=e:
Σ M (1) + Σ (1s(1))
Σ Ms Qu, + + us Qc5((1 + Σ 21
5=1
Σ (1) (u, + Σ (1) Qcg())))))) +
3
-2 + Σ (1) (es,)es(1)) +
Ag
where the second equality is by the Assumption that different μ are all orthogonal to each other; the last equality is because (1, e1 always hold and further by Assumption 2 we have
(1) (u,, (1) (Mr (1, (1,;(1) = 1.
Similarly we have
(2) T x y=eı
Σ (2) + Σ (2,s(2)
Σ nv (1, + (.5000001))
Q15(1) + Σ 2
j=1
(2, Q1, +
(a,((2) (2) (Hs, (a.))) +
3
=0 +
Ms.j (2) (Mu. Qu. (1)) +5,
1=1
B,
39

---

Published as a conference paper at ICLR 2024
where the last equality is because (2, (H,,:Qr,((11) (2,,(2) (e, 0. Similarly, also have
w33) x y= =
(3) (Mas Qo,( +
j=1
C2
It is easy to see that, for k2, k2 = 2.
0, k = k2,
(ks( (Ho.) =
11 otherwise.
Since in the OOD distribution, Q1,(1) can be any of el, and e3. we have Aj. Bj. C E (0.
and A, + B, + C2 = for j 1,2,3. Specifically, Aj = 1, = 0, 0 if Qa. = E1,
A, 0, B, = 1, C3 0 Qa, e2, and A₂ 0, B, 0, O2C, 1f E3. We then have
1, if if ` 11 Q
E2) 3,
(w(1) T x w2) x ly=e,
0, if شリ UQQ nn (1)= = 2 and Σ 3 1 (Qaa(1)) =e3) =
otherwise.
Recall Definition T in the OOD distribution, have
E1, with probability
3P:
(1,
C22 with probability
e3, with probability g.
Combing Lemma 3 with the results above we have
(J) E [0,c] when Σ₁= (1) e2) (equivalent to (T(Q.,j(1) e2))
holds simultaneously)
Σ [(Q
..)( (1) = e3) = 3, the probability is 2p²/27.
Aood (f) E [1/2 c, when Σ3 I[Q ., (1) = e2) = 2 and Σラ (1Q e3) = lor (1) = =ea) = 2 and
Σ (1) = =e2) = 1). the probability
of which is C2pp"227 25pp///7=pp//..
(ood(f) E [1 e, 1] otherwise, the probability of which 1 8p22/7. /27.
So the overall expected OOD acuracy is Aood(f) -2pp//9- 1/2 (1 -1-8p'/211 1) [1- |p- /27 e, 11 - 5p /27 +e]. We have Aood (f) E [
/27 +e] following
the same proof.
(b) Output space ensemble and weight space ensemble.
Similar to the proof above, for weight space ensemble we have
(wi1) + +()) x + @))u=ee
Σ (1) + 5100(1)
Σ nv, (v, + Σ
Eus +
Σ 24
j=1
(1) (me, (1,( +
(L) (Hs, (1.()) +
-4 +
Ma (1) (Ma. Qu. (1)) +g
40

---

Published as a conference paper at ICLR 2024
We also have
(20(2) + (2()) x v= Σ (2)
(ps +h,
,=1
B3
(17(3) + an)))) 20 + 6o
Σ
(3))(3)
(Ms, Qs,(1)) +
j=1
C2
Then
2, if Σ
1 (Q, e2)
-1, 1, if 2 1ー- (Q., =e2) = 5 and if 6 -0,)) = 1,
(wo(1) Ta uo(2) xx)) l-es
0, if [E (Q. = = and
Σj= [Q,,())) -1)= =1)
Ej= (Q, = and Σ レ==
=2)
2 otherwise.
Then by Lemma
we have
[0, c). with probability 2((p/3) + (p/3)6) 14p0 /729.
[ c, 3 + c), with probability
(/wse) E
2(6 (p/3)² (1 2p/3) + 6C2 (p/3)9) = 2/ /243/44--2 + 4pp/81
[1 - c, 1], with probability 1 - 20ppp /729 4p"/81.
Then the overall expected OOD accuracy Aood (/wse) is in
[1- [1-2p' /81 /7pp /729 - E, 1 2p /81 - /7p ht/p//720 722 + +
The accuracy of the model ensemble and model averaging are the same in Example since
(w(k) + (k)) x(4 + x)lu=e, =
(m))(k) (p. +
(k) (uen) +
and
4 ( (((k)((x)))ue))))) Σ (k)( (u., + (k) (a.)Qa.((1) +
jml
F.2 PROOF OF PROPOSITION 2
Before starting the proof process, we restate Proposition 2 and Definition for K (K 2 3) class
situation as follows:
Definition (Data Generation Process). The whole data generation process are as follows:
y Unif E2, .... (x) x Concat ( dd U (
(o v) = N Qu,yy o² Po Po y) (Mu Qusy. oIa Vi,
(8)
where Qu,i, Qs,j E {0, 10 Further, Qo Is [er, cz.... exl always hold. the ID distribution D id Qa, Ixi and in OOD D ood the kth column of Q, i.e Qa,(k), is as follows for k = 1,2, K:
(ex; with probability 1 p-p Uniifee1, e2,..., ek), with probability p.
(k)(k)
41

---

Published as a conference paper at ICLR 2024
Proposition 7 (General Results for OSE). Consider Definition 1 nite ID and OOD samples. Omitting small constants involving we have
Assumption 2 hold, and infi-
(1 n) + Tle
(ood( Fp
v ns
p) p) + n4
(fo Pp
Vile
(m,, + (1)) + (nl + nw))
Aood(Sose) Pp
Vng + ng + 2n
In the proof process, here we take notation first:
I(I,...,,kk)) P=~N(0.0PIR_) (a + >0Vi 1...., --1)
in which
a - 2,
at aj
for any i i j-
Consider the extracted features both models as
( U { - U (a Re- U - u (er U (as,))
in which Thyos aoo are the numbers of overlapped invariant features and spurious features respec- tively.
Then for each single model, have
Ryo
+
Engi
Us,ir
3=1
i=1
Av Nuo
+ Σラ 1i=1
n1oo
U (k)
Σ ii=1 (,,((k)
Ene (s,i(h) = 1, 1,..., K
(k)
i=1
Vnl + n2
Ruo
120
nue
x
D +
2, 2 Dutt
Av Av-oovo
(k,
+ Σ
Ar nuo 7=1
(,,;(k)
Eizi (e,(k) Eie (k)
iik)
k= == K.
Vi + f.
Then we analysis the forecasting accuracy for both averaging model and ensemble model spectively.
F.2.1 PROOF FOR SINGLE MODEL
Considering the extracted features Algorithm as
{
10 U
1
for convenience, denote
R₂ +
= xΦ
Eo x
xz.j.
iml
3=1
then according Lemma 5 can obtain the estimated classifier on label ex:
1
ih (k)
Vio + n Ms
Nop
ng
2=1
j=1
42

---

Published as a conference paper at ICLR 2024
Based on this classifier, the forecasting accuracy on ID case is
P(ú - v)
Ex F {1(z T (((k) > Tkk)), AK' + h1]
K
P2..100 ((0(()) - T(k')²² + (z0(k)
0, *
K
1=
Σ
P
P=~ ((σ((k) uk(K))²² 8n VK' +
K
in which we denote that
Av
OE no + ng
Σ
1+
Σ
L,
3=1
for any k + k. And considering Assumption 2 we have
(w)k) wkk)) (wo(k) wi (kkk)) 1,
kk()) w(k)) 112-2,
for any l: K + k" Then with Lemma 2 the IID forecasting accuracy can be expressed
Ply D) ,))...,,
which can be influenced by ng Rg.
Then we turn to the OOD forecasting accuracy. For class k we suppose there are spurious features maintaining their parameters, and rk-*k refer to the number of spurious features flipping to the class K, the corresponding probability is
P([K,, [r--s,, AK' + kI]) rellll
n.
(1
p
P nm K
K
1 p)
K
and the conditional OOD forecasting accuracy on label e₁ is
Plù ex [FM, (r VK' + k]], = ex)
Ine WW Wk])) (1(6" w ((k)) x² (k:)) vk' + k)}
(2 (w(k) -okk))-((k))) wkk)) + n. Tkk TR-KK Wk' + k)
Vu, + nz
L( Ro + rh rk n.o + nr
vk' k),
according to this, the OOD forecasting accuracy can be expressed as
PP = y) = F.y PD = y v)) = Pgú el y = =
Σ
P(r [ri Ak' * [[ Ty +1 -I-K
4k' 2 1).
The +
P V22
with Lemma we can get related properties about L(-), then take upper and lower bounds respec- tively.
Considering the close form of G(.) in equation 13 we denote n, = nog Flg Rlv 70
0, C' = then the OOD forecasting accuracy can lower bounded as
PDù = v) 2 P(A)(1 6) + P(C(N))(((N) -
2 P(A)
Σ P(C(N))A(N)-C= - G(no, 0,0 c,
N=1
and the other hand, it can also be upper bounded by
K-1
P(g - y) < P(A)+
P(C(N))M(N) + cP(B) < Gim,,,,,,,00
ng, 0,0,0) +
43

---

Published as a conference paper at ICLR 2024
Similar with Algorithm 1, also get the ID and OOD forecasting accuracy in Algorithm 2, which is related to another ny invariant features and n, spurious features.
For the OOD forecasting accuracy, we'd like take some intuitive approximation for (n,, R, 0, 0, 0). As the number ner n2 are large enough, we can take approximation by multi- variate Gaussian distribution. To be specifie, denote (r1, -2...-.-.--.... -1-kk, then can regard them as r ~ (Yy, E), in which
[m, p pppp/)) K), n.p/ ,..,, n.p/ n.p/K]r,
Σ (i 7i)
-177
Di,
ns
If we denote new (K 1) K matrix
1
0
-1
0
T
And the new (K - d random variable, i.e,
T + n
still Gaussian, be specific, if we denote its distribution v (a M), then have
a (n, p) + n2) 1,
M n2, p(K + pK)
K
p(K 1 pKK)
Mij = 11.
K
G( n,, 0, can be approximated as
P(n: 0, .... MK-1 > 0),
which is equal to F, ((na(1 p) + n./)/ /n.), and (-) defined in Appendix F.2.5
F.2.2 PROOF FOR WEIGHT SPACE ENSEMBLE
For averaging model, denote
a x
Rv
Σ
Ryo
Rso
1
ne Ben
Ruo
8
2 0,3
2
0g,5
2
2
2
1=1
j=1
i=1
i=1
1=
i=
then after scaling, the averaging classifier on label ex:
v : (w + 2w)
T. nh nue (k) + 2 1-1
110 11o (k, + Li=1
Tiu-no Ruo (k, + -i=1
n Reo (k, + +2 2nn + (k) (k)
Ti, Tia nh + no 2n 2n
Based on this classifier, if we denote ñ = (n. + no + ni + n. + 2n + 2n the forecasting accuracy on case
P(O = v) = K
Em Bnyy=e, [ w(k) > a TTw(k))) Ak' # k)}
1=1
Pa)V(0,A0011) ((w(()) w(k')) + (w²(k) wo(k)) ex *
K
Σ
P (kb(k)) - TT())))T= z + 5,. > 0, #k).
K
44

---

Published as a conference paper at ICLR 2024
in which denote that
8 E
Rv
- 1 R. no 1 Σ 1 Eis 4 > 2i=1 4
2==1
1,
noe + ng ng + ne + 2n + 2no
for any W'' # k. And considering Assumption 2 have
(w)(k) 2 (k')) (w(k) - (((k) 1, @((k) - (kk) 113=2.
for any k * # k" Then with Lemma 2 the IID forecasting accuracy can be expressed as
Pyy Vwse) L 1) -
which can not be influenced by fu, Ra. nvs nug Ruo: Rsa-
Then we turn to the OOD forecasting accuracy and for each I,.... K, we take some notations as follows:
Fk [I(Ms,s (k) (s, n. Rgo
Pg {I(Ha, (k) us (k:)) R
r (T(Ms,s(K) (k) ps (k:) [na
(9)
FR-E { (k) Mo, (kk)) Rao
FR-* (1( (ks,(k))) 1ã,
k [(
(k)))
To be specific, for class k we suppose there are FR, Fx spurious features (no overlapped) maintaining their parameters, related to Algorithm 2, and correspondinlly, FR-s', fk- refer the number of spurious features flipping to the class k', and rf. Th are defined similar in overlapped spuri- ous features. Then denoting R& corresponding probability as
[T.. rR, rhe [ F Th Vkd of k]], we obtain the
(1 (1-p+ P y F(Rx(r)) (Fx + F ( uu (Fr fk 1r
(m. + n. Mm
K --1
K p),,, +h, -Pa-----
and the conditional OOD forecasting accuracy on label ej
Plù = ex (T), .. ex)
(e)R.()).v=e (1(#"<x(() (b()) > n" 1 (k'),, Vk' f
PP=-N(0,bo (z0(k) w(kk)) T nr + no + 2n + F + Fk + +rr Fk- Pk - 4r
AK +
Vī + ns n. ni + 2n + 2n
=L
No n. 2n + Fk + Fx + 4r ēk** PK-sk 4rt-w. -M VN *h), The + n5 + n. + n. + 2n + 2nso
according to this, the OOD forecasting accuracy can be expressed as Pg = y) Fy [P(r = y v)] = (p e, y = =1)
PRR n. + n. + 2n + F1 + P1 + 4r F1 - P1nsk 4r 1r1-k vk' # k).
Fhe + n. + n nu + 2n + 270
with Lemma we can get related properties about L (-), then take upper and lower bounds respec- tively. Still recalling the expression equation 13 with n,, n. + N.y n1 n, + Nu Thyo = Tyo: To No C 4, we can obtain the lower bound for OOD forecasting accuracy as
K-1
P - v) 2 (1))(1 - -) + P(C(N))(A(N)- e)
K-1
2 P(A)) +
Σ
P(C(N))(N)) - -c G( + No, nh + Mey Noo, Dso: 4)
N=1
and the other hand, it can be upper bounded by
K-1
P(g = v) < P(A)
P(C(N))((N)+ + cP(B) < Gūū + nv ng + ngg Revo Ason 4) +
45

---

Published as a conference paper at ICLR 2024
Similar to the analysis before, for ID forecasting accuracy, have
Ind = 0 I 3c,
and for OOD forecasting accuracy, we can draw conclusion that
Food 2 G( + uyy fz + nay Beoy 4) max R,, G(o,, Ra 0,0 3e.
Similar to the analysis above, we'd like to take some intuitive approximation for OOD forecast- ing accuracy in the ensemble model. As the number Rs, up. ns. Roa, Aso are large enough, can take approximation by multivariate Gaussian distribution. To be specific, then can r = [11, Fi-k]] & = (11. [1-kk] and r =
denote
regard them as F ~ N5. Σ), F ~ (Vr Σ) and re ~ N(r?; ,O) (they are independent), in which
7 = [m - m00) p + p/ K), (ī. pp K, ..., (m, - /so)p// k/]
Σ 3: [m, N0 7:)
-73
Σn
n. no
n1g - -os0
7 [( no0 ) (1 p + K), (īg -neo)p/ K K...., (mz m0o) KJT.
E. 7, 20
- 51)
-98313
Egg ñig 12:0
n. no
20 = [n (1 p p/ K) //oD/ K, ..,,.., Rlao K)T,
2/ (M 7
10.0
Uso
If we denote a new (K - 1) x K matrix as
1
0
1
0
T
0
-1
And the new (K - 1)m random variable, i.e,
n = (T, T, (F, (f, FO + (n. + n. + 2m )1 is still Gaussian, to be specific, if we denote its distribution
~ N( M), then we have
((ñ, + n. + 2n (1 p) + n. + nr + 2n 1, p(k + 2 pk)
M = (f. + n. + 14n
K
p(K + 1 - -pK)
M. = (7. + n. + 14
K
Gi + ny. nl + Rgy Roay Dsay 4) can be approximated as
P(m > .,.... 7K- > 0),
which is equal to Fp (( + n. -2 - p) + nr + no + 2n / + n. + 14m and
F,()) is defined in Appendix F.2.5
F.2.3 PROOF FOR OUTPUT SPACE ENSEMBLE
For ensemble model, we also denote
(k) Mu R (k. + Ein (k) + Σm
n. us (k)) 2 n00 (k.((k) + 2200 (k.)
+
Vn. + no + n. + no + 2noo + 2n.0
then the forecasting accuracy IID case is
P(ù = y) =
Em Emiy=ey (1(x0 w(k) + xτ" ((kk) > xΦ "x(k())) + x " # *b)
K
k=1
k
P P2 (lσ(k) w(k')) z 80,0r 0, # #k).
K
46

---

Published as a conference paper at ICLR 2024
in which
2i=1 Mow Mou + 2==
B, 10 1 -i=1
Mo0 1 プラ=1
n 20 20o 2 -i=1
1=1
v + ni ny ns nn + 20 + ns ny nss nloo Nlso
n. n. n. + nis
v no
no + n. 2n 20 v n.
+ n. Rsoo
while Tuo + Tiso < (fh + ne + n. + n.) for any + k. And considering Assumption we have (wb(k) (('))) (w(()) )(k*))
|l (o(k) w(kk) ) 11 2,
for any * k/ * k²,"" Then with Lemma 2 the IID forccasting accuracy can be expressed as 1-c,
P(y 1) L( (s.........
which can be influenced by Re, Rl, Re, Tex, Pooo Than- Then we turn to the OOD forecasting accuracy. Similar to the notation in equation 9 for class k we suppose:
Fs [[I(,,, (k) = (k)( Reoo
[[[H,,, (k) us,, (k)) "e
r = (KT(Hs,(()) = M4., (k.((k)))
F (kll(ps.. (k, (k) 1s0
F ((l((ss. (k. 1)
r ((T((ss; (k }) [arl
Then denoting (R(r)= = [Fx, FRi E (Fe-s Sk r se Co Ak' * ki]. we have the corresponding probability
((Rx(r) = (Fk + Fx frllllll ex (Fk sk Fx Fx (1-p+ P t K K-1 and the conditional OOD forecasting accuracy label is
(.. + n. 2m !n
D) nr tū noo -
K
P(ý = Ex RR(r)), ex)
B { (1
(lo (k()) + x (i(k) ad wo(k') ) + x (k'k)) Vk' * k))
+ to Fk nx + 2 Fo PR 2r
=L( Vn, + n. + 2n.o 20 v no f. n. + nr Nuo 1l20
vk' + k).
According this, the OOD forecasting accuracy can be expressed as
P(ğ = 1)) = Ey = #)] = Pi e1 = e1)
nhe A. F1 F1 + 2r9 - F1 FI 2r w
Σ (())(
wk' + 1).
Vnu n. Toe 2noo 2n Th Ti noo neo 1100
(b)
with Lemma
3 can related properties about L(-). then take upper and lower bounds respectively.
To be specific, recalling the expression in equation 13 with n. R. + nu, n. no Roo Buoo Uso
Mso; C = we lower bound the OOD forecasting accuracy as
P(ツ = 3) 2 P(A)(1 -) c) + Σ
P(C(N))(((N) -c)
K-1
2 P(A)
Σ
([CN)))(N)--= G(A, + no ns ne Reo, nso. 2)
N=1
and on the other hand, it can be upper bounded by
F = (P) + Σ P(C(N))A(N) + <P(B) S G(üy + No. nhs + its Rvo, Nooo 2) +
Similar to the analysis before, for ID forecasting accuracy, we have
Thd 3
47

---

Published as a conference paper at ICLR 2024
and for OOD lorecasting accuracy, draw conclusion that
Food 2 G(n, Re., + Naa Reas Roo, 2) max{G((n))) Rzx, 0, 0, 0), G( n., 0,0,0) 3e.
Similar to the analysis above, we'd like to take some intuitive approximation for OOD forecasting accu- racy in the ensemble model. As the number nu Ra, ne, n. Reo, Rso are large enough, we can take ap- proximation by multivariate Gaussian distribution. To be specific, we denote f11: (f--g [1, [11,P1-22,... f1-5k] and r [ri, (frr,.2,...... ri. then can regard them f (F, Σ) and r" ~N(r"; ; Σ") (they are independent), in which
2),
[( no00 (1 p P/K), (n, nnoolp/K K.... (n, Nso)p/K]²
Σg -M.
7%))
Enn -7.55
71 10.0
n. nl50
[n, )1
p\ K), , (ī, n.o)p/K. K,
(7. m/o)p/KK]"
Σ. (m, Ns50
- %1
Σnj -9.53
N1 Pb.n
Nue 11x0
7 = [1 - p P/ K), nsop/ / ,,..., K)²;
E1 = 7f (m 7?)
Ei
11s0
denote new (K 1) K matrix as
-1 ...
T
And the new (K- 1) dim random variable, i.e.
n (T, F")"" + (no + fl, + 2m
still Gaussian, specific, if we denote distribution as v ~ N(x, M). then we have the OOD forecasting accuracy can he approximated as
a ((ns 7.)(1 p) + ny + Nt) 1,
M = (īx + n. + 2 p(K KKK- + 2 2-p
K
p(K + pk)
M.,- (f. n, 2m.a)
K
Pm > 0, 7K 1 0),
which is equal to
+) + R.. Vng + n. + 221zo) and F,()) is defined in Appendix F22.5
F.2.4 CASE STUDY FOR 3
To interpret the improvements on OOD accuracy of model average model ensemble, here we set 3, and further take an insight on the representation function G(.).
Recalling the results calculated above, the OOD accuracy for single models can be approximated as
G( n. 0, O, 0), G(ü,, n. 0, 0, 0),
and for average model and ensemble model, we could focus on
G(n, to. Al 7s, Neo, Rso. 4), G(nv Ne. + Ns. Tevo Nso, 2).
To take specific calculations, we denote the random vector as [ri, rf. P1-33, r--22 rf r- and approxi- mate them probabilities related the -oo-dmenniionl Gaussian random vector 7 ~ (O, H). ), in which the covariance matrix H has components as
H: pl5 3p))
p(4 3p)
Hij
3
3
48

---

Published as a conference paper at ICLR 2024
Then could obtain
G(f Rr 0,0,0) = P((m 2 (1 p/mp the
,122 -1
p] ny
Thy
v
G( R. 0,0,0) = P((m 2 (1 p)ñg ne
112 2 -- p/ ng
Vin
v
G( + Nu, nh + ni, Yovo: Nhan 4)
=P(m 2 (1 - (m, + nr + 2n + No + no + 2noo
112 2 (1- (f., (1-p) + no + 2n + no n. + 2noo
Vnc. + nea + 147
Vn. + n. + 14
G( + Nv Rs + ngg neo. nooo 2)
P (1 (n, ū.) + + no
122 (1 (n, n.,) + Fer + no
Vn, iz + 2m2o
V + feg
Here we denote a new function
F(c) -2, -2),
which implies that F is monotonically increasing with respect And shows that average model and ensemble model could obtain higher OOD accuracy compared with single models due to
(1 (n, n. 2M,.) n. no 2n
p)n, th (1 p)ñ,, + No
2 max
Vn, + 14n
Vnu
Vin
(1 - (n. n.) n. + nh 2 max p)n, + no
p)ū, + any
Vn. n, 20
Vnu
Vmo
F.2.5 CLOSE FORM OF Fp(-)
Here we provide the explicit expression of function Fa(x:) in K class situation, which is monotonically increas- ing with æ.
We denote a K 1-dim random variable 1 ~ (N, M).. in which
pl K
2 pk)
M., = p(K + 1 pK)
M
K
K
then (x) is defined as
Fx(x) P(m ,,..,,77--11 0).
F.2.6 CLOSE FORM OF G(n ng Noor Nooo C)
First, denoting a random vector (r) = [Tk, The [TK- VK + k]] E R2K we have the correspond- ing probability
P(R+(r) = re!r!!lI, w 11
(n, 2n Maz!!
-pp 2rttt2
K p) -
wITh
then define several sets:
A (Ra(r) in + Crî -I-M Cri-x + ni > 0, O, = =2, 2..... K},
(10)
(Ru(r): min ri + Cri P1-1 Cri n, <0}.
(11)
2.....,KK
C(N) = (r)
min n + Cri - F1+# Cri + nu = 0, #=2...KK
the minimum can be achieved by N values):
(12)
and related functions as G(us, Nu, Teso, neo. C) the probability defined as following:
((N)= = P P=-N(0.0313) (a (,.=>0,W-=......
in which a. a, 1 and || a, | 1 for any i j.
G(ne; Dhyy Boos Truo C) PPA)
P(C(N))(()) N=1
Σ
(13)
where set A and C(N) are two sets of (r(r)) defined in Equation equation 10 and equation 12
respectively.
Note that (P (Rx (r)) and the set and C(N) all depend on ng. Tey Bsay Beo and C.
49

---

Published as a conference paper at ICLR 2024
F.3 PROOF OF PROPOSITION 4
By the proof in Proposition we have
(Tk) Σ Mu, (k) +
Σ (k)
j=1
6
((k) Σ (k)
Σ (k)( (k).
Then we consider averaged mode about the value of th" a.""""" where w+t 142
and $
2-10
T+A
We first have:
(k(k)
M),(k) + Σ Mo)(k)+
Σ Mag(k)+ +
E Mag(k))
1 1+1 A
1.2
633.4
5-1,2,3
1-4,5,6
(r \ob")
y=1
1
Mo Qu, (1)+ + E
Me
1=3,4
Σ
MajQes +2 E (15Q + Σ 2; Σ
2))
1=1,233
$=4,5,8
④ (x)
1==e
(1 + X)2
(k)) pe, Qu. X
Σ (k))
Ma (1)
1=3.4
Σ
Σ (h) pr (a.Q..(11))))
1=1,23
3=4,5,6
Then for class k = 1, we have
(1) ( 2 ly=ei
+
+222 +
Σ (ss(1)P
Ms, (s, X2 Σ (e,00)"ps)900000
(1 + X)2
>12
1,2.3
25 3-4,5.6
For the other two classes, we have
(2)) "="""" lo=e 2
Σ
Ma, (2)
ua + X2 Σ (2) Q..
(1 x)2
f=1233
>6 1=4,5,6
w6)) -Φ y=c1
Σ (3) 1=,,33
Mo
Qa.
x²
Σ (3) (s.Qaa(())
X)2
>5
4,5,6,6
For simplicity of the discussion, will ignore the constant factor xt
(1+A]Z> when
v5, we discuss the
value of (o(1) x - 22) x lymer
When
Σ I(Q., (1) e2) it suffices consider comparing:
$=4.5.6
2 + Σ م Moy (1) Mr (a.) + X X² j- (es Mo. (1,(1)
Σ E1=123 (2) p. (1)
(11) "6""" "0="""=""
ly=e1 S (2)" x T
only when
Σ T(Q = e1)
0. that is,
f=4.5.6
Σ (1Qa =a) 1.
4=66566
The probability of the aforementioned scenario is 3 (p/3)3 =p²/3 ///
Then
(1)" " ly=es < <0) . y=e, if Σ
I(Q=,(1) e2) =3
j=1,233
((1) Z ly=e; = =T xΦ" ywer if
Σ
I(Q ex) = 2 and
Σ [(Qa,(1) ex) =
2m1,2,3
J 1,233
50

---

Published as a conference paper at ICLR 2024
Therefore, the probability of is" (w/3)² /244,,"""""""""""""""""""""" (p/3)² = /811
When E,=45, e2) = 3, suffices to consider comparing
Ms, (1)² Me. Ma. 2 +
-- Now (c. X²
Trivially we have 1,=12, (a 22)²2 pa,, Qn,,
52 52 (a)(1)² Mss M.. Q1,(1), therefore,
<<" holds under this case. the probability is (p/3)² = /27.
When En=i,s UQQ.. (1) e2) 0/1,
Σ (2) Qa. (2, X² Σ
(n, 3 2 2x + +.. Therefore,
$=1,5.6
< is impossible to hold in this case.
To sum up, for comparing the first class the second class, the " probability ps/22233 + /27, the probability is p"/81.
Generally, the "<" probability is 2p" /213 2p' /27, the "=" prohability is 2p /81, the otherwise prohability
is 8p /213 2p" /27. Then the total accuracy approximately 1/2 /81 8p /243 2pp /27) 1 5p /243 2p /27, that is, accuracy lies - 5p", /213 2p / - 5p /213 2p 2 + a].
F.4 PROOF OF PROPOSITION 5
Proof. (a)Two individual models. The accuracy two individual models are the same with Example |
following the same proof. Specifically, we have Aood (f) E [1 5pp /54 5p²/54 +).. Aood (f)
5p /27 - e, - 5p /27 +
(b) Weight space ensemble. We first solve the and a on the infinite ID samples. Lemma 5 for h 1, 2,3, have
((k)
Mns (k) + Σ (k)()))
j=l
((k)
(k) + Σ
Ma. (k))
j=3
So we have
n (k) + mb(k) = Σ (k) + 2k),,2(k) + Σ
(ko(k) + (kk
j=1,2,4,5
For samples from the first class, we also have
x ④b 01/== Σ Mo. (o. 2p. 20.2))11111
Σ ps. 24. 2
1=12244
where 2: ~ NN0, Ta), Vi. have
(w(k) + w(k))
x + مل1
Σ (k) (u..Qu.:(1)) + 4p).2(k) (u2:2Qu2(1))
1,333
Σ (ks(h)
(H., (.s.))as(1)) (k)) (H,,s) (H,,3..s(1)) +
j=1,2,455
So
(wi(1 + x(1))
=6+
Σ (1) (H. Apn,((1) Q#,,)Q,,311) +
1=,,
51

---

Published as a conference paper at ICLR 2024
Similarly, we have
(#x(2) - (2)) ( x m)lu=e =
Σ (2) (Ms, (Ms,)Qss(1)) + 4ps,,(2)) (Me,) + +
5=1,2,455
(w(3) (3)) x D)lv=
Σ Mar(3) (M.,Qaa(1)) 43),,(3) (Ha,)..a(1)) +
1-,,2,,5
Then
(00(1) + w(1)) (6+ ((6(2) (2)) ( + +)
-2, if E1=1,2.55 (Q),((1) = ex) 4, and ((Q).(1)) e2) =
-1, [(Q),,(1) = 62) = 1, 1, -
= =3, and
(Qn((1) es) =1):
O. if (I(Q,,(1)== = =2) = 1, j= 1112445 = =3 =3 and E;=1244 (Q+, = = =1)
(I(),,(1)= = ex) 1, 1. = =2) = and Es e3) 2
2 otherwise.
Then we can compute the probability respectively,
For -2 case, the probability is given by 2 (p/3) = 2p 2p222222 /243
For 1 case, the probability is given by 4 (p/3)² = 8p /243
For 0 case, the probability given by 2- (4 (1 2p/3) (p/3)*
4 4C2 (p/3)²) 8p"/81 4p /243
Otherwise, the probability is: 8p" /81 2y /81
Then the total accuracy can be computed as approximately 1 1- 4p" /81 11888-188 -8p' /243, the interval is 1 - 4p /81 8p /243 e, 4p /81 8p /243 cc
4p
(c) Output Space Ensemble. Similar to the derivation of model averaging, we have
w(k) xΦ + =k)) x0lp-er
Σ (k)(k) Qu.(1)) (k):2(k) (u.:2Qu:(1))
Σ (n)(k) (4.) (1,3)..3(1)) +g.
Then we consider fist class:
w1) x + (1) 20//mme
Σ pms(1)) (Ho, (1) + 21),)(1)) (1,22Qu,(1))
Σ (T)(1) (Ms, Qa., (1)) (1) (H,)) +
1=1,2,,55
44
Σ (1)
(Hr,Qun + (1)
(M.3Q.,(11) + g.
1=1,,455
Similarly we have,
(2)) xΦ (2)
x0ly=e
y=e j=1.2,4,5
Σ (2) (Mo Q1) 22,,,(2) (pa:aa21)))) +
w(3) x T t033 x0/y=en
Σ (35(3) (M.. 23..(33) (4.. +
1=1,2,46
Then
(w0(1
x + x(1)
xΦ w x \=
if j=1.2,4.5 I(Q+,((1) ((Q_0)) e2) and Q.,((1 e2
1, if |=
((0_(1)( 62) 1 (1) =1 (In ea
[(Q+,(1)
0, if (Σ 1,,2,44555 (1Q = = =2 and Σ TQQ, ea) =) and Q..((1) = =2)
or
5=,2,445 (QQ,(() e2) 4 and Q..(1))= (s)
1. e2] 3, 3, 10 (1) or
= 1, (L)) eअ)
1 otherwise.
52

---

Published as a conference paper at ICLR 2024
Then compute the probability respectively:
For case, probability given by (p/3)² 224
For 1 case, the probability is given by 4- (p/3)' (p/3) = 8p8
For O case, the probability is given by 2- 8p" /81 /21/ Otherwise, the probability is given
0/33--0033--00/33---22211010-110 (p/3)) =
8p" 582/811 8p'/213
Then the probability can be computed as approximately [] 4-0 1 /2/ - E, - +
F.5 AUXILIARY LEMMAS
Lemma 1. For any N i.d random variables { ~ N(0,0²)) and > 0, with probability at least
1 Ne 20?, we have
+, for any i= i
Proof. For any index i, according Markov inequality, with any À O, have
Asd Eh expl X=00
P(#: < 0) P(e ee1)
Xt},
taking the minimum value on the right handside, with respect we can get
P(z: +) S expí- 20
Then considering the random variables through index ,,...,..
,.
P(min + 0) = P(e de; 2 e 1) & Blexp(maxx dzs - A()),
while
Eexp (max )z,} = E8 max 6 drt $ Σ Be s Ne 23
we can take the minimum value the right hand side, with respect with A, then further obtain P(min + 1 0) & Ne -8/22
Lemma Suppose that (O, o2 La). k vectors {a.,..., a and R for I...., then by Sraa--Shmidt process, the probability of
a x + 61 0,
0g 6b-
equivalent to the probability of
81
C1
+
0.
a1
2
a
U1
a 01
82
222
C1
0,
a2
v1
1|
a2
[|
11
1|
a2
4T Vs
8
ee
C4 +
>
0.
a
|2l|
Vi
a
1|
which {er} are ii.dd NN0, 02) and
{v,
1- 1=1
are orthogonal vectors span on {a {a. 1--1 as
a
V1-
a 11
(a U1
)
a2
1 (a} m1)
a
-a 1
0k-1
V a -k-- (ap_ v.)
53

---

Published as a conference paper at ICLR 2024
Lemma' 3. Just consider two classes K 1.2 and denote T1-42 the number the events (I(Q,,(1) 02]} } that holds. Suppose Assumption hoold, then
when n, + nhv 2r1->2 > the related probability P((w(11 un(2)²² 2 x lymer) larger than vI-
2. when n2 + +ny 2r1-b2 = O, the related probability P((w(() = 2 xΦ > 0 (y=cn) in 11/2 6, 1/2 + tc.
when n, + The 2 < O, related probability (P((w(1)- w (2)) xΦ > Iy=e)) than 6
Proof.
This can be directly deduced by using Lemma which is the more general version.
Lemma 4. Denote t as the number of the events (T(Q.a(k)- = ed) that hold. Suppose Assumption hold, then for class k.
when n2 + nhy Eieeh M/ > maxi4k Tk-t the accuracy is larger than
when + -Eift Ph-t1 maxi4i Tk-+ denote N the number of the events that holds
[T(x=>> = n1 + Ev Fk-21)) the accuracy in [1/(N + - 1/(N + +1)+ e),
when nı + nh Eize Mh-si < maxias Th-b the accuracy is less than e.
Proof. Considering the conditional forecasting accuracy on class k. with respect to Tk-+1,... K is equivalent with G[[n= + nhe Eifle Th-st Pk- Vs of k)), which is defined previously. Then we can take analysis case by case:
n1 + n2g Eiju Tk-t > maxise Mi-ti
In this case, elements function G(-) are larger than which means that no smaller than 1/(N++ NY.. With Assumption we have
G([ +
E Th-A.- k)) F 1yh
21 -
ONN + N!)
+2 Eieu Tk-N maxi Tk-b1
In this case, all elements in function G(-) are no smaller than 1/(NV, + NY), except N zero elements. With Assumption I have
G(in The Σ -
2 c) 2 2
n, + n2 - -iife Tk-t! < maXiys Pi-H1
In this case, there is at least one element G(-) no larger than -1/(N3 + N:), still considering Assumption [
we have
G((n, + nu
Σ
Ph-st Pk-aa 4 k+) F(- o(N + +N)+ S.
14
Lemma With features ( Vhe and (xag D2 the classifier trained on infinite samples is equivalent with the mean value of Din + yn Rz.. R...
Proof. Considering the classifier, should max.. P(y (xx) T "). From Definition we have y Mi y. (r., + nu))" oo)) in which fa = En Mat + E Ma,j- For simplicity, we denote z æ $49 and is independent of y..
54

---

Published as a conference paper at ICLR 2024
Given samples {(x, ):) maximum likelihood estimation (MLE), have
expl (as jo a. (w y:)}
arg max I Ee, v expl (o n (w ez)}
arg Max then taking derivative for each we have
(ny + noJo (wo )
Σ log Σ
expl
(no + n.)oz (w ( (woes))
oj
expl (m, n ms)o Tq wk)
2. 0.
K.
(nw + oo
(no ns)o? Ex, expl (notm.)o= a (w ow))
On the other hand, using Bayesian formula to consider the conditional expectation of æ, we have
Ex ex))
BxxE 1 ex) x))
Ta(:: y ex) ply ek)
ply Ek)
(n ooz x (1 es)] exp(
E(æ et) KE
x - expl
to 2. (μ e,))
implies that sample size goes to infinity, the following
maximize the likelihood function:
k) pa o ex Mud
Σ
1 eb:
J=1
for k = 1, 1,.... K. And by scaling the classifier, can get the estimated classifier as
w(k) VTe nhe
Ck. C.k.
for any class & 1,..., K, which is the same as (1/vn, + ne. Ew w=e, [
ex..
G ILLUSTRATING THE THEORY OF WISE-FT
Recall Definition that, learns ñ. invariant features and n. spurious features, as well as another single model f has n. invariant features and π. spurious features. Further, f and learns n.. overlapped invariant features and niso overlapped spurious features. Let f denote the pre-trained model and f denote the fine-tuned model. WiSE-FT specifically is the following: has good OOD bad ID, has bad OOD but good ID, and weight space ensemble of f and ff has excellent OOD performance. These be expressed as:
Aid(f) < Aia(f)) Aood(f) > Aoodlf),
Amad(furc ) > max(Annae) (f), Anoa(f))
It straightforward that the ID accuracy satisfies the following inequality due to Assumption Aie(F) < if The n. nop n..
Intuitively, if a model learns more feature, it can predict the label better in the ID setting. As for the OOD accuracy, by Proposition
have
Abod Fb (1 p) n. + Bu ). Aooa(f) = F, ( nL + n.
Vn,
Vnx
Furthermore, by Proposition 3 the OOD accuracy of weight space ensemble (WSE)
Aood(Swec) = Fp ((1 (1- (n. + n, + 2n + nh. + ne 2n Vn, + n, + 14Tco
55

---

Published as a conference paper at ICLR 2024
Then the WiSE-FT phenomenon can be effectively explained if the following conditions holds:
Fly + Fig Ai + 18g: (l [1-p)agthhe
Kiy [1-p](R, +2n..) 44, Aa 20
max (1 t (1-p)R.-Bi Wig
V
The theoretical results above characterize the conditions for the WiSE-FT phenomenon. To gain better derstanding, we use concrete example for illustration: p = 0.9, there no overlapped features learned by two models, i.c., Rso Rioo = O. The pretrained model f learns some invariant and spurious features, i.c., Yhe 2. nh, 4. The fine-tuned model learns more spurious features and less invariant features, i.e. te = 1, A, 6. In this example, the fine-tuned model f has better ID performance than the pre- pre-trained since ñ. + n, = > Rg + n, = 6. The fine-tuned model ff has worse OOD performance than the pretrained model f since focuses more spurious features. Specifically, we have Aooa (fo Aood (f) since
p)ñ, +
Aooa(f) = Fi
VTg = (y.20.
Aood( (f) Fp
pln, + No
(0.97.)
vn.
Based on Proposition
3 OOD performance of wse is
Aood Fp (1 - (+. te, + 2TE + The + n. 2nooo
= Fp(1227).
Vng + ngg + 1dn,o
Recall that Fp(-) is monotonically increasing, we can see that Acod(fuse) > maxt (oodl f), Aood(f))
H ILLUSTRATING THE EFFECTIVENESS OF BANG THROUGH THE LENS "ACCURACY ON THE CURVE"
Considering that Mixup and Label Smoothing (I.S) enhance the OOD performance of fine-tuned model, we investigate whether the improvement achieved by BANG is primarily due to better calibration or the fine-tuned model's enhanced OOD performance. In Appendix E.6 we present our findings. which include the following observations:
Dividing the weight of the vanilla fine-tuned model by multiple scalars significantly enhances the performance of weight averaging, closely approaching the performance of BANG.
BANG demonstrates ability to correct substantial number misclassified samples compared to the fine-tuned model.
To further investigate the performance of BANG, examine the concept of "Accuracy on the Line" (Miller al.] 2021] Liang al. (2023}. We generate many checkpoints of vanilla fine-tuning by using different hyper-parameters, Specifically, we fine-tune the model using various hyperparameters, including learning rates (le -8 .e -5 3e 5e "), training epochs (4, 8, 10, 12, 16, 20), and learning rate schedules (cosine, step de- cay). Notably, the default hyperparameters used in Section 4 and mentioned in WWrtsman et al. (2022) are a learning rate of 3e 10 training epochs, and cosine scheduler. Weight averaging is applicd to each fine-tuned checkpoint with the pretrained model.
Figure 21 illustrates the OOD performance each fine-tuned model, as as averaged model com- bines the fine-tuned model with the pre-trained model. Interestingly, we observe that the OOD accuracy of the averaged model forms a quadratic function with respect to the OOD accuracy of the fine-tuned model Liang et al. (2023). rather than linear relationship described in Miller et al. 2021).
Furthermore, BANG demonstrates significant robustness in OOD scenarios, surpassing the of expected performance.
56

---

Published as a conference paper at ICLR 2024
65.5
OOD Perfromancc of the Averaged Model
64.5
BANG
WISE-FT
63.5
62.5
61.5
-0.016xx* 2.0045x 2.803 RR 0.9637
60.5
59.5
59
OOD Perfromance of the Fine-tuned Model
Figure 21: Illustrating the effectiveness BANG through the lens of "accuracy on the curve".
57


Reviews:
[{"review_id": "9b0nyAqJsm", "reviewer": "Reviewer_kd2C", "summary": "The paper analyses ensembling to improve OOD robustness. They find that the individually incorrect models can average out to be correct when, loosely the models do depend on the invariant feature sufficiently in both models. They setup a linear data generating process and provide some evidence of when linear", "questions": "See weakneses.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper studies an interesting phenomenon where individually incorrect models can be made correct. The idea is related to boosting, but the different models should rely on different spurious features.\n\n - The experiments are direct and encouraging, although not directly connected to the theory (WiSE-FT and WSE seem to not be compatible because the methods that work for WSE cannot be applied directly to WISE-FT.\n\n - The theoretical setup is interesting and does confirm some of the intuition of ensembling models that are diverse in their spuriousness.", "weaknesses": "The paper's weight space ensemble is too simple, when considering overconfident models. WiSE-FT ensembles all the weights, not just the linear layer. So it is unclear how the scaling can be assumed to be a simple multiplier on the linear layer. \n\nThis is a sizeable gap because WiSE-FT is weight space ensembling of very specific models (good OOD/bad ID and good ID/bad OOD). This is not reflected in the paper except loosely when they say low overlap is good weight space ensembles. The authors acknowledge this even when they talk about their method \"However, this method can not be directly applied to WiSE-FT since WiSE-FT ensemble model weights instead of the outputs.\"\n\nThe paper does focus a lot of output ensembling, which is not known to work as well as WiSE-FT. In that case, there is no discussion of boosting, which seems to have the same flavor (especially when forcing models to rely on different features).\n\n\nIt is also unclear that Mix-up and label-smoothing have singular effects on the confidence, they also regularize. So BANG is really just finetuning with MIXup/LS and then doing WiSE-FT. A more direct correction of confidence would be the right one, like doing temperature scaling for each checkpoint during finetuning and then running WiSE-FT", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eznsHBij7d", "reviewer": "Reviewer_KdWP", "summary": "The paper examines WiSE-FT to understand how ensemble based models improve OOD performance. First, a “FalseFalseTrue” phenomenon is observed, where the ensembled model predicts samples correctly even when both the pretrained and finetuned model that are ensembled predict the samples incorrectly. The paper suggests an explanation for this related to ensembled models making use of diverse sparse features, and this reducing the influence of any given sparse feature below that of invariant features. This is then shown to be the case theoretically, and on a synthetic empirical setting. Finally, the paper makes the related observation that successful ensembling requires balanced models (so that the ensembled model benefts from diverse sparse features). This motivates a recommendation to finetune models with mixup and label smoothing to decrease confidence, and the resulting ensemble model is called BANG. BANG shows empirical performance stronger than that of WiSE-FT.", "questions": "* In tables 2 and 7, we see that on the in-distribution ImageNet test set, finetuned(Mixup+LS) models outperform standard finetuned models (by ~1.5%), but BANG does not outperform WiSE-FT. Is there intuition for what BANG is doing in the in-distribution setting that explains this?\n\n* It would be good to see “FalseFalseTrue” analysis (like figure 2 left) of BANG models compared to WiSE-FT models. It appears from appendix E.6 that BANG models may be helpful in a “TrueFalseTrue” setting where the finetuned model was originally incorrect and overconfident. Does calibration help in the FalseFalseTrue setting?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Weight space ensembled models have been of great interest for OOD generalization and an explanation of how these models improve OOD performance beyond both pretrained and finetuned models—which this paper seeks to provide—is valuable.\n* The FalseFalseTrue phenomenon is unexpected and interesting, especially since the FalseFalseTrue ratio is comparable to the total overall improvement provided by WiSE-FT. \n* The analysis provides a partial mechanistic explanation for how EBM improves OOD performance beyond pretrained/finetuned models via diverse spurious features.\n* The MultiColorMNIST experiment clearly demonstrates the value of diverse spurious features for the OOD performance of EBM.\n* The observation that imbalanced models hurt EBM is actionable, and the algorithmic recommendation of BANG leads to OOD accuracy improvements over WiSE-FT.", "weaknesses": "I am not convinced that the performance gains of BANG are mainly due to better calibration as claimed in the paper and not because mixup/LS significantly improves OOD performance of finetuned models. While appendix E.6 shows that there are “oracle weights” to scale standard finetuned models when ensembling, it may not be that BANG models are bringing improvements for the same reason.\n\nOne way to demonstrate this would be analysis similar to “accuracy on the line” where there are models finetuned with mixup and LS at the same OOD accuracy as standard finetuned models. It would strengthen the claims of Section 4 if the BANG models in this setting outperform WiSE-FT even with finetuned models at the same OOD performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xeZfvmMZdI", "reviewer": "Reviewer_qFv2", "summary": "This paper investigates the WiSE-FT method's ability to enhance OOD performance in machine learning and identifies a notable \"FalseFalseTrue\" phenomenon that shows WiSE-FT correcting both individual model errors. Motivated by this, the authors conduct theoretical analysis in a multi-class setting. The theoretical results suggest that using diverse spurious features in ensemble methods improves OOD generalization, contradicting the traditional emphasis on invariant features. This theory is validated with experiments on the MultiColorMNIST dataset. Meanwhile, the authors also identify a weakness of WiSE-FT due to its overconfidence in fine-tuned models under OOD data. Based on that, the authors proposed a new method BANG that addresses the overconfidence problem and reaches better OOD performance.", "questions": "See in weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written with a clear structure. Motivations are well-explained on why the authors study the problem and the contributions of this study are well discussed. The illustrative examples are helpful in understanding the concepts. Overall, it is easy to follow the logic and flow of the paper.\n2. Theoretical results are solid and well-organized. The authors made the theoretical settings clear: definitions are well-explained and assumptions are clear. Proofs of the theory are sound as far as I read into.\n3. Empirical results support the theoretical findings. The proposed method is also shown to be effective on the ImageNet dataset.", "weaknesses": "I have the following questions regarding the empirical results\n1. The paper considers the setting of ensembling two individual models (FalseFalseTrue) and show that such ensemble indeed improves over single model's OOD performance on MultiColorMNIST. Is this finding generalizable to ensembles of more than two models? If we increase the number of models in Table 1, could the ensemble's performance potentially see further improvement?\n2. For datasets that specifically focus on spurious correlations (Waterbirds), there will be only one strong spurious feature (background) in the training data. In such a case, will the ensemble method still be effective? If not, could we manually adding new spurious features to the training data (color, rotation, scale, ect.) and apply ensemble methods like BANG to improve the model's performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)