Feature Reconstruction From Outputs Can Mitigate Simplicity Bias in Neural Networks
Authors: Sravanti Addepalli
Anshul Nasery
Venkatesh Babu Radhakrishnan
Praneeth Netrapalli
Prateek Jain
Venue: ICLR 2023 poster (2023)
Keywords: Simplicity Bias
Out-of-distribution robustness
OOD Generalization
Deep Learning

Abstract: Deep Neural Networks are known to be brittle to even minor distribution shifts compared to the training distribution. While one line of work has demonstrated that \emph{Simplicity Bias} (SB) of DNNs -- bias towards learning only the simplest features -- is a key reason for this brittleness, another recent line of work has surprisingly found that diverse/ complex features are indeed learned by the backbone, and their brittleness is due to the linear classification head relying primarily on the simplest features. To bridge the gap between these two lines of work, we first hypothesize and verify that while SB may not altogether preclude learning complex features, it amplifies simpler features over complex ones. Namely, simple features are replicated several times in the learned representations while complex features might not be replicated. This phenomenon, we term \emph{Feature  Replication  Hypothesis}, coupled with the \emph{Implicit Bias} of SGD to converge to maximum margin solutions in the feature space, leads the models to rely mostly on the simple features for classification. To mitigate this bias, we propose \emph{Feature Reconstruction Regularizer (FRR)} to ensure that the learned features can be reconstructed back from the logits. The use of \emph{FRR} in linear layer training (\emph{FRR-L}) encourages the use of more diverse features for classification. We further propose to finetune the full network by freezing the weights of the linear layer trained using \emph{FRR-L}, to refine the learned features, making them more suitable for classification. Using this simple solution, we demonstrate up to 15\% gains in OOD accuracy on the recently introduced semi-synthetic datasets with extreme distribution shifts. Moreover, we demonstrate noteworthy gains over existing SOTA methods on the standard OOD benchmark DomainBed as well.

Published as a conference paper at ICLR 2023
FEATURE RECONSTRUCTION FROM OUTPUTS CAN MITIGATE SIMPLICITY BIAS IN NEURAL NETWORKS
Sravanti Addepalli
Anshul Naseryf
Praneeth Netrapalli
Venkatesh Babu R.
Prateek Jain
f Google Research India
Indian Institute of Science, Bangalore
ABSTRACT
Deep Neural Networks are known to be brittle to even minor distribution shifts compared to the training distribution. While one line of work has demonstrated that Simplicity Bias (SB) of DNNs - bias towards learning only the simplest fea- tures a key reason for this brittleness, another recent line of work has surpris- ingly found that diverse/ complex features are indeed learned by the backbone, and their brittleness is due the linear classification head relying primarily on the simplest features. To bridge the gap between these two lines of work, we first hy- pothesize and verify that while SB may not altogether preclude learning complex features, it amplifies simpler features over complex ones. Namely, simple features are replicated several times in the learned representations while complex features term Feature Replication Hypoth-
might not be replicated. This phenomenon, esis, coupled with the Implicit Bias of SGD to converge to maximum margin so- lutions in the feature space, leads the models to rely mostly on the simple features for classification. To mitigate this bias, we propose Feature Reconstruction Reg- ularizer (FRR) to ensure that the learned features can be reconstructed back from the logits. The use of FRR in linear layer training (FRR-L) encourages the use of more diverse features for classification. We further propose to finetune the full network by freezing the weights of the linear layer trained using FRR-L, to refine the learned features, making them more suitable for classification. Using this sim- ple solution, we demonstrate up to 15% gains in OOD accuracy on the recently introduced remi-synthetic datasets with extreme distribution shifts. Morcover, we demonstrate noteworthy gains over existing SOTA methods on the standard OOD benchmark DomainBed as well.
INTRODUCTION
Despite the remarkable success of Deep Neural Networks (DNNs) in various fields, they are known to be brittle against even minor shifts in the data distribution during inference, which are not uncom- mon in a real world setting Siinonero-Candeed et al. 2008; Torralba & Efros 2011). For example, a self-driving car that works well in normal weather may perform poorly when it is snowing, leading to disastrous outcomes. The need for improving the robustness of such systems against distribution shifts has sparked interest in the area of Out-Or Distrriiibttion or OOD generalization (Hendrycks & Dietterich] 2019 Gulrajani & Lopez-Paz 2020).
In this work, we aim tackle the problem of OOD generalization of Neural Networks in covariate- shift (Shimodaira) (2000) based classification setting, by addressing the fundamental cause of their brittleness, rather than by explicitly enforcing invariances in the network using domain labels or data augmentations. More specifically, we aim to mitigate the issue of Simplicity Bias, which is the tendency of Stochastic Gradient Descent (SGD) based solutions to overly rely on simple features alone, rather than on a diverse set of features (Arpit et al. 2017 Valle-Perez et al. 2018). While this behavior was earlier used to explain the remarkable generalization of Deep Networks, recent works suggest that this is indeed a key reason behind their brittleness to domain shifts (Shah et al. 2020).
"Equal Contribution. Correspondence (sravantia, anshulnasery } google.com

---

Published as a conference paper at ICLR 2023
The extent of Simplicity Bias seen in models is a result of two important factors - diversity of features learned by the feature extractor" "I and the extent to which these diverse features are used for the task at hand, such as classification Recent works suggest that generalization to distribution shifts can be improved by retraining the last layer alone, indicating that the features learned may already be good enough for the same (Rosenfeld et al. 2022; Kirichenko al.) 20226). Does this imply that brittleness of models can be attributed to the learning of the classification head alone? If this is the case, why does SGD fail utilize these diverse features despite its Implicit Bias converge to maximum margin solution in a linearly separable case (Soudry et al., (018)) To answer these questions, we firstly hypothesize and empirically verify that Simplicity Bias leads to the learning of simple features over and over again, as compared to other, more complex features. For example, among the 512 penultimate layer features of ResNet, 462 of them might capture a simple feature such as color, while the remaining 50 might capture a more complex feature such as shape we refer to this as (Simple) Feature Replication Hypothesis. Assuming feature replication hypothesis, we further show theoretically and empirically that maximum margin classifier in the replicated feature space would give much higher importance to the replicated feature when compared to others, highlighting why the linear layer relies more on simpler features for classification.
To mitigate this, we propose a novel regularizer termed Feature Reconstruction Regularizer (FRR), to enforce that the features learned by the network can be reconstructed back from the logit or pre- softmax layer used for the classification task. As shown in Fig 2 linear classifier alone by freezing the weights of the feature extractor. This formulation enables the learning of an Invertible Mapping in the output layer, specifically for the domain of features seen during training. This further allows the logit layer to act as an information bottleneck, encouraging all the factors of variation in the features to be utilized for the classification task, thereby improving the diversity of features used, We theoretically show that adding this constraint while finetuning the linear layer can learn a max-margin classifier in the original input space, disregarding feature replication. Consequently, the learnt linear classifier also gives more importance to non replicated complex features while making predictions. We further explore the possibility of improving the quality of features learned by the fcature extractor, by using FRR for finctuning the backbonn as well. In order do this, we freeze the linear classification head, and further finetune the backbone with FRR. We find that this encourages the network to indeed learn better quality features that are more relevant for classification. We list the key contributions of this work below
we firstly propose to train the
Key Observation: We provide a crisp hypothesis of "feature replication" to explain the brittleness of ERM trained neural networks to OOD data 3.1). Using this, we further provide theoretical and empirical evidence to justify the existence of Simplicity Bias in maximum margin classifiers. Novel Algorithm based on the Observation: Based on this, we introduce a novel FRR regularizer to safeguard against the feature replication phenomenon (Sec 3.2). We also provide theoretical support for FRR in an intuitive data distribution setting. Furthermore, we introduce a simple FRR- L method to only regularize the linear head with FRR, and then introduce FRR-FLFT training regimen to train the feature extractor for improved OOD robustness (S
Empirical validation of the hypothesis and the proposed algorithm: We demonstrate the effective- ness of FRR-FLFT and FRR-L by conducting extensive experiments on semi-real datasets (Ta- ble 2 constructed to study OOD brittleness, as well as on standard OOD generalization bench- marks, where FRR-FLFT can provide up to 3% gains over SOTA methods for OOD generaliza- tion(Table 3
2 RELATED WORKS
Learning diverse classifiers to counter simplicity bias: Recent works have shown that ERM trained models learn diverse features, however, the linear layer fails at capturing and utilizing these diverse features properly. There have been several attempts at training classifiers which can make use of such diverse features. [eney et al. (2022) train a number of linear classifiers on top of a pre-trained network with diversity regularizer. which encourages the classifiers to rely on different features. Xu et al. (002) and [Bahng et al. (2020) propose to train debiased classifiers which are statistically independent from trained biased networks, but these need careful design and prior knowledge of the
In this paper, we refer to the penultimate layer's activations as features. iic. by the final classification layer.
2

---

Published as a conference paper at ICLR 2023
biases in trained networks. Kirichenko et al. (202aa) show that reweighting train set examples and retraining the last layer of a pre-trained deep network can alleviate spurious correlations, provided one can access a balanced dataset. In contrast to these methods, our method can work simply on the training set data, and produce a single classifier which is debiased. |Huang et al. (2000) propose to mute the features with highest gradients, and use only the other features to make prediction. While this method suppresses the maximally used features, it does not encourage the learning of hard-to- learn features, which is directly realized using our loss formulation, Kumar et al. (2022) suggest that finetuning the final linear layer first before finetuning the entire network can make it more robust to OOD shifts, and we utilize this insight in the FRR-FLFT phase of our method. A complementary approach to this problem is to learn features that are more diverse (Zhang et al. 2222 Wang et al.) 2019). We note that applying our proposed method on top of such techniques would encourage the classifier to use the diverse features effectively, and this can further benefit the performance.
Domain Generalization and OOD robustness: The performance of neural networks is known to drop when there is a mismatch in the train and test distributions (Hendrycks & Dietterich] 2019), and methods to mitigate this have been gaining a lot of attention in recent years. The problem has been studied under various assumptions main generalization (Gulrajani & Lopez- Paz, [020 Li et al.] (2018)) assumes that the train distribu- tion consists of mixture of distinct distributions (called domains), with each train sample having domain label associated with it. The stronger setting of aggregate domain generalization (Thomas) al. 2021] Matsuura & Harada (020) assumes training data to be drawn from a mixture of distri- butions, but does not assume the availability of domain labels. Finally, OOD robustness (Hendrycks) & Dietterich 2019 Koh al. 202T) drops all of these assumptions. Most works tackling the do- main generalization problem attempt to train model whose predictions are invariant to the domain label (Li al. [0018|| Arjovsky et al. 2019). or try to align the features of the model for examples from different domains (Shi et al.) 202 Shankar et al. 2018). However, since we aim to tackle the stronger setting of OOD generalization, we do not use domain labels. Tackling the OOD robustness problem. Thomas et al. (2021) and Matsuura & Harada (2020) first cluster training examples into "pseudo-domains" after which standard domain generalization techniques are used. Another recent line of works propose using model averaging (Cha et al., [2021: Li et al.. 2022) and/or ensembling (Arpit et al., (021) for better OOD generalization. These techniques are complementary to our con- tribution, and we demonstrate how they can benefit each other in our empirical evaluation.
distribution shift. The commonly studied setting of do-
3 FEATURE REPLICATION HYPOTHESIS
Prior works have shown that neural networks trained with SGD exhibit simplicity bias (SB), even when initialized with pre-trained models that can capture complex features. Our Feature Replica- tion Hypothesis FRH- states that: SB is observed because the simpler features of the input are replicated multiple times in the feature space of neural networks.
When trained using SGD, the final linear layer then learns the max margin classifier on these repli- cated features, which leads to over-reliance on simpler features in the input. Hence, the outputs of the network are brittle to distribution shifts that change such replicated features. In this section, we provide empirical and theoretical evidence for FRH, and propose a new regularizer-FFR FRR- to mit- igate this effect.
We first introduce some useful notations. Let fo(x) : R2d + Tm be the feature extractor of a neural network parameterized by weights 0, and W E m be the weight matrix of the linear classifier. For input æ E Rd, the output of the network is W fa()) E g².
3.1 EMPIRICAL VALIDATION OF FEATURE REPLICATION HYPOTHESIS (FRH) IN ERM
Coloured MNIST dataset. To empirically demonstrate feature replication, we use binarized version of the coloured MNIST dataset (Gulrajani & Lopez-Paz 2020). We construct this dataset by first assigning two digits of the MNIST dataset, namely "1" and "5", to classes and respec- tively. While training the network, we super-impose images of "1" onto colours of range Ro [1115, 0) - (256, 141, 0)] (i.e. red), and images of "5" onto colours of range R1 = [(0. 115, 0) (141, 256, 0)] (i.e. green). The dataset is constructed such that the simple feature, namely colour, is weakly correlated with the labels, while the complex shape features are strongly correlated with la- bels. See Appendix D. for more details about the dataset.
3

---

Observations: In Table ! we report the number of colour features, shape features, and the average correlation of each of these with the final prediction. We observe that the ERM trained model learns both shape and colour features, but the number of learnt colour features (26) is much higher than Published as a conference paper at ICLR 2023
Table 1: Features replication in Coloured MNIST: We observe that ERM learns more colour fea- tures than shape features, and the prediction is less correlated with the shape features. Adding FRR makes the network depend more on shape and less on colour, leading better OOD performance.
Number
Averuge correlation with input
Correlation with output
Algorithm
Colour
Shape
Colour
Shape
Colour
Shape
Accuracy
OOD Accuracy
ERM
26
0.76
0.26
0.81
0.61
99.9%
59.1%
ERM+FRR-L
26
4
0.76
0.26
0.71
0.65
99.6%
64.9%
Training setup: We train a model on this dataset, and test it on images which do not have any cor- relation between the label and the colour, i.e. images where the digits "I" and "5" are superimposed on randomly coloured backgrounds. We construct this test distribution to see how well different al- gorithms learn simple (i.e. colour) and complex (i.e. shape) features, since an algorithm which de- pends only the spurious colour features would not have good performance on the test domain. We train a four layered CNN on this data. If a feature in the penultimate layer fo(z) has more than 90% correlation with the color shape of the input, then call it as color feature or shape fea- ture, respectively. We also compute the correlation of these features with the output of the network (W" fo())) over inputs from the test domain. This gives information of the learnt features, and their contributions to the final prediction of the network. Note that the feature dimension is m 32, and the output dimension is k
the number of shape features (4), despite their weaker correlation with labels, thus validating our Feature Replication Hypothesis. We also visualize the inter-feature correlation of the learnt features in Fig 5 which shows blocks highly correlated features, further validating our hypothesis. We note that correlation of the output with the shape features is lower, leading to OOD accuracy of 59%.
3.2 FEATURE RECONSTRUCTION REGULARIZER (FRR)
To alleviate simple feature replication issue, we propose Feature Reconstruction Regularizer (FRR) to enforce that the learned features can be reconstructed from the output logits. We propose to retrain the final linear layer using this regularizer to allow the model to utilize diverse features to compute the final output. We implement this by introducing another neural network with the objective of reconstructing the features of the network from the output logits, i.c. features fa() should be recoverable from the predictions of the network through transform To(.) parameterized by ó. That is, FRR is given by:
CERR(Z: 0, W. Ø) (1fo(z) - TW
(I)
where |1-Ilp denotes the l, norm. We set this norm be foo or E, in our experiments. In the simplest case, (v) = dy, where & E Rm** Note that order to find the appropriate d, we jointly optimize W and ø using gradient descent based optimizers. We also experiment with being more complex neural network.
We empirically validate FRR on Coloured MNIST, where using FRR with the linear layer leads to lower correlation with Colour compared to standard ERM (Table T]. Consequently, OOD accuracy improves by 5% over ERM.
3.3 FRH & FRR: THEORETICAL ANALYSIS
We now present a simple and intuitive data distribution with feature replication that highlights the OOD brittleness of standard ERM, and also demonstrates FRR can be significantly more robust.
Data Distribution: Consider a linearly separable distribution consisting of two factors of variation as shown in Figure That is, consider the following distribution (x, v) D, where,
y 11 with probability 0.5, x == [v. y)
[M1, n2) E R.? n4 ~ UnifUn 0.5), i [2]. (2)
4

---

Published as a conference paper at ICLR 2023
SVM, O-Rep
SVM, 5-Rep
FRR (Ours), 5-Rep
FRR (Ours), 5-Rep (OOD)
(ID)
(ID)
(ID)
1.0
0.5
-0.5
-1.0
(a)
(b)
(c)
(d)
Figure We demonstrate the brittleness of SVM (a, b) and effectiveness FRR (c. d) based
classifiers toy dataset comprising of 2 factors of variation, sampled from uniform distribution. We consider d (= 0 or 5) feature replications (Rep) along the y-axis. FRR converges to a maximum margin solution in the non-replicated feature space, resulting in improved OOD robustness (d)
Also consider feature extractor fo(-) which captures feature replication in the first feature, i.e. for every data point (z. y), the new, feature replicated data point will be (Z, y), where,
fo(x) = ã = [æ1, 11 T22 E Rd++1
(3)
i..., is repeated times. The joint distribution of features and labels is denoted by D. Finally, we define the l2 max margin classifier over a distribution D as WMM == arg min ||ll subject to y. (w, x²) 2 A (xc y) Supp (D). Then we have the following results:
Claim 3.1 (Brittleness due to Feature Replication). Consider the data distribution given in Equa-
tion 2 3 Then, the following holds: (1.) The max-marp max-mnrgin classifier WMM over D given by WMM =
[1, 1I. and (2.) The max-man max-margin classifier ÜMM over given by WMM -
2 Rd+1
The above claim implies that when there are replicated features to the input of the linear layer, the max- max-margin classifier would give much more importance to the feature that is replicated. Hence, even a minor change in this replicated feature in the input space would be amplified in the output of the classifier. This is especially concerning in light of the observations in Table T which validate the Feature Replication Hypothesis in Coloured MNIST.
Proposition 3.2 (Robustness of FRR). Denote the average feature reconstruction loss x;)²] and consider any (w",Ø* satisfying:
LFRR(W, δ) = max E 1gi$d41
(B. ((ū,,))
(w*,0* Φ*) E arg min LFRR(u, Φ) subject to (1, F) 2 A00 (Z, v) E Supp (D).
(0,+)
We have that: wi +... w U.+1- Consequently, we have (wi . X (WMM, x) for all x E TR...
implement the above as l2 batch. Above result shows that the feature reconstruction regularizer will produce a linear classifier that gives equal weights to the replicated and non-replicated features. This equivalent feature space, thereby resulting in enhanced robustness to distribution shifts. Same is reflected in Figure I (c). (d) which show impact of FRR on the trained boundary in the non-replicated feature space. We defer the proofs of the above to Appendix A We also provide more general result by assuming correlated feature representations in Appendix
Practically,
maximum margin classifier in the non-replicated
TRAINING PROCEDURE
Pretraining In order learn features which are relevant to the train distribution, we first pre- train our model using standard ERM with the cross-entropy loss ( 0, (2, v)).
FRR-I Since ERM training is known to learn several rich and diverse features, we freeze the backbone parameters 0, and retrain the final layer as the following-
(WTRR, Φtrr) min Lus (W, 0, (ss, v)) + CrRR (I. 0, W, W.
where ',. is hyperparameter weighing the two losses. We train W and ø jointly. We refer to this step FRR-L, 1, i.e. Feature Reconstruction Regularizer Linear, since we only train the linear layer.
5

---

Published as a conference paper at ICLR 2023
ERM Training
FRR-L
FRR-FLFT
Crax*
Frwara Bacsbone
Freezn spot
Lutd
Figure 2: Our training procedure: Dotted fill indicates that the parameters are trainable.
FRR-FLFT Following the suggestions of Kumar al. (2022) follow up the linear layer training with the finetuning of the feature extractor 0 with a weighted combination of the cross- entropy loss and FRR, weighted by hyper-parameter AFLFT. In this step we freeze the weights of the linear layer to improve the stability of training. We do this since naively using this constraint during network training could amplify the Simplicity Bias in networks in degenerate cases. For example, the backbone could learn to output single replicated simple feature which is predictive enough on the training data. Reconstructing such feature from logits would also be easy. but such network might not generalize well. Formally, the optimization problem for this step is
FLLFT = min Laa (FFR 0, (zr, y)) + CFRR 0, WERR, ΦFRR)
(5)
0
We view this step "sharpening" the features for more accurate predictions. Freezing the linear head makes sure that the features do not collapse to a degenerate solution. Our training algorithm is summarized in A the training pipeline illustrated in Figure
EXPERIMENTAL RESULTS
5.1
UNDERSTANDING HOW FRR MITIGATES SIMPLICITY BIAS
To empirically illustrate the extent of Simplicity Bias in Neural Networks, Shah et al. (2020) intro- duced several synthetic and semi-synthetic datasets, where some features are explicitly simple, re- quiring simpler decision boundary for prediction; while others are complex. In this section, demonstrate the effectiveness of the proposed Feature Reconstruction Regularizer towards mitigat- ing Simplicity Bias, by evaluating the same 10-class variant of the proposed semi-synthetic MNIST-CIFAR dataset, as discussed in the following section.
5.1.1 MNIST-CIFAR-I DATASET
We extend the simple binary MNIST-CIFAR dataset proposed by Shah et al. (2020) to a 10-class dataset, in order evaluate the impact of the proposed Feature Reconstruction Regularizer more complex scenario when compared to the binary ColOred-MNIST dataset presented in Section-
We refer to this dataset as MNIST-CIFAR-II The higher complexity of this dataset allows for more reliable evaluation of various settings such as linear probing, full network finetuning and fixed- linear finetuning, with better granularity of results.
To construct this dataset, we first define correspondences between the classes of CIFAR-10 and MNIST, Each image from class MNIST is mapped with an image from class of CIFAR-10, with the label being set to k. Thus, every training data sample (I1, which are images from CIFAR-I0 and MNIST respectively, along with their ground truth class y. to be noted that for both CIFAR-10 and MNIST, labels are always correlated with the respective images. In such a scenario, although a classifier can achieve very good performance by relying solely the simple (MNIST) features. the goal of to rely on the complex (CIFAR- 10) features as well. This dataset represents the toughest setting of OOD generalization, where there is differentiation between important features and spurious
consists of and 22,
(OOD) robustness requires

---

Published as a conference paper at ICLR 2023
Table ID and OOD accuracy (%) by training on MNIST-CIFAR-IO in various training regimes.
Initialization
Layers trained
Exp ID
Training Less
Training Datnset
In-Diaribation (ID) MNIST-CIFAR-10
MNIST-ArgCIFAR
Ons-Of--ieriittiio (OOD)
CIFAR-AVgNNIST
EI
Mi: FRM (CF)
MIIST-CIFAR-ID
99.84
97.44
51
Random
All layers
F2
Cros-Fatopyy
CIFARRRMIMNISST
88.53
9.77 sm
88.52 2m14
Cros- Entropy
MNIST RandCIEAR
99,68
94,34
10.02
E4
M2: FRM-L
MNIST-CIFAR-IC
99.80
97.06
52.77 2m
E5
Cross Eatropy
CIFAR- BMFAM-RR
65. 14
10.35
65. 10 zom
MI-ERM
Linear layer
D6
Cro-- Entropy
MNIST RandCIARR
99.71
96,84
10:33
F7
CE Fail-Rank Reg
CIIST-CIFAR-IO
86
97.04
52.87
MR: (Ours)
MNIST CIARR 10
99.85
96.31
59.13
All layers
E9
Cros- Entropy
99.58
97.67
53.33 2m1
M2-ERM-L
Pesture
All Feature extractors
FRR IT
Cross Entropy
MNIST-CIFAR-
99.84
99.84
97.67
97.32
53.67
54.12
D12
TRR ILPT
99.81
98.44
60.02
All layers
E13
Cross Entropy
99.87 -0X
97.03 1:
61.75 -oa
MEIPRR-L
All lyyes
Peature extractirs
E15
E14
Cross Entropy
FRR-FT
MIIST-CIIAAR1I
99.85 -000
99.85 -04
99.30 2:
97.35 133
63.73 1012
62. =0.0
Festure extractors
E16
M4: FRI-FLFT (Ours)
99.84 -00
99.45 10
68.12 go.
MALFRR FLFT
Linear layer
E17
E18
Eine-Eatropt
Ente-Eatropy
CIFAR-RMMMNNIST MINISTRRnNCIFARR
99.71 =00
79-92 20
99.46 3m
11.03 am
77:35 gore
10.27 2013
correlations. A real-world example of such a case is the classification of swans versus bears, with the training dataset consisting of only white swans and black bears. Here the model could either rely on shape or color for classification. A classification network that relies solely on the simplest feature color, fails to generalize to the test set consisting of black swans and polar bears.
5.2 TRAINING AND EVALUATION SETTINGS
We consider two separate ResNet-18 18 (He et al. 2016) feature extractors for CIFAR-10 10 and MNIST respectively. The outputs of the Global Average Pooling (GAP) layers in each of the feature extrac- tors are concatenated to form a 1024 dimensional vector, which is given as input to the linear clas- sifier. This architecture allows the computation of accuracy based on either a combination of both CIFAR-10 and MNIST features, or based on features of only one of the datasets. For example, to evaluate the performance of the classifier based on CIFAR-10 features alone, we replace the 512 di- mensional MNIST feature vector of each data sample with average feature vector computed from all images in the MNIST dataset. We refer to this as the responding one for MNIST is refered to the MNIST-AvgCIAAR dataset. Similar to the work by Shah et al. 22200), we define two additional datasets, CIFAR-RandMMn RandMNNST and MNIST-RAndCIIAAR where images from one of the datasets (MNIST and CIFAR- 10 respectively) are randomly shuffled with respect to their corresponding labels. The base training (E1, E2, E3) is done for 500 epochs, and
dataset, while the cor-
the linear layer training/ / finetuning (E4 EE8) is done for 20 epochs, without any augmentations.
5.3 EXPERIMENTAL RESULTS IN VARIOUS TRAINING REGIMES
We present the results of training the ANIST-CIFAR-ID dataset using different algorithms in The mean and standard deviation across five runs have been reported for each case.
Table 2
By training randomly initialized model on the MNIST-CIFAR-10 dataset using the cross-entropy loss (E1), we obtain an accuracy of 99.84% on its corresponding test split. While the accuracy of this model on the MNIST-avgCIFAR dataset is high (97.44%) its performance on the CIFAR-avgMNIST dataset is poor (51.92%) indicating that the model chooses to rely more on the simpler MNIST features, rather than combination of both CIFAR and MNIST features.
ERM Training:
While the performance on the -IFAR-avgMNIST and MNIST-avgCIFAR datasets is sufficient to understand the extent of CIFAR/ MNIST features used by the classification head, it does not give a clear picture on the features learned by the two feature extractors. To understand this, we reinitialize the linear classification head randomly, and train the same using CIFAR-RandMNY MNIST-RandCIFAR datasets (E6) respectively. We obtain an accuracy of 65.2% on the CIFAR- avgMNIST dataset in the former case, indicating that although the CIFAR features learned can possibly achieve 13% higher accuracy (w.r.t. El), the bias in the classification head prevents them from participating in the classification task. The MNIST-avgCIFAR accuracy of the latter case is high expected. An upper bound on CIFAR-10 and MNIST accuracy that can be achieved with the selected architecture and training strategy (without using any augmentations) can be seen in E2 (88.53%) and E3 (8%.68) respectively.
(E5) and
7

---

Published as a conference paper at ICLR 2023
Training the Linear Classification Head: As discussed, while ERM training (E1) learns features that can be used for better OOD performance (E5), it does not effectively leverage these features for the classification task. We firstly explore the possibility of bridging the difference in the CIFAR- avgMNIST accuracy between E1 and E5 by merely retraining the linear layer. By reinitializing and naively retraining the linear layer with Cross-entropy loss, the accuracy on CIFAR-avgMNIST improves by less than 1% (E4). Using the proposed Feature Reconstruction Regularizer (FRR) for training the linear layer alone, the CIFAR-avgMINIST accuracy improves by 7.21% as shown in E8, demonstrating the effectiveness of the proposed regularizer in mitigating Simplicity Bias. We penalize the fo norm of difference in original features and their reconstruction in addition to the minimization of cross-entropy loss. The reconstruction based regularizer enforces the network to utilize both CIFAR and MNIST features for classification. Since this regularizer resembles an orthonormality constraint on the linear classification head, we additionally check the effectiveness of explicitly enforcing a full-rank constraint on the linear layer by minimizing the following: |[ WWW--
Ill (E7). We find that this is not effective in improving the overall accuracy, possibly because it enforces a very stringent constraint on the final classification layer. Contrary to this, the proposed Feature Reconstruction Regularizer allows more flexibility by imposing this constraint only on the domain of features seen during training. This accounts for the simple feature replication as well, enabling view the logit layer as an information bottleneck in the reconstruction.
Finetuning (FT) and Fixed Linear Finetuning (FLFT): We explore the finetuning of a given base model in two settings firstly by finetuning all layers in the network (denoted as FT Funenuning), and secondly, by freezing the parameters of the linear classification head and finetuning only the feature extractors, which we refer as FLFT or Fixed Linear FuneTuniig, By finetuning an ERM trained base model using either of the two strategies (E9 and E10), we observe gains of less than 1%. We observe similar gains even by finetuning the full network with FRR (E11). Contrary to this, by using FRR-FLFT even the ERM trained network (E12), we obtain 7.29% improvement over the base model. This shows that, by allowing the full network to change while imposing the FRR constraint, the network can continue to rely on simple features, possibly by reducing the number of complex fcatures Iearned by the feature extractor. However, by freezing the weights of the linear layer and further imposing this constraint, the network is forced refine the CIFAR features that are already being used for prediction.
Combining FRR-L and FRR-FLFT: While we obtain similar order of gains (~ 7%) using both FRR-L and FRR-FLFT individually, the former improves the diversity of features being considered by the classification head, while the latter improves the quality of the features themselves. We therefore propose a training strategy that combines both FRR-L and FRR-FLFT. Using this, we obtain gains of 16.2% over the ERM baseline as shown in E16. indicating that the combination of FRR-L and FRR-FLFT has a compounding effect by firstly selecting diverse features, and further refining these features to be more useful for classification. Although FRR-L followed by FRR-FT (E15) is also effective, it has about 6% lesser gains when compared to the proposed approach of FRR-L + FRR-FLFT. We note that following up FRR-L with ERM-FT (E13) or ERM-FLFT (E14) also refines the learned features, making them more suitable for the classification task, yielding 2.6% and 4.6% gains respectively over FRR-L.
We verify the quality of features learned by the feature extractors after the proposed training strat- by reinitializing and retraining the lincar classifier on CIFAR RandMNIST egy FRR-L+ + RandCIFAR (E18) datasets respectively. We observe considraabee gains of (E17) and MNIST- around 15% on MINIT-CIFAR-II 10 accuracy using ERM (E5), demonstrating that the proposed approach not only results in more CIFAR features be- ing used for classification, but also leads to the learning of better CIFAR features.
training when compared to
5.4 OOD GENERALIZATION IN A REAL WORLD SETTING
We show the efficacy of FRR towards improving OOD generalization on the DomainBed (Gulrajani & Lopez-Paz 2020) benchmark. We use the performance of the model on in-domain validation data (i.e. the in- no-domain strategy by Gulrajani & Lapez-Paz] (2020) to select the best hyper- parameters, and report the average performance and standard deviation across random seeds.
Baselines We compare method against standard ERM training, which has proven to be
frustratingly difficult baseline (Gulrajani & Lopez- Paz 2020). and also against several state of the
art methods on this benchmark SWAD (Cha et al. 2021 MIRO (Cha et al. 2022) and SMA (Arpit)
8

---

Published as a conference paper at ICLR 2023
Table Results on DomainBed: The bottom partition shows results of methods that perform model weight averaging. In both cases, with (top) and without (bottom) model weight averaging, the proposed approach outperforms existing methods.
Algorithm
PACS
VLCS
TerraIncognita
DomainNet
Avg.
ERM
85,5 +.11
77.5 04
66.5 +02
46.1 +.0
40,9 =
63.3
IRM
83.5
78.5
64.3
47.6
33,9
61.6
CORAL
86.22
78.8
68.7 03
47.6 47.62
41.5 =01
64.5
MIRO
85.4 04
79.0 00
70.5 004
50.4 11
44.3 =2
65.9
ERM+FRR-L
85.7 :01
76.6
68.4 202
53.7 1:06
44.2
65.7
ERM+FRR
87.5
77.6 03
69.4 2::1
54.1 1:06
45.1 =01
66.8
SMA
87.5 0:02
78.2
70.6 1:1
50.3 1:05
46.0
66.5
SWAD
88.1 01
79.1 =01
70.6 #0
50.0 0:03
46.5 -11
66.9
SWAD+FRR
89.2 0.4
80.0 02
70.3 #
53.2 #3
46.2 on
67.9
et al. 2021). Finally, we show that our approach can be effectively integrated with stochastic weight averaging to obtain further gains. See Appendix for further experimental details.
The main results of our algorithm are reported in Table 3 We find that our pipeline of training and finetuning with FRR, when combined with ERM achieves improved performance with respect to the state of the art methods that do not use model weight-averaging, and in fact achieves comparable performance
Main Results:
methods that use model weight averaging as well. Further, our method obtains substantial gains of more than 3% over ERM across datasets. The gains are especially pronounced for the larger datasets including DomainNet and Terralncognita (8% and 5% resp.), indicating the efficacy and scalability of our method. Further, it is clear from Table that finetuning the feature extractor once the linear layer is fixed provides boost of over 1% on average over FRR-L. This empirically validates Finally, using our method in tandem with SWAD helps us achieve a new state-of-the-ar on the BemainBed benchmark, outperforming other methods three datasets while achieving comparable performance on two, and being better than existing SOTA by close to 1% on average. We report detailed results and further ablations in Appendices H J
finetuning paradigm which denote ERM+FRR.
6 CONCLUSION AND DISCUSSION
In this work, we consider the problem of OOD generalization through the lens of mitigating Sim- plicity Bias in Neural Network training. To unravel the paradox pertaining to the existence of Sim- plicity Bias in learning only the simplest features, and the observation that the features learned by large practical models may already be sufficiently diverse, we put forth the Feature Replication Hy- pothesis that conjectures the learning of replicated simple features and sparse complex ones. Com- bining this with the Implicit Bias of SGD to converge to maximum margin solutions, we provide a theoretical justification to the high OOD sensitivity of Neural Networks.
To specifically overcome the effect of simple feature replication linear layer training, we propose the Feature Reconstruction Regularizer, that penalizes the l, norm distance between the features and their reconstruction from the output logits, thus improving the diversity of features used for classifi- cation. We further propose freeze the weights of the linear layer thus trained, and use the FRR reg- ularizer for finetuning the full network, to refine the features to be more useful for the downstream task. We justify the proposed regularizer both theoretically and empirically on synthetic and semi- synthetic datasets, and demonstrate its effectiveness in a real world OOD generalization setting.
We believe and hope that this work can pave the way towards obtaining better understanding on the underlying causes for OOD brittleness of neural networks, and inspire the development of better algorithms for addressing the same. We believe the proposed regularizer can potentially work effectively in several other settings that involve the use of linear layer training/finetuning, such as domain adaptation and transfer learning. While the regularizer works effectively in a scenario where the network is first trained using an algorithm such ERM to learn features that relevant to the task at hand, the robustness the proposed algorithm in the presence severely non- non-relevant features yet be explored.

---

Published as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 00779,22,, 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, al. A closer look at memorization in deep networks. In International conference on machine learning, pp. 233- 242. PMLR, 2017.
Devansh Arpit, Huan Wang, Yingho Zhou, and Caiming Xiong. Ensemble of averages: Im- proving model selection and boosting performance in domain generalization. arXiv preprint 10.0022110.10832, 2021.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Scong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning, pp. 5-8-5528-99 539. PMLR, 2020.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24, 2011.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. arXiv preprint arXiv:2203.10788, 2022.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 17(1):2996-038) 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wicland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:1811.1222 2018.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. /226/, 2019.
Zeyi Huang, Haohan Wang, Eric Xing, and Dong Huang. Self-challenging improves cross-domain gemeralization. In European Conference Computer Vision, pp. 124-140. Springer, 2020.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937, 2022a.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations, arXiv preprint arXiv:2204.02937 2022b.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference Machine Learning. pp. 5637-5664. PMLR, 2021.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202. 10054, 2022.
10

---

Published as a conference paper at ICLR 2023
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. Proceedings of the AAAI conference on artificial intelligence, vol- ume 32, 2018a.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference Computer Vision (ECCV), pp. 624-633 2018b.
Ziyue Li, Kan Ren, Xinyang Jiang, Bo Li, Haipeng Zhang, and Dongsheng Li. Domain generaliza- tion using pretrained models without fine-tuni tie-ttning. arXiv preprint arXiv:2203.04600 2022.
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using a mixture multiple latent domains. In AAAI, 2020.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1355-135, 2009.
Joaquin Quinonero Caanolero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Em may already learn features sufficient for out-of-distributtin generalization. arXiv preprint wrXiv:2202.06856 2022.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateck Jain, and Praneeth Netrapalli, The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9588 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint arXiv:1804.1074 2018.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.099 09900099997..... 2021.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-
likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19 (1):2822-2878, 2018.
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplic- ity bias: Training diverse set of models discovers solutions with superior ood generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16761-16772, 2022.
Xavier Thomas, Dhruv Mahajan, Alex Pentland, and Abhimanyu Dubey. aggregated domain generalization, 2021.
Adaptive methods for
Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. 1528, 2011. doi: 10.1109/CVP..211..0 1.599534777
CVPR 2011, pp. 1521-
Guillermo Valle-Perez, Chico Camargo, and Ard A Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. arXiv preprint S.Ki:1805000522 2018.
11

---

Published as a conference paper at ICLR 2023
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL
neurin
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994, 2020.
Yilun Xu, Hao He, Tianxiao Shen, and Tommi Jaakkola. Controlling directions orthogonal to classifier. arXiv preprint 1 2022.
Jianyu Zhang, David Lopez-Paz, and Léon Bottou. Rich feature construction for the optimization- generalization dilemma. arXiv preprint arXiv:2203. 2022.
12

---

Published as a conference paper at ICLR 2023
A PROOFS OF THEORETICAL RESULTS
In this section, we present generalization of Claim 3.1 and Proposition 3.2 under weaker assump- tions on the featurizer, and use them to prove the claims made in Sec 3
more general setting Consider the dataset distribution D as defined in eq equation Now, let fe Oz. where O Edx2, Further, vi 0,, = 1. In simple words, the feature extractor maps the imput to a d-dimensional feature representation, where each feature a convex combination of [ 07
0
input feature variations. Also note that
corresponds to feature replication.
0 1
We now rephrase the results from Sec 3.3 in this setting and provide proofs for the same.
Claim A.1. (Restating Claim 3.1 The max margin classifier w minimizing [|]]|² and satisfying
(wv, Ox) 2 is given by w La d
21
Proof. Consider the point (0.5, 0.5). Then, due to the constraint the max-margin classifier, we have y,u, Ox) 2 1 i.e.
2 Σ wy (0,,1 J,1 + +0,2) 21
d
Σ wj 21
2
The minimizer of & norm under this constraint would be when all wy for all j.
Note that the "effective classifier" t in the input space in this case is (w, θ), i.e. slope of the classifier in the input space th Σg O,2 In particular, for the case of feature replication, Σ5 0n
Ej 0/2 = d and Σ 0,1 = 1, leading to a skewed classifier.
Now we restate and show the robustness of FRR in this setting.
Proposition A.2. (Restating Prop 3.2) Denote the feature reconstruction regularizer for this setting
as
FRR(w) min max [[(( Oz) (0z)))²1
1sidd
Let wrnn be the minimizer of F RR(w) satisfying y(urnn Oz) 2 0 (i.e. is perfect classifier). Then, WFRR satisfies
(WFRR, O.2)) 1 1-(0,,1 - 0a,2)
(WFRR, O.,1) 1 (0b,1 0,,2)
where a arg max; 622, arg max; 0.1-
Proof. Consider the FRR for this dataset
FRR(w) = min max (ww Ox) - (Ox).)²)
1sisd
13

---

Published as a conference paper at ICLR 2023
For each i, by computing the minimum u, by considering the quadratic in u4 we obtain
[ (+x))=
FRR(W); B[(Oz))
We now consider each term in this expression
E[(@x) [] [E[(Or, 11 + 02,2
13
12 (0%, 02,2) + 100,,00.2
This is because E[ = 1 and [[x]]] = -[ =
1
T2:
Ww, Oz)) E[([n (w, θ,1) (u 2. 02
= fu (1))))
0.2)
+ (w, 2)).).))
13
(un, (
12
132
(w (ul + (t.
12a ((w, O 20
13
(1
+ 6
022))
Finally,
(w Ox: = E[ (w. θ.1) + ww (22))
13
(w, . )² + (w, 2) 2(w, I,, O.2)
12
Putting it together,
133 - (u O
FRR(w),
12a
13
133
122
13
((w, (
0.
12
13a
12 1)((w, (mur, θ (w, 2.,))
13 (w, 0,,1 + (un, O 2) + 2 2(w, θ (w, O
Let (w.
Further, Let a, be such that arg (a 02,1²
and
(w.
arg max, 0, 2 Then,
FRR(a)+ & max (a 0,,1),, 21 0,
1+ +2+(1-a)
To minimize the above expression w.r.t. a, compute the derivative of the above expression for each component of the max function
i (a (3)
14 a (1 of
(a - (a.1)(a(20.,) 1) 0₂ + 13)
12
x
da
(2a 20..1 + 13)
and
I (1-a-0m.)3 0,
(1 a + (a(2)0) 1) 0b.1 - 12) (2a 20a, 13)2
do
14

---

Published as a conference paper at ICLR 2023
For 1-0,,1-0..2 the second term is greater and the derivative is positive. Similarly, for Σ 1+0..-0b.2 the first term is greater and the derivative is positive. Hence, the minima is obtained at 1 1+om 06.2 Assuming 0.5 1, fb, S or 0.0 1,,1, 0b, S 0.5.
Now, in order to compute b, we can look at the maximization problem again-
arg max (6t,1 0b,2))²
b arg max(1 (1- (0a,1 + 0c,2))²
Since 1. 012 are bounded between [0, 1] the function to maximize is monotonically decreasing in 0,,,0.,, 0. Hence, arg min, 0z,1 and b arg min; 0z,2. Conversely, arg max, 02.2. b = arg max, 0ú,1 and a = (+, 1 0).)) Hence,
(w. O.2) (w. O.,)
(0o,1 - 0a.2)
1 + (0,,1 6,,2)
Assuming that the maximum correlation of both the features is close, FRR will lead to solution which gives roughly equal weights to both the features.
Note that for the case of feature replication, 0,,1 and 8,,2 1. Hence, (w, 1. (w, .1
B JUSTIFICATION OF FEATURE REPLICATION HYPOTHESIS (FRH)
In practical scenario where features are not disentangled, our hypothesis translates to the following:
Conjecture: Simpler features of the input are represented more in the feature space of neural net- works, while complex (hard-to-leamn) features are sparse.
Assumptions:
We consider simple features such as background to be spurious, and complex features such as shape to robust.
We consider an overparameterized network that has the capacity learn more features than what exist, resulting in feature repetition.
Justification: We justify the conjecture by showing that all other possibilities discussed below cannot be true.
Assumption: DNNs learn only Simple Features
Contradiction: Prior works (Rosenfeld et al.) 2022, Kirichenko et al., 2022b) show that features learned by ERM are diverse, and last layer training on target domain is good enough to obtain robustness to spurious features. This cannot be possible if the network has learned only spurious features.
2. Assumption: DNNs learn only Complex Features
Contradiction: The dominance of Simple features in the learning of DNNs is shown by Shah et al.] (2020). Moreover, the existence of texture bias (Geirhos et al.) (018)) and background-bias (Xiao et al. 2020) have been demonstrated in prior works, which show the dominance of Simple features.
3. Assumption: DNNs learn uniform distribution of both Simple and Complex Features.
Contradiction: SGD converges to an SVM solution due to its implicit bias (Soudry et al. 2018). From Claim-3.1 (1), in the presence of balanced features that correlated with the label, SVM solution gives equal weight to all features to maximize margin. This contradicts
the existence of Simplicity Bias (Shah et al. 2020).
4. Assumption: DNNs learn more Complex Features and less Simple Features.
Contradiction: Since Complex features are indeed more robust and are better correlated with the labels, the classifier would rely more on these features. This contradicts the exis-
tence of Simplicity Bias (Shah al. 2020
15

---

Published as a conference paper at ICLR 2023
Therefore, the only feasible option which supports the empirical observations in the prior works discussed above is that DNNs learn more Simple Features and Complex features are sparse, which justifies conjecture.
C EMPIRICAL EVIDENCE FOR FEATURE REPLICATION HYPOTHESIS (FRH)
C.I SYNTHETIC DATASETS
We present empirical validation to support the Feature Replication Hypothesis (FRH) several semi-real datasets and describe them in detail below:
1. Coloured-MNNIST-2 In this dataset, we use images of digits superimposed on either of the two colours- red green. The difference from Coloured-MNISTT is that consider only two colours for the background, rather than a range. We notice extreme simplicity bias in this case, with the network learning 32 colour features and 0 shape features.
2.
This is similar to the Coloured-MNIST dataset described in Section-3.1, with the exception that each of the classes is now composed of two digits. More specifically, the digits 'I' and '7" and mapped to Class O and digits "5" and '8' are mapped to Class 1. We note that '1' and '5' are chosen from the original Coloured-MNIST dataset, while the second digit (e.g.' '7' ) in each class is selected to be one that is similar to the first digit ('1') in the same class. This dataset is constructed specifically to show that the issue of Simplicity Bias and FRH exists even when there is higher variation in the shape feature, and is reported as GoloureMMNSST-MuliiDD shape features are learnt as compared to Coloured-MNIST. the network still relies more on colour to make its decisions.
below. We see that while more
3. Digit- Sogit-Coloure-MNIIST This is similar to the CoNSST dataset described in Scction-3.1, with the exception that the digit is coloured rather than the background. This dataset is constructed specifically to show that the issue of SB and FRH exist even when the region that is coloured, which is the extent to which simple features exist in the im- age is much lesser, and is reported also demonstrates the presence of SB, we note that the average correlation of features with shape is higher when compared the above datasets.
DigitColouredMNIIST below. Although this dataset
Table 4: Features learnt by an ERM trained model on synthetic datasets.
Number Colour
Average correlation with input Colour
Correlation with octput Colour
Algorithm
Shape
Shape
Shape
TD Accuracy
OOD Accuracy
ColouredMNIST
0.76
99.9
59.1
CooColoueIIMNIST
32
0.90
0.82
99.9
49.5
17
64.2
DigiiColouceMNIST
26
3
0.76
0.36
0.79
0.45
99.9
62.1
C.2 REAL WORLD EXAMPLE
We attempt demonstrate feature replication in model trained with ERM on the Real domain of OfficeHome. We train ResNet-50 on this domain, and perform PCA on the features leart by this network. The network learns 2048 features per example, and we compute the 2048 2048 sized covariance matrix of the features over samples from a test domain (Clipart). We then compute the eigenvalues of this matrix, and find that 500 principal components can explain about 97.5% of the variance, i.c. the matrix is extremely low rank, shown in Fig 3 This points the fact that a lot of the learnt features are linearly dependent and highly correlated with each other. This trend is similar to what we observed ColouredMNIST, where large number of features were highly correlated with the colour, and in turn with each other (Fig. 5 the appendix).
We note that in all the additional datasets considered, simpler features represented more in the network while complex (hard-to-leam features are sparse. This empirically justifies our hypothesis in Section-3.
16

---

Published as a conference paper at ICLR 2023
of Explained
0.8
04
500
1000
1500
2000
Index eisenvalue
Distribution of eigen values of covariance of learnt features: A small fraction of prin- cipal components can explain most of the variance features, indicating that features are highly correlated with each other.
Figure 3:
D SYNTHETIC DATASETS
D.1 COLOURED MNIST
binarized version of the coloured MNIST dataset (Gulrajani & Lopez-Paz 2020). To construct this dataset, we firstly assign two digits of the MNIST dataset, namely "I" and "5", to classes 0 and respectively. For the in-domain training distribution, we associate colours in the range Ro = [115, 0, 0) (256, 141, 0)) (i.e. red) label (i.e. the digit "1") and the range R₁ [(0, 115, 0) (141, 256, 0)] (i.e. green) the label 1 (i.e. the digit "5"), where colors are represented in the RGB space. To summarize, while training the network, we super-impose images of "I" onto colours of range Ro. and images of "5" onto colours of range Ri It is to be noted that the choice of colour ranges defined above introduces an overlapping range between [(115, 115, 0) (141, 141, 0))] where images are associated with labels 0 and with equal probability. This overlap reduces the correlation of colour features with labels, while shape features have correlation of with the labels. In Figure we show examples of images from the train and test distributions of this dataset. In Figure pictorially depict the
In order to empirically demonstrate feature replication,
5
5
I
5
S
Figure 4: Random images from the coloured MNIST dataset: The top row shows examples from the train distribution, while the bottom row has images from the test distribution. Here, colour red corresponds to the digit I and green corresponds to the digit 5 in the train data, while this correlation destroyed in the test data.
correlations between the 32 features learnt by the network. We can see block structure emerging, indicating that there is high amount of feature replication.
D.2 TOY DATASET
In line with the theoretical formulation described Section 3.3 we further empirically validate the brittleness of VVM models and the highlight the effectiveness of the proposed Feature Reconstruc-
17

---

Published as a conference paper at ICLR 2023
10
15
20
25
30
1.0
0
5
0.8
10
0.66
15
0.4
20
25
0.2
30
Figure 5: Correlation of the features learnt on coloured MNIST
tion Regularizer in the presence of replicated features. We consider a linearly separable toy distri- bution consisting of two factors of variation shown in Figure [ We define the means of the two classes at (1,1) and (-1,-1) and construct 500 data points in each class by adding noise sampled from Unif[-0.5,0.51 independently along each dimension to the respective means. We sample an Out-of- Distribution (OOD) test set from a Uniform distribution with means centered at (1,0) and (-1,0) re- spectively, and similar noise along each dimension as the train set. Therefore, while the train dis- tribution can be classified by considering either the features aligned with the X-coordinate or the Y-coordinate, the test set performance crucially depends the variation along X-coordinate alone. We consider feature replication along the y-axis, and hence construct this OOD dataset to verify the extent which the other feature considered for classification. To select the best hyperparameter for both SVM and FRR, we consider the presence of validation set whose distribution is similar the test distribution. As shown in Figure we observe that the SVM model starts relying more on the replicated features alone in case of feature replication, compromising its performance on the OOD data. The proposed regularizer on the other hand, gives equal importance to both features even in the presence of feature replication, resulting in improved OOD generalization.
E ALGORITHM
Our training procedure is detailed in Alg
F DETAILS ON THE OOD GENERALIZATION SETTING CONSIDERED
The problem of improving robustness to distribution shifts has been studied in several settings, where, in addition to labeled source domain data, varying levels of access to the target domain data assumed. Some of the well-researched settings include Unsupervised Domain Adaptation, with
access to only unlabeled target domain data (Pan & Yang 2009 Ganin et al. 2016), and Domain Gemeralization, where typically data from several source distributions is assumed to be available, and the target domain in unseen during training (Blanchard al. 2011 Li et al. 2018a Gulrajani & Lopez-PPar Paz 2020)). In the latter case, it is assumed that traning data samples are annotated with domain labels as well, that training algorithms explicitly impose invariance attributes
18

---

Published as a conference paper at ICLR 2023
Algorithm 1: Our training algorithm
Data:
Training data Ds = [(2,, 9) ii [n]], model (0, W), feature reconstruction model o,
Xpnn. APT
Bud Wad +- Adam (ming,w E, (0, W, (s))).
/* Standard training of model parameters 0 and W.
Freeze 0 to be Oatd
Initializing model for training with FRR.
x/
WERRS PFRR
+ Adam (minw.6 à E, (Baa W, (z4,8e) + ALCrRa Ssid. Ø))))
/ FRR-L: Training W, with FRR defined in eqn.
*
+FLFT <- Adam (ming Σ; Lusa (0, WIRR, (zi, ve)) + XPLETCrRR(M; 0, WrRn, ΦFnR))
/* FRR-FIFT: Finetuning 0 with FRR accordi ng to eqn.
*/
Result: Trained model (OFLFT, WFRR).
that cause distribution shift in input data without change in their label distribution (Muandet al. 2013 Ganin et al. 2016 Li et al. 2018b Arjovsky et al. 2019; Shi et al. 202T).
A more challenging case is when the training data belongs to several distributions that may not even be sufficiently discernable to have explicit domain annotations, or may contain multidimensional distribution shifts, such as weather, time of the day and geographical location, that cannot be easily annotated or clustered. We investigate this crucial setting which has been relatively less researched, and refer to it as Aggregated Domain Generalization, as introduced by [Thomas et al. (202T). We note that this setting is different from the case of training on data from a single domain such as Im- ageNet, and evaluating on distribution shifts (Hendrycks & Dietterich) 2019), due the availability of an aggregate of source domains during training, which can enable the effective use of in-domain validation set for hyperparameter selection.
While there have been several approaches to improve the performance of models in the setting of Do- main Generalization, Gulrajani & Lopez-Paz (2020) show that when evaluated fairly, that is, with- out assuming access to the test domain data even for selecting the best set of hyperparameters, none of the approaches perform consistently better than standard training using Empirical Risk Minimiza- tion (ERM). Furthermore, we consider the setting of Aggregated Domain Generalization, which is more challenging due to the absence of domain labels during both training and validation.
G EXPERIMENTAL DETAILS ON DOMAINBED
We test our approach on the DomainBed benchmark (Gulrajani Lopez-Paz] 2020) comprising of five different datasets, each of which have & domains. For each dataset, we train model & 1 domains, and test it on the left out domain. The average out-of-domain performance across the held-out domains is then reported. In this section describe the hyper-parameter selection strategy and the ranges considered for our approach. In line with the DomainBed testbench, we use ImageNet pretrained ResNet-50 models for all algorithms. We use random search select hyperparameters for our algorithm, and use the suggested hyperparameters for the other baselines. We train for 3000 (5000 for DomainNet) steps in the FRR- - phase, and 5000 (10000 for DomainNet) steps in the FRR- FLFT phase. The batch size is fixed to 32, and SWAD hyper- hyper-paramette parameters are the same as those used by Cha et al. (2021). We use the in-domain accuracy protocol from Guirajani & Lopez-Paz) (2020) to select tlyper-paramete of hyper-parameters for each. The range of the hyperparameters experiment with two implementations of Loo norm: f1, where we first compute the loc of feature reconstruction for each example in a batch and then average across the batch, and & where we compute the average & reconstruction norm of each feature across the batch, and then apply foo norm on this m dimensional vector. All our experiments were done on single V V100 GPUs.
for each domain of each dataset, and search over 8 random combinations
shown in Table 5 Note that we
19

---

Published as a conference paper at ICLR 2023
Table 5: Ranges of hyperparameters considered for DomainBed
Hparam
Range
Learning Rate
loguniform 10 10-1)
ÄFrR
loguniform(100 (10-6 100)
AFT
loguniform{ (10-6, 10%)
Norm
(Ei, L1,00 3
H ABLATIONS ON DOMAINBED
Comparing the choices for & In Table we experiment with various architectures for the decoder ó when computing FRR according to equation We consider using a two layer neural network as the decoder & (FRR-LDeeper), and also consider setting 6 = W (FRR-LShared), i.c. explicitly tying the weights of the decoder and the classifier layer. Overall, both these variants are worse than the default single layer, free parameterization of Φ. We believe that this happens because the latter approach enforces much stricter constraint on W, leading to poorer in-domain accuracy, while the former approach enforces weaker constraint, potentially enabling reconstruction of more complex features from a smaller amount of information about them the logits. Both these have detrimental effect the overall performance of the model.
Table 6: Effect of different design choices on OOD accuracy:the rows shows different architecture choices for
Algorithm
PACS
OfficeHome
TerraTncogr ta
Avg.
ERM
85.5 t.
66.5 + 0.2
46.1 # 0.6
65.3
ERM+FRR-LShared
85,2 0.
68.2 0.1
49.4 0.5
67.6
FRM+FRR-LDeepee
84.6 0.07
65.6 0.2
52.5 0.5
67.6
Sensitivity Analysis : We vary APrR and plot out the OOD performance in Fig 6 the performance is stable for wide range of the hyper-parameter on most domains.
We find that
I PSEUDO-CODE FOR FRR
Below we provide the python code for FRR-L the DomainBed framework.
c class ERMWithFRR_LA (A Algorithm
def -iiniitt-astt input-shape. num classes num domains,ns, hparams)
super (ERMWIthRRRLL self)..iinit_- input_shape, num_classes, num num_domains, hparams
self. featurizer networks. Ceaturizer(input... self classifier = networks. Classifier
self hparams)
self. featurizer n.outputs. num classes,
self hparams nonlinear classifier") '],
for params in self. featurizer. parameters (): params. aequires-grad False
self, classifier.inv = networks. Classifier num_classes,
self. featurizer. n outputs self. hparams nonlinear classifier" J
20

---

Published as a conference paper at ICLR 2023
PACS
(b) VLCS
(c) OfficeHome
Figure 6: Variation of OOD accuracy with varying XPRR
bias=False if exxactrrcoonstruction in self hparams
and helf.hparams/ "exact.recoosstuuctioo"" else True,
self. network mn. Sequential ( (self. featurizer, self classifier)
self, optimizer torch. optim Adam
list (self network. parameters ())
list (self. classifier.inv parameters ())
Ir=self hparams| I'Ir'. weight hparams weight.decay" '].
self. reconstruction.c wt self. hparams| reconstruutionn.ww self. norm float (self. h "noomm)]
def update (self, minibatches. unlabeled=None) )
all_x = torch. ([x for x, y in minibatches })
all-y torch. torch.cat(f3 cat for x. y in minibatches ])
pred, rec. feat = self. get feats and rec (all x)
loss = cross entropy (pred, all-y)
reconstruction _lss =
torch sum sum(torch torch max max (torch. abs feat rec), dim=1))01)
/ x
shape
)
loss loss self. reconstruction.we reconstrrucion.loss
self. optimizer. zero grad ()
loss backward
21

---

Published as a conference paper at ICLR 2023
self. optimizer...... step ()
return
loss": loss item O) "reconstruction_loss"" reconstruution item ()
def (sed, x):
return self network (x)
def get feats and rec ( (seff, x):
feats = self. network [0](x)
pred = self network [1](feats)
rec = self. classifier inv (pred)
return pred, rec, feats
J DOMAIN WISE ACCURACIES
this section, we show detailed results Table 3 in the main text. The numbers for the baselines taken from Gulrajani & Lopez- Paz (2020), Cha al. (2021) and Arpit et al.) (2021) while the results for MIRO (Cha et al. 2022) were reproduced using their code-base.
Table 7: Out-of-domain accuracies (%) on PACS.
Algorithm
C
Avg
CDANN
84.6 1.8
75.5 0.9
96.8 + 0.3
73.5 0.6
82.6
MASF
82.9
80.5
95.0
72.3
82.7
DMG
82.6
78.1
94.5
78.3
83.4
IRM
84.8 1.3
96.7 :: 0.6
1.0
83.5
MetaReg
87.2
79.2
97.6
70.3
83.6
DANN
86.4 + 0.8
77.4 0.8
97.3 + 04
73.5 2.
83.7
GroupDRO
83.5 0.
79.1
96.7 0.3
78.3 2:
84.4
MTL
87.5 0.
77.1 0.5
96.4 + 0.8
+ 18
84.6
I-Mixup
86.1 1.5
78.9
97.6
75.8 1 1.8
84.6
MMD
86.1 1.4
79.4 = 0.9
96.6 02
76.5 1.5
84.7
VREx
86.0 1.6
79.1
96.9
77.7
84.9
MLDG
85.5 4
80.1 + 1.7
97.4 0
76.6 + 1.1
84.9
ARM
86.8
76.8
97.4
79.3
85.1
RSC
85.4 +
79.7 1.8
97.6 +
78.2 + 1.2
85.2
Mixstyle
86.8
79.0
96.6
78.5 +
85.2
ER
87.5
79.3
98.3
76.3
85.3
PAdaIN
85.8
81.1
97.2
77.4
85.4
ERM
84.7 4.04
80.8
97.2 03
79.3 L.0
85.5
EISNet
86.6
97.1
78.1
85.8
CORAL
88.3 02
80.0
97.5 03
78.8
86.2
SagNet
87.4 + 1.0.
80.7 0.6
97.1 0.1
+ n.4
86.3
DSON
87.0
80.6
96.0
82.9
86.6
SMA
89.1 0.1
82.6
97.6
80.5
87.5
MIRO
87.5
79.0
98.3
76.2
85.3
SWAD
89.3 0.02
83.4 =
97.3 03
82.5 1 0.5
88.1
SWAD+FRR
89.9 4.22
83.9 0.7
98.2 03
84.8 0:04
89.2
22

---

Published as a conference paper at ICLR 2023
Table 8: Out-of-domain accuracies (%) on VLCS.
Algorithm
L
V
Avg
GroupDRO
97.3 +00
63.4 = 09
69.5 = 0.8
76.7 #
76.7
RSC
97.9 +01
62.5
72.3 12
75.6 0.
77.1
MLDG
97,4 0.22
65.2 0.7
71.0 1
75.3 1 1.0
77.2
MTL
97.8 1.04
= 0.3
71.5 0.
75.3 1 1.7
77.2
I-Mixup
98.3 4.66
64.8
72.1 0.
74.3 0.8
77.4
ERM
97.7 0.4
64.3 + 0.9
73.4 0.5
+ 1.3
77.5
MMD
97.7 0.1.1
64.0
72.8 02
75.3 4.3.
77.5
CDANN
97.1 0.3
= 1.2
70.7 0.8
77.1 + 1.5
77.5
ARM
98.7 202
63.6
71.3 112
76.7 206
77.6
SagNet
97.9 1.0.1
64.5 0.0.
71.4 13
77.5 0:0.
77.8
Mixstyle
98.6 1:03
64.5 1.11
72.6 105
75.7 1
77.9
VREx
98.4 +
64.4
74.1 00
76.2 + 1.3
78.3
IRM
98.6
64.9
73.4
77.3 +
78.6
DANN
99.0
65.1 =
73.1 0:0
77.2 0:
78.6
CORAL
98.3 101
66.1
73.4
77.5
78.8
SMA
99.0 0.02
0.2
74.5 0.3
1.1
78.2
MIRO
99.3
65.2
74.9
76.0
78.9
SWAD
98.8 0.1
63.3 = 0.3
75.3 0.5
79.2 n.6
79.1
SWAD+FRR
98.9 0.4.4
66.3
75.9 1.
79.0 1:
80.0
Table 9: Out-of-domain accuracies (%) on OfficeHome.
Algorithm
C
R
Avg
Mixstyle
51.1 +
53.2 = 04
68.2 + 07
69.2 + 066
60.4
IRM
58.9
52.2
72.1
74.0 4:25
64.3
ARM
58.9 +.0
51.0 0.5
74.1 0.1
75.2 0:
64.8
RSC
60.7 +
51.4
74.8 1.1
75.1
65.5
CDANN
61.5 1.4
50.4
74.4 09
76,6 0.8
65.7
DANN
59.9
73.6 0.
0.5
65.9
GroupDRO
60.4 00
52.7
75.0 0.
76.0 2
66.0
MMD
60.4 0.
= 0.3
74.3 0.11
77.4 + 0.6
66.4
MTL
61.5 007
52.4
74.9 +.9+00.
76.8 + 0.4
66.4
VREx
60.7 4.09
53.0 0.9
75.3 0.1
76.6 0.5
66.4
ERM
61.3 107
52.4
75.8
66.5
MLDG
61.5 0.09
53.2 0.6
75.0 12
77.5 + 0.4
66.8
I-Mixup
62.4
54.8
76.9
78.3
68.1
SagNet
63.4 02
54.8 04
75.8 04
78.3 0:
68.1
CORAL
65.3
54.4
76.5
78.4 +
68.7
SMA
66.7 0.05
::: 0.1
78.6
1
70.6
MIRO
66.0
54.5
78.9
81.7
70.3
SWAD
66.1 0.4
57.7 + 0.4
78.4 0.1
+ 02
70.6
SWAD+FRR
65.2 1:03
57.7 0.5
78.2 0.2
80.2 0.1
70.3
23

---

Published as a conference paper at ICLR 2023
Table 10: Out-of-domain accuracies (%) on Terra Teerrancconita..
Algorithm
L,100
I,88
L.33
T,46
Avg
MMD
34.8 to
57.0 19
35.2
42.2
GroupDRO
2.7
21
56.7 00
36.4 2.1
Mixstyle
54.3 1.1
34.1
55.9 11
31.7 2.1
44.0
ARM
49.3 0.7
38.3 2:44
53.8 1.88
38.7 = 1.3
45.5
MTL
1.2
39.6 6:
11.6
37.8
45.6
CDANN
47.0 1.
41.3 4:88
54.9 17
39.8 + 2.3
45.8
ERM
49.8
42.1 1:4
56.9
35.7
46.1
VREx
48.2
41.7 1.
56.8 208
38.7 3.1
46.4
RSC
20.2
14
56.3
40.8
46.6
DANN
51.1
40.6 1:06
57.4
37.7
46.7
IRM
54.6
39.8 19
56.2 118
39.6
47.6
CORAL
51.6
42.2
57.0
39,8 2:29
47.7
MLDG
54.2
44.3 11
55.6 +
36.9 2.2
I-Mixup
59.6
42.2 1:4
55.9 0.8
33,9
47.9
SagNet
53.0
43.0 2:
57.9 006
40.4 = 1.3
48.6
SMA
54.9 0.4
45.5 0.06
60.1 + 1.5
40.5 + 0.4
50.3
MTRO
59.6
41,1
60.2
40.4
50.3
SWAD
55.4 = 0.0
44.9 +
59.7 0.4
39.9 = 0.2
50.0
SWAD+FRR
60.13 = 1.05
47.89
60.76
42.34
53.2
Table 11: Out-of-domain accuracies (%) on DomainNet.
Algorithm
clip
info
paint
quick
real
sketch
Avg
MMD
32.1 + 13.
466
26.8 11.3
8.7 2.1
32.7 138
28.9 : 119
23.4
GroupDRO
47.2
17.5 04
33.8 05
9.3 0.
51.6 04
40.1
33.3
VREx
47.3 +
1.
+ 466
10.9 0.
49.6 +
42.0 + 3.
33.6
IRM
48.5
15
38.3 4.
10.9 20
48.2 12
42.3
33.9
Mixstyle
51.9 = 0.4
13.3 0.02
37.0 2:
12.3 0.1
46.1 1.03
43.4 - 04
34.0
ARM
49.7 = 0.3
16.3 0.
40.9
9.4 2.1
53.4 20
43.5 -
35.5
CDANN
54.6 à 0.4
17.3 0.00
43.7 4 0.9
12.1 07
56,2 0.4.
45.9 4: 0.5
38.3
DANN
53.1
18.3 1.
44.2 0.77
11.8
55.5 4.04
46.8
38.3
RSC
55.0 = 1.2
18.3 4.05
44.4 266
12.2 022
55.7 0.07
47.8 = 0.
38.9
I-Mixup
55.7
0s
44.3 00
12.5
55.8 +0
48.2
39.2
SagNet
57.7 2 0.3
1.
13.3
12.7 205
58.1 1:05
48.8 1
40.3
MTL
57.9
18.5
46.0
12.5
49.
49.2 -
40.6
ERM
58.1 0.3
18.8 4:03
46.7 4.03
12.2 04
59.6 4
49.8
40.9
MLDG
59.1 +
03
0.7
13.4
4.22
50.2 +
41.2
CORAL
59,2
19.7 4:02
46.6 0.3
13.4 04
59,8 4.02
50.1
41.5
MetaReg
59.8
25.6
11.5
50.1
43.6
DMG
65.2
22.2
50.0
15.7
59.6
49.0
43.6
SMA
64.4
22.4
53.4
15.4 0.
64.7
55.5 -
46.0
MIRO
61.9
20.9
50.3
13.0
65.2
52.7
44.2
SWAD
66.0
22.4 0.3
53.5 0.101
16.1 0:
65.8 1
55.5 -
46.5
SWAD+FRR
65.9 0.1
22.3 2:00
52.8 0.1
14.8 0.
66.2 0:0.1
55.0 :: 0.1
46.2
24


Reviews:
[{"review_id": "mYtFhMjCf4", "reviewer": "Reviewer_Rne6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tries to shed light on the cause and nature of simplicity bias and proposes a regularization strategy to mitigate the same. \nWhile we know that deep neural networks (DNNs) are prone to learning simple features (simplicity bias), recent work provides evidence \nthat the penultimate features in the DNNs contain a diverse and complex set of features.  The authors try to bridge the gap between these seemingly contradictory observations. They try to show that the features learned by the DNN are highly biased toward simple features. The simple features are replicated to a large degree causing the SGD to converge to max-margin solutions (due to its implicit bias) that primarily rely on simple features. The authors further devise a regularization scheme (FRR) to mitigate simplicity bias. They use FRR-L to retrain only the last layer (while freezing the backbone), and this is followed by retraining the backbone while freezing the last linear layer. This approach enforces the model to learn a rich and diverse set of features, leading to better OOD generalization. They show SOTA results on standard OOD benchmark datasets.", "summary_of_the_review": "The idea of the paper is to impose a regularization scheme to somehow alleviate SB, but why their method works is not very clear, and some of the claims, like how the stochasticity part of SGD (which is what leads to max-margin solutions) is responsible for SB. And are the simple features actually replicated a lot? The simple colored-MNIST eg in the table-1 is dissatisfactory as the authors assume that the backbone or DNN feature extractor disentangles shape and color for free during training. A single neuron in the feature layer could be correlated with both color and shape. How do the authors justify that each neuron is majorly responsible only for a single attribute (color or shape). Even if this holds true for colored-MNIST, it is seldom the case in more realistic datasets of high resolution like ImageNet, CelebA or AFHQ (especially since there is no explicit enforcement of any such disentanglement constraint). So the core hypothesis in the paper, which is feature replication, itself is questionable and not fully justified. I feel that the paper lacks an in-depth analysis of the problem, and certain observations and claims seem superficial and maybe not fully true. The authors should better justify all this. I'm open to changing my review after the rebuttal if the authors convince me of the overall story of the paper. \n", "strength_and_weaknesses": "# Pros:\n1. Their Feature Replication Hypothesis is easy to understand and forms a good attempt to explain simplicity bias (SB)\n2. Their Feature Reconstruction Regularizer (FRR) scheme to mitigate SB is also easy to understand and implement\n3. Overall, the experiments seem promising\n\n# Cons\n1. The authors say that the main reason for SB is the implicit bias of SGD (more specifically, the stochastic component of it) to converge to max-margin solutions. I am not sure that this is the only main reason. Can the authors show an experiment using full-batch gradient descent (which should be easily possible on toy datasets used in the paper like MNIST/CIFAR, colored-MNIST etc) and its impact on SB? If SGD is the main cause for SB, then full-batch gradient descent should perform better in this regard. If full-batch gradient descent also shows SB, then it's questionable whether the feature replication hypothesis sufficiently explains the SB issue observed in DNNs.\n\n2. I'm skeptical about the results and conclusions derived from Table-1. How the correlation analysis was performed is not clear. What is the metric used? The thing that bothers me in Table-1 is there is no reason to believe features in the last layer are disentangled wrt color and shape. There is no reason to assume that color and shape will be disentangled by the feature extractor for free (without any explicit constraints to ensure this). Can you better justify why features must be disentangled here? Apart from this toyish setup, do you have any more concrete realistic datasets/experiments/results to support the feature replication hypothesis? This is concerning as its one of the main\npoints of the paper, and it's not well justified according to me.\n\n3. Another concern is why Eq-1 in the paper (L_FRR) regularizer results in diverse features. The authors do not provide much reasoning or justification in this regard. Intuitively it appears that this regularizer is trying to enforce the full rank of the W linear weight matrix. I suggest authors explain this in more detail to help the readers understand this. And it's also not clear why the explicit enforcement of a full-rank constraint doesn't work. The authors say its a very stringent constraint, but their regularizer also is trying to learn a transformation\nthat inverts the W matrix, which is essentially trying to enforce the full rank of W. Some more clarity is needed in this regard; why is there such a stark difference between these two seemingly similar strategies?\n\n\n4. The authors should mention the effect of lambda in Eq-4,5. An ablation study in this regard will help know the sensitivity of the method wrt lambda. How dependent is the method on the choice of this value, and what is the possible range of suitable values for this hyperparameter? Maybe I missed this point in the main paper. If this is already present, please point me to it.\n\n5. I think there is a lack of in-depth analysis of the cause and reasoning for the simplicity bias and the other observations the authors make in the paper. And, in the end, the authors simply seem to propose a regularization strategy. The results seem promising, but I feel it is not well supported or backed by in-depth analysis or theory, and hence I'm not fully convinced of the overall story being portrayed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ht_Zo5uJqUO", "reviewer": "Reviewer_4rL6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission hypothesizes that one of the underlying reasons behind OOD failures related to the simplicity bias in neural networks is that the features learned tend to include several replicates of simple features, while complex features are not similarly replicated. This mechanism is claimed to lead to greater emphasis on simpler features, which is responsible for corresponding OOD failures. To prevent this, the submission proposes to add an invertibility criterion to the features-to-logits mapping — imposing invertibility would discourage the nonidentifiability implicit in replicated units in the feature dimensions. This regularizer is shown to result in improvements for OOD benchmarks.", "summary_of_the_review": "While I am not really convinced by the empirical demonstration nor the hypothesis, it seems that the idea of imposing a form of identifiability on the final-layer mapping has a positive effect. It is not very clear to me why this is the case, but I am happy to treat this as an empirical discovery with positive consequences. My initial rating is borderline, while I wait for the authors to figure out the discrepancies in the tables — if it turns out that the results are indeed state-of-the-art, I shall revise my scores.", "strength_and_weaknesses": "The feature replication hypothesis is novel and intriguing, and somewhat intuitive. The empirical results seem to indicate improved performance, suggesting this is an effective method on the whole.\n\nWhile the hypothesis is interesting, I find the empirical validation to be quite weak. There is just one experiment demonstrating it (corresponding to one dataset, one network architecture, one round of training). In effect, the dataset introduces significant variation in the easier-to-learn colour-features (because of the use of ranges instead of monochromatic colouring, and colouring the entire background instead of just the digit) coupled with restriction of shape-variation by using only two digits, which perhaps naturally encourages a network to learn multiple features for different colour-buckets (to account for cases when the correlation does not hold). Of course, the invariant feature is still shape, and this mode of failure might well arise in real life datasets as well. However, this single, highly-contrived experiment does not really serve as a sufficiently reliable model of reality, in my view, and therefore does not really constitute as empirical validation of the hypothesis.\n\nSince the proposed method results in longer training overall, what happens if the baselines are also trained for just as long? It seems likely from the results in Table 2 that ERM by itself might not improve much, but the other baselines (such as RSC) might.\n\nThere seems to be some inconsistencies in the aggregated results in Table 3 vs. the expanded numbers in the Appendix. For example,\n\n — PACS: Table 3 lists SMA as 87.5 while Table 6 lists it as 95.5\n\n — VLCS: Table 3 lists SMA as 78.2 while Table 7 lists it as 80.7\n\n — OfficeHome: Table 3 lists ERM as 66.5 while Table 8 lists it twice, once as 67.6; Table 3 lists SMA as 70.6 while Table 9 lists it as 82.0\n\n — TerraIncognita: Table 3 lists ERM as 46.1 while Table 9 lists it twice, once as 47.8; Table 3 lists SMA as 59.7 while Table 9 lists it as 59.7\n\n — DomainNet: Table 3 lists ERM as 40.9 while Table 9 lists it twice, once as 44.0; Table 3 lists SMA as 46.0 while Table 9 lists it as 60.0\n\n — MIRO numbers aren’t shown in the Appendix-tables\n\nThe SMA discrepancies in particular might invalidate the claim that a new state-of-the-art is set.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ePJrO_SxC1L", "reviewer": "Reviewer_PdgB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inspired by the previous observation that the network is able to learn complex features even when it is biasing to simple features. This paper propose to further investigate how much the complex features are learned compared with the simple features. They find that the simple features are replicated multiple times over complicated ones. Based on this observation, they propose a novel regularization term and verify the effectiveness with some experiments.  ", "summary_of_the_review": "Overall, the authors observe a very interesting phenomenon, based on the observation, a regularization term is proposed. But the reviewer feels more comprehensive analysis can be done and included, and the quality of the current version of the paper is not yet ready for publication in ICLR. ", "strength_and_weaknesses": "**Strength**: \n- The paper spot an interesting observation that even though the complex features are learned, the amount of it is significantly less than the simple ones. \n- A feature reconstruction regularization is proposed to enhance robustness.\n\n**Weakness**:\n- Lack of comprehensive analysis on the observed phenomenon. \n- The writing of the paper is not clear. \n- More experiments need to be done such as on larger datasets and comparing with other SoTA methods. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Hv0dqWx3FF", "reviewer": "Reviewer_GBf7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of countering the existence of bias features in the data, with a hypothesis that the classifier learns the simple features (that are not generalizable in OOD) settings with too many weights, validating this hypothesis in simple MNIST experiment, and then offer some analytical discussions with correspondence to SVM, and compare to some SOTA line experiments. Each part of these above discussions seems reasonably good, but there seem some concerns when these pieces are combined into a coherent manuscript. ", "summary_of_the_review": "the overall paper is interesting, especially the several major components. However, the internal connection of these components seems missing. ", "strength_and_weaknesses": "- strength\n    - the flow of the paper (a hypothesis, validation, theoretical discussion, empirical results) goes very well, and logic of the contributions are expanded nicely. \n    - the paper studies an important problem and build another layer to the current knowledge of the problem. \n\n- weakness\n   - how each pieces of the main logic flow are connected together is elusive to me (see detailed comments below)\n   - the authors use DomainBed as an empirical testbed for the results, but not following the standard test protocol that DomainBed introduces: it seems there are missing datasets from the DomainBed protocol. \n     ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster