Simplicity bias in $1$-hidden layer neural networks
Authors: Depen Morwani
Praneeth Netrapalli
jatin batra
Karthikeyan Shanmugam
Prateek Jain
Venue: Submitted to ICLR 2023 (2023)
Keywords: Simplicity Bias
Neural Network
Gradient Descent

Abstract: Recent works \citep{shah2020pitfalls,chen2021intriguing} have demonstrated that neural networks exhibit extreme \emph{simplicity bias} (SB). That is,  they learn \emph{only the simplest} features  to solve a task at hand, even in the presence of other, more robust but more complex features. Due to lack of a general and rigorous definition of \emph{features}, these works showcase SB on \emph{semi-synthetic} datasets such as Color-MNIST, MNIST-CIFAR where defining features is relatively easier. 

In this work, we rigorously define as well as thoroughly establish SB for \emph{one hidden layer} neural networks. More concretely, (i) we define SB as the network essentially being a function of a low dimensional projection of the inputs (ii) theoretically, we show that when the data is linearly separable, the network primarily depends on only the linearly separable ($1$-dimensional) subspace even in the presence of an arbitrarily large number of other, more complex features which could have led to a significantly more robust classifier,  (iii) empirically, we show that models trained on \emph{real} datasets such as Imagenette and Waterbirds-Landbirds indeed depend on a low dimensional projection of the inputs, thereby demonstrating SB on these datasets, iv) finally, we present a natural ensemble approach that encourages diversity in  models by training successive models on features not used by earlier models, and demonstrate that it yields models that are significantly more robust to Gaussian noise.

Under review as a conference paper at ICLR 2023
SIMPLICITY BIAS IN 1-HIDDEN LAYER NEURAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent works Shah et al. (2020); Chen et al. (202I) have demonstrated that neural networks exhibit extreme simplicity bias (SB). That is, they learn only the sim- plest features to solve a task at hand, even in the presence of other, more robust but more complex features. Due to the lack of general and rigorous definition of features, these works showcase SB on semi-synthetic datasets such as Color- MNIST, MNIST-CIFAR where defining features is relatively easier.
In this work, we rigorously define as well as thoroughly establish SB for one hidden layer neural networks. More concretely, (i) we define SB as the network essentially being a function of low dimensional projection the inputs (ii) the- oretically, we show that when the data is linearly separable, the network primarily depends on only the linearly separable (1 (1-dimensional) subspace even in the pres- ence of an arbitrarily large number of other, more complex features which could have led to a significantly more robust classifier, (iii) empirically, we show that in-
models trained deed depend on low dimensional projection of the inputs, thereby demonstrating SB on these datasets, iv) finally, we present a natural ensemble approach that en- courages diversity in models by training successive models on features not used by earlier models, and demonstrate that yields models that are significantly more robust to Gaussian noise.
real datasets such as Imagenette and Waterbirds-Landber
INTRODUCTION
It is well known that neural networks (NNs)
are vulnerable to distribution shifts as well as
to adversarial examples (Szegedy et al. 2014
Hendrycks et al., (021).
recent line of
work (Geirhos et al. 2018 Shah al. 2020
Geirhos et al. 2020)) proposes that Simplicity Bias (SB) aka shortcut learning-i.e.. the ten- dency of neural networks (NNs) to learn only the simplest features over other useful but more complex features, is a key reason behind this non-robustness. The argument follows: for example, in the classification of swans vs bears, as illustrated in Figure [ there many features such as background, color of the animal, shape of the animal etc. that can be used for classification. However using only one or few of them can lead to models that are not robust to specific distribution shifts, while using all the features can lead to more robust models.
Figure 1: Classification of swans vs bears. There
several features such as background, color of the animal, shape of the animal etc., each of which
roughly as
sufficient for classification but using all of them will lead to more robust model.
Several recent works have demonstrated SB on a variety of semi-real constructed datasets (Geirhos) et al. 2018 Shah et al. [2200] Chen et al.) 2021). and have hypothesized SB to be the key reason for NN's brittleness to distribution shifts (Shah et al. 2020). However, such observations are still only for specific semi-real datasets, and general method that identify SB given dataset and given model is still missing in literature. Such a method would be useful not only to estimate the robustness of a model but could also help in designing more robust models.
Image source: Wikipedia swa

---

Under review as a conference paper at ICLR 2023
A key challenge in designing such a general method to identify (and potentially fix) SB is that the notion of feature itself is vague and lacks a rigorous definition. Existing works like Geirhos et al. (0188; Shah et al.] (2020); Chen et al. (202I) avoid this challenge of vague feature definition by using carefully designed datasets (e.g., concatenation of MNIST images and CIFAR images), where certain high level features (e.g., MNIST features and CIFAR features, shape and texture features) are already baked in the dataset definition, and arguing about their simplicity is intuitively easy.
Contributions:
One of the main contributions of this work is to provide a precise definition of particular simplicity b - LD-SB- --hidden layer neural networks. In particular, we characterize SB as low dimensional input dependence of the model. Concretely,
Definition 1.1 (LD-SB), A model R + R with inputs R and outputs f(z) R (e.g., logits for e classes), trained on a distribution (z. y) ~ D satisfies LD-SB if there exists a projection matrix P E Rdxd satisfying the following:
* rank (P) 1k]=k<> d,
[(Px1 + P122) Re (21 V(21, MI), (22, D, and
. An independent model g trained on (Piz,y) where (I, y) ~ DD achieves high accuracy.
Here p- is the projection matrix onto the subspace orthogonal to P.
In words, LD-SB says that there exists a small k-dimensional subspace (given by the projection matrix P) in the input space H" which is the only thing that the model f considers in labeling any input point x. In particular, if we mix two data points I1 and 22 by using the projection of i onto P and the projection of 22 onto the orthogonal subspace Pl, the output of f on this mixed point Pri + Pi 22 is the same as that on æ1, This would have been fine if the subspace P1 does not contain any feature useful for classification. However, the third bullet point says that P. indeed contains features that are useful for classification since an independent model trained on (P_x,y) achieves high accuracy.
Furthermore, theoretically, we demonstrate LD-SB of 1-hidden layer NNs for fairly general class of distributions called independent features model (IFM), where the features (i.e., coordinates) are distributed independently conditioned on the label. IFM has a long history and is widely studied, especially in the context of saive-Bayes classifiers Lewis [1998). For IFM, we show that as long as there is even a single feature in which the data is linearly separable, NNs trained using SGD will learn models that rely almost exclusively on this linearly separable feature, even when there are an arbitrarily large number of features in which the data is separable but with non-linear boundary. Empirically, we demonstrate LD-SB on three real world datasets: binary and multiclass version of Imagenette (FastAl 2021) well as anterbirds-landbird. (Sagawa et al. 2020a) dataset. Compared to the results in Shah et al. (2000), our results (i) theoretically show LD-SB in a fairly general setting and (ii) empirically show LD-SB on real datasets.
Finally, building upon these insights, we propose a simple ensemble method OrthoP - that se- quentially constructs NNs by projecting out principle input data directions that are used by previous NNs. We demonstrate that this method can lead to significantly more robust ensembles for real- world datasets in presence of simple distribution shifts like Gaussian noise.
Why only 1-hidden layer networks?: One might wonder why the results in this paper are restricted to l-hidden layer networks and why they are interesting. We present two reasons.
1. From a theoretical standpoint, prior works have thoroughly characterized the training dynamics of infinite width 1-hidden layer networks under different initialization schemes (Chizat et al.) 2019) and have also identified the limit points of gradient descent for such networks (Chizat & Bach 2020). Our results crucially build upon these prior works. On the other hand, we do not have such a clear understanding of the dynamics of deeper networks.
From a practical standpoint, the dominant paradigm in machine learning right now is to pretrain large models on large amounts of data and then finetune small target datasets. Given the large and diverse pretraining data seen by these models, it has been observed that they do learn rich features (Rosenfeld et al. 2022 Nasery et al. 2022). However, finetuning on target datasets might not utilize all the features in the pretrained model. Consequently, approuches that can train robust finetuning heads (such 1-hidden layer network on top) can be quite effective.
2

---

Under review as a conference paper at ICLR 2023
Extending our results to deeper networks and other architectures is exciting direction of re- search from both theoretical and practical points of view.
Paper organization: This paper is organized as follows. Section presents related work. Section presents preliminaries. Our main results on LD-SB are presented in Section Section presents results on training diverse classifiers. We conclude in Section
2 RELATED WORK
Simplicity Bias: Subsequent to Shah et al. (2020). there have been several papers investigating the presence/absence of SB in various networks as well as reasons behind SB Scimeca et al. (2021). Of these, Huuh al. (2021) and [Gllant & Poggio 2022 are the most closely related works to ours. Huh et al. (2021 empirically observe that on certain synthetic datasets, the embeddings of NNs both initialization as well after training have low rank structure. Galanti & Poggio (2022) provide a theoretical intuition behind the relation between various hyperparameters (such as learning rate, batch size etc.) and rank of learnt weight matrices, and demonstrate it empirically. In contrast, we prove LD-SB theoretically on the IFM model as well as empirically validate this on real datasets. Moreover, we also show how to use LD-SB to train second diverse model and combine to obtain robust ensemble. PPezeshki et al. (021) proposes that gradient starvation at the beginning of training is a potential reason for SB in the lazy/NTK regime but the conditions are hard to interpret. In contrast, our results are shown for any dataset in the IFM model in the rich regime of training. Finally Lyu et al. (202T) consider anti-symmetric datasets and show that single hidden layer input homogeneous networks (i.e., without bias parameters) converge to linear classifiers. However, such networks have strictly weaker expressive power compared to those with bias parameters.
Learning diverse classifiers: There have been several works that attempt to learn diverse classifiers. Most works try to learn such models by ensuring that the input gradients these models do not align (Ross & Doshi- Velez, 2018 Teney al.) 2022). [xu et al. (2022) proposes a way learn diverse/orthogonal classifiers under the assumption that a complete classifier, that uses all features is available, and demonstrates its utility for various downstream tasks such as style transfer. Lee
it al. (2022) learns diverse classifiers by enforcing diversity on unlabeled target data.
Spurious correlations: There has been large body of work which identifies the reasons for spu- rious correlations in NNs (Sagawa et al. 2020b) as well as proposing algorithmic fixes in different settings (Liu al. 2021, Chen et al.) 2020p.
There is also large body of work understanding the implicit bias of gradient descent dynamics. Most of these works are for standard linear (Ji & Telgarsky] (2019) or deep linear networks (Soudry et al., [0188] Gunasekar et al., 2018). For nonlinear neural networks, one of the well-known results is for the case of 1-hidden layer neural networks with homogeneous activation functions (Chizat & Bach 2020), which we crucially use in our proofs.
Implicit bias of gradient descent:
3 PRELIMINARIES
In this section, we provide the notation and background on infinite width max-margin classifiers that required to interpret the results of this paper.
3.1 BASIC NOTIONS
1-hidden layer neural networks and loss function. Consider instances x E Rd and labels y E {+1} jointly distributed as D. 1-hidden layer neural network model for predicting the label for given instance x, is defined by parameters (w E mx b E RT ā E RmT). For a fixed activation function Φ given input instance x, the model is given as f( b, a). x) = (a, d(wx + b)). where ((+) is applied elementwise. The cross entropy loss L for a given model f. input x and label is
given as L (J(x), v) del log(1 + exp(-yff((u) b, b,a), x)).
Margin. For data distribution D, the margin of model (xx is given as min (z)).
3

---

Under review as a conference paper at ICLR 2023
Notation. Here is some useful notation that will use repeatedly. For matrix A, A(i, denotes the äth row of A. For any E N, sh-1 denotes the surface of the unit norm Euclidean sphere in dimension k.
3.2 INITIALIZATIONS
The gradient descent dynamics of the network depends strongly on the scale of initialization. In this work, we primarily consider rich regime initialization.
Rich regime. In rich regime initialization, for any i E [m]]. the parameters (vi(i,-), ((.().)))))))) (()) of the first layer are sampled from a uniform distribution on St. Each a(i) is sampled from Unifl -1, 1}, and the oulput of the network is scaled down by 1 (Chizat & Bach 2020). This is roughly equivalent Xavier initialization Glorot & Bengio (2010), where the weight parameters both the layers are initialized approximately as . [O, m when m >> d.
In addition, we also present some results for the lazy regime initialization described below.
Lazy regime. In the lazy regime, the weight parameters in the first layer are initialized with with./(0 N( 4) those of second layer are initialized with (0, ) and the biases are initialized to 0 (Bietti & Mairal Lee et al. 2019). This is approximately equivalent Kaiming initialization (He et al. 2015)
2019
3.3
THE INFINITE WIDTH LIMIT
For 1-hidden layer neural networks with ReLU activation in the infinite width limit i.e., m - oc, Jacot et al. (2018); Chizat et al.] (2019); Chizat & Bach (2020) gave interesting characterizations of the trained model. As mentioned above, the training process of these models falls into one of two regimes depending on the scale of initialization (Chizat et al. 2019):
Rich regime. In the infinite width limit, the neural network parameters be thought of distri- bution y over triples (w, 6, a) E gd+1 where w E Rd b, E R. Under the rich regime initialization, the function f computed by the model can be expressed
f(v, x) (wbb)))~
(a(o (w, x) + 6)]
Chizat & Bach (200)) showed that the training process with rich initialization can be thought of as gradient flow on the Wasserstein-2 space and gave the following characterization of the trained model under the cross entropy loss ((a.)>>(L<(r) (1,(,,(,))) (z),
Theorem 3.1. Chizat & Bach 2020) Under rich initialization in the infinite width limit with cross entropy loss, if gradient flow on 1-hidden layer NN with ReLU activation converges, it converges to a maximum margin classifier v² given as
1/" arg max min yf(v, x). (EP(St+1)(aaa)) (a-i))-
(2)
where P(g++1) denotes the space of distributions over Sd+1.
This training regime known as the 'rich' regime since it learns data dependent features (w, +).
Lazy regime. Pacot et al. (2018) showed that in the infinite width limit, the neural network behaves like a kernel machine. This kernel is popularly known as the Neural Tangent Kernel(NTK), and is given by K(x, _²) (f(=)) af(²)), where W denotes the set of all trainable weight parameters. This initialization regime is called 'lazy' regime since the weights do not change much from ini- tialization, and the NTK remains almost constant, i.c, the network does not learn data dependent features. We will use the following characterization of the NTK for 1-hidden layer neural networks. Theorem 3.2. Bietti & Mairal ( (2019) Under lazy regime initialization in the infinite width limit, the NTK for I-hidden layer neural networks with ReLU activation i.e., Φ(u) max(u, 0), is given
aw aw
(x, x²)
K(x, = ||
||x
TT0
where
k(u) = (2u(T - cos (u)) + v1 - u?)
4

---

Under review as a conference paper at ICLR 2023
Figure 2: Illustration an IFM dataset. Given a class 11 represented by blue and red respectively, each coordinate value is drawn independently from the corresponding distribution. Shown above are the supports of distributions on three different coordinates for an illustrative IFM dataset, for positive and negative labels.
Lazy regime for binary classification. SSuddyy et al. 2018 showed that for linearly separable datasets, gradient descent for linear predictors on logistic loss converges to the max-margin sup- port vector machine (SVM) classifier. This implies that, any sufficiently wide neural network, when trained for finite time in the lazy regime dataset that is separable by the finite-width induced NTK, will tend towards the L2 max margin-classiinn given by
arg min |l s.t. (f(r) 2 1Y (x, D, feH
where H represents the Reproducing Kernel Hilbert Space (RKHS) associated with the finite width kernel (Chizat 2020). With increasing width, this kernel tends towards the ffnntte-width NTK (which is universal (Ji et al., (0200).. Therefore, in lazy regime, we will focus on the L2 max- margin-classifier induced by the infinite-width NTK.
CHARACTERIZATION OF SB IN 1-HIDDEN LAYER NEURAL NETWORKS
In this section, we first theoretically characterize the SB exhibited by gradient descent on linearly separable datasets the independent features model (IFM). The main result, stated in Theorem 4.1 is that for binary classification of inputs in Rd, even if there is single coordinate in which the data is linearly separable, gradient descent dynamics will learn a model that relies solely on this coordinate, even when there are arbitrarily large number d - 1 of coordinates in which the data is separable, but by a non-linear classifier. other words, the simplicity bias of these networks is characterized by low dimensional imput dependence, which denote by LD-SB. We then experimentally verify that NNs trained on some real datasets do indeed satisfy LD-SB.
4.1 DATASET
We consider datasets in the independent features model (IFM), where the joint distribution (I, y) satisfies p(z,g y) r(y) IT 9: (z:ly), i.e, the features are distributed independently condi- tioned on the label y. Here r(y) is distribution over 1-1,+11 and (z;ll)) denotes the conditional distribution cooordiine v; given y. IFM is widely studied literature, particularly in the con- text of maive-Bayes classifiers Lewis (1998). We make the following assumptions which posit that there are at least two features differing complexity for classification: one with linear boundary and at least other with non-linear boundary. See Figure 2 for illustrative example.
One of the coordinates (say, the 1 coordinate WLOGG) separable by linear decision boundary with margin (see Figure 2 i.e. By > O, such that Supp(gi (r1 ly +1)) S [7: oo) and -? E Supp(qi (xyly = - 1)) S (-x0, -7))), where Supp(-) denotes the support of distribution.
None of the other coordinates is linearly separable. More precisely, for all the other coordinates -1 -)) and +1) S Supp(g (z+ y= +1)).
E [d] \ [1), E Supp(g
The dataset
be perfectly classified 3i # 1, such that (x,ll)) has disjoint support for +1 and y = -1
without using the linear coordinate. This means,
Though assume axis aligned features, results also hold for any rotation of the dataset. While our results hold in the general IFM setting, in comparison, current results for SB e.g., /Shhh et al. 2020). obtained for very specialized datasets within IFM, and do not apply IFM in general.
5

---

Under review as a conference paper at ICLR 2023
4.2 MAIN RESULT
Our main result states that, for rich initialization (Section 3.2). NNs demonstrate LD-SB for any IFM dataset satisfying the above conditions. Its proof appears in Appendix A..].
Theorem 4.1. For any dataset in the IFM model satisfying the above conditions and 2 1. if gradient flow for 1-hidden layer FCN under rich initialization in the infinite width limir with cross entropy loss converges, it converges to V 0.5dg, 0.5dg, Sut), where
-1/v2 -1/V2) and er an [1.0, ........ .0 denotes first standard basis vector. This implies f(v", Px1 + P1 = f(v", A (01, V1), (I2, U2) D, where P represents the (rank-1) 1) projection matrix first coordinate.
81 (211492) C1 V2(14~2) 1/v2),82
2((++))) C1 (2(1~)))
Moreover, since at least one of the coordinates 12....dd d} has disjoint support for ( -1) and q: (2 1). P_ (xr) can still perfectly classify the given dataset, thereby implying LD-SB.
It is well known that the rich regime is more relevant for the practical performance of NNs since allows for feature learning. while lazy regime does not (Chizat al. 2019). Nevertheless, in the next section, we present theoretical evidence that LD-SB holds even in the lazy regime, by considering much more specialized dataset within IFM.
4.3
LAZY REGIME
In this regime, will work with the following dataset within the IFM family:
For E {+1} we generate (I, y) E as
X1 TV
+ for y =
Vi E 2,,,, d,
0 for
y -
Although the dataset above is point mass dataset, it still exhibits an important characteristic in common with the rich regime dataset only one of the coordinates linearly separable while others are not. For this dataset, we provide the characterization of max- max-margin NTK (as in Eqn. (3))
Theorem 4.2. For sufficiently small e >> O, there exists an absolute constant N such that for all > N and 7 E [7,eVd), the L2 max-margin classifier for joint training of both the layers of 1-hidden layer FCN in the NTK regime on the dataset D, i.e., any J satisfying Eqn. (3) satisfies:
pred(f + P1x2)) = ( V (11, (x2, V2) E D
where P represents the projection matrix on the first coordinate and pred(f(x) represents the predicted label by the model f on
The above theorem shows that the prediction on mixed example Pæ1 + Plæy is the same as that on 3ı, thus establishing LD-SB. The proof for this theorem provided in Appendix A.2
4.4 EMPIRICAL VERIFICATION
In this section, will present empirical results demonstrating LD-SB on 3 real datasets: Imagenette (FastAl 202T), a binary version of Imagenette (b-Imagenette) and waterbirds- (Sagawal al.] (020a) well as designed dataset MNIST-CIFAR (Shah al. 2020). More details about the datasets can be found in Appendix B.T
4.4.1 EXPERIMENTAL SETUP
We take Imagenet pretrained Resnet-50 models, with 2048 features, for feature extraction and train 1-hidden layer fully connected network, with ReLU nonlinearity, and 100 hidden units, for classi- fication on each of these datasets. During the finetuning process, we freeze the backbone Resnet-50 model and train only the 1-hidden layer head (more details in Appendix B.1).
Demonstrating LD-SB: Given model f(),, we establish its low dimensional SB by identifying a small dimensional subspace, identified by its projection matrix P, such that if we mix inputs æ1 and

---

Under review as a conference paper at ICLR 2023
100
MESTCC Irageraca D- b-mugensta
80
80
Effective rank
Effective rank
60
60
4.0
40
Weartires
100
200
300
400
500
100
200
300
400
500
Epochs
Epochs
(a) Rich regime
(b) Lazy regime
Figure 3: Evolution of effective rank of first layer weight matrices in rich and lazy regimes.
Table 1: Demonstration of LD-SB in the rich regime: This table presents P1 and P randomized accuracies (RA) as well as logit changes (LC) on the four datasets. These results confirm that projection of imput x onto the subspace spanned by P essentially determines the model's prediction on x. t (resp. 4) indicates that LD-SB implies large (resp. small) value.
Dataset
rank (P)
Acc(f)
P1-RA (t)
P-RA())
PI-LC
P-LC (f)
b-Imagenette
93.05+0.26
89.94+0.22
49.53+0.24
28.57+0.26
92.13 + 0.24
Imagenette
10
79.5210.13
75.89:0.25
9.00
33.64#1.21
106.2910.53
Waterbirds
3
91.88
1 0.1
91.47+0.11
62.5110.07
102.35 25.24
MNIST-CIFAR
99.69 1 0.0
94.15:0.21
5.12
38.97110.76
101.98
I2 as Px1+ + +P P1222, the model's output on the mixed input I del Pxi 12++P. PP P122. f()) is always close to the model's output æ1 i.e., (11). We measure closeness in four metries: (1) P1 P_-randomized curacy (PI-RA): accuracy on the dataset (Px1 + P1 22, V1) where (æ1, V1) and (222. 1/2) are sampled iid from the dataset, (2) P-randomizzed accuracy (P-RA): accuracy on the dataset (Pz) +P P1æ2, y2),
(3) Pi logit change (LC::C): relative change wrt logits of x i.e., [f()) f(x1)] (x IlI and (4)P logit change (P-LC): relative change wrt logits of x2 i..., llf(z)
(212)l/ll((x)) (x2((l
As described in Sections 4.2 and 4.3 the training of 1-hidden layer neural networks might follow different trajectories depending on the scale of initialization. So. the subspace projection matrix P will be obtained in different ways for rich vs lazy regimes. For rich regime, we will empirically show that the first layer weights have low rank structure as per Theorem 4. while for lazy regime, we will show that though first layer weights do not exhibit low rank structure, the model still has low dimensional dependence on the input as per Theorem
4.4.2 RICH REGIME
Theorem suggests that asymptotically, the first layer weight matrix will be low rank. However, since we train only for a finite amount of time, the weight matrix will only be approximately low rank. To quantify this, we use the notion of effective rank Roy & Vetterli (207)) measure the rank of the first layer weight matrix.
Definition 4.3. Given a matrix M, its effective rank is defined as: Eff-rank (M) value E. (MM)) (M)? where (MM denotes the th singular value of M and (M def a.(M)²
Σ. E.
One way interpret the effective rank is that it is the exponential of von-Neumann entropy Petz MM 2001 of the matrix TrMM where Tr (-) denotes the trace of a matrix. For illustration, the effective rank of a projection matrix onto & dimensions equals
Figure Ba shows the evolution of the effective rank through training the four datasets. We observe that the effective rank of the weight matrix decreases drastically towards the end of training. To confirm that this indeed leads to LD-SB, set P be the subspace spanned by the top singular
7

---

Under review as a conference paper at ICLR 2023
Table 2: Demonstration of LD-SB in the lazy regime: This table presents P1 and P randomized accuracies as well as logit changes on the four datasets. These results confirm that the projection of input z onto the subspace spanned by P essentially determines the model's prediction on æ.
Dataset
rank (P)
Aco())
PI-RA (1)
P-RA (1)
PI-LC
P-LC (f)
Imagenette
92.7510.06
90.07
52.0911.34
36.9411.01
138.41#1.62
Imagenette
15
79.97+0044
68.25+1.18
11.92+0..2
55.99+3.86
133.86+5.422
Waterbirds
90.46+00.07
89.67+0.42
62.44+4.48
36.89+5.18
105.41+7.06
MNIST-CIFAR
2
99.74 + 0.0
99.45:0.17
49.83+0.67
2..9
141.12
Table 3: Mistake diversity and class conditioned logit correlation of models trained independently (Mist-Div (f.find and CC-LogitCorr (f: fima) resp.) vs trained sequentially after projecting out im- portant features of the first model (Mist-Div (J: (pmij) and CC-LogitCor (J, Sproj) resp.). The results demonstrate that and Sroj are more diverse compared to f and find.
Mist-Div (f. fina) (f)
Mist-Div
CC-LogitCor (fd) (4)
CC-LogitCorr
Dataset
(pro)) (t)
(poj) (L)
B- B-Imagenette
3.87 + 1.54
21.15 1 1.57
99.88 + 0.01
90.86 + 1.08
Imagenette
6.6 + 0.46
11.44 + 0.65
99.31 + 0.12
91 + 0.59
Waterbirds
2.9 + 0.52
14.53 + 0.48
99.66 = 0.04
93.81 + 0.48
MNIST-CIFAR
0.0 0.0
5.56 7.89
99.76 0.17
78.74 # 2.28
directions of the first layer weight matrix and compute P and P1 randomized accuracies well as the relative logit change. The results, presented in Table Table confirm LD-SB in the rich regime on these datasets.
4.4.3 LAZY REGIME
For the lazy regime, it turns out that the rank of first layer weight matrix remains high throughout training. as shown in Figure 3b However, we are able to find low dimensional projection matrix P satisfying the conditions of LD-SB (as stated in Def as the solution to optimization problem. More concretely, given pretrained model f and rank r, we obtain a projection matrix P solving:
1
Σ (C(F(Px)) AL ((PP=;), min TI
wheree/[L] represents a uniform distribution over all the L labels, (æ1, y1). (Zn: yn) are training examples and L (:5.) -) is the cross entropy loss. We reiterate that the optimization only over P. while the model parameters I are unchanged. In words, the above function ensures that the neural network produces correct predictions along P' and uninformative predictions along Pl. Table presents the results for Pı and P-RA as well as LC. As can be seen, even in this case, we are able find small rank projection matrices demonstrating LD-SB.
TRAINING DIVERSE CLASSIFIERS USING OrthoP
Motivated by our results on low dimensional SB, in this section, we present a natural way to train diverse models, so that an ensemble of such models could mitigate SB. More concretely, given an initial model f with a low dimensional projection P that captures its input dependence, we train another model Speoj by projecting the input through P1 i.e., instead of using dataset (æ;, ys) for training, we use (P_ F__x,, ys) for training the second model (denoted by fproj). We refer to this training procedure OrthoP for orthogonal projection.
Given any two models I and J. we evaluate their diversity using two metrics. The first is mistake diversity: Mist-Div (J,J) def (i
((:f(())) f(x))/H
where we abuse notation by using
8

---

Under review as a conference paper at ICLR 2023
b-Imagenette (f)
(b) b- I-magenette Upen)
(c) Waterbirds (f)
(d) Waterbirds Uproy)
Figure 4: Decision boundaries for f and fpmoj for B-Imagenette and Waterbirds datasets, visualized in the top 2 singular directions of the first layer weight matrix. The decision boundary of froj is more non-linear compared to that of f.
08
0.85
acouracy
0035
5td dev
ded_dev
std del_dev
(a) Binary-Imagenette
(b) Imagenette
(c) Waterhirds-Landdiid
Figure 5: Variation of test accuracy vs standard deviation of Gaussian noise added to the pretrained representations of the dataset. Model 1 (i.e., f) is kept fixed, and values for both the ensembles averaged across runs. Standard deviation is shown by the error bars.
f(;)) (resp. f(zz)) to denote the class predicted by (resp f) x;, Higher Mist-Div (f.f) means that there is very little overlap in the mistakes of f and f. The second class conditioned logit cor- relation i.e., correlation between outputs of f and f. conditioned on the class. More concretely, CC-LogitCorr (f.f) E.sy (I
= y) rep- resents the empirical correlation between the logits of f and f on the data points where the true label is 3 the diversity of two independently trained models (f and fina) with that of sequentially trained models (f and fpoo) above. The results demonstrate that and fpoi are more diverse compared to f and fin- Figure 4 shows the decision boundary of f and fpouj on 2-dimensional subspace spanned by top two singular vectors of the weight matrix. We observe that the decision boundary of the second model is more non-linear compared to that of the first model.
where corr([f(x,). [f(x;)]
[>]
Finally, Figure shows the variation of test accuracy with the strength of gaussian noise added to the pretrained representations of the dataset. We can see that ensemble of f and fpmo is much more robust as compared to an ensemble of f and find (where an ensemble is obtained by averaging the logits of the two models).
DISCUSSION
In this work, we characterize the simplicity bias exhibited by one hidden layer neural networks in terms of the low-dimensional input dependence of the model. We provide a theoretical proof for linearly separable datasets, and validate empirically on real datasets. Based on this characteriza- tion, we also propose simple way to train diverse models and show that it leads to models with significantly better Gaussian noise robustness.
This work is initial step towards rigorously defining simplicity bias shortcut learning of neural networks, which is one of the major challenges to their real-world deployment (Geirhos et al., 2020). Providing similar characterization for deeper nets and other architectures is an important research direction, which, in our opinion, requires a better understanding of the training dynamics and limit points of gradient descent on these networks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Bear image from
beear
/media/File:Gr1 htt.://eeee.cccaaaaaeeeeeeeeeeee Accessed: 2022-09-26.
Swan image from wikipedia.
a wwt..//.w///wlleeee......
(Cymms__olor__end_ceeeeeee
Accessed: 2022-09-26.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep ma- trix factorization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL /t///tppp///ppooeeedng...coo
pdf
Alberto Bietti and Julien
On the inductive bias of neural tangent ker- Advances Neural Information Processing Systems, volume 32, URL https: http://oooeeiiinn......eeeeeeeee
Mairal.
nels.
2019.
Paper. pdf
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 14:11834-11845, 2021.
Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. Advances in Neural Information Processing Systems, 33:21061- 11071,20200
Lenaic Chizat. Gradient descent for wide
two-layer neural networks
ii: Generalization and implicit bias.
fattp:://rancissaach.. com/
gradient -dsceet-- for ffor-wide- two layer newral new.aal-eewwors---nn
2020.
Lénaïc Chizat and Francis R. Bach, Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, 2020, volume 125, pp- 1305-1338. 2020.
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differen- tiable programming.
In Advances in Neural Information Processing Systems, vol- wtass.cc/pppeee cttp:///s../ppp///1///...e. pdf
32, 2019. URL
Matthew Cook, Alina Zare, and Paul Gader. detecttion through null space analysis of neural networks, 2020. uw..http../axxxx..rr///////00000
FastAl. Imagenette dataset. http:s///gtthtb..o//aaaaaaaaaaaa
2021.
Tomer Galanti and Tomaso Poggio. Sgd noise and implicit low-rank bias in deep neural networks, 2022.
02799 05794
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-traing trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2018.
Robert Geirhos, Jörn- Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 663,-673, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett
10

---

Under review as a conference paper at ICLR 2023
(eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL ottp://s///cccea.....o....a.eas. httpp//pss.cccpppoo////////////
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient de- scent on linear convolutional networks, In Advances in Newral Information Processing Systems, volume 31, 2018. URL 55411Cassccaaae511ccccccellll2-2
pdf
Guy Hacohen and Daphna Weinshall. Principal components bias in wver-aaaaettrrize linear mod- els, and its manifestation in deep neural networks. Journal of Machine Learning Research, 23 (155):1 46, 2022. URL http: jmlr. org/ papers/ v2 212/3//2199911111 html
Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree to agree: Neural networks share classification order on real datasets. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3950-3960, PMLR, 13-18 Jul 2020. URL htttps:/
mlr. www.ppress/vee Sttesss/l19/hacchee222 html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribus pp. 3340-8349, 2021.
generalization. In International Conference on Computer Vision,
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprini arXiv: 100. 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Informaiion Processing Systems, pp- 8580-8589, 2018
Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory, pp. 1772-1798, 2019.
Ziwei Ji, Matus Telgarsky, and Ruicheng Xian. Neural tangent kemels, transportation mappings, and universal approximation. In International Conference on Learning Representations, 2020.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complex- ity. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019, URL htttps::/procceeiingg/ www.:gs..e.uiuscoooooooo htt.rriis..cc/pppp////////1/ 7053225565a9777777777066688338
.dff
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems, volume 32, 2019. URL
pdf
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from under- specified data. arXiv preprint arXiv:2202.03418. 2022.
David Lewis. Naive (bayes) at forty: The independence assumption information retrieval. In European conference machine learning, pp. 4-15, 1998.
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations, 2021.

---

Under review as a conference paper at ICLR 2023
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference Machine Learning, pp. 6781-6792, 2021.
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent two-layer nets: Margin maximization and simplicity bias. Advances in Newral Information Processing Systems, 34:12978-12991, 2021.
Anshul Nasery, Sravanti Addepalli, Praneeth Netrapalli, and Prateek Jain. Daft: Distilling adversar- ially fine-tuned models for better ood generalization. arXiv preprint arXiv:2208.091: 19939, 2022.
Ibrahima Ndiour, Nilesh Ahuja, and Omesh Tickoo. -uttof-distri detection with subspace techniques and probabilistic modeling of features, 2020. URL htw.hhtsss//eeiii..rrrrees abs/
2012. 04250
Dénes Petz. Entropy, von neumann and the von neumann entropy. In John von Neumann and the foundations of quantum physics, pp. 83-96. 2001.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34: 125--1222 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. 2018. doi: /ttsttts://arxxv wwtp://w..rr/ab/////e/e/ 08734
Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 21174-21187. Curran As- cc
sociates, Inc., 2020. URL BtLLattps//ppccccddn..g..eer httppsscccppppppepe. pptppa/pcp//ELLL../
pdf
Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL hntpp:://proceednnsss newriis.eeu.pppp.pp wwtpuurissccpppppeeeeee pctpppppee/219//1eeee.. 4
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint 2rXi::2202.06856. 2022.
Andrew Ross and Finale Doshi- Velez. Improving the adversarial robustness and interpretabilitty of deep neural networks by regularizing their input gradients. In AAAI Conference on Artificial Intelligence, volume 32, 2018.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In European signal processing conference, pp. 606-610, 2007.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference Learning Representations, 2020a.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparametcrization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-8356, 2020b.
Luca Scimeca. Scong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shortcut cues will dnns choose? a study from the parameter-space perspective. In International Conference on Learning Representations, 2021.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan. Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
12

---

Under review as a conference paper at ICLR 2023
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data. J. Mach. Learn. Res., 19(1):282-2285 jan 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16761-16772, 2022.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribttion with virtual-logit matching. In IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 4911-4920. IEEE, 2022. doi: 10.1 PTW.///VPS2288.02222222 2022.000 00487
URL https: http:///oooooooo /ttp://...../....111////////..
Yilun Xu, Hao He, Tianxiao Shen, and Tommi Jaakkola. Controlling directions orthogonal to classifier. arXiv preprint 12X1 2022.
Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural net- works. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737. PMLR, 18-24 Jul 2021. URL https://groceednggss www.:innnssm....p......m v139/yang2lc.htm.
Alireza Zaeemzadeh, Niccolo Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. Out-o--disti detection using union of 1-dimensional subspaces. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9452-9461, June 2021.
13

---

Under review as a conference paper at ICLR 2023
A PROOFS FOR RICH AND LAZY REGIME
A. RICH REGIME
We restate Theorem 4.1 below and prove it.
Theorem A.1. For any dataset in IFM model satisfying the conditions in Section 4.1 2 and x) as in Eqn. (/). the distribution v² = 0.5d0, + 0.580, on S the unique max-margin classifier satisfying Eqn. (2). where 01 (2214477] e1, /2(4413 1/v2),111///))/ 1/v2), 02 =
(2(1+22) C1 /2114-2) -1/v2) and e1 def [1.0..... ,0] denotes first standard basis vector In particular, this implies that if gradient flow for 1-hidden layer FCN under rich initializa- tion in the infinite width limit with cross entropy loss converges, it converges v² satisfying f(v", Pxi + P1 = J(v",, ;x1 21)((a113))) (2) E D, where represents the (rank-1) pro- jection matrix on the first coordinate.
Proof of Theorem A.7 Chizat & Bach 2020) showed the following primal-dual characterization of maximum margin classifiers in eqn. (2)
Lemma A.2. (Chizat & Bach 2020) v² satisfies eqn. if there exists data distribution p" such that the following two complementary slackness conditions hold:
Supp(v") S
arg max bl((o(t,) + b]]] and
(4)
(w.baa)E84+1
Supp(p") C arg min ( (()((ww + b))]
(5)
(aaa)~>
The plan to construct a distribution p" that satisfies the conditions of the above Lemma.
Uniqueness. Note further that for fixed p", (( ( (v) is an upper bound for the margin (2,g)) vf(,,)) x) of any classifier v. Hence, for uniqueness, it suffices to show that ōØ, 5g, are the unique maximizers of the objective the RHS of eqn. 4 and that the unique maximum margin convex combination of 6g, 5ø, over DD is v²"
We first describe the support D of p² For y E {+1} generate (I,y) D
X1 = yy
1 for 1
Wi 2E, d,x;
0 for y = -1
Now for (I, y) E D, define
y=1 0.5d for -1
0.5 for
p (2,
Note that p' is supported on 24-1 positive instances and negative instance. We begin by showing eqn. (5)
Claim A.3. p¹ as in eqn. 16) satisfies eqn. (5). Further, the unique maximum margin convex combi- nation of da. 8b, is v*.
Proof. Let find the minimizers (x, v) of yf x) = -E b)]] for any =dd,, (1 Ajdog. 0 < X < 1.
()) for (x. y) with -1 (denoting #1 by -0n, where on 2 is
vf(,,z) = -1 -1[2 tb
7
oj eT([a1e)) + /2(1+)2)
1
12 + +72)
V2
1
-1
+ (1 X)
cT [[(-110))+ +
[22(1+72)
22(1+
V2.
14

---

Under review as a conference paper at ICLR 2023
and for (x, with y (denoting 21 by 02, where dg 7) is
1
uf(u,z)= = 1 x 112 777)
(v211+T3)))
ej (a2e1)) + +2(1+
V22
+ x)
7
ei (aze1)
1
-1
2 72)
22 + +
As 2 1, the expressions above equal VraytI and (1 A) 1) //aat1 respectively, and hence are 2
minimized a1 (r2 7 Hence, the margin of is min(), A) vi+z² which is uniquely maximized at À = 1/2. Further for = 1/2,, all points in have the same value yf(,
In the rest of the proof we show eqn. 4 Let us denote by g(,,,, b, == (a v/a00 +b))] We show that 0g, d0, are the only maximizers of g(w, b, a) over Sd+1
We first find g(0)), (02)
((1)) = Pr((y = 1) 1.
Φ
7
c (70))) +
1
V2
12((1 + +?2)
/2(1 + 72))
+ Pr(y=
V22 o
eT (-11)) + /2 /2(1+
1
72
/2 +
4
3211
where the first term is because W2, ........ are zero for 01. Similarly, g(02)
We now
show that g(w, a, b) < V for (w.a,b) a, b) & (01, 02) We begin by showing the following simple but useful claim.
Claim A.4. All maximizers of g(w, b, a) over gd-1 satisfy |a| = 1/v2.
Proof.
The proof essentially follows from the 1- homoogennytt of the ReLU function ø and sepa-
rability of g(w, b, a). Note that g(w, b, V| baa g(w', b), where | + 82 = 1. Maximizing g(w, a) is equivalent to maximizing g( "b, over S and / w| + D2 S respectively. The second of these has its unique maximum |a| = 1/v2, completing the proof.
Now express g(w, b, a) as
(w. b, a) (Pr(y 1)E(O( xx b)ly - -Pr(y - 11)E(d(w +bly -1])
E, [o(rw] b Σ a,w;)] - d(b -yw1) >wn))
where o, are independent Rademacher random variables. We have two cases on a:
Case 1: 1/v2. By eqn.
7 we have
b.v.b1//22 < E, (0((1) +b b Σ arw)))
2/2
2=2
To simplify the above, define the random variable X Σ daw;; and denote ywi + b by Note
that |a| Iyw1 + bl $ /
7++1 which follows from |lm|2 + 12 1/2. The expectation in the last
expression above becomes
E(o(X + a)] = E + - - > + a 2 2aa)
=EX1 2 at + Pr(X 2 a)) - E[X1{X 2 atl ta,
15

---

Under review as a conference paper at ICLR 2023
where the last equality follows from symmettr of X. Note that Var
Σi= w which is most
14-22 (using ywu + = a and ||l|2 + b² = 1/2). Using A.5 to upper bound E [ 2 atl
a²
we have
B + +a]] sat
1 min
14
a²
(
a²
2
2a
1 +
We can check that the RHS of the above has its unique maximizer at a =
1 for |a| S V
1ta2
2
Hence g(w, a, S VI+y² in this case. We are now done since any (w1, satisfying yw + b
/ and wp b² 1/2 has b v2(1+>))
2
Case 1/v22. Using eqn. we have g(w, b, 1//22 S @(b - 721)/2/2 which for 6² + w < 1/2 attains its unique maximum
1 at b (2(1+)))
Finally, note that the weights of the trained network (w, b, a) sampled from v/" Hence, the final claim in the theorem about [VV", Pxi + Pi 122 follows since the distribution of w only has support on e and -e1-
A.1.1 AUXILIARY LEMMAS FOR RICH REGIME
Lemma A.S. For any symmetric discrete random variable X with bounded variance, for >> 0,
E[XT(X 2 a)) V min (1 Var(X)
Var(X).
2a²
Proof.
EXXI(X 2 a)]= Σ Exp(x)
Vylz)Vp(r)t p( a) Σ x2p(x)) 22a
Σ
where the last inequality by Cauchy-Schwartz. Also by Chebyshev's inequality, p( 2 < V (ar(X)/20) Combining this with eqn. (8) and using symmetry of X and non-negativity of a gives the required lemma.
A.2 LAZY REGIME
Theorem 4.2 is corollary of the following more general theorem.
Theorem A.6. Consider point E D. For sufficiently small O, there exist absolute constant N such that for all d > N.y < ed and 2 7, for the joint training of both the layers of 1-hidden layer FCN the NTK regime, the prediction of any point of the form (C, 12:a) satisfies the following:
1. For Ç 0.73, the prediction is positive.
2. For Ç < -0.95%, the prediction is negative.
The above theorem establishes that perturbing æ1 by O(7) changes pr)d(f(x)) for D (whereas classifier exists that achieves a margin of ((vd) on D, D has margin for coordinates (2...d)). As = o(d), this shows that the learned model is adversarially vulnerable.
The idea of the proof is to obtain an explicit expression for f(x) by applying standard kernel max-margin SVM theory the NTK kernel 3.2
Proof of Theorem A.6
16

---

Under review as a conference paper at ICLR 2023
We begin with some preliminaries. We will refer to the first coordinate of the instance the 'linear' coordinate, and to the rest as 'non-linear' coordinates. Also, henceforth we append an extra coordi- nate with value 1 to all our instances (corresponding to bias term) is standard for working with unbiased SVM without loss of generality.
Explicit expression for f. Using representer theorem for max margin kernel SVM, know that can be expressed as
f(z)
E (01)EE
Aay K (x, x0)),
for some X, 2 0 (that are known as Lagrange multipliers). Further by KKT conditions, a function possessing such representation (that correctly classifies D) has maximum margin if y f(x(t))= = 1 whenever X > 0 (training points t satisfying X > O are called support vectors).
We begin with useful claim.
Claim A.7. The max margin kernel SVM for D with the NTK kernel has all points in D as support vectors.
Proof. By the above discussion, it suffices show that the (unique) solution RIPI Ka y satisfies sign(a;; ()) for all i, where K is the |D] |D| Gram matrix with (i,j)th entry ((x(0) x(6) and y/: (d (the Lagrange multipliers X, then given by yaa,).
Structure of Gram matrix. Order D so that the positive instances appear first. Then the Gram B C. where B E 1 1x2" and R R are the matrix K has block structure of the form (Cr (
(R)
x1 represents the Gram matrices for the positive and negative instances respectively, and C E R 2 K K(x(0), o D)) values for i < [D].
Recall that for the NTK kernel, K ((x:(9,, has the form (1)l))))))))((((()()))()))()))) ()))) Note all the positive instances have the same norm (denoted by P1 vd+ and the inner product between two positive instances depends only on the number i of non-matching non-linear coordinates (de- noted by 8 for 0 < i S d 1). Hence, the rows of B are permutations of each other, with the entry 3 appearing ( times. Similarly, the entries in C' are all equal and are denoted by PIP25a where Ba denotes ( DI) for any & < |D] and P2 = ||a| / + 72 The only entry in is pāx(1). In particular,
8, - 2i +1-2i+7²)
1 1- 2
for [ID] - 1},
and Ba=
d d+
Vd+
Now are ready to solve Ka y. By symmetry in the structure of K, a looks like [a a ..... 6). where the first [D] 1 entries are the same.
Expanding Ka y, we get two equations given by
A-1 1
api
Σ
1-11
Bi +
bpiP28a= =1
and 2d=1 apipp22 + p²xx(1b
i
Solving, we get
P2t((1) + P1
and b= -1 2d- ap1P28a
p2p2 Ei
(1-)) [s{(1) -39
(3x(11)
We now show that a 0 and b < Note that for sufficiently large d, 34 can be made arbitrarily close to x(0) 1/x (since k is smooth around 0). Hence, a > 0 implies b O. We in fact give the following estimate for a:
a = 21-d (2k + P1
2
+ 0 ( $
where
(9)
Epp
π
z
17

---

Under review as a conference paper at ICLR 2023
For the lower bound on write
d-1
[d/2] (d-1)
1
[r((1]3;
-331= x(1) Σ
(3: + BB-11) - 23
i
==
Id/2|
Σ
d 1 i 2,30/2-2
- 2
220 32
2d
(<())<(0)~
where for the first inequality we used convexity of and for the second inequality we used 3a/2 <<0) ((1d)) Ba x(0) + O(1/vd). For the upper bound on &, write
1
2i
(n((1)
331 S R(1)
Σ
K
d 72
1=0
2i
k(1)(d 1)2d-1
x(1
2
x11)2
d + 72
d + 72
where for the second inequality we used n²u) S + u (which holds by convexity and x(-1) (11) 2).
Now analyze predicted labels for points of the form (2.02. where E D. We make two cases depending on the label of x.
Predicted label for point (2.42 where 2.(1) D has positive label
Our point (denoted by x)) has the form (S, S1, .2..., 1) where & E #1. The idea of the proof is to write I' explicitly as a function of and work with its first order Taylor expansion around 7 with some additional work to take care of non-smoothness of f.
Explicit form for f. Let Tú def x²) have exactly i non-matching non-linear coordinates (for 0 < S d - 1). Similarly denote by Ta the quantity (z, / [[[x=|||>= []. In particular,
([1== (I|=||] ||=" (]) for positive instance x' E D, where x and x'
d 2i + (0
75 pz |||
Pi ||>|||
and
Ta
By the above discussion, we have
1
d-1
f(x) = Σ K(ora(0)) + bK (,, = apıl |2:l
Σ
m((T)) + (b02)(=((<((T)))
Substituting and denoting [ (x)//|=]]| by g(S) we get
g(<) = ap1 Σ d --11 ((=)) ((T)(())
20-1 so(1) m(Ta(()))
k(Ta(()) P2x(1)
(10)
Now try to expand g() using the Taylor series around Ç = 7 (note that g(7) = 1/P1). Note that k' can however be unbounded around -1 and 1. To get around this, write g h + where h has bounded first and second derivative, and q has lower order than for Ç interest. In particular,
3d/4
d m((((())
2d-13a
a(Ta(()) (2n(1)
(() = apı
Σ imd/4
(((F(())) x(1)
and
i
(d-1) M((T(3))
q(C) = ap1
Σ
1:/4/2-1/2d/1
18

---

Under review as a conference paper at ICLR 2023
Observe that g(L) o(c*)) for 1 using the estimate eqn. (9) for and concentration for sums of independent Bernoullis. By Taylor's theorem,
g(C) h(7) ((())-(())()) + (()() 7)2
g(C),
(11)
for some 9 E [), where h(7) = 1/Vd. It will turn out that [1'(7)] = 0(1VVd), [W((() = o(1/v This will allow us to complete the proof using the linear approximation of g({) by ne- glecting the second order term and g((). We now compute h', h², treating ||ж||| vd + (2 constant for exposition (the proof works without this approximation or the reader may think of 7 as o(v))). Using TC( x pil Ta(()=
((()) ap1
d 1
Re' (T:(S))
2d
k2 (Fa((())
7
+
n'(r((()) y
PI p1||z11
1(1)
P2 p2|1zr1|
Pen(1)) pel|all
1) K6" (T((S)
22
2d-13a
72
(F(a(()) 72
h²(() = ap1
p
-
x(1)
R
(Ta(S))
p 2012
(2n(1) -31
Plugging ||=|||| 2 2 d and substituting a from eqn.
n'(c) = (1 + (2/()d((Ta(())
k²(4) = o (1) and
+
(3x(1)va
which substituted in eqn.
11 with ra(S) 2z 0, 3, = x(0), 16 (Ta(S)) = x (0) gives
1 (1 1+ (1 (1) (0)) (S (5- + (v2) .(() va
x(1)p3
Hence, g(C) > 0 whenever the coefficient of 1/Vd above is bounded above zero, and similar
condition holds for 9(() < 0. Using the estimates of from eqn. (9) and k'(0) 1,
1/T, x1) = p2 = 1 + ?2 in the above gives that g({) > 0 for > -0.88 - 1.68/7 and g(C) 0 for Ç 0.905 1.905/7-
Predicted label for point (C, x 22d41) where (1) E D has negative label
(1)
Following the same plan, write our point (denoted by x) as (C, 0, ...,...... 0, 1).
Explicit form for f. Begin by finding
15 75
1 (1-29 75 Ta
T:
||0|||
and
P2 |2:||
eqn. T0
now gives
g(() 1dppppp [x(To(()) Ban(Ta((S)] x(1)
k(Ta(()) p2*(1)
Expanding k(To(()) using Taylor series around <=
<(ro({)) xx() + n² (TT )
+
for some 0 E [-1, 1]. For large d, (0) 0 and ()()) 0(1/Vo). Hence we have
g(C)= = p2x(1) 1) + +1
( 1 Ban(Ta(si)] m((a({)) 1
x(0) + +
EPiPs
Vi x(1)
p2n(1)
1 (0)
(²)(0)
+ x(I) M(Ta(()) + o[1).
=
P2 E
Ex(1)
As before g({) > whenever the coefficient of 1/p2 above is bounded above zero which happens
for < 2 0.73 (for 7 2 3). Similarly, g(S) < 0 for < S O.
19

---

Under review as a conference paper at ICLR 2023
B EXPERIMENTS
In this section, we provide experimental details, including hyperparameter tuning setup and some additional experiments.
B. DETAILS THE EXPERIMENTAL SETTING
We will first describe the four datasets that have been used in this work.
1. Imagenette (FastAl 2021): This is a subset of 10 classes of Imagenet, that are compara- tively easier to classify.
2. b-Imagenette: This a binarized version of Imagenette, where only subset of classes (tench and English springer) is used.
3. SiterbirdssLanddiiee (Sagawa et al. 2020a;: This is a
group dataset, consisting of waterbirds on water and land background, well landbirds land and water background. This dataset serves as a baseline for checking the dependence of model on the spurious background feature when predicting the bird class, most of the training examples have waterbirds on water and landbirds on land background.
4. MNIST-CIFAR (Shah et al.) 2020): This collage dataset, created by concatenating MNIST and CIFAR images along axis. This is synthetic dataset for evaluating the simplicity bias of a trained model.
Setup Throughout the paper, we work with the pretrained representations of the above datasets, obtained by using an Imagenet pretrained Resnet 50. We finetune 1-hidden layer FCN (hidden dimension 100) on top of these representations (keeping the backbone fixed) using SGD with a momentum of 0.9. Every model is trained for 20000 steps (large enough for convergence) with warmup and cosine decay learning rate scheduler. For each of the runs, we tune the batch size, learning rate and weight decay using validation accuracy. Below the hyperparameter tuning details:
Batch size e (128, 256]
Learning rate:
Rich regime: E {0.5, 1.0) (as learning rate in rich regime needs to scale up with the hidden dimension)
Lazy regime: E {0.01, 0.05]
Weight decay: {0, 1e=4)
The final numbers reported averaged across independent runs with the selected hyperparame- ters.
Evaluation For Imagenette, b-maggenette and MNIST-CIFAR, we report the standard test aceu- racy in all the experiments. For waterbirds, we report train-adjusted test accuracy, reported in Sagawa et al. (2020a). Precisely, accuracy for each group present in the test data is individually calculated and then weighed by the proportion of the corresponding group in the train dataset.
B.2 ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present few additional experimental results.
Accuracy of fproj In Table we show the test accuracy of fpmg- As can be seen, even after projecting out the principal components used by f. Sprij attains significantly high accuracy. Note that, in these experiments, model 1 was kept fixed and the accuracy of fproj is averaged across runs.
Results on Imagenet We trained 1-hidden layer FCN (with 2000 hidden neurons) Imagenet dataset, using rich regime initialization, with learning rate selected from the set - (5, 10} (as learning rate in rich regime needs scale up with hidden dimension). The evolution of effective rank of the
20

---

Under review as a conference paper at ICLR 2023
Table Trained accuracy of froj in rich regime
Dataset
Acc(f)
Acc proj
b- b-Imagenette
93.35
91.35 + 0.32
Imagenette
79.67
71.93 0.12
Waterbirds
90.29
89.92 + 0.08
MNIST-CIFAR
99.69
98.95 0.02
Imagenet
72.02
69.63 + 0.08
1250
Imagend:
1000
Effective rank
500
10
20
30
40
Epochs
Figure 6: Evolution of effective rank of first layer weight matrix (dimension 2048 x 2000) for Imagenet dataset in rich regime.
first layer weight matrix is shown in Figure As be seen, the weight matrix becomes sufficiently low rank as the training progresses, Also, P and P. randomized accuracy (RA) and Logit change (LC) are shown in Table 5 As be seen, the model's prediction almost determined by the projection along the top 150 singular vectors of the weight matrix.
We also train model2 on representations obtained by projecting out the top 150 singular vectors of the weight matrix of model 1. In Table (6) we show the mistake diversity (Mist-Div) and class- conditioned logit correlation (CC-LogitCorr) between model and model projected out model has higher diversity and lower correlation as compared to an independently trained model. In Table we also show that the second model achieves comparable accuracy to model 1.
can be seen, the
Singular value decay In Figure 7 we provide the singular value decay of the weight matrix for the first model trained in rich regime. As be seen, the top few singular values capture most of the Frobenius norm of the matrix.
MNIST-CIFAR In Figure we show that an ensemble of and frooj has better gaussian robust- ness than an ensemble of and find on MNIST-CIFAR dataset.
Quantitative measurement non-lincarity of decision boundary this section, we report quantitative measure of non-linenn con-liicarity of the decision boundary along the top singular vectors for and froj. Basically, we fit linear classifier to the decision boundary and report its accuracy. As shown in Table the test accuracy obtained by the linear classifier for fpeij is less than f.
Variation of LD-SB with depth In Figure 9 and 10 we show the evolution of effective rank of weight matrices for depth-2 and ReLU networks. As can be seen, the rank still decreases with training. however the effect is less pronounced for the initial layers. Note that the initialization used these runs was the feature learning initialization as proposed Yang Hu (202T).
21

---

Under review as a conference paper at ICLR 2023
Table 5: Demonstration of LD-SB in the rich regime: This table presents P1 and P randomized accuracies (RA) as well as logit changes (LC) on the Imagenet dataset. These results confirm that projection of input x onto the subspace spanned by P essentially determines the model's prediction on I. f (resp. 4) indicates that LD-SB implies a large (resp. small) value.
Dataset
rank (P)
Ace(J)
PI-RA (t) P-RA (J)
PIC
P-LC
1901.27 =
199958 +
Imagenet
150
72.15+0.09 68.3540.04
0.25 + 0.0
0.2
44.51
Table 6: Mistake diversity and class conditioned logit correlation of models trained independently (Mist-Div (f, fina) and CC-LogitCor (f. fina) resp.) vs trained sequentially after projecting out im- portant features of the first model (Mist-Div (f. fyeo)) and CC-LogicCor (f+ fpoo) resp.). The results demonstrate that f and spos are more diverse compared to f and fima-
Mist-Div (f, fima) (1)
Mist-Div (puj) (t)
CC-LogitCor fma) (4)
CC-LogitCorr
Dataset
(poj) (L)
Imagenet
6.97 0.06
12.31 + 0.16
99.5 = 0.0
92.52 = 0.01
NNIST CHRAR
Fraction of frobenius nom
0.6
Waterbiids
Imagenetie
P: h-imagenethe
0.4
0.2
5
10 15
Singular value index
Figure 7: Fraction of Frobenius norm captured by the top 8th singular value i.e., of ///Ei=1 03 vs of the first layer weight matrix trained in rich regime for various datasets.
1.00
mado ensomble pri
ensemble-nn Ind
0.95
accuracy
0.90
0.85
0.80
0.75
2
3
std_ dev
Figure& 8: Variation of test accuracy with the standard deviation of Gaussian noise added to the pretrained representations of MNIST-CIFAR dataset. Model 1 is kept fixed, and values for both the ensembles are averaged across runs.
22

---

Under review as a conference paper at ICLR 2023
Table 7: Quantitative measurement of non-linearity of decision boundary accuracy fitted linear classifier to the decision boundary
Dataset
Linear-Classifer-Acco(]
Linear-Classifee--Acco/om
b- Imagenette 96.12
95.28 + 0.2
Waterbirds
97.28
93.24 0.24
MESTIRAA Miteniess b-maggestt
Eftective rank
Effective rank
40
40
20
20
Epochs
Epochs
Layer
(b) Layer 2
Figure 9: Evolution of effective rank the weight matrices for a depth-2 ReLU network on Resnet- 50 pretrained representations of the dataset
1.0
MAB CAR
130
MUTOOO ouanda
1.00
MESTORO
203
6.00
250
5G2 Epochs
110
1000
200
450
512
Epachs
Epocihs
(a) Layer I
(b) Layer 2
(c) layer
Figure 10: Evolution of effective rank of the weight matrices for depth-3 ReLU network Resnet- 50 pretrained representations of the dataset
23

---

Under review as a conference paper at ICLR 2023
C EXTENDED RELATED WORKS
In this section, we provide an extensive literature survey of various topics that the paper is based on.
Low rank Simplicity Bias in Linear Networks Multiple works have established low rank sim- plicity bias for gradient descent on linear networks, both for squared loss as well as cross-entropy loss. For squared loss, Gunasekar et al. (017) conjectured that the network is biased towards find- ing minimum nuclear norm solutions for two-layer linear networks. [Arora et al. (2019) refuted the conjecture and instead argued that the network is biased towards finding low rank solutions. Razin & Cohen (2200) provided empirical support to the low rank conjecture, by providing synthetic exam- ples where the network drives nuclear norm to infinity, but minimizes the rank of the effective linear mapping, Li et al. (202I) established that for small enough initialization, gradient flow on linear networks follows greedy low-rank learning trajectory. For binary classification on linearly separable data, Ji & Telgarsky (2019) showed that the weight matrices of a linear network eventually become rank- I as training progresses.
Low rank Simplicity Bias in Non-Linear Networks For non-linear networks, the work related to low--wram wowk simplicity bias is rather sparse. Two of the most notable works are Hre et (202T) and Galanti & Poggio (0022) Huh et al. (202T) empirically established that the rank of the embeddings learnt by a neural network with ReLU activations goes down as training progresses. Galanti & Poggio 2022) provided an intuition behind the relation between the rank of the weight matrices and various hyperparameter such as batch size, weight decay etc. In contrast to these works, for layer nets, we theoretically and empirically establish that the network depends on an extremely low dimensional projection of the input, and this bias can be utilized to develop a robust classifier.
Relation to OOD Many recent works in OOD detection (Cook et al., 2020 Zaeemzadeh et al.,
T explicitly create low-rank embeddings so that it is easier to discriminate them for an OOD
Other works also implicitly rely on the low-rank nature of the embeddings. Ndiiur et al. use PCA on the learnt features, and only model the likelihood along the small subspace spanned by the top few directions. Wang et al. (2022) utilise the low rank nature of the embeddings to estimate the perpendicular projection of given data point to this low rank subspace and combine it with logit information to detect OOD datapoints. While there have been works implicitly utilizing the low rank property of embeddings, we note that our paper (i) demonstrates low rank property of
point.
2020
the weights, rather than that of embeddings, and (ii) shows that it is consequence of SB.
Other Simplicity Bias There have been many works exploring the nature of simplicity bias in neu- ral networks, both empirically and theoretically. Kalimeris et al. (2019) empirically demonstrated that SGD on neural networks gradually learns functions of increasing complexity. [Rahaman et al.) (0118) empirically demonstrated that neural networks tend to learn lower frequency functions first. Ronen et al. (2019) theoretically established that in NTK regime, the convergence rate depends on the eigenvalues of the kernel spectrum. Hacohen et al. 2020) showed that neural networks always learn train and test examples almost in the same order, irrespective of the architecture. Pezeshki et al.) (202T) proposes that gradient starvation at the beginning of training is potential reason for SB in the lazy/NTK regime but the conditions are hard to interpret. In contrast, our results are shown for any dataset in the IFM model in the rich regime of training. Lyu al.] (2021) consider anti- symmetric datasets and show that single hidden layer input homogeneous networks (i.e., without bias parameters) converge to linear classifiers. However, such networks have strietly weaker ex- pressive power compared to those with bias parameters. Hacohen & Weinshall (202) showed that for deep linear networks, in NTK regime, they learn the higher principal components of the input data first. Most of the previous works used simplicity bias as reason behind better generalization of neural nets. However, (Shah et al. 2020) showed that extreme simplicity bias could also lead to worse OOD performance.
Learning diverse classifiers: There have been several works that attempt to learn diverse classifiers Most works try to learn such models by ensuring that the input gradients of these models do not align (Ross& & Doshi- Velez, 2018 Teney et al. 2022). Xu et al. (2022) proposes a way to learn diverse/orthogonal classifiers under the assumption that complete classifier, that uses all features is available, and demonstrates its utility for various downstream tasks such as style transfer. Lee
et al. (2022) learns diverse classifiers by enforcing diversity unlabeled target data.
24

---

Under review as a conference paper at ICLR 2023
Spurious correlations:
There has been large body of work which identifies the reasons for spu- rious correlations in NNs (Sagawa et al. (020b) as well as proposing algorithmic fixes in different
settings (Liu et al. 2021 Chen et al. 2020).
There is also a large body of work understanding the implicit bias of gradient descent dynamics. Most of these works for standard linear (Ji & Telgarsky (019) or deep linear networks (Soudry et al., 2018] Gunasekar et al., 2018). For nonlinear neural networks, one of the well-known results is for the case of 1-hidden layer neural networks with homogeneous activation functions (Chizat & Bach 2020), which we crucially use in our proofs.
Implicit bias of gradient descent:
descent:
25


Reviews:
[{"review_id": "rBO6fMG-ULg", "reviewer": "Reviewer_Hiqj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper rigorously defined as well as thoroughly established simplicity bias for one hidden layer neural networks, as a function of a low dimensional projection of the inputs. The authors theoretically proved that the network primarily depends on only the linearly separable subspace when the data is linearly separable, and emprically showed that the models trained on real datasets depended on a low dimensional projection of the inputs. Moreover, they proposed an ensemble approach to combine the orthogonal direction learning.", "summary_of_the_review": "This paper rigorously defined as well as thoroughly established simplicity bias for one hidden layer neural networks, as a function of a low dimensional projection of the inputs. The low-rank idea is NOT new for defining simplicity bias, therefore, the paper needs to be polished to distinguish from previous works.", "strength_and_weaknesses": "Strengths:\n\nS1. The paper is very well-written and I enjoyed reading it.\nS2. Both theory and experiments look great.\n\nWeaknesses:\n\nS1. The low-rank bias is NOT new. Please refer to [R1] and [R2] as well as related works in the two papers.\n\n[R1] The Low-Rank Simplicity Bias in Deep Networks. https://minyoungg.github.io/overparam/resources/overparam-v2.pdf\n[R2] Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networks. JMLR 2022\n\nS2. A better overview of previous related work is suggested. E.g., as mentioned in Shah et al. (2020), simplicity bias is related to Out-of-Distribution (OOD) detection. In OOD detection, the low-rank bias and ensemble learning along the orthogonal projection has also been extensively studied.\n\n[R3] Outlier detection through null space analysis of neural networks. arXiv:2007.01263, 2020.\n\n[R4] Out-of-distribution detection with subspace techniques and probabilistic modeling of features. arXiv:2012.04250, 2020.\n\n[R5] Out-of-distribution detection using union of 1-dimensional subspaces. CVPR 2021.\n\n[R6] ViM: Out-Of-Distribution with Virtual-logit Matching, CVPR 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gyyLhq9cZ1", "reviewer": "Reviewer_3SUe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes the phenomenon of \"simplicity bias\" in neural networks. Simplicity bias is defined here as the tendency of models to learn \"simple\" features (low-rank linear projections of the input space) that enable the model to solve the task even when different non-linear projections could also be used to solve the task with high accuracy. It is argued in the paper that this can result in \"shortcut learning\" where the model can learn simple features that correspond to \"spurious correlations\" and are not stable w.r.t. distribution shift.\n\nThe paper focuses on 1-hidden layer ReLU binary classifiers.\n\nA theoretical analysis is carried out in the infinite width limit, with two different initialization regimes: a \"lazy\" regime that corresponds to the NTK framework, and a \"rich\" regime with enables training of the hidden layer and is therefore more representative of practical training regimes. Under the assumption that the data comes from a distribution where the input features are independent conditional on the output, they show that the model will project the input in a rank-1 linear manifold if it suffices to solve the problem, even when other non-linear solutions exist.\n\nAn empirical analysis is done on classification heads on top of frozen pre-trained image classification models for various image classification tasks, showing results consistent with the hypothesis.\nThe paper also briefly discusses how to use this simplicity bias property to train an ensemble of diverse models by sequentially projecting the data on the orthogonal space of the linear projections that the model tend to learn.", "summary_of_the_review": "Good paper, I would give a higher score if my concerns about reproducibility were addressed.\n\nEDIT: I am increasing my score.", "strength_and_weaknesses": "Strengths:\n- The paper is well-written\n- The result is interesting, important, and as far as I know, novel\n \nWeakness:\n- No discussion of why is \"simplicity bias\" supposedly leads to poor OOD generalization. Wouldn't a more complicated decision boundary be even less robust than a quasi-linear one?\n- How are the projection matrices computed in the experimental section? There is very little discussion and no pseudo-code. Is the weight matrix used in the rich regime? How do you learn the projection matrix in the lazy regime? Are there any constraints?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ay5ynMv7vOw", "reviewer": "Reviewer_EewH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper elaborates on a specific simplicity bias called LD-SB. Theoretical proofs of LD-SB are provided for 1-hidden layer neural networks on the IFM distribution under different situations. Experiments on four real datasets demonstrate that LD-SB is of practical importance. The authors further propose a new robust ensemble method based on the understanding of LD-SB.", "summary_of_the_review": "The proposed LD-SB is interesting and backed by both theoretical and empirical justifications. However, according to my understanding, the experiments on real datasets do not fully establish the LD-SB (see point 1 in weaknesses). Hence, I give a score of 5, e.g., marginally below the threshold.\n\n------------------------20221122-----------------------\n\nas the authors had addressed my concern about establishing LD-SB in real datasets, I decided to raise my rating from 5 to 6.", "strength_and_weaknesses": "Strength:\n\n- The concept of LD-SB is clearly and rigorously defined, along with a good presentation of related background knowledge.\n- Both theoretical proof (for 1-hidden layer net and IFM) and empirical justification (for four real datasets) are provided to demonstrate the LD-SB.\n- The paper further proposed a potentially useful ensemble method, OrthoP, and show this method is more robust under Gaussian noise.\n\n\nWeaknesses:\n\n- In the experiments on real datasets, the current four metrics do not reflect the third point of the definition for LD-SB. Hence, the current experiments do not fully “show LD-SB on real datasets.”  As the projection operator is ready, why not train an independent model to show that a model using the remaining features in the orthogonal sub-space can achieve high accuracy? \n- Despite the authors' arguments that fine-tuning on 1-hidden layer network is effective in many practical cases, I do think for real datasets, experiments could involve more sophisticated network architectures. It is interesting to see whether the LD-SB exists in more complex and practical settings.\n- In subsection 4.4.3 for Lazy regime, it said that “able to find a low dimensional projection matrix P which satisfies the conclusions in Theorem 4.2. This statement seems improper since Theorem 4.2 applies to a specific dataset rather than those real datasets.\n- Can you discuss why the low dimension character is also present when applying to real datasets rather than IFM? Is there any other mechanism involved?\n- Missing details of experiments referred to in Fig.5. For example, how is the output of the ensemble model computed? Average over both member models?\n- In Fig.4, it is claimed that the decision boundary of the second model is more non-linear. It would be better to provide a quantitative measure of the non-linearity.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "97iCRrLphc", "reviewer": "Reviewer_iV59", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the simplicity bias (SB) for 1-hidden layer neural networks (NNs). The paper formulates SB as a concept (the NN learns a simpler feature than those available and informative for classification) and then prove that this might happen in 1-hidden layer NNs. Lastly, the paper proposes a heuristic way to train an ensemble of models to avoid SB. ", "summary_of_the_review": "At the moment, there are certain unclear things to me, e.g. whether the simplicity bias is the same as shortcut learning, or why the results cannot be extended to deep networks or the finite width regime. In addition, I find the experiments are rather weak. Having said that, it is possible that this paper has value for the community if the paper is strengthened during the rebuttal period. ", "strength_and_weaknesses": " * [+] Understanding of neural networks and their (learning) properties is an important task. \n\n * [+] The definition of simplicity bias in Definition 1.1 is clear to me.\n\n * [-] The results are currently limited, e.g. only one-hidden layer network. Despite the authors explicitly identifying this, I am still not convinced that one-hidden layer nets are a practical consideration. \n\n * [-] The core theorem 4.1 is only applicable for an infinite width network. \n\n * [-] The paper mentions that we do not have a clear understanding of the dynamics of deeper networks (page 2), however this is not true. The papers of “Training Integrable Parameterizations of Deep Neural Networks in the Infinite-Width Limit” or “Gradient Descent Finds Global Minima of Deep Neural Networks” already study the dynamics of deep networks (or their proof applies for deep nets). I would recommend revising this paragraph to avoid any confusion from the readers. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject