On the Foundations of Shortcut Learning
Authors: Katherine Hermann
Hossein Mobahi
Thomas FEL
Michael Curtis Mozer
Venue: ICLR 2024 spotlight (2024)
Keywords: shortcut learning
spurious correlations
architectural inductive bias
Primary area: visualization or interpretation of learned representations

Abstract: Deep-learning models can extract a rich assortment of features from data. Which features a model uses depends not only on *predictivity*---how reliably a feature indicates training-set labels---but also on *availability*---how easily the feature can be extracted from inputs. The literature on shortcut learning has noted examples in which models privilege one feature over another, for example texture over shape and image backgrounds over foreground objects. Here, we test hypotheses about which input properties are more available to a model, and systematically study how predictivity and availability interact to shape models' feature use. We construct a minimal, explicit generative framework for synthesizing classification datasets with two latent features that vary in predictivity and in factors we hypothesize to relate to availability, and we quantify a model's shortcut bias---its over-reliance on the shortcut (more available, less predictive) feature at the expense of the core (less available, more predictive) feature. We find that linear models are relatively unbiased, but introducing a single hidden layer with ReLU or Tanh units yields a bias. Our empirical findings are consistent with a theoretical account based on Neural Tangent Kernels. Finally, we study how models used in practice trade off predictivity and availability in naturalistic datasets, discovering availability manipulations which increase models' degree of shortcut bias. Taken together, these findings suggest that the propensity to learn shortcut features is a fundamental characteristic of deep nonlinear architectures warranting systematic study given its role in shaping how models solve tasks.

Published as a conference paper at ICLR 2024
ON THE FOUNDATIONS OF SHORTCUT LEARNING
Katherine L. Hermann', Hossein Mobahi", Thomas Fel', and Michael C. Mozer!
Google { (DDeepMind, "Research), Mountain (hermannk, hmobahi, thomasfel, memozer ( @google.com
Mountaiin View, CA, USA
ABSTRACT
Deep-learning models can extract a rich assortment of features from data. Which features a model uses depends not only on predictivity- reliably a feature indicates training-set labels-but also on availability- how easily the feature can be extracted from inputs. The literature on shortcut learning has noted examples which models privilege one feature over another, for example texture over shape and image backgrounds over foreground objects. Here, we test hypotheses about which input properties are more available a model, and systematically study how predictivity and availability interact to shape models' feature use. We construct a minimal, explicit generative framework for synthesizing classification datasets with two latent features that vary in predictivity and in factors we hypothesize to relate availability, and we quantify model's s shortcut bias its over-reliance on the shortcut (more available, less predictive) feature at the expense of the core (less available, more predictive) feature. We find that linear models are relatively unbiased, but introducing single hidden layer with ReLU or Tanh units yields a bias. Our empirical findings are consistent with a theoretical account based on Neural Tangent Kernels. Finally, study how models used in practice trade off predictivity and availability in naturalistic datasets, discovering availability manipulations which increase models' degree of shortcut bias. Taken together, these findings suggest that the propensity to learn shortcut features is fundamental characteristic of deep nonlinear architectures warranting systematic study given its role in shaping how models solve tasks.
INTRODUCTION
Natural data domains provide rich, high-dimensional input from which deep-learning models can extract a variety of candidate features. During training, models determine which features to rely Following training, the chosen features determine how models generalize. A challenge for machine learning arises when models come to rely on spurious or shortcul features instead of the core or defining features of a domain (Arjovsky al., 2019; McCoy et al., 2019; Geirhos et al., 2020; Singla & Feizi, 2022). Shortcut "cheat" features, which are correlated with core "true" features in the training set, obtain good performance the training set as well as an iid test set, but poor generalization on out-o--distiibution inputs. For instance, ImageNet-trainee CNNs classify primarily according an object's 's texture (Baker et al., 2018; Geirhos et al., 201 Sa; Hermann et al., 2020), whereas people define and classify solid objects by shape (e.g., Landau et al., 1988). Focusing on texture leads to reliable classification on many images but might result in misclassification of, say, a hairless cat, which has wrinkly skin more like that of an elephant. The terms "spurious" and "shorteut" are largely synonymous in the literature, although the former often refers features that arise unintentionally in a poorly constructed dataset, and the latter to features easily latched onto by model. In addition to a preference for texture over shape, other common shortcut features include propensity to classify based on image backgrounds rather than foreground objects (Beery et al., 2018; Sagawa et al., 2020a; Xiao et al., 2020; Moayeri et al., 2022), or based on individual diagnostic pixels rather than higher-order image content (Malhotra et al., 2020).
The literature examining feature use has often focused on predictivity how well a feature indicates the target output. Anomalies have been identified in which networks come to rely systematically on one feature over another when the features are equally predictive, or even when the preferred feature has lower predictivity than the non-prefereed feature (Beery et al., 2018; Pérez al., 2018; Tachet

---

Published as a conference paper at ICLR 2024
et al., 2018; Arjovsky al., 2019; McCoy et al., 2019; Hermann & Lampinen, 2020; Shah et al., 2020; Nagarajan et al., 2020; Pezeshki et al., 2021; Fel et al., 2023). Although we lack a general understanding of the cause of such preference anomalies, several specific cases have been identified. For example, features that are linearly related to classification labels are preferred by models over features that require nonlinear transforms (Hermann & Lampinen, 2020; Shah et al., 2020). Another factor leading to anomalous feature preferences is the redundancy of representation, c.g., the size of the pixel footprint in an image (Sagawa et al., 2020a; Wolff & Wolff, 2022; Tartaglini al., 2022).
Because predictivity alone is insufficient to explain feature reliance, here we explicitly introduce the notion of availability to refer to the factors that influence the likelihood that model will use a feature more so than a purely statistical account would predict. A more- availablil feature is easier for the model to extract and leverage. Past research has systemtically manipulated predictivity; in the present work, we systematically manipulate both predictivity and availability to better understand their interaction and to characterize conditions giving rise to shorteut learning. Our contributions are:
We define quantitative measures of predictivity and availability using a generative framework that allows us to synthesize classification datasets with latent features having specified predictivity and availability. We introduce two notions of availability relating to singular values and nonlinearity of the data generating process, and we quantify shorteut bias in terms of how learned classifier deviates from an optimal classifier in its feature reliance.
We perform parametric studies of latent-feature predictivity and availability, and examine the sensitivity of different model architectures to shortcut bias, finding that it is greater for nonlinear models than linear models, and that model depth amplifies bias.
We present a theoretical account based on Neural Tangent Kernels (Jacot et al., 2018) which indicates that shortcut bias is an inevitable consequence of nonlinear architectures.
We show that vision architectures used in practice can be sensitive to non-core features beyond their predictive value, and show a set of availability manipulations of naturalistic images which push around models' feature reliance.
2 RELATED WORK
The propensity of models to learn spurious (Arjovsky et al., 2019) or shortcut (Geirhos et al., 2020) features arises in variety of domains (Heuer al., 2016; Gururangan et al., 2018; McCoy et al., 2019; Sagawa et al., 2020a) and is of interest from both scientific and practical perspective. Existing work has sought to understand the extent to which this tendency derives from the statistics of the training data versus from model inductive bias (Neyshabur et al., 2014; Tachet et al., 2018; Pérez et al., 2018; Rahaman et al., 2019; Arora et al., 2019; Geirhos et al., 2020; Sagawa et al., 2020b;a; Pezeshki et al., 2021; Nagarajan et al., 2021), for example bias to learn simple functions (Tachet et al., 2018; Pérez et al., 2018; Rahaman et al., 2019; Arora et al., 2019). Hermann & Lampinen (2020) found that models preferentially represent one of a pair of equally predictive image features, typically whichever feature had been most linearly decodable from the model at initialization. They also identified cases where models relied mooe-prreiittve feature that had a nonlinear relationship to labels. Together, these findings suggest that predictivity alone is not the only factor that determines model representations and behavior. A theoretical account by Pezeshki et al. (2021) studied a situation in supervised learning in which minimizing cross-entropy loss captures only a subset of predictive features, while other relevant features go unnoticed. They introduced formal notion of strength that determines which features are likely to dominate a solution. Their notion of strength confounds predictivity and availability, the two concepts which we aim to disengangle in the present work.
less-predictive feature that had linear relationship task labels over
Work in the vision domain has studied which features vision models rely on when trained on natural image datasets. For example, ImageNet models prefer to classify based on texture rather than shape (Baker et al., 2018; Geirhos et al., 201 Sa; Hermann et al., 2020) and local rather than global image content (Baker et al., 2020; Malhotra et al., 2020), marking difference from how people classify images (Landau et al., 1988; Baker et al., 2018). Other studies have found that image backgrounds play an outsize role in driving model predictions (Beery et al., 2018; Sagawa et al., 2020a; Xiao et al., 2020). Two studies manipulated the quantity of a feature in an image to test how this changed model behavior. Tartaglini et al. (2022) probed pretrained models with images containing shape consistent
2

---

Published as a conference paper at ICLR 2024
A
= 0.6, 0.9
0.85, 0.9
B
Pwerib lead nort)
ineriii (nci heart
N
St,
0,W:
2s
Z5
ft
Figure 1: Synthetic data. A: Two datasets differing in the predictivity of 2s- B: Schematic of the embedding procedure manipulating availability via the mapping from z to æ. Dashed boxes are optional.
with one class label and a texture consistent with another, where the texture was present throughout the image, including in the background. They varied the opacity of the background, and the the texture became less salient, models increasingly classified by shape. Wolff & Wolff (2022) found that when images had objects with opposing labels, models preferred to classify by the object with the larger pixel footprint, an idea we return to in Section 5. The development of methods for reducing bias for particular features over others is active area (e.g. Arjovsky et al., 2019; Geirhos et al., 2018a; Ra; Hermann et al., 2020; Robinson et al., 2021; Minderer et al., 2020; Sagawa et al., 2020b; Ryali al., 2021; Kirichenko al., 2022; Teney et al., 2022; Tiwari & Shenoy, 2023; Ahmadian & Lindsten, 2023; Pezeshki et al., 2021; Puli et al., 2023; LaBonte et al., 2023), important for improving generalization and addressing fairness concerns (Zhao et al., 2017; Buolamwini & Gebru, 2018).
GENERATIVE PROCEDURE FOR SYNTHETIC DATASETS
To systematically explore the role of predictivity and availability, construct synthetic datasets from a generative procedure that maps a pair of latent features, z = (Za; ze). to an input vector x E IRd and class label y E {-1, +1}. The subscripts and denote the latent dimensions that will be treated as the potential shoricut and core feature, respectively. The procedure draws from a multivariate Gaussian conditioned class,
( y Uis [ylec]
zIy~ N
Usc 1
[se
with la < 1. Through symmetry, the optimal decision boundary for latent feature i is 2, = 0. allowing us define the feature predictivity P Pr(y = sign( For Gaussian likelihoods, this predictivity is achieved by setting μ44 = V2 erf (2ps - 1). Figure 1A shows sample latent-feature distributions for Pe = 0.9 and two levels of ps.
Availability manipulations. Given a latent vector z, we manipulate the hypothesized availability (hereafter, simply availability) of each feature independently through embedding procedure, sketched in Figure 1B, that yields an input x E IRd We posit two factors that influence availability of feature i, its amplification ag and its nesting 7/- Amplification a; a scaling factor on an embedding, e = w, where w; E Rd is a feature-specific random unit vector. Amplification includes manipulations of redundancy (replicating feature) and magnitude (increasing feature's dynamic range). Nesting 77 is a factor that determines ease of recovery of a latent feature from an embedding. We assume a nonlinear, rank-preserving transform, e fr, (e). where fr, is a fully connected, random net with 7), E N tanh layers in cascade. For f) = O, feature i remains in explicit form, e/ = e;; for v): > 0, the feature is recoverable through an inversion of increasing complexity with ?)-- To complete the data generative process, we combine embeddings by summation:
e', + e'y
Assessing shorteut bias. Given a synthetic dataset and a model trained on this dataset, we assess the model's reliance on the shortcut feature, i.e., the extent to which the model uses this feature as basis for classification decisions. When & and 2g are correlated (a.c > 0), some degree of reliance on the shorteut feature is Bayes optimal (see orange dashed line in Figure 6B). Consequently, need to assess reliance relative to that of an optimal classifier. We perform this assessment in latent space, where the Bayes optimal classifier can be found by linear discriminant analysis (LDA) (see
3

---

Published as a conference paper at ICLR 2024
1.00
optimal classifier
model
0.7 0.85
0.50
0.25
0.00
-0.25
0055
-0.75
2
16 128
-1.00
Figure 2: Deep nonlinear models can prefer a less-predictive but more-available feature more-predictive but less-available one. The color of each cell in the heatmap indicates the mean bias of model as a function the availability and predictivity of the shortcut feature, 2g. The inset shows in faint coloring the decision surface for an optimal Bayes classifier (LDA) and trained model. Overlaid points are subset of training instances. The model obtains a shortcut bias of 0.53.
Figure 2 inset). The shortcut bias is the reliance of a model on the shortcut feature over that of the optimal, L.DA:
bias reliancemodel reliancespiimal
Appendix C.1 describes and justifies our reliance measure in detail. For given model M, whether trained net or LDA, we probe over the latent space and determine for each probe z the binary classification decision, (M((2) (see Figure 2). Shortcut reliance is the difference between the model's alignment (covariance) with decision boundaries based only on the shortcut and core features:
2 (
reliance,
I.M. sign(ze,) (Z+,)
Σ
j.M sign sign(Ze.)
where n is the number of probe items. Both the reliance score and shorteut bias are in [-1, +1].
4 EXPERIMENTS MANIPULATINE FEATURE PREDICTIVITY AND AVAILABILITY
Methodology. Using the procedure described Section 3, we conduct controlled experiments examining how feature availability and predictivity and model architecture affect the shorteut bias. We sample class-balanced datasets with 3200 train instances, 1000 validation instances, and 900 probe (evaluation) instances that uniformly cover the (=s, zc) space by taking Cartesian product of 30 2, evenly spaced 3ps, +3 and 30 z. evenly spaced [-3/ec, - 3/pe]- In all simulations, we set 100, T)e 7/s- 0. Pe 0.9, asc 0.6. We manipulate shortcut-feature predictivity with the constraint that it is lower than core-feature predictivity but still predictive, i.e., 0.5 < Ps < P. 0.9. Because only the relative amplification of features matters, we vary the ratio ag//a.. with the shorteut feature more available, i.e., a./a. 2 1. We report the means across 10 runs (Appendix C.3).
Models prefer the more available feature, even when it is less predictive. We first test whether models prefer 2, when it is more available than 2c, including when it is less predictive, while holding model architecture fixed (see Appendix C.3). Figure/ 2 shows that when predictivity of the two features is matched (p.- Ps = 0.9), models prefer the more-available feature. And given sufficiently high availability, models can prefer less-prediction but more-available shortcut feature to the more-predictive but less-available core feature. Availability can override predictivity
Model depth increases shorteut bias. In the previous experiment, we used a fixed model architec- ture. Here, we investigate how model depth and width influence shortcut bias when trained with as, a./a, = 64 and P. = 0.85, previously shown to induce a bias (see Figure 2, gray square). As shown in Figure 3A, find that bias increases function of model depth when dataset is held fixed.
Model nonlinearity increases shortcut bias. To understand the role of the hidden-layer activation function, compare models with linear, ReLU, and Tanh activations while holding weight initializa- tion and data fixed. As indicated in Figure 3B, nonlinear activation functions induce larger shortcut bias than their linear counterpart.
4

---

Published as a conference paper at ICLR 2024
A
B
relu
0.7
933
300
675
0.00
933
0.75
013
140g
3.80
wndti
Figure 3: A: Model depth increases shorteut bias. The color of each cell indicates the mean bias of an MLP with ReLU hidden activation-funcii functions, for various model widths and depths, trained on data with a shortcut feature that is more available (a./a, = 64) but less predictive (p= = 0.85) than the core feature. Model nonlinearity increases shorteut bias. B: Shorteut bias for three hidden activation functions for a deep MLP with width 128 and depth 2, trained on datasets where predictivity matched (p, = Pe 0.9), but shortcut availability is higher (a. ae 32). A shorteut bias is more pronounced when the model contains a nonlinear activation function. C: Shortcut bias for MLPs with single hidden layer and hidden activation function that is either linear (left) or ReLU (right), for various shortcut feature availabilities (as/ aa)) and predictivities (px). See B. for Tanh.
A
0.5
B
0.5
Pc
0.4
0.4
0.3
0.3
bias
0.2
0.1
4x
0.1
0.0
0.0
-0.1
1 2
0.7
0.8
ag/ac
P.
Figure ResNet-18 prefers shortcut feature when availability is instantiated as the pixel footprint of an object (feature), even when that feature is less predictive. A: Sample images. B: Shortcut bias increases as a function of relative availability of the shortcut feature when features are equally predictive (p.- Pr 0.9), consistent with Wolff & Wolff (2022). C: Even when the shortcut feature less predictive, models have a shortcut bias due to availability, when a,/a. = 4.
Feature nesting increases shortcut bias. The synthetic experiments reported above all manipulate availability with the amplitude ratio a./a., We also conducted experiments manipulating a second factor we expected would affect availability (Hermann & Lampinen, 2020; Shah et al., 2020), the relative nesting of representations, i.e., 7/c 17s 2 1. We report these experiments in Appendix C.3.
5 AVAILABILITY MANIPULATIONS WITH SYNTHETIC IMAGES
What if we instantiate the same latent feature space studied in the previous section in images? We form shorteut features that are analogous texture or image background features previously noted to preferentially drive the behavior of vision models (e.g. Geirhos et al., 201 8a; Baker et al., 2018; Hermann et al., 2020; Beery et al., 2018; Sagawa et al., 2020a; Xiao et al., 2020). Building on the work of Wolff' & Wolff (2022), we hypothesize that these features are more available because they have a large footprint in an image, and hence, by our notions of availability, a large a.
Methods. We instantiate latent vector from our data data-generation procedure feature becomes an object (z, a circle, že a square) whose color is determined by the respective feature value. Following Wolff & Wolff (2022), manipulate the availability of each feature in terms of its size, or pixel footprint. We randomly position the circle and square entirely within the image, avoiding overlap, yielding 224 224 pixel image (Figure 4A, Appendix C.4).
image. Each
Results. Figure 4B presents the shortcut bias in ResNet18 as a function of thortcut-featuur availability (footprint) when the two features equally predictive (p.- Pc 0.9). In Figure 4C, the availability
5

---

Kernel Spectrum. Consider two-layer network with the first layer having possibly nonlin- ear activation function. When the width of this model gets large, learning of this model can be approximated by kernel regression problem, with a given kernel function -(.,-) that depends on the architecture and the activation function. Given a distribution over the input data p(æ), we Published as a conference paper at ICLR 2024
ratio is fixed at as/ac = 4, and the shorteut bias is assessed function of P.. 8 is biased toward the more available shortcut feature even when it is less predictive than the core feature. Together, these results suggest that a simple characteristic of image contents-the pixel footprint of an object-can bias models' output behavior, and may therefore explain why models can fail to leverage typicalty-smaller foreground objects in favor of image backgrounds (Section 7).
THEORETICAL ACCOUNT
In our empirical investigations, we quantified the extent to which trained model deviates from statistically optimal classifier in its reliance on the more-available feature using a measure which considered the basis for probe-instance classifications. Here, we use an alternative approach of studying the sensitivity of a Neural Tangent Kernel (NTK) (Jacot et al., 2018) to the availability of a feature. The resulting form presents crisp perspective of how predictivity and availability interact. In particular, we prove that availability bias is absent in linear networks but present in ReLU networks. The proof for the theorems of this section is given in the Supplementary Materials.
For tractability of the analysis, we consider few simplifying assumptions. We focus on 2-layer fully connected architectures for which the kernel of the ReLU networks admits a simple closed form. In addition, to be able to compute integrations that arise in the analysis, we resort to asymptotic approximation which assumes covariance matrix is small. Specifically, we represent the covariance matrix as s [1, 012: ; 012 1. where the scale parameter > is considered to be small. Finally, in order to handle the analysis for a ReLU kernel, we will use a polynomial approximation.
define a (linear) kernel operator one that acts on function produce another function g as in g(x) = .g. k(x, x) (z) p(z) dz. This allows us to define an eigenfunction o of the kernel operator as one that satisfies,
xo(x) Sq. k( 2) ((2) p(z) dz. The value of X will be the eigenvalue of that eigenfunction when the eigenfunction ş is normalized as Jan o² (az) p(x:) da = 1.
(1)
Form of p(=). Recall that in our generative dataset framework, we have pair of latent features &e and 2g that are embedded into a high dimensional space via Txx1 = Og + a.= = U Uxx222x222x11 With this expression, A. and therefore A is diagonal matrix with positive diagonal entries, and the columns of U are (approximately) orthonormal. Henceforth, we also refer to features with indices 1 and 2 instead of and c. An implication of orthonormal columns on U is that the dot product of any two input vectors and æl will be independent of U, i.e., (x, ) = (Az, Az Consequently, we can compute dot products in the original 2-dimensional space instead of in the d-dimensional embedding space. On the other hand, we will later see that the kernel function k(æ1, (x2)),)) of the two cases we study here (ReLU and linear) depends on their input only through the dot product (x1, 22) and norms ||z>|| || and | (self dot products). Thus, the kernel is entirely invariant to U and without loss of generality. we can consider the input to the model as Az. Therefore,
switch terminology such that w; - U and a
x ~ N
[aiu]] 02/22
ai
0102012
1) x- ~ N ( az
(-
(-
[eiue]] [aaHla)
of a1020112
0102012 oz
11122112
Linear kernel function. the activation function is linear, then the kernel function simply becomes a standard dot product,
k( x2) A (a11,
The following theorem provides details about the spectrum of this kernel.
Theorem Consider the kernel function k( x22) A (m), 22). The kernel operator associated
with k under the data distribution p specified above has only one non-zero eigenvalue À = =||Ap|| and its eigenfunction has the form v(x)) (Au.æ)
AAul

---

Published as a conference paper at ICLR 2024
ReLU kernel function. If the activation function is set to be ReLU, then the kernel function is known to have the following form (Cho & Saul, 2009; Bietti & Bach, 2020); k( x2)
|
211 at2
||>||| )) la(w) 1
u ( aarcco(()) V u²
(3)
||ا|||
In order to obtain an analytical form for the cigenfunctions of the kernel under the considered
data distribution, we resort to quadratic approximation of h by h as h(u) A 815 3072 (1 + u)². This
approximation enjoys certain optimality criteria. Derivation details of this quadratic form are provided in the Supplementary Materials, is a plot showing the quality of the approximation. We now focus on spectral characteristics of the approximate ReLU kernel. Replacing h in the kernel function & of Equation 3 with h, we obtain an alternative kernel function approximation for ReLUs:
|||||||||| h ( The following theorem characterizes the spectrum of this kernel.
||>||| || a" (1
kx 2)
|
(4)
Theorem Consider the kernel function k(æ,, 2) A |||||| a" (1+
22 |=:||
The kernel
|=
operator associated with under the data distribution p specified above has only two non-zero || with associated eigenfunctions given by
eigenvalues \1 A2 -0
Φ (xx)
2
1+
TAp
Alc
and Φ2(x)) = (
Ap
A
2
Aull
A
6.1 SENSITIVITY ANALYSIS
We now assign target value y(œ) to each input point By expressing the kernel operator using its eigenfunctions, it is easy to show that f(x) = (x))(()) fg 6.(z) y(z) p(z) dz) where runs over non-zero eigenvalues ); of the kernel operator (see also Appendix G). We now restrict our focus to a binary classification scenario, y X - {0, 1). More precisely, we replace p with our Gaussian mixture and set yæ)) to I and O depending on whether æ is drawn from the positive or negative component of the mixture.
f(x) = E. (o) Saa 01(2) (1) + 1 (z) (z) 1 E, (A)))
where pt and p denote class-conditional normal distributions. Now let us tweak the availability of each feature by diagonal scaling matrix B e diag(b) where b - [bi, b2). Denote the modified prediction function as 9B. That is, (x(x) g is their normalized dot product,
Σ (, The aligmment between f and
Yb)
A SEa Vhe (e))((e) di Vhe (t)p(( p(x) dx.
f(m)
2a (
(5)
We define the sensitivity of the alignment to feature i for i = 1, 2 as its m'th order derivative of n w.r.t. b, evaluated at b = 1 (which leads to identity scale factor B = I).
G
2
gm 0860
Va ( (E, (A)))) de V (E, (())), Sga
E, (A))
Σ, (M)
dæ
b=1
b=1
G indicates how much the model relies on feature to make prediction. In particular, if whenever feature i is more available than feature j. i.e. a; >> aj,, we also see the model is more sensitive to feature i than feature j. i.c. Ics| > 1Ggl the model is biased toward the more-available feature. In addition, when a; ag but IGl 15,. the bias is toward the less-available feature. One can express the presence of these biases more concisely as sign (ISi) ( - =2)). where values of +1 and
-1 indicate bias toward more- and less- available features, respectively. To verify either of these two cases via sensitivity, we must choose the lowest order m that yields non-zero [G1 |[2)- On the other hand, if [4 - [C2l = 0 for any choice of a and a2, the model is unbiased to feature availability,
The following theorems now show that linear networks are unbiased to feature availability, while ReLU networks are biased toward more available features.
Theorem In a linear network, Isil |S2l is always zero for any choice of m 2 1 and regardless of the values of an and 02
7

---

Published as a conference paper at ICLR 2024
0.1
142 0.5
P2 = 2
A2 10
Figure 5: Plot of ( (aa))(a a2]) for ReLU network function of and az. We fix ln and vary U2 E {0.1, 2, 10). Yellow and blue correspond values +1 and -1 respectively.
Theorem In a ReLU network, - for any 1 S m S The first non-zero |si1 huppens at = 9 and has the following form,
1c21
5670
152
|||p|| (4102/1/42) (a? -2z22)
(6)
straightforward consequence of this theorem that sign ((Kal-1<al16) -e2) = sign
Amllas (01020102) (c2n2-o2 -w)) sign ( ciu3)(ar-an -au)
5670
Recall from Section that feature predictivity P is related to μ4 via Pt = (1 + erl( (11 Observe that Pj is an increasing function in μı; thus, bigger 14 implies larger pi- That together with (7) provides crisp interpretation of the trade-off between predictivity and availability. For example, when the latent features are equally predictive (μ1 = μ2), the sign becomes +1 for any (non-negative) choice of availability parameters 01 and 0g. Thus, for equally predictive features, the ReLU networks are always biased toward the more available feature. Figure shows some more examples with certain levels of predictivity. The coloring indicates the direction of the availability bias (only the boundaries between the blue and the yellow regions have no availability bias).
7 FEATURE AVAILABILITY IN NATURALISTIC DATASETS
We have seen that models trained on controlled, synthetic data are influenced by availability to learn shortcuts when a nonlinear activation function is present. How do feature predictivity and availability in naturalistic datasets interact to shape the behavior models used in practice? To test this, we train ResNet18 models (He et al., 2016) to classify naturalistic images by binary core feature irrespective of the value of a non-core feature. We construct two datasets by sampling images from Waterbirds (Sagawa et al., 2020a) (core: Bird, non-core: Background), and CelebA (Liu et al., 2015) (core: Attractive, non-core Smiling). See C.5 for additional details.
Sensitivity to the non-core feature beyond statistical account. Figures 6A and B show that, for both datasets, as the training-set predictivity of the non-core feature increases, model accuracy dramatically increases for congruent probes and decreases for incongruent ones. In contrast, a Bayes optimal classifier is far less sensitive to the predictivity of the non-core feature. Thus, models are more influenced by the non-core feature than what we would expected based solely on predictivity. This heightened sensitivity implies that models prioritize the non-core feature more than they should, given its predictive value. Thus, the absence of predictivity as an explanatory factor, we conclude that the non-core feature is more available than the core feature.
Availability manipulations. Motivated by the result that Background more available to models than the core Bird (Figure 6A), we test whether specific background manipulations (hypothesized types of availability) shift model feature reliance. As shown in Figure 6C, we find that Bird accuracy increases we reduce the availability of the image background by manipulating its spatial extent (Bird size, Background putch removal) or drop background color (Color), implicating these as among the features that models latch onto in preferring image backgrounds (validated with explainability analyses in C.5). Experiments in Figure B.9 show that this phenomenon also occurs in ImageNet- pretrained models; background noise and spatial frequency manipulations also drive feature reliance.
8 CONCLUSION
Shortcut learning is of both scientific and practical interest given its implications for how models generalize. Why do some features become shorteut features? Here, introduced the notion of
8

---

Published as a conference paper at ICLR 2024
Bird
A
land
Waterbinds
B
CelebA
100
H
Bird accuracy (%)
attractive accuracy
80
Background
congruent (model)
70
incongruent (model)
8
congrvent (opkimal)
incongruent (optimal)
water
30
20
0.5
0.6
0.7
0.8
0.9
0.6
0.8
1.0
Background predictivity
Smillng predictivity
sze
Background patch removal
Color
100
100
100
Bird accuracy (%)
80
80
60
60
60
40
40
40
20
20
0.2
0.4
0.5
0.8
Scale
Background patches
Color
Greyscale
chance
p=0.55
p=0.65
p=0.75
p=0.85
p=0.95
p-0.5
p-0.6
p-0.7
p-0.8
p-0.9
Figure 6: Availability as well as predictivity determines which features image classifiers rely on. A: Models (ResNet18) 8) were trained to classify Birds from images sampled from Waterbirds. We varied Background (non-core) predictivity while keeping Bird (core) predictivity fixed (= 0.99), and show Bird classification accuracy for two types of probes: congruent (blue, core and non-core features support the same label) and incongruent (orange, core and non-core features support opposing labels). As Background predictivity increases, the gap in accuracy between incongruent and congruent probes also increases. The model is more sensitive to the non-core feature than expected by Bayes-optimal classifier (optimal): predictivity alone does not explain the model's behavior. B: Similar to the Waterbirds dataset, models trained to classify images from CelebA as "Attractive" exhibit an effect of "Smiling" availability. C: Bird accuracy for incongruent Waterbirds probes influenced by both Background predictivity (p) and availability when we manipulate the latter explicitly (see also B.9).
availability and conducted studies systematically varying availability. We proposed generative framework that allows for independent manipulation of predictivity and availability of latent features. Testing hypotheses about the contributions of each to model behavior, find that for both vector and image classification tasks, deep nonlinear models exhibit a shorteut bias, deviating from the statistically optimal classifier in their feature reliance. We provided theoretical account which indicates the inevitability of a shortcut bias for a single-hidden-layer nonlinear (ReLU) MLP but a linear one. The theory specifies the exact interaction between predictivity and availability, and consistent with our empirical studies, predicts that availability can trump predictivity. In naturalistic datasets, vision architectures used in practice rely on non-core features more than they should on statistical grounds alone. Connecting with prior work identifying availability biases for texture and image background, we explicitly manipulated background properties such as spatial extent, color, and spatial frequency and found that they influence model's propensity to learn shortcuts.
Taken together, our empirical and theoretical findings highlight that models used in practice are prone to shorteut learning, and that to understand model behavior, one must consider the contributions of both feature predictivity and availability. Future work will study shortcut features in additional domains, and develop methods for automatically discovering further shorteut features which drive model behavior. The generative framework we have laid out will support a systematic investigation of architectural manipulations which may influence shortcut learning.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Pieter-Jan Kindermans for feedback on the manuscript, and Jaehoon Lee, Lechao Xiao, Robert Geirhos, Olivia Wiles, Isabelle Guyon, and Roman Novak for interesting discussions.
REFERENCES
Amirhossein Ahmadian and Fredrik Lindsten. Enhancing representation learning with deep classifiers in presence of shortcut. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 02893, 2019.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip Kellman. Deep convolutional networks do not classify based on global object shape. PLOS computational biology. 14(12):
e1006613, 2018.
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip Kellman. Local features and global shape information in object classification by deep convolutional neural networks. Vision research, 172:46 61, 2020.
Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Alberto Bietti and Francis R. Bach. Deep equals shallow for relu networks in kernel regimes. ArXiv, abs/2009. 14397, 2020.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp. 77-91. PMLR, 2018.
Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Y. Bengio, D. Schuurmans, Lafferty, C. Williams, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 22, Curran Associates, Inc., 2009.
Thomas Fel, Remi Cadene, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Thomas Fel, Victor Boutin, Mazda Moayeri, Rémi Cadène, Louis Bethune, Mathieu Chalvidal, Thomas Serre. et A holistic approach unifying automatic concept extraction and concept importance estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-traime trainned cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: /8/ 12231, 2018a.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018b.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):66)6666666 673, 2020.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts natural language inference data. arXiv preprint arXiv: 1803, 02324, 2018.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33:9995-10006, 2020.
Katherine Hermann, Ting Chen, and Simon Korbblith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 33: 19000-19015, 2020.
Hendrik Heuer, Christof Monz, and Amold WM Smeulders. Generating captions without looking beyond objects. arXiv preprint arXiv:1610.03708, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Jason Jo and Yoshua Bengio. Measuring the tendency of enns to learn surface statistical regularities. arXiv preprint arXiv: 1rXiv:171..11611, 2017.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re- re-training is sufficient robustness to spurious correlations. arXiv preprint arXiv:2204.02937, 2022.
Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards last-layer retraining for group robustness with fewer annotations. arXiv preprint arXiv:2309.08834 2023.
Barbara Landau, Linda B Smith, and Susan Jones. The importance of shape in early lexical learning. Cognitive development, 3(3):299-3211 1988.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Gaurav Malhotra, Benjamin D Evans, and Jeffrey S Bowers. Hiding a plane with pixel: examining shape-bias in cnns and the benefit of building in biological constraints. Vision Research, 174: 57-68, 2020.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint 1921v01902.01007, 2019.
Matthias Minderer, Olivier Bachem, Neil Houlsby, and Michael Tschannen. Automatic shortcut removal for self-supervised representation learning. In International Conference on Machine Learning, pp. 6927-6937. PMLR, 2020.
Mazda Moayeri, Phillip Pope, Yogesh Balaji, and Soheil Feizi. A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. Proceddings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. arXiv preprint arXiv:2010. 2020.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of out-of-distribution generalization. In International Conference on Learning Representations, 2021.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization deep learning. arXiv preprint arXiv: 14/2.66/4, 2014.
Guillermo Valle Pérez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the farameter-funntion map biased towards simple functions. stat, 1050:23, 2018.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018.

---

Published as a conference paper at ICLR 2024
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34: 14::1266-122 1272, 2021.
Aahlad Puli, Lily Zhang, Yoav Wald, and Rajesh Ranganath. Don't blame dataset shift! shortcut learning due to gradients and cross entropy. arXiv preprini arXiv:2308./2555 2023.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning. pp. 5301-5310, PMLR, 2019.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems, 34:4974-4986, 2021.
Chaitanya K Ryali, David Schwab, and Ari S Morcos. Characterizing and improving the robustness of sulf-super learning through background augmentations. arXiv preprint AXI::2103.12711 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference Learning Representations, 2020a.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine PMLR, 2020b.
Learning, pp.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, sims 33:9573-9585, 2020.
Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. Workshop on Visualization for Deep Learning, Proceedings of the International Conference on Machine Learning (ICML), 2017.
Ajay Subramanian, Elena Sizikova, Najib Majaj, and Denis G Pelli. Spatial-frequency channels, shape bias, and adversarial robustness. arXiv preprint 2309::2300.13190, 2023.
Remi Tachet, Mohammad Pezeshki, Samira Shabanian, Aaron Courville, and Yoshua Bengio. On the learning dynamics deep neural networks. arXiv preprint 4rxiv:1889.006848, 2018.
Alexa R Tartaglini, Wai Keen Vong, and Brenden M Lake. A developpeentally-nnsirdd examination of shape versus texture bias in machines. arXiv preprint arXiv:2202.083340 2022..
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton Van den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16761-16772, 2022.
Rishabh Tiwari and Pradeep Shenoy. Sifer: Overcoming simplicity bias in deep networks using a feature sieve. arXiv preprint 33X 2023.
Yusuke Tsuzuku and Issei Sato. On the structural sensitivity of deep convolutional networks to the directions of fourier basis functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 51-60, 2019.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
Max Wolff and Stuart Wolff. Signal strength and noise drive feature preference in cnn image classifiers. arXiv preprint arXiv:2001.8889, 2022.
12

---

Published as a conference paper at ICLR 2024
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09999, 2020.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective model robustness in computer vision. Advances in Neural Information Processing Systems, 32, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2014.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv: 2017.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6)) 2017.
13

---

Published as a conference paper at ICLR 2024
Supplementary Material for "On the Foundations of Shortcut Learning"
A BROADER IMPACTS
Our work aims achieve principled and fundamental understanding of the mechanisms of deep learning-wher wwhin succeeds, when it fails, and why it fails. As the goal is to better understand existing algorithms, there is no downside risk of the research. The upside benefit concerns cases of shortcut learning which have been identified as socially problematic and where faimess issues are at play, for example, the use of shortcut features such as gender or ethnicity that might be spuriously correlated with core feature (e.g., likelihood of default).
B SUPPLEMENTARY FIGURES
Methods details accompanying these figures appear in Section C.
tanh
635
625
-025
-0.5
Figure B.1: Bias as function of availability and predictivity for single-hidden-layer MLP with a tanh hidden-layer activation function. Settings match those described in Figure 3C.
2-rellance:
Syrellance. -1.0
anreelance ance: 0.0
z-tellance 0.0
2x
Figure B.2: The z,-reliance (Sections 3 and C. of hypothetical classifiers which rely exclusively on 2, (top left) or (top right), or rely equally on both features (bottom row). Color indicates the predicted label (red: pos, blue: neg) for probe items plotted by base probe (2.. 2g) values
14

---

Published as a conference paper at ICLR 2024
0B5 09
0.8 07 0.5 08 085 0.9
36
train accuracy (%)
val accuracy (90)
0U
80
80
0.85 0F 0.75
70
0.5 08 015 0.
0.50
optmall A- a.telll-.eaaee
085 025 08 085
0.50
Acrellance
0.25
025
0.00
0.00
-0.25
-0.25
-0.50
-0.50
085
-0.75
-0.75
-1.00
-1.00
Bulli
Figure B.3: Supplementary data for the results presented in Figure 2. TOP ROw: Model performance on the train (left) and validation (right) sets. BOTTOM ROW: zg-reliance for the optimal classifier (left) and models (right).
A
B
1.00
0.9
0.75
0.85
0.85
0.50
0.8
0.25
0.00
0.65 0,7 0.75
-0.25
0.7
-0.50
-0.7
-1.00
0g/0c
ag/a;
Figure B.4: The results shown in Figure are not specific to choice of ose. Results for the same experiment using datasets with osc 0.4 (A) and asc 0.8 (B).
100
100
90
90
train accuracy
val accuracy
depth
depth
70
70
60
60
128 width
256
128 width
256
Figure B.5: Supplementary data for the results presented in Figure 3. Model performance on the train (left) and validation (right) sets.
15

---

Published as a conference paper at ICLR 2024
A
B
bird predictivity
P
validation accuracy (%)
validation accuracy (%%
70
70
validation
60
60
60
50
50
50
0.8 background predictivity
0.6
1.0
0.7
0.8
0.9
0,/d,
Ps
Figure B.6: Supplementary data for image experiments. Model performance underlying A: Experi- ments with the Waterbirds datasets presented in Figure 6 and described in C.S, and B: Experiments with image instantiations of the base dataset manipulating pixel footprint presented in Figure (left) and C (right).
A
B
-550
08
10
val accuraty (96)
025
frain accuracy
0.00
0.75
-20
70
07
68
0.5
11.00
-50
-50
18
1.00
Ferellance
0.25
0000
0000
065 07 075
0.25
0225
-0..0
-0.50
16
Figure B.7: Models prefer shallowly embedded nonlinear feature (2,) to a deeply embedded nonlinear one (z.). A: The color of each cell of the heatmap indicates the mean bias of models as function of the degree of nesting of 2e (πc), and the predictivity of 2, (px). 2g is always shallowly embedded (1), 0), and the predictivity of is held fixed (p.- = 0.9). See C.3 for additional details. B: Model performance. Cizanceliance of the optimal classifier (left) versus the model (right).
16

---

Published as a conference paper at ICLR 2024
varying bird size
varying the number of background cells
varying the noise intensity (uniform and perlin)
Removing the high frequencies
Color to Greyscale
Figure B.S: Illustration of studied Background availability manipulations. For each of the six perturbations influencing the availability of both the background and the bird (in the case of bird scale), we provide visual representations of the potential perturbations applied to the training dataset. For instance, the first row demonstrates variations in the dataset resulting from changes in bird size, ranging from 5% 95%.
17

---

Published as a conference paper at ICLR 2024
RechaeTE
Longon
10
Pretrained Rea0
3000000
Figure B.9: Manipulations of image backgrounds show that availability well predictivity determines feature reliance in vision models, including in models pretrained on ImageNet. Expanding on the results shown in Figure 6, we plot Bird accuracy for incongruent probes (opposing Bird and Background labels) as function of Background predictivity (p) and hypothesized types of availability. TOP ROW: ResNet18 models. BOTTOM ROW: IN ResNet50 models (see Section C.5 for additional details).
18

---

Published as a conference paper at ICLR 2024
ResNet18 Bird size 40%, p= 0.8
Saliency
SmoothGred
RISE
Sobol
ResNet18 Bird size 80%, p = 0.8
Saliency
SmoathGrad
RISE
Saboll
Figure 1010: Explainability sanity check: Attribution maps corroborate Background availabil- ity study, In Figures 6 and B.9, we saw that Bird size manipulation affects models' tendency to classify incongruent probes consistent with their Bird labels. Here, we see that attribution maps for probe images (top row) reflect a difference focal point for models trained on images with Bird size 40% (preference for image background) versus Bird size 80% (preference for foreground bird). See C.5 for additional details.
19

---

Published as a conference paper at ICLR 2024
C SUPPLEMENTARY METHODS AND RESULTS
C.I ASSESSING BIAS
In Section 3, we described how we determine reliance, M a score which quantifies the extent which a model (optimal classifier or trained model) uses feature 2, as the basis for its classification decisions. In Figure B.2, for intuition, we show the 2g reliance of hypothetical classifiers.
C.2 SYNTHETIC DATA
Optimal classifier. To obtain optimal classifications, use Linear Discriminant Analysis (LDA, as implemented in Sklearn with the least-squares solver). In all but the experiments on the effect of nesting as part of the generative process (C.3 and Figure B.7), the optimal classifier was fit to and probed with the same embedded base inputs used to train the corresponding model in a given experiment. In the nesting experiments, LDA was fit to and probed with base inputs directly.
C.3 VECTOR EXPERIMENTS
Default model architecture and training procedure. Unless otherwise described, we train multi- layer perceptron (MLP, depth 8, width 128, hidden activation function = ReLU) with MSE loss for 100 epochs using an SGD optimizer with batch size 64 and learning rate- le 02. We use Glorot Normal weight initialization.
Models prefer the more-available feature, even when it is less predictive. In these experiments, given a base dataset with (ps: Pc), for each availability setting (a,, ag), we manipulate the availability of the dataset, sample a random intialization seed, train the default model architecture, and evaluate the bias of the model. We repeat this process for each 10 runs, computing the biases for individual (model, optimal classifier) pairs, and then averaging the results.
Effect of nesting on availability. Figure IB illustrates how our generative procedure for synthetic datasets supports nested embedding of latent features, as described in Section 3. In this experiment, we test whether a model prefers a shallowly embedded nonlinear feature (2,) to a deeply embedded nonlinear one (zz) when the amplitude of both features is matched (a,, 0g 0.1) and the nonlinearity is a scaled tanh (tanh(Az), (Az), with À = 100). After embedding each feature as e; = a,W; we apply the nonlinearity. For nesting factor nj > we pass e; through a fully connected network (depth 77, width = d, no bias weights) with scaled tanh activations on the hidden and output layers. We initialize weight matrices to be random special orthogonal matrices (implemented as s scipy. st st special ort ho group). In Figure B.7, we show bias as function of ne and Ps: 2, is fixed to be shallowly embedded with 7/s = 0 and has Pr: 0.9. We report the results as the mean across 3 runs.
C.4 PIXEL FOOTPRINT EXPERIMENTS
Model architecture and training procedure. In these experiments, we train a randomly initialized ResNet18 architecture with MSE loss using an Adam optimizer with batch size = 64 and learning rate = 1e 02, taking the best (defined on validation accuracy) across 100 epochs of training. For each (p., predictivity setting, we repeat this process for 10 (sampled dataset, random weight initialization) runs.
Analyses. In Figure 4, we report results for experiments in which we use a base 2, pixel footprint of 400 px, and manipulate 2, pixel footprint to be 1 - 5x larger (a, = 1, a, E [1, 5)). To determine the bias of a model. we compute the 2 ze-reliance of the trained model given image instantiations of the base probe items, and compare to the 2, relianeee of the optimal classifier (LDA) fit to and probed using vector instantiations of the same base dataset with the same as, and Qg. We repeat this process for 5 runs, computing bias on a per-run basis as in Section C.3.
C.5 FEATURE AVAILABILITY IN NATURALISTIC DATASETS
Datasets. Waterbirds. Waterbirds images (Sagawa et al., 2020a) combine birds taken from the dataset (Wah et al., 201 1) and backgrounds from the Places dataset
Caltech-UCSD
20

---

Published as a conference paper at ICLR 2024
(Zhou et al., 2017). To construct the datasets used in Figure 6A, we sample images from base Waterbirds dataset generated using code by (Sagawa et al., 2020a) (github. com/ www/koop//aaaeeccooeeeeeooeaaaaa kohpangweil group_ _Ro) with val frac 0.2 and confounder strength 0.5, yielding sets of 5694 train images (224 x 224), 300 validation images, and 5794 test images. We then subsample these sets to 1200 train, 90 validation (x2 sets), and 1000 probe images (x2 sets), respectively, such that the train and validation sets instantiate target feature predictivities, and the probe sets contain congruent or incongruent feature-values, as described in Section
To construct the datasets used in Figures 6C and B.9, for a given predictivity setting, we subsample 8 bird types per category (land, water) and cross them with land and water backgrounds randomly sampled from the standard Waterbirds background classes, to generate blas-bbllaned train sets ranging in size from 800 1200, and incongruent probe sets of size 400.
CelebA. The CelebA (Liu et al., 2015) dataset contains images of celebrity faces paired with variety of binary attribute labels. In our experiments, we cast the "Attractive" label as the core feature, and the ""Smiling" label as the non-core feature. To construct the datasets used in Figure 6B, we sample images consistent with the target predictivities.
Waterbirds availability manipulations. In the datasets depicted in Figures 6C and B.9, we aim to manipulate various aspects of background availability. To do so, we use CUB-200 masks to exclusively modify the background, and apply five distinct types of perturbations: altering bird size, removing background patches, changing background colors, applying low-pass filtering, adding Perlin noise, and adding white noise. These perturbations test hypotheses about specific image properties than may make an image background more available to a model than the foreground object. We vary Background predicitivity while holding bird predictivity fixed (= 0.99). We report results for at least 500 runs (model training) for each perturbation type. Figure B.8 shows sample image to which the perturbations have been applied. In detail, the perturbations are as follow:
Bird size: We manipulate the pixel footprint of the bird by scaling the square mask. Note that an increase in bird size corresponds decrease in background pixels in the image.
Background patch removal: This manipulation removes patches of the image background while preserving the foreground bird. To apply the manipulation, we partition the image into a 3 x 3 checkerboard, position the bird in the center, and then remove in [1, 8) background cells.
Color: To assess the influence of color information, we use the original image with its colored background ("Color") or convert the background to grayscale ("Grayscale").
Low-pass filtering: We apply a low-pass filter to the frequency representation of the background, with the cutoff frequency varying from 224 (original image) to 2 (retaining only components at 2 cycles per image).
Uniform noise: Uniform noise introduced to the image with an amplitude ranging from 0 (original image) to 255. It is essential to note that image values are clipped within the (0, 255) range before preprocessing.
Perlin noise: We apply Perlin noise, which possesses spatial coherence, various intensity levels along with uniform noise. We employ 6-octave Perlin noise to match the frequency distribution of natural backgrounds (wavelengths 2, 4, 16, 32, 64, and 128).
Model architecture and training procedure. For the experiments shown in Figure 6A and B, we normalize all images by the ImageNet train-set statistics (RGB mean = [0.485, 0.456, 0.406)), std = [0.229, 0. 0.224, 0.225),, and then use the same settings as described in 5.
In the experiments shown in Figures 6C and B.9, we preprocess images by normalizing to be in [-1,1], 1]. apply random crops of size 200 | 200, resize to 224 224, and randomly flip over the horizontal axis with p- = 0.5 We train models with MSE loss using an Adam optimizer with batch size- 32, cosine decay learning rate schedule (linear warmup, initial learning rate- le 03), and weight decay = le - 05. For the ResNet18 experiments in Figures 6C and B.9, we train a randomly initialized ResNet18 trained for 30 epochs. For the ResNet50 experiments in Figure B.9, we train the randomly initialized readout layer of a frozen, ImageNett-retrained ResNet50 (IN ResNer50) for 15 epochs.
21

---

Published as a conference paper at ICLR 2024
Vision models, including ImageNNet-pretrained ones, are sensitive to Background availability manipulations. Figure B.9 displays the results of the six manipulations which we hypothesized affeet Background availability. The top row shows accuracy for ResNet18 models trained from scratch, while the bottom row shows the results for IN-ResNet50. Our findings reveal several key observations:
Results:
First, as the size of the bird increases (Col.1), as well as when the background becomes noisier (Col. 4,6), filtered of its high frequencies (Col 5), simply masked (Col 2), the model tends to rely more heavily on the bird itself for classification. Conversely, when the background is clear and informative (available), the model utilizes the background features to predict the class. Second, the spuriousness of the background, as quantified by p. also has discermible effect on the model's behavior. However, only partially accounts for the observed variations, demonstrating that availability factors are at play. Last, these results indicate that pretraining has beneficial impact and can modulate feature availability. This effect may be attributed to the learned invariances during the pretraining process, enabling the model to better adapt to different availability conditions. Our spatial frequency experiments complement existing and concurrent work studying what models learn as a function of spatial frequency manipulations at training (Jo & Bengio, 2017; Yin al., 2019; Tsuzuku & Sato, 2019; Hermann et al., 2020; Subramanian et al., 2023) or test time (Geirhos et al., 2018b).
Results: Explainability analyses corroborate Background availability study. The experiments presented in Figures 6C and B.9 showed that the degree of Background availability influences model classifications of incongruent probes. Here, a model predominantly relies on the image background, or on the foreground bird, according to the experimental results, its focal point reflect this. Figure B.10 shows the attributions maps of two ResNet18 models trained from scratch, one using a bird size of 40%, and the other using bird size of 80%. For the same set of probe images, the model trained on larger birds exhibits strategy shift in classification, seemingly having his focal point on the birds, whereas the model trained on smaller birds mainly relies on the background. The methods employed encompass combination of black-box techniques (Rise (Petsiuk et al., 2018) and Sobol indices (Fel et al., 2021)) and gradient- based approaches (Saliency (Zeiler Fergus, 2014) 4) and Smoothgrad (Smilkov et al., 2017)).
attribution methods to verify that when
D QUADRATIC APPROXIMATION OF h
Since is the dot product of two normalized vectors, its range is limited to -1 < < 1. Observe that the derivative h has the form,
1 π
N' f2(u)= 1
arccos(u).
The only place within E [-1, that h' vanishes is u -1, suggesting the critical point of our quadratic must lie at this point. Furthermore, observe that h--1) = 0. Forcing these two constraints onto our quadratic, leaves us with the form,
Tu()) a(1 + u)²
where a is a parameter to be determined. Our aim is to find a so that the function h and looks similar over the entire domain E [], We choose Eg error between the two functions defined as,
Ca = [ (M(w) -((u) du.
(10)
22

---

Published as a conference paper at ICLR 2024
0.8
0.8
0.4
Figure D.I: The plots for h (blue) h (orange) within [-1,1].
Replacing the definitions for h and ìe into (h(u)
- -i) one can
obtain
163
32
32
Eg
a
+
a²
(18)
48
5
277²
Since ea is a convex quadratic in o, its minimizer can be determined by zero-crossing the derivative of ea w.r.t. This yields,
815
a" - arg mine, ea
(19)
3072
This leads to the claimed quadratic form,
815
T((u)
(1 u)?
(20)
3072
Figure D.I shows plot of the original h and the quadratic approximation h visually get sense of the approximation quality.
E PROOF OF RELU KERNEL THEOREM
Recall that the kernel function & for ReLU is expressed using / as in (3). Replacing h with h thus gets us an alternate kernel function which approximates that of ReLU's.
((1 | (11
>
((12))
||2:|| ||||| 1))
k(x,, 2)
|>|||| ||
|| |||| a" (1+
(21)
We first claim that the indefinite integral of (N(u) - Ti(u))² has the form,
(u)
/ ( (
(((u((=
( arccos(u))
/11 - u² a(1 u))) 2
du
(11)
21607 432 2 2 + 80 (9 (6a2 4a + 1) 17) u²
(12)
+240 2 + 11) + 1080 2 a(2a 1)u2 + 2160 2 a(2a 1)u²
(13)
- 155 / - u² (a (90n² + 256u + 207u 64) - 128u + 32)
(14)
+120 (mma (3n²² + Su + 6) 12² 12 + 4 (1 4x²² V1 - cos (u)
(15)
-1215ma - 720m² cos (n)²²
(16)
This can easily shown by taking the derivative both sides and arriving equality. The definite integral can now computed by evaluation the indefinite integral at the boundary points.
163
32
32
ea = (1) (-1) =
+
0
(17)
48
5
27
23

---

Published as a conference paper at ICLR 2024
E. EIGENFUNCTIONS An eigenfunction & must satisfy,
J/n a" ||
||||
(1
(1+
x
|| ))² 2
(z))(z)) Ad(x))
(22)
|
or equivalently,
2 ||||| )
a"
||=|||
|||||
((>)p(z) = )o(x),
(23)
|
We first focus on the above integral.
|
1 +
|| 6(z)p(z) dz
(24)
Rm
||>|||
S/a
(xz, ,²)
(ar, 2)²
|||||
++2 2
(z))(()d
(25)
||a=|| ||=||
||0:|22 ||
L/a
(ar, 2)
(ax, ) ?
|||||
+ 2
6 ((z)p(z) dz
(26)
||=|||
||0||² ||
l
||=|| + 21 + 12
x722 + x323 20
d(z)p(z) dz
(2
Vz + x9
(o} x22)
21 + #2
J
/201
V2r²
27
x7
||||| + V221
V222
(28)
V + 222
V 11 + 122
2
+ 2/
+ x33)
2/
2223
V22122
V22102
+
((>)p(2) dz
(29)
v 2 + 2
(x² + x
+
(2) + x2))
V20
V2.2
+7
22
22 (30)
= do + 01
+ 22
+
+
03 21 +
22
+
2 + 22
2
+
as
x7 + 23
where dg to as are the result of the definite integral for each term. Observe that each a; is thus constant in I and z. Plugging the above expression into (23) yields,
v /2
x7
20122
a" ||||||||||| tai
+ V
ta2
2
+
22
2 + 27
22
04
+
x2
+ a5
x
+
x2
Ad(x),
(31)
Thus the eigenfunction has the form,
<xx) = a' ||ج||| (ao + A
v V2n
V202
x
x2
V2 x2
V + 2
02
+
22
+
03
2 11
+
42
+
01
21
2
+
2
+
05
+
22
7
(32)
In order to figure out the values of ag to as we plug the obtained eigenfunction form into (23) again, for both d(x)) and (0z), That takes us from
2 o0
a" ||||||
||=||
[1+
(2))((2) = )Φ(x),
(33)
|
|
to,
a" |ل||| [ || + Vēn
V2n1
V21a
zî
of
+ V222
+
(34)
Vo7+ + oz
Vo + xラ
V27 +
2
(af + xz))
2/
/20122 V2n
(35)
V V + 23 (2) +3
/ 2 +z))
V221
V222
2
3/
V22122
2
(00 01
/ +
2
+ 02
/=2 + 22
03
+ 2
04
27 + 2
+ 05
2 + 2/
xp(2)dz
(36)
x a" ||||||| (00 + 01 /2 + x2
V2u1
+ 02
V + x2
/222
+ 03
x
1
+
3
+
04
x + 5
12
+ 0s
v 22 02
x + 5
(37)
24

---

Published as a conference paper at ICLR 2024
Dividing both sides by a" |||////AA /A yields,
a
|
|=|
1 +
|| || 1= p(>)p(z)dz Ad(a)
*
(38)
v221
V282
2||
2221
V222
x2
V of x
V o #
V
(4f +
22
V2
V20122
(uf
z3)
22 #3
(2) +g))
//21
V22
41
/22122
x||=|| (do + d1
V27 +
+ 82
V
+
84
o1
+
+ as
$3 +
p(z)
V252
22
(ag +
23
02
as
23
04
23
(42)
For the above identity hold at any point æ we need to equate the coefficients for each term involving æ. That is, we need to have.
||=||
22
/222
of
G1
01
2
|I|||
V2s1
V22a
V=++o3
V=++
p(x) ds
022
0.3
+
or in a more compact form,
a" (fpe (vx))) p(z)
Xa,
(44)
where,
Uz
=
221
222
|=
22 1
(45)
We now focus on the form of p(z). Recall that we assume p(z) mixture of / Gaussian sources. Denote the selection probability of each Gaussian component by n; and the corresponding component- conditional normal density function by g(z; 1((), c(O). Thus,
p(z) Σ T:9(2; (())
(46)
Since we assume that the covariance matrix of each gi is small element-wise, we can resort to Laplace-type approximation for the integrals as below.
[ [())
Σ t. ((!))..
(47)
Under this approximation, the linear system can expressed as,
Σ Foioa
a = a
(48)
where,
2
||
22
(
22
(1)
(49)
O
Thus, any solution must be eigenvector of C. In the sequel discuss how to obtain the eigenvectors of C. In particular, in our 2-d problem with Gaussian mixtures, where म = 72 = 1/2 and (())) = Au and (2) A44, the linear equation can be expressed as below.
X
a
(50)
2
2
a
25

---

Published as a conference paper at ICLR 2024
where,
||all
Vanym
(20211
(axpi)² [|
(azpz)²
T
(51)
[Aul
|A
Apll
-V201m1
-V202u2
(omm)) )
(02u2)
V2a1 [[
(52)
||
[|ppll
We now seek the cigenvectors of C' in this specific setting. A key observation is that || = 110 because the two are different only in the sign of their components, We now claim that, for any two
vectors v* and e- such that ||=+|| 1 ||. the eigevectors of the matrix C vtv + Lvo have the form,
v = c (v v")
4/2 co (v*
(53)
where ej and C2 are normalization constants. We prove that ab is an eigenvector of C: similar argument applies 1/2-
(v+ +T + vov (v+ +
Ct
(54)
C1 (utllot 12 u (ut u + (ut v + v 11 ||]1)
(55)
e1 (u + (ut +
+ 10 11²)
(56)
=
,
+ c
1 (ut ( v
(57)
= a(Il=t1²+00 v ))²² + ( (ot, 1)0
(58)
= (100
c + v2)
(59)
$1-
(60)
The above shows that u satisfies C'ty, & 1 and thus 1/, must be an eigenvector of C. Plugging the definition for v and v gives us the cigenvectors of our matrix, and thus obtain the solution in the equation (48). Since are not constraining the length of the eigenvectors here, we can absorb all the constants in equation (48) into one and express the solution pair a as below.
a
v+ o-
||A
0
0
(a )
[Apl
(aapg) ) [A
v Vea 1y 110aa22 Ap
(61)
b+ -0
a
10
2an/1
v 202222
0
T
(62)
Recall from (32) that,
a"
/2m1
V2.2
x²
V2
(xx)
x (a0 + a1
02
+
a3
04
a5
(63)
||>|||
[
x
1
x
2
||
Plugging the pair of solutions for a into the above yields the two eigenfunctions,
||pp|| + (a
(agu2)²
x2
/2 p1 a2
22x122
(x)) x ||ق|||
(64)
AAul
|
|Apil
|
||Ap
x
12
/2.1
V2.ra
(x)) x ||ج|||
2201m1
+ /222112
(65)
|
|<:||
or more simply,
o (x) & ||ق||| |uull
1+
x
Au
2
(66)
||0|||
A|u
(2(x) x (x, Api)
(67)
We convert the x to equality by applying proper scaling factors dj and d2 as follows.
(1())
1111=11|A/
(1+
[
( ||
Au
2
(68)
Φ2(m) d2 (1. Al1)
(69)
26

---

Published as a conference paper at ICLR 2024
In order to find the factors d1 and d2, require Φ1 and Φ2 to have unit norm in the sense of (6).
Starting with @1 we proceed as,
10:11
df ||ul1
J
||=1|
Ap
p(a)
(70)
|Aul
( ||01
/ Aj
))²
di ||ul1|
1+
11
71)
|||||
[|Ap
||=||
||Au
wm
Aj
of ||41|1 ||||"
||
|[
(72)
d ||AM| (1+
Ap
Aμ
|
||
(73)
(1+17)².
(74)
Setting the above to and solving in d yields,
1 di
(75)
A/
and consequently,
)
)
||||||
1 1
| Tap
x
$ (x())
(76)
2 2||A
Similarly for d2 we proceed as,
d2 ( (I, (x, Api p(z) dx
||x2|
(77)
R2a
d3 (2 (z, Apjj x=Aa + ((, AHi a=-AP Ar
(78)
d3 (Ap, Ap))
(79)
(80)
Setting the above to and solving in d2 yields,
1 ||
d2 =
(81)
and consequently,
Aμ
(2())
(82)
[Aull
[Apl
27

---

Published as a conference paper at ICLR 2024
E.2 EIGENVALUES
To find the eigenvalues, we simply put the eigenfunctions in their defining equation (1). Starting with Φ1 we proceed as below.
k(x, 2) ((2)
(83)
22
))
/ R2
a
||a|||
|||
1 1
(84)
|
||]||
|
1+
|
2
Au ||Am
a"
||a||]
||
1
+
p(z) dz
(85)
||a=||
||=||
AApll
R2
||=|||
1+
T
Ajill
Au
||>||||
|||||
+
(86)
||
|
==A
+ 2
||a|||
||=|||
1
+
x
2
||
1
||
Apall
A₁
(87)
||0|||
||
A
2
App
Ap
1
A
Au
Aμ
||<|||
||4
1
(88)
||,||||
||Ap|||
+
a" ||>|||
Apll
(1+
/ ||=0|1 || -A4 ||
-Au
2 21+
A-
AAp
[[
(89)
(1+
Ap
))
Au
a"
||>|||
||||||
1+
(90)
||=|||
||Au||]
|
A₁u
2
a'
| Axall
1+
(91)
||>|||
A
||=|||
||Au|] )*)
Aμ
2a" A|p1 2/| 2||Ap|l
1+
(92)
||=|||
2a"
|
112 01 (z))
(93)
Therefore,
A
2aa|||AAa] ||
2a
(94)
28

---

Published as a conference paper at ICLR 2024
Similarly for d2 we have,
/
k(x, z) p(=)
(95)
R2
/ a
2
|
|||||
22 02(2) p(z) dz
(96)
||a=||
||=|
R2
2
Au
||2|||
||=||
p(z) dz
(97)
||a=||
||=||
|
||
2
||Ap|]| )
Af
||0||
|||||
(98)
||>||
||
||
z=AP
2
Ap
+
a
|
||=||
1
+
(99)
2
||2|||
||
||
|
Aμ
Ap
Aμ
||||||
|
1+
(100)
||=|||
||Ap|||
||pI||
||Au||
AA
|| ))
2
Ap
Au
$2
a"
||0|||
Aull
1
||~||| ||
AAp
Apll'
||
(101)
||pl (1+ ||>||| |||p||| ))²
Ap
a"
||0|||
(102)
(1+ ||=||| || |1-Aμ
AAp
+
a" ||>||| |1
Apell
(103)
(1+
Ap 1) 2
(1-
Ap
|||||| ||Aμ||
(104)
||00||
||A
||00
||A
\|=l| |A )
||>||| ||ll
x
(105)
2a
||
x
Aμ
(106)
A|
||A
2a
||u1| (z)x).
(107)
Therefore,
A = 2a [Au]]
(108)
F PROOF OF LINEAR KERNEL THEOREM
An eigenfunction of the kernel operator associated with kernel k(x,, 2) = (ax, 2) must satisfy,
J (x, z) ((z)p(z) dz Ad(x)
(109)
Following our earlier setup, we proceed with a 2-dimensional input this becomes.
(x, z) ((=)p(z) dz
(110)
(x1 + x222) (())p(()) dz
(111)
0 + 0232,
(112)
where aj and ag are the result of the definite integrals of form fg² xz)(()) dz, for = 1,2. Observe that each a; is thus constant in and z. Plugging the above expression into (109) yields,
a1 + 0232
A (x))
(113)
29

---

Published as a conference paper at ICLR 2024
Thus the eigenfunction has the form,
1 d(x)) X (0131 + 0222))
(114)
In order to figure out the values of 01 and az plug the obtained eigenfunction form into (109)
again, for both ((x) and ((z),
(c, z) ((z)p(z)dz
(115)
R2a
/ (2121 + x2222 (a121 + a222) p(z) dz = X (a121 + a222)
(116)
/ (112) + x222) (0121 0222) 0222)
À (013) + 02.2)
(117)
Σ (21
( + x2H2²") (a144)
(i) + azيu? (0)
(a1 + 0gT2),
(118)
where in the last line we use the assumption that the covariance matrix of each Gaussian source is small, and thus we can resort Laplace- like approximation of the integral. To have the above identity hold true at any æ we need to require that,
Σ TipE
(4) (aip1 02f2
(8)
(i)
A01
(119)
E Tapl? (aipl) o2222)
Aa2
(120)
(121)
or in matrix form,
(
Σ
M1
P1
p
a1
=
a1
(122)
[g
02
Mi
H2
H2
The equation can be compactly expressed as,
Ca = Xa
(123)
Thus, any solution must an eigenvector of C. In the sequel we discuss how obtain the eigenvectors of C. Observe that each matrix C; is rank-one symmetric matrix that can be written
as 1u (0) (0) In particular, in our 2-d problem with Gaussian mixtures, where कं T2 = 1/2 and (1))) Aμ and 2(2) = AA,, the above equation can be expressed as below.
(1 (AW) + 1 (-An)
Aa
(124)
+ (AI)( aaa
(125)
Thus,
a x Aμr
(126)
Recall from (114) that,
(xx) = = (a. x).
(127)
Plugging the solution for into the above yields the eigenfunction,
<(x) (Aμ, x),
(128)
We convert the x to equality by applying proper scaling factors c as follows.
Φ(x)) c (A,,
(129)
30

---

Published as a conference paper at ICLR 2024
In order to find the scale factor c, require @ have unit norm in the sense of (6).
||
0²
(A)
p(x) da
(130)
c²
+ 2 2 (AAH,
((Au, x)) Ap
(131)
Ap
è x)) e² (I)Au)???
(132)
(133)
Setting the above to and solving in yields,
(134)
A
and consequently,
(Au, x²) ||μi|
<(x)
(135)
For finding the eigenvalue associated with Φ we plug this form into (109) for both ((x) and ((z) and then solve in X.
x 2) ((z)p(z) dz = A
(136)
R
(Aps, 2)
Ap,
(xe >)
|A/u| p(2) dz
(137)
Apull
(Apr, 2)
{
1
(Ap, 2)
(Au, x)
(x, 2)
(x, x)
(138)
|| ==
2
||A Az=AA Ap
|| A ||
(Ap, Au)
(Au, x))
(x. Au)
(139)
Ap
Aup
| 1
X
(140)
G OPTIMAL PREDICTION
Consider the task of finding function (ax) that best approximates given function y(x) via the following regression setup.
(1())-((a)) 1 - gm L -= f
u(z)) p(æ)dd
(141)
Suppose we are interested in expressing f using the bases induced by the kernel k, that is the eigenfunctions of the linear operator associated with whose eigenvaules are non-zero. That is,
f(x) = Σ (x))(x),
(142)
where Oe is an eigenfunction, and & runs over eigenfunctions with non-zero eigenvalue. We now show that the function f which minimizes the regression loss L has the form,
((x) Σ (x))(())
(143)
Σ
L 2x (2(x)g(=)
(144)
We start from,
L gg (f(x) - -v(o)) p(x)
(145)
Σ (x)) - u(z))) p(x)dxx.
(146)
31

---

Published as a conference paper at ICLR 2024
Thus,
0
(x)x(x) - v(xx) (e)d)(()) p(a))dx
L
(147)
Da,
am
(1)0))(())))(0)) (2 p(ardex
(148)
k
as [ ()(2))((Z) p(x)dax
apaj
(z)
(149)
h
R
= d² 0j
(150)
Zero-crossing,
aj = / ()(())((a)) ((a))))
(151)
Thus,
[(x) Σ (x)
(152)
= Ele (x()))(())(()ddzo)()
(153)
In particular, we replace p(z) by our pair of Gaussian densities with small covariance, and set y to and 0 depending on whether æ is drawn from the positive or negative component of the mixture, we arrive at,
[(x) = Σ ( /
(0) = 2 Σ si (z)) (Ap),
(154)
where pt and p denote class-conditional normal distributions.
H SENSITIVITY ANALYSIS
H.1 LINEAR KERNEL
Theorem In a linear network, [Gi| - [C2l is always zero for any choice of m 2 1 and regardless of the values of aj and 02
Recall the definitions,
f(x)
2 Σ ((()))((AA))
(155)
(B(x) (x)
4
2 Σ (((
(156)
f(x)
(B(
(b)
p(x) dx.
(157)
g2
Viaa f²(t) dt Vh (t(tt((t)))) dt
For linear kernel, we had only one eigenfunction with the form,
(Ap, x)
t (x)
(158)
AAu|
Replacing that, in the above definitions yields,
f(x) = (Ap, x))
(Au, Ap}
1 (A,,, x)
(159)
2 ||ul
Au
2 ||u
(Au, x))
(Au, BAp)
(B(x) = 2 A
(160)
A
32

---

Published as a conference paper at ICLR 2024
It is easy to obtain,
[ f (x))p(z)
4 |Aull J ((Aps, x))² (0)))ppaa p(x)) dx
(161)
1
1
2((AH, Ap)))² 2
(162)
A
and,
Jec (x))p(a))
((Au, BA
[ ((A)I, x p(az) da
(163)
|A
1 ((Au, BA
2((AH, Ap))
(164)
||
((Ap, BA
(165)
||pp|
Plugging these results into the definition of yields,
f(x)
(B(x)
7(b)
p(æ) da
(166)
g2
VIso p2(t) dt V. (t) dt
(Au,æ)
(AH.))
(An.BAu)
AA
AM
p(ac) da
(167)
R2
BAn)
V2
V2
|A
(Ap..))
(Ap..))
sign(
(An, BAp))
2 ||ul
|Aulli
p(x:) da
(168)
V2
sign( (Ag,, BA
((Au,, p(x) do
(169)
2| Apll
sign( (Au,, BA
(170)
= sign(braïuι + b20g13)
(171)
Recall that,
8m
C1 A
(172)
0b
b=1
is easy to see that when for any m 2 1,
3m
0
biafut + braguz + 0
(173)
abm
Naa
biafni + brazuá = 0
To find Çi and (2 need evaluate the above the point bj b2 1. Since by definition, 14: # and 0g * 0, it easy follows that (i = C2 = and thus K4 = [C2 for any m 2 0.
H.2 RELU LINEAR KERNEL
Theorem In ReLU network, IGI- K2l for any 1 S S The first non-zero [4 11al
happens at m = 9 and has the following form.
5670
K51 - 1Sal
| (a (2 -2z16).
(174)
Recall the definitions,
1 Σ (1())d((A))
f(x) = 2
(175)
(n(x)
2 Σ (i(
(176)
f(x)
7b))
p(x) dx.
(177)
J
Vheo (2t() dt Vis (t)) dt
33

---

Published as a conference paper at ICLR 2024
For ReLU kernel, we had only two eigenfunctions with the form,
[
Aμ
Au
Φ (a)
(2())) =
(178)
Apal
2
||ul|
|
Replacing them in the above definitions yields,
A/
Ap
|
||
||A
|Apal
A
||
1 (A,,æ)
(Ap, Au
[(x)) = 2 AAull
+
AAplll
2
2 AA
||
||>||
Aμ
2 )1
Aμr
)) (AH,, a))
(180)
|
||00|
||Ap||
||Aμ||
|Api|
||ppl
||0||
Au
)1 1) + 1 (Ap, x)
1+
(181)
8 A
||00|
||u
||
||a||
Aju
1(A (Ap,, x))
=
Apul
+
||=|||
||Ap|]
+ 2 |
(182)
and,
||||||
1+
Aje TART
BAu
1+
TAATT
BAR
TAPT
Apt
12
(Ap,,
(Ap, BAY
(B(
(183)
Aμ
Aull
2
2 |AU
AAN
o BA
(1+
Asa ||ll
(1+
BA
An
1 (An.
(AN BA
(184)
Auul
||ll|
|B
|
AA
AAN
It is easy to obtain,
(xxx)
(
1 |
(1+
Aμ
))
+
(Ap, σ)
(185)
Ayal
|
||
2
AApl
||a|||
A/
[Ap, x))
2
(1+
(186)
4 ||p
||=||
||Ap|]|
|Apll
and therefore,
[ (())p(z)
2 ||Al|]
|||
Ap
Ap
12 )
2(2 (Ap, An1)
(1+
*187)
||Apu|l' ||Aμ||
||Ap||
= (1+1 +
1
(188)
1.
(189)
in a similar way, one can compute fgn (x()((( dx. Plugging these into the definition of y yields,
7(b)
2 (a +3
2v B8
(b) bb2) 10b, 150
65
(190)
It is messy but straightforward write the derivatives w.r.t. by and by evaluated at b 62 = 1 (which gives (1 and (2) to see that for any derivative of order < 8 the above yields (1 C2 and at m one obtains,
5670
G (aa
|lulal 18 (4102)a)2)) -2212)
(191)
I QUADRATIC APPROXIMATION VS RELU KERNEL
The paper presents an analytical form that characterizes the trade-off between predictivity and availability in single-layer ReLU model for classifying two Gaussian sources.
In order to keep the theoretical analysis tractable, we the following approximations. First, we used asymptotic approximation to the covariance matrix by having the size of the covariance going
34

---

Published as a conference paper at ICLR 2024
Quadratic
ReLU
Figure 1.1: Plot of sign((Ka] - K22 (a1 az)) for ReLU and Quadratic NTK models trained on 50 training samples from positive (μ, C) and negative N -1, C)) classes. Here, (M1, 12),, covariance is C o² where μ 1 and 0.01. Top and bottom figures respectively correspond to μ22 μ 0.1.
to zero. Second, use replaced the ReLU kernel function by a quadratic approximation. A natural question is, whether the predictions made by theory sensitive these approximations. More precisely, if we work with the actual ReLU kernel and a real covariance matrix, whether observe similar predictions.
We verify this here by learning a model from finite training data in the NTK regime. The latter amounts to performing kernel regression with kernel specified by the NTK. The result of the simulation is provided in Figure I.1 and confirms that the approximations used in the theory are not affecting the actual predictions, in the sense that it matches the trend of the predictions made by the theory from Figure 5.
The left panel in Figure 1.1 still uses quadratic kernel. However, it is generated by actual predictions from model trained by finite training data, whose distribution is real covariance (as opposed to an asymptotically small one). This confirms the closed-form expression derived from our theory (predictions from Figure 5) match the simulation result.
The right panel of Figure 1.1 goes further and replaces the quadratic kernel by the actual ReLU kernel. Observe that this does not affect the trend of the predictions, hence showing there is not much lost by switching between the ReLU kernel and quadratic approximation.
NUM POINTS 20
Number of points used to plot the result (larger better but slower)
N TRAIN SAMPLES
[__TRANNSSAPLEE
50
Number of training points per
Gaussian source
USE_QUAD
True
# When to use ReLU or its quadratic
approximation
MU_1
1.0
MU List
[10, 0.1]
SCALE_COM
0.0001
# Scale parameter s from the paper Abs [COR] must be less than Epsilon identity added before
COR
0.0
EPS
SCALE_COV/
/100
4 inverting kernel matrix
DELTA
0.05
# delta to approximate derivatives
with finite different
import numpy as np from matplotlib import pyplot as plt
def kernel (x1, x2)
norm x1=mp.------- sqrt (np. dot (x1,x1)) faam_x2=np.se (np. dot (x2, x2))
u=np.dot (x1/norm_xl) x2 20/norm__x)
35

---

Published as a conference paper at ICLR 2024
if np. abs (u) >1.0:
u-np.sign sign
if USE QUAD:
h=815.0/3011 0* (1 + x**x2
else:
h- (u* (np. paapp--pppp.. np arccos (u) inp. =+=++22) /np.pp
return norm xorm_x1=norm_x22
def
(A, Y, X, kernel_func)
do linear ridge regression to obtain coefficients c_ to 2_2n
crmp.empty.. (22=N((22A))))) TRAIN SAMPLES, TRAIN SAMPLES))
for i in range (2+N_ TRAIN SAMPLES)
for j in range 12=N_TRRAII SE___TRRINSSAMPPPPPS SAMPLES)
c = <[i.j]=kernel_fune (Aex [i] A@x (j1)
C-N_TRRIN_SAMPLES) ,y)
compute prediction vector
np. (2*N_TRAIN TRAIN SAMPLES, 2*N_TRAIN (AMPL_S)
for i in range (2+N_TRAIN TRAIN T S_MPLAEMMMS for range TTRAN_SAMPLERS
func (A8X [i] Aex()])
return K_pred
### MAIN 4## sensitivity
пр. empty ( (len [MU____iiaa NUU_PILTTPPOINTSS NUM_PIOTTFOIINTS)
for 2J_ indx in range (len (MD_2_nist))
MU 2 _U_ List
2__stiist [MU_ [MU_2_indx]
generate 2d-gaussian data (n samples per class) for two
~ classes: x0 and
wean_closs_00Dp.arrayMMM)M MU MU]
mean_class._1-meenncclss....
cop-SSClE__oovmyy.arrra(((1.1., COR), [COR, 1.01])
rng np. mgg.random.ddfaaullt_nnggg
x0 attg.aautttaaarrttteooaaaaaaaee (mean_class_)) cov,
4 BLieeN_TRANNSSMPPPP
y0 np. ze.zeros (N W__TRAINN_SMPLEE3
x1 = mnng.multiivarriat_nnmmme
cov,
cize=N__RAAN_SAMMILA
yl = pp. oneones (N_TRAIN_GAMPLES)
x=ap concatenate (x0,x1), x1), axis=0)
y= concatenate (y0, yl), axis=0) X=x
for i_al range (NUM_
print ( "'completionn",i_ala11 "of' PUM_PIIOT)) al (4.0x (i_aa+1)) WWTM_MMLOTPPOIINCPP
for a_a2 in range (NUM (NUM_PLOT, PLOT_ POINTS):
a2 (4.0+ (4..0-1__222)))MUU_Pp0LL.0000000 /NUM__POTTPOOIITTTT
(=mp.diagg(a1.:2))
pred = kernel_predict (A, x, kernel) x,,
A_Perturbed
[DELTAA abs (al), 01, []]
pred_inc_al = kernel_predict (A (A_rerrurbe,, kernel)
A.Perturbed A 1 DEb. [a2)
pred_inc__d = kernel_predice (A_Perturbed, x, X, kekeeel)
normalize prediction vectors
36

---

Published as a conference paper at ICLR 2024
pred pred/ap. (np. dot (pred, pred)
pred_inc__d
(np. dot (pred inc al, pred pred inc_in_____ all)
pred_inc_a22 =
sensitivity setsttiiitty_L1_2___aa___LLaLLL | - np.sign (
np. np. dnp (pred, -11 /
(DELTA. abs
dot (pred, pred_inc_aapeee a2) -1.0)
(DELTA+ap.co abs (a2)) (al-a2)
Plot Sensitivity
xv, yv = meshrid (mp. linspace (0, 4, NUM_PHOT_POINTS) 4 np.linspace (0, 4, NUM WUUM P_OT__PIITSS)
fig, (ax1, ax2) plt.subplots
axl.pcolormest (xv, yv, [0n) vmin -1, vmax 1);
5x2.pcolorment (xv, yv, sensitivity[1], vmin -1, vmax 1);
axl stl.xet__ox__spect (1)
sel.set_box_aspect (1) A.t.show ()
37


Reviews:
[{"review_id": "XSM1qXN2iv", "reviewer": "Reviewer_QCsi", "summary": "This paper studies why shortcut learning happens. It focuses on two characteristics that input features may have: availability and predictivity. Availability refers to how frequently that type of feature is available in the data, while predictivity means how useful it is in predicting the target. Shortcuts are described as overly relying on available features that are less predictive. Based on this interpretation of shortcuts, the paper experimentally shows on synthetic and natural image datasets that availability explains shortcut learning where predictivity cannot. These findings are supplemented with a theoretical analysis that concludes that adding a single hidden layer already biases the model to rely on shortcuts.", "questions": "The paper points out that availability can explain (some) occurrences of shortcut learning. The availability can be determined by looking at the training data distribution, but could one also identify these shortcuts stemming from availability by directly looking at the train neural network weights? For example, if multiple filters in the same CNN layer share the same pattern?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and the main message that availability is a key driver to shortcut learning is convincingly conveyed.\n\nBoth empirical support and theoretical support are provided to underline the effect of availability on shortcut learning.", "weaknesses": "The theoretical analysis is limited to a single hidden layer MLP which does not reflect the type of architectures used in practice. It is difficult to conclude whether this also holds for other types of architectures like Transformers or CNNs.\n\nExperiments are limited to supervised classification settings. Self-supervised training or other tasks like object detection would be interesting to consider in the context of shortcut learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eKgVo54Ef8", "reviewer": "Reviewer_mfV7", "summary": "The paper investigates the learning of shortcut / spurious features. Mainly, the paper looks at the interplay between predictability and availability (that defines how easy it is for a model to extract a feature). First, experiments on simple, synthetic data, generated from two variables for which we can control the predictability and availability are shown. A notion of shortcut bias is defined as the additional reliance on spurious features for a learned model, compared to an optimal model. It is shown that availability determines the learning of spurious features, even when the predictivity of spurious features is lower than that of the core features. Non-linearities are shown to induce more bias for shortcuts. Theoretical analysis shows that linear networks are not biased to feature availability while ReLU networks are. Experiments on image datasets show that they are biased for background and object size.", "questions": "Could the curves given by making an intervention on the background, be use to benchmark the robustness of different method? E.g. more robust models would have curves that are less sensitive to interventions on the background. Whould this offer any additional insight as opposed to comparing the accuracy of the model on balanced data, without foreground-background spurious correlations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- S1. The paper deals with an important aspect of understanding neural networks, the bias for learning shortcuts.\n\n- S2. Using the notion of shortcut bias, based on the reliance of an optimal predictor is a good idea.\n\n- S3. Analysing based on a notion of availability, that can be computed produces some good observations. \n\n- S4. Interesting to see that theoretically, ReLU networks are more biased than linear networks.", "weaknesses": "- W1. The notion of availability is very closely connected with the concept of simplicity of neural networks. The simplicity bias has been pointed out as a cause of non-robust learning. The paper gives multiple references (e.g. Shah et al., 2020, etc.) for works dealing with simplicity bias, but they are not discussed in detail. The authors must clarify what is the difference between the notion of availability presented in this work, and the simplicity bias previously introduced. What new observations does the proposed framework bring?\n\n\n- W2.The experiments are not that strong. The image datasets do not seem to bring any interesting observations. It is already known that the background influences the prediction to a high degree. It is expected that alterating the background will improve the reliance on the core features. All methods that use these datasets, try to learn how to not rely on the background. There doesn’t seem to be any novel observation in this regard. \n\n- W3. For image datasets: “a Bayes optimal classifier is not comparably sensitive to the predictivity of the non-core features” How do we know this? What experiments give this conclusion? \n\n- W3.2 Also, how is the Bayes optimal classifier created in the case of real image datasets (WaterBirds, CelebA)?\n\n- W4. The paper defined a shortcut as a feature that is more available, but less predictive. Differently, a shortcut is usually defined by good predictivity in distribution, but poor predictivity in some other distributions (OOD).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GiQWCqnNnH", "reviewer": "Reviewer_KL3Q", "summary": "The paper studies shortcut learning, i.e., why networks use to learn shortcut/spurious features over intended semantic features. The authors suggest that the network prefers a feature that is more available to quantify shortcut bias in terms of how a learned classifier deviates from an optimal classifier in its feature reliance. The authors study the neural network preference toward shortcut features using synthetic datasets with varying predictability and availability of the shortcut features. They empirically observed that networks prefer to learn shortcut features when they are more available, and ReLU is biased toward shortcuts. They also theoretically show using the NTK that linear networks are less biased towards the shortcut compared to the ReLU networks.", "questions": "* Do you think the observations will be similar for other non-linearities?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and easy to follow. The problem of shortcut learning, which is not extensively studied, is important to understand.\n\n* The paper presents interesting insights into neural networks’ preference toward shortcuts. The paper studies shortcut learning empirically using controlled datasets and theoretically using the NTK.\n\n* The derivation for the bias of linear and ReLU networks using NTK would be helpful for future work in this domain.", "weaknesses": "* The main observation that the model depth and non-linearity increase bias towards the shortcut features is intuitive. Both depth and non-linearity allows the model to learn a rich representation.\n\n* Theoretical analysis using NTK is problematic as they don't learn feature and thus cannot necessarily explain model's preference towards the shortcut feature. \n\n* Only vision tasks are explored in the paper, it is not clear if the observations will hold true for other domains.\n\n* It would be interesting to see experiments with the vision transformers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J711UT4PUu", "reviewer": "Reviewer_G1nt", "summary": "This paper presents a systematic study on the trade-off between predictivity and availability in deep learning, proposing a theoretical foundation for shortcut learning in neural networks. The key contributions are:\n1.The paper introduces quantitative notions of predictivity and availability using a generative model to synthesize classification datasets where two latent features vary in terms of these attributes. A measure of \"shortcut bias\" is proposed to quantify a model's over-reliance on more available but less predictive features.\n2.Through controlled experiments, it is shown that nonlinear models exhibit greater shortcut bias compared to linear models, and model depth amplifies this effect.\n3.A theoretical analysis based on Neural Tangent Kernels proves the inevitability of availability bias for nonlinear architectures like ReLU networks, while linear networks are unbiased.\n4.Experiments on natural image datasets demonstrate that widely used vision models are sensitive to availability, not just predictivity, of non-core features. Explicit availability manipulations of images are shown to alter models' reliance on different features.\nTaken together, the empirical and theoretical findings reveal shortcut learning as an inherent characteristic of nonlinear deep models that needs to be studied systematically. The framework presented lays the groundwork for further investigating architectural choices, dataset factors and methods to mitigate shortcut bias.\n5.Overall, this is a thorough, well-executed study that makes important theoretical and practical contributions towards understanding shortcut learning in deep neural networks. The paper is clearly written and the empirical methodology is sound. The theorems and proofs rigorously analyze the interplay between predictive value and feature availability. This work provides key insights into model failures related to over-reliance on spuriously predictive features, and tools to improve model robustness.", "questions": "Here are some questions and suggestions to further improve the paper:\n\n- The quadratic approximation for the ReLU kernel greatly simplified the theoretical analysis. Could you provide some empirical verification that this does not alter the core findings? For example, compare the availability bias for the true ReLU kernel versus the quadratic approximation.\n\n- Have you experimented with any architectural manipulations beyond depth and activation functions that could potentially reduce shortcut bias? Things like skip connections, normalization layers etc. Exploring this could provide practical guidelines. \n\n- For the image experiments, it would be interesting to also show the impact of availability manipulations on a model pretrained on Imagenet. Does pretraining affect sensitivity to shortcuts?\n\n- The measures of reliance and bias involve probing a model in latent space. For vision experiments, could you provide some visualizations of model decisions before/after availability manipulations to build intuition?\n\n- The notion of availability is central but still somewhat conceptual. Are there any additional experiments you could do to further validate the manipulations proposed to affect availability?\n\nI hope these suggestions help further refine the work. The key results seem solid and demonstrate the importance of availability bias. Some additional experiments and discussion along the lines above could make it even more convincing and applicable across domains. I look forward to the authors' response.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The proposed generative framework for synthesizing datasets with controllable levels of predictivity and availability is creative and enables controlled experimentation.\n2.The empirical methodology is thorough and sound. The datasets, models, and evaluation metrics are carefully designed. Results are reported over multiple random seeds to ensure significance.\n3.Shortcut learning is a pivotal concept in deep learning with implications for generalization and fairness. This work significantly advances our understanding of why models fail in this manner.\nIn summary, this is a paper of exceptional quality and scientific merit that offers significant theoretical and practical value to the field. The novel concepts, thorough empirics, and rigorous theory set a new standard and open up many promising research directions.", "weaknesses": "This is an good paper overall, but a few minor weaknesses could be addressed to further improve it:\n\n- The theoretical analysis makes approximations to obtain tractability (small covariance, quadratic approximation of ReLU kernel). It would strengthen the analysis to discuss the impact of these approximations. Are the core insights still valid without them?\n\n- The proposed notion of availability is intuitively reasonable but remains a hypothesis. Additional ablation studies that directly validate the choice of manipulations affecting availability could make this more concrete.\n\n- The measures of reliance and shortcut bias, though well-motivated, are indirectly quantified through alignment with idealized classifiers. More analysis could be provided to justify these metrics over alternatives.\n\n- There is scope for further investigating architectural manipulations that could mitigate shortcut bias, beyond just model depth and activation function. This could lead to practical guidelines.\n\nOverall these are minor limitations that do not diminish the quality of the work. The paper thoroughly delivers on its core promises. Addressing the above points where possible would make it even stronger. The work clearly advances our understanding of an important problem and provides a solid foundation for reducing shortcut reliance in deep learning models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ki5MwsFCp4", "reviewer": "Reviewer_gtPN", "summary": "This work studies the relationship between predictivity and availability of input features, and how a model depends on them. The work starts out with a 2D synthetic scenario where these two dimensions are explicitly characterised. In this scenario, they find that models can rely on more available features, in spite of others being more predictive, and that model depth and nonlinearity increases this effect. The authors further test a synthetic, high-dimensional image dataset and natural images which are manipulated. They find that the number of pixels which correspond to shortcut features, which is viewed as availability of a feature, can lead to the model relying more on it, even when said feature is less predictive. They also provide theoretical results relating to the Neural Tangent Kernel.", "questions": "In section 3, the two main parameters that govern availability are $\\alpha_i$ and $eta_i$. $\\eta_i=0$, so $\\alpha_i$ remains, and its ratios are manipulated. Could you please explain your intuition why $\\alpha_i$ controls “how easily the feature can be extracted, or leveraged, from inputs”? Why is a feature with a large latent amplitude more readily produced by a neural network? This important premise of your experiment in characterising availability is unclear to me. Any further evidence that makes this clearer would be appreciated.\n\nI don’t fully understand why your definition of reliance is useful. $\\hat{y}_\\mathcal{M}(z)$ is the “binary classification decision”, i.e. -1 or 1. Then, when taking the expected value over z, I don’t understand why taking a sign difference of z_s and z_c is reasonable.\n\nThe novelty of this work relative to related work, which I am not very familiar with, is unclear to me and not discussed in the paper. Could you please delineate your work, and clarify your novel contributions, or point me to appropriate positions in the paper which describe this, if missed? \n\nI would be interested in more complex synthetic scenarios than the one outlined. Did you think about these? How could the robustness of your conclusions in the synthetic scenario be tested?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "This work is a creative way of understanding shortcut learning, and I enjoyed reading the paper. The paper clearly defines central concepts and terms, and quantifies them. The experimental analyses are likewise rigorous, plentiful, and well-supported, even though I have important concerns which are listed below. The paper is well-written, mathematically precise, and clear. Overall, this paper will be of interest to the research community.", "weaknesses": "The work has several weaknesses, and I have the following concerns: \n\n* The novelty of the work is unclear (see my comment below).\n* The synthetic experiment in section 3 is—as much as I enjoyed reading it—highly constructed. It is a good opener for the work, and it draws intuitive, story-supporting conclusions, but I would like to ask the question to which degree the stated conclusions will generalise to high-dimensional datasets in the wild, and generally to which degree these conclusions are generally true. I recognise that this a first step towards this goal, but further evidence or explanation on why these conclusions may be generally true would be useful. Also, it should be considered if section 3 should be given less prominence.\n* The naturalistic data experiment in section 7 is most questionable to me. In my view, decreasing the “availability” of the background image by setting its entropy to 0 (black color) is directly biasing the classifier to learn from the remaining available, foreground signal where you know the label is highly predictive. The observed relationship of higher model accuracy is hence non surprise. The same applies to the other scenarios looked at.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)