Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamics
Authors: Nihal Murali
Aahlad Manas Puli
Ke Yu
Rajesh Ranganath
kayhan Batmanghelich
Venue: Accepted by TMLR ()

Abstract: Deep Neural Networks (DNNs) are prone to learning spurious features that correlate with the label during training but are irrelevant to the learning problem. This hurts model generalization and poses problems when deploying them in safety-critical applications. This paper aims to better understand the effects of spurious features through the lens of the learning dynamics of the internal neurons during the training process. We make the following observations: (1) While previous works highlight the harmful effects of spurious features on the generalization ability of DNNs, we emphasize that not all spurious features are harmful. Spurious features can be "benign" or "harmful" depending on whether they are "harder" or "easier" to learn than the core features for a given model. This definition is model and dataset dependent. (2) We build upon this premise and use instance difficulty methods (like Prediction Depth) to quantify "easiness" for a given model and to identify this behavior during the training phase. (3) We empirically show that the harmful spurious features can be detected by observing the learning dynamics of the DNN's early layers. In other words, easy features learned by the initial layers of a DNN early during the training can (potentially) hurt model generalization. We verify our claims on medical and vision datasets, both simulated and real, and justify the empirical success of our hypothesis by showing the theoretical connections between Prediction Depth and information-theoretic concepts like $\mathcal{V}$-usable information. Lastly, our experiments show that monitoring only accuracy during training (as is common in machine learning pipelines) is insufficient to detect spurious features. We, therefore, highlight the need for monitoring early training dynamics using suitable instance difficulty metrics.

Published in Transactions on Machine Learning Research (10/2023)
Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamics
Nihal Murali
nihal. edu
Intelligent Systems Program University of Pitlsburgh
Aahlad Puli
edu
Department Computer Science New York University
Ke Yu
yu.ke@pitte edu
Intelligent Systems Program University of Pittsburgh
Rajesh Ranganath
mnjeshr@iims. nyu. edu
Department of Computer Science New York University
Kayhan Batmanghelich
butman Wru.. edu
Deportment of Electrical and Computer Engineering Boston University
Reviewed OpenReview: https:// openrrveeee...v.eee. net/ ttr/ffrmm? id= Tioumt 9rDmm
Abstract
Deep Neural Networks (DNNs) are prone to learning spurious features that correlate with the label during training but irrelevant to the learning problem. This hurts model generalization and poses problems when deploying them in safety-critical applications. This paper aims to better understand the effects of spurious features through the lens of the learning dynamics of the internal neurons during the training process. We make the following observations: (1) While previous works highlight the harmful effects of spurious features on the generalization ability DNNs, we emphasize that not all spurious features are harmful. Spurious features can be " "benign" or "harmful" depending on whether they are "harder" or "easier" to learn than the core features for a given model. This definition is model and dataset dependent. (2) We build upon this premisc and use instance difficulty methods (like Prediction Depth (Baldock et al.] (021)) to quantify "easiness" for given model and to identify this behavior during the training phase. (3) We empirically show that the harmful spurious features can be detected by observing the learning dynamics of the DNN's early layers. In other words, easy features learned by the initial layers of a DNN early during the training can (potentially) hurt model generalization. We verify our claims on medical and vision datasets, both simulated and real, and justify the empirical success of our hypothesis by showing the theoretical connections between Prediction Depth and concepts like V-usable information (Ethayarajh et al. 202T Lastly, our experiments show that monitoring only accuracy during training (as is common in machine learning pipelines) is insufficient to detect spurious features. We, therefore, highlight the need for monitoring early training dynamics using suitable instance difficulty metrics.

---

Published in Transactions on Machine Learning Research (10/2023)
1 Introduction
DNNs tend to rely on spurious features even in the presence of core features that generalize well, which poses scrious problems when deploying them in safety-critical applications such as finance, healtheare, and autonomous driving (Geirhos et al.)
2020 Oakden-Rayuer et al. 2020 DeGrave et al., 2021 A feature is termed spurious is correlated with the label during training but is irrelevant to the learning prob- lem (Saab et al. 2022; Izmailov et al., 2022). Previous works use a distribution shift approach to explain this
phenomenon (Kirichenko et al. 2022 Wiles et al., 2021; Bellamy et al., 2022 Adnan et al. 2022). However, we get additional insights by emphasizing on the learnability and difficulty of these features. We find that not all spurious fcatures are harmful. Spurions features can be "benign" or "harmful" depending upon their difficulty with respect to signals that generalize well. We show how monitoring example difficulty metries like Prediction Depth (PD) (Baldock et al., 2021) can reveal the harmful spurious features quite early during training. Early detection of such features important as it can help develop intervention schemes to fix the problem early. To the best of our knowledge, we are the first to use the training dynamics of the model to detect spurious feature learning-
The premises that support our hypothesis are as follows: (P1) Spurious features hurt generalization only when they are "easier" to learn than the core features (see Fig. (P2) Initial layers of a DNN tend to learn easy features, whereas the later layers tend to learn the harder ones (Zeiler & Fergus 2014 Baldock iet al. 2021). ). (P3) Easy features are learned much earlier than the harder ones during training (Mangalam & Prabhul 2019 Rahaman et al. 2019). Premises (P1-3) lead us to conjecture that: "Monitoring the easy features learned by the initial layers of DNN early during the training can help identify the harmful spurious features."
We make the following observations. First, we show that spurious features can be benign (harmful) depending upon whether they are more challenging (easier) to learn than the core features. Second, we empirically show that our hypothesis works well on medical and vision datascts (sections Peections4.210 4.2 A.3). both simulated and real, regardless of the DNN architecture used. We justily this empirical success by theoretically connecting prediction depth with ceammaaiinnthhooreee (sections 3 A. Lastly, our experiments highlight that monitoring only accuracy during training, as is common in machine learning pipelines, is insufficient to detect spurious features. In addition, we need to monitor the learning dynamics of the model using suitable instance difficulty metrics to detect harmful spurious features (section-433 reliable models that do not rely on spurious features.
concepts like V-usable information (Ethayarajh et al. [022))
4.3 This will not only save time and computational costs, but also help develop
2 Related Work
Not all spurious features hurt generalization: Geirhos et al. (2200) deffne spurious features as features that exist in standard benchmarks but fail to hold in more challenging test conditions. Wiles et al. (2021) view spurious feature learning as a distribution shift problem where two or more attributes are correlated during training but are independent in the test data. [Bellayy et al. 2022) use causal diagrams to explain spurious correlations as features that are correlated with the label during training but not during meptoyment.
All these papers characterize spurious correlations purely as a consequence of distribution shift; methods
exist to build models robust to such shifts (Arjovsky et al. 2019 Krueger et al. 2021 Puli et al. 2022) The distribution shift viewpoint cannot distinguish between benign and harmful spurious features. In contrast, we stress the learnability and difficulty of spurious features, which helps separate the benign spurious features from the harmful ones (see Fig.D.. Previous works like (Shah et al. (2020); Scimeca et al. (2021) hint at this by saying that DNNs are biased towards simple solutions, and Dagaev et al (2021) I use the 1oo--godd-to-b true" prior to emphasize that simple solutions are unlikely to be valid across contexts. Veitch et al. (2021) distinguish various model features using tools from causality and stress test the models for counterfactual invariance. Other works in natural language inference, visual question answering, and action recognition, also assume that simple solutions could be harmful in nature (Sanh et al., 2020 Li & Vasconcelos] 2019 Clark et al. 2019 Cadene et al. 2019 He et al.) 2019). We take this line of thought further by hypothesizing that simple solutions or, more explicitly, easy features, which affect the carly training dynamics of the model can be harmful in nature. We suggest. using suitable example difficulty metrics measure this effect.
2

---

Published in Transactions on Machine Learning Research (10/2023)
Training Data w/ Spurious Correlation
Testing Data w/o Spurious Correlation
13
15
7
8
y
y
8
7
Scenario 1: Difficult Spurious Feature (prime number)
S
L
S
L
8
7
Common Test Data (no Spurious Feature)
Prediction
Data Generation
Scenario 2: Easy Spurious Feature (white patch)
(A)
(B)
Figure 1: An illustration of how the distribution shift viewpoint. cannot distinguish between the benign and the harmful spurious features. This viewpoint suggests that training and testing are different graphical models between input (z), output (v), and the spurious feature (s). If z can predict s, and is not correlated to on the test data, then is viewed as spurious feature. (A) The figure shows two scenarios for even-ode oven-odd classification. Scenario shows a dataset where all even numbers have a spurious composite number (located at the top-left), and odd numbers have prime number. Scenario 2 shows a dataset where all odd numbers have a spurious white patch. The spurious white patch is harmful in nature it is an easy feature that the the test data. Whereas classifying prime numbers, shown
model can learn, causing poor performance in scenario 1, is challenging. So the model ignores such benign spurious features that do not affect test data performance. This shows not all spurious features hurt generalization.
Estimating Example Difficulty: There different metrics in the literature for mcasuring instance- specific difficulty (Agarwal et. al. 2022 Hooker et al. 2019 Lalor et al. 2018). Jiang et al. (2020) train many models on data subsets of varying sizes to estimate a consistency score that captures the probabil- ity of predicting the true label for a particular example. Toneva et al. (2018) define example difficulty as the minimum number iterations needed for a particular example to be predicted correctly in all subse- quent iterations. |Agarwal et al. (2022) propose a VoG (variance-of--raddinnt) score which captures example difficulty by averaging the pre-softmax activation gradients across training checkpoints and image pixels. Feldman & Zhang (2020) use statistical viewpoint of mcasuring example difficulty and develop influence functions to estimate the actual leave-one-out influences for various examples. Ethayarajh et al (2021) use an information-theorede approach to propose a metric called pointwisc V-usable information (PVI) to compute example difficulty. [Blldock et al. (2021) define prediction depth (PD) as the minimum number of layers required by the DNN to classify a given input and use this to compute instance difficulty. In our experiments, we use the PD metric to provide proof of concept for our hypothesis.
Monitoring Training Dynamics: Other works that monitor training dynamics have significantly dif- ferent goal than ours. While monitor training dynamics to detect the harmful spurious features, Rabanser et al. 2022) uses neural network training dynamics for selective classification. They use the disagreement between the ground truth label and the intermediate model predictions to reject examples and obtain a lavorable accuracy /couraccc trade-off. Feng & Tu (2021) use a statistical physics-based approach to si udy the training dynamics of stochastic gradient descent (SGD). While they study the effect of mislabeled data on SGD training dynamics, we study the effect of spurious features on the carly-time learning dynamics of DNNs. Hu et al. (2020) use the early training dynamics of neural networks to show that simple linear model can often mimic the learning of two-layer fully connected neural net nerw Adnan et al. (0022 have
3

---

Published in Transactions on Machine Learning Research (10/2023)
similar goal as ours and use mutual information to monitor spurions feature learning. However, computing mutual information is intractable for high-dimensional data, and hence their work is only limited to infinite- width neural networks that offer tractable bounds for this computation. Our work, on the contrary, is more general and holds for different neural network architectures and datasets.
3 Background and Methodology
In this section, we formalize the notion of spurious features, task difficulty, and harmful vs benign features. Let Ptr and Pte be the training and test distributions defined over the random variables X (input), y (label), and s (latent spurious feature).
Spurious Feature (s): A latent feature is called spurious if it is correlated with label y in the training data but not in the test data. Specifically, the joint probability distributions Por and Pte can be factorized as follows.
Per (x = P.r s. Pr Pie (x (s)
The variable s appears to be related to but is not. This is shown in F We also introduce notation for task difficulty. The difficulty a task depends on the model and data distribution (X.y).
Task Difficulty (V): Let V (X,,)) y) indicate the difficulty of predicting label y from input for a model M, such that X,y ~ P.
We give examples of two metrics that can be used for measuring TM-
Example-1 for DM (Prediction Depth): Baldock et al. (2021) proposed the notion of Prediction Depth (PD) to estimate example difficulty. The PD of input is defined as the minimum number of layers the model requires to classify the imput. The lower the PD of input, the casier it is to classify. It is computed by building k-NN classifiers on the embedding layers of the model, and the earliest layer after which all subsequent k-NN predictions remain the same is the PD of the input. Figure illustrates how we compute prediction depth. PD can be mathematically defined as follows:
PD(x) = min fkin (3r) = Sknn (x);; k}
(1)
k
feom is the k-NN classifier (see Appendix-A.0 A.6 for details), ø² is the feature embedding for the given input at layer-i, and N is the index of the final layer of the model. We also use the notion undefined PD to work with models that are not fully trained. We treat k-NN predictions close to 0.5 (for a binary classification setting) as invalid. 3 illustrates how to read the PD plots used in our experiment.
Example-2 for M (V-Usable Information): The Mutual Information between input and output, Y), is invariant with respect to lossless encryption of the input, i.c., I(r(X);Y) = (X;Y). Such definition assumes unbounded computation and is counter-intuitive to define task difficulty as heavy encryption of X does not change the task difficulty. The notion of "Usable Information" introduced by Xu et al. (2200) assumes bounded computation based on the model family V under consideration. Usable information is measured under a framework called predictive V-information (Xu et al. 2020). Ethayarajh et al. 2021) introduce pointwise V-information (PVT) for measuring example difficulty.
PVI(x + y) = loog, 1ogg (v) + logz g [x),
(2)
8.t. 9.9' E
The function g is trained on (o.y) input-label pairs, where o is a mull input that provides no information about the label y. g' is trained on (x. y) pairs from the training data. Lower PVI instances are harder for V and vico-versa. Since the first term in Eq corresponding to g is independent of the input x, we only consider the second term having g' in our experiments.
4

---

Published in Transactions on Machine Learning Research (10/2023)
conv2
conv3
convé
CONVS
1000
14 512
28 28 5122
7x7 512
3
Input Query
x12x 128
PD( x 4
224 224
Class-1 (Cats) Class-2 (Dogs)
Figure 2: The prediction depth of image for given model tells us the minimum number of layers needed by the model to classify the image. A subset of the training data is used to collect feature embeddings at each layer. The KNN classifier uses these feature embeddings at each layer to classify the input query. Prediction depth for the input query is the number of layers after which the KNN predictions converge.
Harmful Spurious Feature: The spurious feature is harmful for model M iff V Pr (X, We use the causal operator "do(s)" denote causal intervention on the spurious feature to break the correlation between spurious features and the labels. This equivalent removing the arrow from y to s shown in Figure In other words, if removing the spurious correlation between and from the training data makes it harder for model M to predict the true label y, then the spurious feature is "harmful".
The above formalization clarifies that a spurious feature may be harmful to one model but benign for another, as its definition closely tied to the dataset, model, and task. Figure illustrates this point.
In what follows, we relate the notions of PD and V-usable information. We use Venn (of finite depth and width) in our proof experiments mainly CNN architectures.
Proposition (Informal) Consider two datasets: D. (X,( with spurious features and D. y)) without them. For some mild assumptions on PD (sce Appendix(A.I). if the mean PD of D2 is less than the mean PD of Di, then the Dran-usable-inoommate for D. larger than the www--uusse-ooommatiinn for Dj: I (X Y) IP: ()-
See proof in Appendix A.1 The proposition intuitively implies that a sufficient gap between the mean PD of spurious and core features can result in spurious features having more Venn-usable information than core features. In such a scenario, the model will be more inclined to learn spurious features instcad of core ones. This proposition justifies using the PD metric detect spurious features during training, as demonstrated in the following experiments.
5

---

Published in Transactions on Machine Learning Research (10/2023)
Epoch
Epoch
Epoch
No. Images
400
No. of Images
400
No. of Images
400
samples hierating
200
200
200
11... mll.
0
80
120
0
120
0
40
80
120
Layer
Layer
Layer
undefined PD
Easy samples in early layers
challenging samples in later layers
Training Progress
Figure 3: Examples of PD plots (for DenseNec-121) at different stages of the training process. The red bar indicates samples with undefined PD, and the dotted vertical line indicates the mean PD of the plot. Notice that the undefined samples (shown in red) slowly accumulate in layer 88 as training progresses. This is because the model needs more time to learn the challenging samples which accunulate at higher prediction depth, i.e. later layers.
Training Data
y = 0 (even digits)
4
8
2
Testing Data
4
y y= 1 (odd digits)
7
9
Case 1: Deep Set
Case 2: MLP
Extract Patches
Image Resize
Prediction = 1 (odd)
Prediction = (even)
Odd digit (test data)
Odd digit (test data)
One model's food is another model's poison! A spurious fcature may be benign for model but harmful for another. Consider the task of classifying digits into "even" "odd" classes. All even (odd) digits in training data have a spurious white patch in the bottom left (right) corner. This correlation doesn's 't exist in the test data as the spurious white patch is randomly placed. Case-1 shows the Deep Set (Zaheer et al. 2017) model, which patchifics the input image, processes each patch separately, and combines the features. The model's output invariant to the location of the white patch. However, the MT.P model shown case-2 is location-sensitive: hence, it learns Consequently, the MLP model is susceptible to learning such spurious correlations and performs poorly on the test data. So the spurious white patch is benign for Deep Sets but harmful for the MLP model.
Figure 4:
associate the location of the spurious white patch with the label.

---

Published in Transactions on Machine Learning Research (10/2023)
4 Experiments
We set up four experiments to evaluate our hypothesis. First, we consider two kinds of datasets, one where the spurious feature is easier than the core feature for ResNet-18 model and another where the spurious feature is harder than the core for the same model. We train classifier each dataset and observe that the ResNet-18 model can learn the easy spurious feature but not the harder one. This experiment demonstrates that not all spurions features hurt generalization, but only those spurious features that are easier for the model than the core features are harmfull Second, we use medical and vision datasets to demonstrate how monitoring the lcarning dynamics of the initial layers can reveal harmful spurious features. We show that an early peak in the PD plot may correspond to harmful spurious features and must raise suspicion. Visualization techniques like grad-CAM can provide intuition about what feature is being used at that layer. Third, we show how harmful spurious features can often be detected relatively carly during training, This is because initial layers which learn such spurious features converge very early during the training, We observe this by monitoring PD plots across training epochs. In all of our experiments, the PD plot reveals the spurious feature within two epochs of training. Fourth, we show that datasets with easy spurious features have more "usable information" (Ethayarajh et al.] 2021) compared to their counterparts without such features. Due to higher usable information, the model requires fewer layers to classify the images with spurious features. We use this experiment to empirically justify Proposition-1 outlined in Appendix-A..] We also conduct additional experiments (see Appendix-A.10 to justify the use of the PD metric in our work.
CIF FMN
MN FMN
Table I: Results for the Dominoes experiment averaged across 4- runs. Numbers in bracket show mean-PD (datasel difficulty). Core-only accuracy indicates the model's reliance on core features. Models achieve high core-only accuracy when spurious features are harder than core features.
Spurious
Core
Dataset
Is spurious harder than core?
Validation Acc.
Test Acc.
(Spurious-orre
(Core-only)
(IF(6.8) FM.N(339)
yos
99, 12-1.7-727%%
96.05-+0.30%
Hard Spurious
Fasy Spurious
Core-only (Test Image)
MN(2.2) FMN(3.9)
DO
90.96.10.05%
50.75-12.96%
CIF(6 8)- KMN(S)
yts
16.991
96. 1.0%
MN(2.2) KMN(5)
09.97:t0.05%
50.48-+..64%%
CIF(6.8) MN(2.2)
yes
00.74:+0007%
90.5++0.66%
Figure 5: Dominoes Dataset
KMNDsteh((1.1) MN(2.2)
99.97:20.044%
68. 78%
4.1
Not all spurious features hurt generalization!
We use the Dominoes binary classification dataset (formed by concatenating two datasets vertically; see Fig-4) similar to the setup of Kirichenko al.] (2022). The bottom (top) image acts as the core (spurious) feature. Images are of size 64 32. We construct three pairs of domino datasets such that each pair has both hard and an easy spurious feature with respect to the common core feature (see Table-1). We use classes } for MNIST and SVHN, {ress, for FMNIST, and {airplane, automobile] for CIFAR1O. We also include two classes from KIzushijiMMNIST (or KMNIST) and construct a modification of this dataset called KMNpatch, which has a spurious patch feature (5x5 white patch on the top-left corner) for one of the two classes of KMNIST. The spurious features are perfectly correlated with the target. The order of dataset difficulty based on the mean-PD is as follows: KMNpatch(1.1) < MNIST((.2) < FMNIST(3.9) < KMNIST(5) < (5H) < CIFAR10(6.8). We use ResNet18 model and measure the validation and core-only accuracios. The validation accuracy is measured on held-out dataset sampled from the same distribution.. For the core-only (test) accuracy, we blank out the spurious feature (top-half image) by replacing it with zeroes (same as Kirichenko et al. (2022)). ). The higher the core-only accuracy, the lesser the model's reliance on the spurious feature.
Table shows that all datasets achieve high validation accuracy as expected. The core-only accuracy stays high (>98%) for datasets where the spurious feature is harder to learn than the core feature, indicating the
7

---

Published in Transactions on Machine Learning Research (10/2023)
(A) MNIST
(B) FMNIST
(C) CIFAR 10
PD Plot (VGG16)
80
PD Plot (V6G16)
8o
PD Plot (VGG16)
/
3
of imagees
of Images
40
80
80
2 5
0 Laver
12
76
%
16
8 Layer
12
16
Layer
Low
High
Axis of Difficulty
PD Plot (VGG16)
PD Piot (VGG16)
80
No. of Images
17
No. of images
60
60
40
20
%
12
16
GradCAM Layer-1
Layar
12
16
(D) KMNIST w/ Patch
(E) KMNIST w/o Patch
Figure 6: Top row shows three reference datasets and their corresponding prediction depth (PD) plots. The datasets are ordered based on their difficulty (measured using mean PD shown by dotted vertical lines): KMN w/ patch(1.1) M21ST(2.2) < FMMNIST(3.9) < KMN w/o patch(5) < SVHN < CIFAR10(6.8). The bottom row shows the effect of the spurious patch the KMNIST dataset. The yellow region the axis indicates the expected difficulty of classifying KMNIST. While the original KMNIST lies in the yellow region, the spurious patch significantly reduces the task difficulty. The Grad-CAM shows that the model focuses on the spurious patch.
model's high reliance on core features. When the spurious feature is casier than the core, the model learns to leverage them, and hence the core-only accuracy drops to nearly random chance (~50%). Interestingly, the KMNpatch-MMY results have a much higher core-only accuracy (~69%) and more significant standard deviation. We explain this phenomenon in Appendix A.7 This experiment demonstrates that not all spurious features hurt gencralization, and only those that are casier than core features are harmful.
4.2 Monitoring Initial Layers Can Reveal the Harmful Spurious Features
Synthetic Spurious Feature in Toy Dataset: To provide a proof of concept, we demonstrate our method on the Kuzushjj-MNNSS (KMNIST) (Clanuwat et al. 2018) dataset comprising Japanose Hiragana characters. The dataset has ten classes and images of size 28 x 28, similar to MNIST. We insert white patch (spurious feature) at particular location for each of the ten classes. The location of the patch is class-specific. We train two VGG16 models, one on the KMNIST with spurious patch (Man) and another on the original KMNIST without the patch (Morig).
Fig. shows the prediction depth (PD) plots for this experiment. The vertical dashed lines show the mean PD for each plot. FD shows that KMNIST with patch has a much lower mean PD than all the other datasets, and the PD plot is also highly skewed towards the initial layers. This suspicious behavior is because the white patch is very easy feature, and hence the model only needs single layer to detect it. The Grad- CAM maps for layer-1 confirm this by showing that Mah focuses mainly on the patch (sce Fig-6D), and hence the lest accuracy on the original KMNIST images is very low (~8%). The PD plot for Morig (see 6E) is not as skewed toward lower depth as the plot for Mak. This is expected as Morig is not looking at the spurious patch and therefore utilizes more layers to make the prediction. The mean PD for Morig suggests that the original KMNIST harder than Rashion-MNISS but casier than CIFAR10. Morig also achieves a higher test accuracy (~98%).
8

---

Published in Transactions on Machine Learning Research (10/2023)
This experiment demonstrates how models that learn spurious features (Mah) exhibit PD plots that are suspiciously skewed towards the initial layers. The skewed PT) plot should raise concerns, and visualization techniques like Grad-CAM can aid our intuition about what spurious feature the model may be utilizing at any given layer.
Semi-Synthetic Spurious Feature in Medical Datasets: We follow the procedure by DeGrave et al. (2021) to create the ChestX-ray14 SaateXXrray4//GHHu-bCOOOO HittHubCCOVID dataset. This dataset comprises Covid19 positive images from Github Covid repositories and negative images from ChestX-ray14 dataset (Wang et al., 2017b addition, we also create the Chex-MIMIC datasct following the procedure by Puli ct al. (2022). This dataset comprises 90% images o Pneumonia from Chexpert (Irvin al. 2019 and 90% healthy images from MIMIC-CXR (Johnson et al., 2019). We train two DenseNet121 models, Meovid the ChestX- ray May14/Gitua SeitHub-COVil dataset, and Mehea the CheX-MIMIC dataset. We use DenseNet121, common and standard architecture for medical image analysis. Images are resized to 512 x 512.
CheX-MIMIC
1enxX-rry1//GiHHHun COVID
PD Plot (Dense121) Grad Cra Layer-4
PD Plot (Dense121)
Grad-CAM Layer
500
200
Images
No. of Images
250
100
40
80
120
0
40
80
120
Layer
Layer
Figure 7: PD plots for two DenscNet-121 models trained Chex-MIMIC and datasets are shown in the figure, along with their corresponding Grad- CAM visualizations. Both PD plots exhibit very high peak in the initial layers (1 to 4), indicating that the models use very easy features to make the predictions.
Fig. 7 shows the PD plots for Meker and Movvid- Both the plots are highly skewed towards initial layers, similar to the KMNIST with spurious patch F. This again indicates that the models are using very easy features to make the predictions, which is concerning as the two tasks (pneumonia and covid19 detection) are hard tasks even for humans. Examining the Grad-CAM maps layer-4 reveals that these models focus on spurious foatures outside the lung region, which irrelevant because both discases are known to affect mainly the lungs. The reason for this suspicious behavior is that, in both these datasets, the healthy and diseased samples have been acquired from two different sources. This creates spurious feature because source-specific attributes or tokens are predictive of the disease and can be easily learned, as pointed out by DeGrave et al. 2021).
Real Spurious Feature in Medical Dataset: For this experiment, we use the NIH dataset (Wag et al. 2017a) which has the popular chest drain spurious feature (for Pneumothorax detection) (Oakden- Rayner et al. 2020). Chest drains are used to treat positive Pneumothorax cases. Therefore, the presence of a chest drain the lung positively correlatod with the presence of Pneumothorax and can be used by the deep learning model (Oakden-Rayy)
al.) 2020). Appendix A.5 A.5 outlines the procedure we use to obtain chest drain annotations for the NIH dataset. We train DenseNet121 model (Mnis) for Pneumothorax detection NIH images of size 128 128.
Fig-SA shows the PD plot for Moih- We observe that the distribution is not as skewed as the plots in the previous experiments. This is because all the images come from a single dataset (NIH). However, we do see suspicious peaks at the initial layers. Pncumothorax classification is challenging even for radiologists; hence, peaks at the initial layers raise suspicion. The Grad-CAM maps Figs-B & & reveal that the initial layers look at irrelevant artifacts and chest drains in the image. This provides evidence that the initial layers
9

---

Published in Transactions on Machine Learning Research (10/2023)
NIH
Prediction Depth (2e
Grad-CAM (artifacts)
Grad- (chesssdraa))
Chestdrain Influence on
Performance
200
1.0
No. of Images
0.8
150
0.6
100
0.4
(AUC 0.84)
50
(AUC
No Tube IAUC 0.74)
0
0.0
40
80
120
0.00
0.25
0.50
0.75 1.00
Layer
(A)
(B)
(C)
(D)
Figure Spurious correlations in NTH dataset. (A) PD plot for DenseNet-121 trained on NTH shows prominent peaks in the initial layers. (B, C) Grad-CAM reveals that the initial layers use irrelevant and spurious artifacts and chest drains for classification. (D) The chest drain spurious feature affects the AUC performance of the model. The X-axis (Y-axis) shows the false positive (true positive) rate.
looking at the harmful spurious features. Fig-8D shows that the AUC performance is 0.94 when the diseased patients have a chest drain and 0.74 when they don't. In both cases, the set of healthy patients remains the same. This observation consistent with the findings of Dakden-Rayner et al. (0220) and indicates that the model looks at chest drains to classify positive Pneumothorax cases.
The above experiments demonstrate how a peak located in the initial layers of the PD plot should raise suspicion, especially when the classification task is challenging. Visualization techniques like Grad-CAM can further aid our intuition and help identify harmful spurious fcatures being learned by the model. This approach works well even for realistic scenarios and challenging spurious features (like chest drain for Pneu- mothorax classification), as shown above. Appendix A.3 includes additional results on vision datasets.
4.3 Detecting Harmful Spurious Features Early
Fig 9 shows the evolution of the PD plot across epochs for Mash (which is the model used in Fig This visualization helps us observe the training dynamics the various layers. The red bar in the PD plots shows the samples with undefined prediction depths.
These plots reveal several useful insights into the learning dynamics of the model. Firstly, we see three prominent peaks in epoch-1 at 1aycrs-4,40,88 (see Fig. The magnitude of the initial peaks (like layers- 4&:40) remains nearly constant throughout the training. These peaks correspond to spurious features, discussed in the previous section. This indicates that the harmful spurious features can often be identified early (epoch-1 1 in this case). Fig shows the PD plots at epoch-2 for other datasets with spurious features. It is clear from Fig 10 that the suspiciously high peak at the initial layer visible in the second epoch itself. The Grad-CAM maps reveal that this layer looks at irrelevant artifacts the dataset. This behavior is seen in all datasets shown in Fig
Secondly, also that accuracy and AUC plots not sufficient to detect spurious features during training. We need to monitor the training dynamics using suitable metrics (like PD) to detect this behavior, Thirdly, the red peak (undefined samples) decreases in magnitude with time, and we see proportional increase in the layer-88 peak. This corroborates well with the observation that later layers take more time to converge (Rahaman et al.) 2019 Mangalam Prabhu 2019 Zeiler & Fergus 2014). Therefore, samples with higher PD are initially undefined and do not appear the PD plot. Nonetheless, samples with lower PD show up very early during the training, which helps us detect the harmful spurious features early. Early detection can consequently help develop intervention schemes that fix the problem early.
10

---

Published in Transactions on Machine Learning Research (10/2023)
Epoch
Epoch
Epoch
Epoch
600
No. of Images
NIH Training
400
0.84
peak AUC
(B)
200
0.82
(B)
D
0
40
1aalt Indl.
80
120
40
80
Mal.
120
0
40
80
120
40
80
120
AUC
0.80
(H))
Layer
Layer
Layer
Layer
(A)
(B)
(C)
(D)
0.78
Epoch 113
Epoch 16
Epoch 19
Epoch 22
600
No. of Images
0.76
400
Validation
(A)
0.74
5
15 Epoch
10
20
25
200
%
40
80
120
0
40
11100
80
126
40
80
120
0
40
80
120
Layer
Layer
Layer
Layer
(E)
(F)
(G)
(II)
Training Progress
Figure 9: Evolution of PD plot across epochs shows the training dynamics of the DNN on the NIH dataset. The initial peaks (layers-4&:40) are relatively stable throughout training, whereas the later peaks (layer-88) change with time. The initial layers learn spurious features, which can detected early during the training. Samples with undefined PD (shown in red) take more time converge and eventually accumulate in the later layers (layer 88 in this case).
KMNIST w/ Patch
Chex-MIMIC
ChestXray 4/GitHub-COVID
PD Piot (VGG16)
GradCAM Laver-]
Plot (Dense121)
Grad-CAM Laver-4
100
PD Plot (Dense121) Grad-CAM Layer-4
of Imbous
120
No. Images
800
No. of Images
83
50
200
00
40
125
0
Layer
Figure 10: Epoch-2 PD plots for various datasets with spurious features. The high spurious peak in the initial layer is visible all the datasets indicating that the harmful spurious features can detected early during the training.
4.4 Prediction Depth ss V-Usable Information
Table-2 measures the influence of spurious features on NIH and KMNIST using PD and PVI metrics. All diseased patients in the "NIH w/ Spurious feat." dataset have chest drain, whereas all diseased patients in the "NIH w/o Spurious feat." dataset have no chest drain. The set of healthy patients is common for the two We use VGG16 for KMNIST
datasets. The KMNIST datasets are the same as those used in Section 42
and DenseNett12 for NIH. Other training details are the same as Section
4.2
Table-Bshows that datasets with spurious features (D₂) have smaller mean PD values than their counterparts without such features (D,). Proposition-1 (see Section 3 Appendix(A.1) shows that sufficient gap between the mean PDs of D, and D, causes the Y-Information of D,, to be greater than D,. Table confirms this
11

---

Published in Transactions on Machine Learning Research (10/2023)
Table Effect Spurious features on Prediction Depth and the negative conditional V-entropy V-Hv (Y
X)). The label marginal distributions are the same with or without the spurious feature, and thus the negative conditional V-entropy is proportional to V-information.
Dataset
PD
Hvonn (Y (X)
NTH w/ Spurions feat.
53.43
-0.1171
NIH w/o Spurious feat.
75.33
-0,2321
KMNIST w/ Spurious feat.
1.06
-0.0024
KMNIST w/o Spurious feat. 5.25
-0.0585
in a medical-imaging dataset with a real chest drain spurious feature, and we see that the mean "usable information" increases when there is spurious feature. This implics that the model learns spurious features as they have more usable information than the core features. We investigate this relationship between PD and V-information in more detail in Appendix A.4 Ethayarajh et al. (2021) also show that V-information is positively correlated with test accuracy. This explains the significant change in AUC observed in Fig DD. Proposition bridges the gap between the notions PD and V-usable information. This connection between V-information and PD indicates that monitoring early training dynamics using PD not only helps detect the harmful spurious fcatures but also bears insights into the datasct's 's difficulty (in information-thcoreee terms) for given model class.
5 Conclusion
We study spurious features by monitoring the early training dynamics of deep neural networks (DNNs). We empirically show that not all spurious fcatures hurt generalization, but only those that are casier than the core features do. So spurious features can be "benign" or "harmful" depending on whether they are harder or easier than core features for a given model. Our hypothesis is: "Movitoring the easy features learned by the initial layers of a DNN early during the training can help identify harmful spurious features." We validate this hypothesis real medical and vision datasets. We show that spurious features are learned quite early during the training and one can detect them by monitoring the early training of DNNs using instance difficulty metrics like Prediction Depth (PD). Further, we show a thcoretical conncction between PD and V-information to support our empirical results. Datasets with spurious features have more V-information as compared to their counterparts without such features causing the model to learn the spurious feature. To conclude, relying only on accuracy plots insufficient for detecting harmful spurious features, and we recommend monitoring the DNN training dynamics using instance difficulty metrics like PD for the early detection of such features.
References
Mohammed Adnan, Yani loannou, Chuan- Yung Tsai, Angus Galloway, HR Tizhoosh, and Graham W Taylor. Monitoring shortcut learning using mutual information. arXiv preprint ar Xiv. 13034, 2022.
Chirag Agarwal, Daniel D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 10368 10378, 2022.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint crXio:1907.02893 2019.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876 10889, 2021.
David Bellamy, Miguel A Hernán, and Andrew Beam. A structural characterization of shortcut features for prediction. Europeon Journal of Fpidemiology, 37(6):563 568, 2022.
12

---

Published in Transactions on Machine Learning Research (10/2023)
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems, 32, 2019.
Kamalika Chaudhuri and Sanjoy Dasgupta, Rates of convergence for nearest neighbor classification. Advances in Neural Information Processing Systems, 27, 2014.
Tarin Clamwwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literalure. arXiv preprint ar N8w:1812.01118 2018
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv:1909.0300 2019.
Nikolay Dagaev, Brett D Roads, Xiaoliang Luo, Daniel N Barry, Kaustubh R Patil, and Bradley C Love. A too-good-to-be-true prior to reduce shorlcut reliance, arXiv preprinl arXiv:2102.06400 2021.
Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 19 detection selects shortcuts over signal. Nature Machine Intelligence, 377):610- 2021.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Information-thoorrti measures of dataset diffi- culty. aiviv preprint (XKx:2110.08442 2021.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881 1219112211111 2020.
Yu Feng and Yuhai Tu. Phases of learning dynamics in artificial neural networks in the absence presence of mislabeled data. Machine Learning: Science and Technology, 2(4))043001, 2021.
Robert Geirhos, Henn-k Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665-66) 673, 2020.
Ingo Gühhinng Mones Raslan, and Gitta Kutyniok. Expressivity of deep neural networks. ar preprint arX1v:2007.04759) 2020.
He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias in natural language inference by fitting the residual. arXiu preprint ar arXic:1908 10763, 2019.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do compressed deep neural networks forget? arXiv preprint 1XX::1911.05244 2019.
Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the carly-time learning dynamics of neural networks. Advances in Neural Information Processing Systems, 33:17116- 17128, 2020.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurca-Ileus, Chris Chute, Henrik Mark- hun, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAT conference on ortificial intel- ligence, volume 33, pp. 590 597, 2019.
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature lcarning in the presence of spurions correlations. Advances in Neural Information Processing Systems, 35:38516-388 1-5322,, 2022.
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew Lungren, Andrew Y Ng, et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint xrxiv:2106 14463, 2021.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXio preprint ar arXi::2002.03200 2020.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Minic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXi::991.00044 2019.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson Last layer re-training sufficient for robustness to spurious correlations. arXie preprint orXiv:2204, 02937, 2022.
David Krueger, Ethan Caballero, Joern-Henrik Jacobson, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 2815-58200 PMLR, 2021.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
John P Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu. Understanding deep learning performance through an examination of test set difficulty: A psychometric case study. In Proceedings of the Confer- ence on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2018, pp. 4711. NIH Public Access, 2018.
Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 9572 9581, 2019.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? 2019.
Jishmu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deep deterministic uncortainty: A new simple bascline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 24384 24394, 2023.
Luke Maldden-Rayeer, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures machine learning for medical imaging. In Proceedings of the ACM confer- ence on health, inference, and learning, pp. 151 119, 2020.
Aahlad Manas Puli, Lily H Zhang, Eric Karl Oermann, and Rajesh Ranganath. Out-of-distribution general- ization in the presence of muisanceindduced spurious correlations. In International Conference on Learning Representations, 2022.
Stephan Rabanser, Anvith Thudi, Kimia, Hamidieh, Adam Dziedzic, and Nicolas Papernot. Selective classi-
fication via neural network training dynamics. arXiv preprint wwXX::2205.13555 2022.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 5301 5310. PMLR, 2019.
Khaled Saab, Sarah Hooper, Mayee Chen, Michael Zhang, Daniel Rubin, and Christopher Ré. Reducing reliance spurious features in medical image classification with spatial specificity. 2022.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. Learning from others' mistakes: Avoiding dataset biases without modeling them. arXiv preprint arXiv:2012. 01300, 2020.
Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shorteut cues will dnns choose? study from the parameter-space perspective. arXiv preprint arXiv:21 100::210000000 10.03095, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishck Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEFE international conference on compuler vision, pp. 618-626, 626, 2017.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateck Jain, and Pranecth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Newral Information Processing Systems, 2020.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Ge- offrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprinl arXio:1812.05153. 2018.
Joost Van Amersfoort, Lewis Smith, Yee Whye Tch, and Yarin Gal. Uncertainty estimation using a sin- gle deep deterministic neural network. In International conference machine learning, pp- 1690-97000 PMLR, 2020.
Victor Veitch, Alexander D'Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious corrclations: Why and how pass stress tests. arXiv preprint arXiv:2106. 0rXiv:2006.00555, 2021.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and local- ization of common thorax diseases. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017a.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and local- ization of common thorax discases. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097 2097-2106, 2017b.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebufi, Ira Ktena, Taylan Cemgil, et al. A fine-grained analysis on distribution shift. arXiv preprint arXi::2110.11398, 2021.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint 1066:20022.1689, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexan- der Smola. Deep scts. Advances neural information processing systems, 30, 2017.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 113. Springer, 2014.
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Nico Towards better benchmarking for domain generalization. Xiv preprind (xXiv:224.080// 2022.
15

---

Published in Transactions on Machine Learning Research (10/2023)
A Appendix
A.1 Proof of Proposition-1
Proposition A.1. Given two datasets, Dg with spurious features and D; without them, we assume the following:
1. (Well-Trained Trained Model Assumption) The part of the network from any representation to the label is one of the functions that compute V-information.
2. (Function Class Complexity Assumption) Assume that there erists K {1,N} such that Venn of depth N K is deep enough to be a strictly larger function class than Venn with fixed neighbor size (29 in this paper). Assume that this Venn is larger function class than linear function.
(Controlled Confidence Growth Assumption) For both datasets D (Do, assume that the for all & E ( (1,"".,N,,
T T Venn (0x) IRamn (dk-11) S
4- (Prediction Depth Separation Assumption) Let. L, be on integer such that, T. S K and I. < N v max, log = =)) Note that p(Y = y) is simply the prevalence of class y. Let there exist a gap in prediction depths of samples in D, and D,: vp E (0, 0.5) such that 1- fraction of D, has prediction depth L and 1 1- - fraction of D, has prediction depth > K.
Then, for model class of N -laayer CNNs, we show that the Venn lin-infoommaton for D2 is greater than Venn information for Dj:
D. (X
I D. (X -1)
Proof. Before proceeding the proof, we attempt justify and reason about the above assumptions.
Assumption-1 states property of trained neural networks in the context of usable information. Let f(X) be a trained neural network. Consider splitting the network into the representation du (X) at the h layer and the rest of the network as function applied to ox i.e,, f(X) = fa o (x(X). Then we assume that fi(.)) is the function that achieves the Venn of sieen-Aj-informaatoon between (x( and Y (Ethayarajh et al. 2021 Xu et al., 2020). The function that computes V-information must achieve a minimum cross-entropy (Ethayarajh et al. 2021). So if we train f(X) by minimizing the cross-entropy loss, fx(()) must converge function that achieves the Venn of size(n-k) size(--k--ifoormmtte -iformmaion between @u (X) and Y.
Assumption-2 implies that the CNN class Venn deep enough such that the network after the Kth layer can approximate a k-NN classifier with 29 neighbors; (K here is same as the K in assumption-4) This is also reasonable assumption (Chaudhuri & Dasgupta, 2014 Gühring al. 2020). Chaudhuri & Dasgupta 2014) lower bounds the error for k--N classifiers for fixed k. and Gühring et al. 2020 shows the depth expressivity of CNN classifiers. Assumption-3 states that the difference in V infe-imafomtiion between intermediate layers does not explode indefinitely and thus can be bounded by some positive quantities J and e.
Assumption-4 is also casily satisfied. For example, the smallest prevalence class in the dataset has a prevalence greater than 1000 then assumption-4 boils down to saying N 0.5 max, l--oop(( y)) N 3.445 where L is the low PD value caused by spurious features in D,, and N is the total number of layers 1000 constraint. Even discases like pneumothorax which are rare, have class prevalence of at least 1 in both NIH and MIMIC-CXR. And L < N - 3.45 is easily satisfied in all our experiments. For e.g., see Fig-5 where 80% (or - 1b = 0.8) of the samples have PD < 16. So L = 16, N = 121(for Densenet-121) easily satisfies 16 < 121 3.45.
in the CNN. All our datasets satisfy the class prevalence > 1
30
Now we claborate on the proof of the proposition given the four assumptions. We proceed in two parts: first, lower bound tnn-iiformation for D., and then upper bound Venn for D,.
16

---

Published in Transactions on Machine Learning Research (10/2023)
Assumption implies:
B Iiim (Φx)
(B A)r - TS Σ= k=A
IRam ( (B--A) (B-A A)c
Note: (A, are just placeholders for the min and max indices over which the summation is defined. They are replaced by (L+ 1, and (K' + N) below while trying to lower bound ID, and upper bound IP: respectively.
PD - PVI connection. Note that by definition, when the prediction depth is k for sample X, then PVIknn (0x (X)) 2 5 but PVI (x)- < 8. This follows from how we compute PD (sce Section 3 in the main paper, and Appendix A.6
Lower bounding IPP....
D
Venn depth N (ok)
[Assumption-1]
Dinn (ok)
(Assumption-2)
K
DD -Ih Vann
(ØL) +
Σ
IPan (Øx) - IBeaa (Ōk-1)
(Tlescoping Sum}
k=1+tl
Venn (ØL) + (K- LDT
[Assumption-3]
2 1 X,Y X,yEDo,dd>=1.
min
PVL (X + Y)
+ (1 - 1))
min X.yED..pdLL
PVI (x+ +Y + (K- D)r
(Prediction Depth Separation]
2 0 v + (1 v) + L)
(Prediction Depth Separation]
Upper bounding Iven
Vonn < IIDr
Denn (NNN)
(Assumption-2)
IP.... (Øx) + Σ
Doan (1xx) - IP (ok-11
(Telescoping Sum}
k=K+1
< Venn (Øk) + (N -K)e
[Assumption-3]
< (N- K)E+ + X,Y X.YS
max
PVID PVIDenn (xx(X)
+ (1 -1-98 +)
max
PVI ( + +Y
[Prediction Depth Separation) }
<N - K)e + + max logoly =))
+ (1 v))
max X
PVI (xK(X) +Y)
(PVI < - log = y)}
< (N- - K)c+ + ab max (- logp(Y = =y) + (/)
[PD-PVI connection for pd > K)
The proof follows by comparing the lower bound on Ve.. and the upper bound on I Intuitively what this means is that when there is a sufficiently large gap in the mean PD between D, and D;, then the V-information of D. exceeds the V-information D,, which is why the model prefers learning the spurious feature instcad of using the core features for the task.
17

---

Published in Transactions on Machine Learning Research (10/2023)
A.2 Grad-CAN Visualization
PD plots help understand the model layers actively used for classifying different images. To further aid our intuition, we visualize the Grad-CAM outputs for the model's arbitrary layer k by attaching soft-KNN head. Let Ienn denote the soft and differentiable version of k-NN. We compute Iknn as follows:
Σ EsENieh exp 110 100--0901 -0//
gknn (on; oi (1.......)))
EjENToOY.. exp -00
This function makes the KNN differentiable and can be used to compute Grad-CAM (Selvaraju et al. 2017) We use the L1 norm for all distance computations. of corresponds to feature al layer-k for query image zyy- Let of (1e(1.2...m)) be the training data for KNN. Let W denote the neighborhood function. N( returns the indices of K-nearest neighbors for 0y N( 1) returns indices of images with positive label (y = 1) from the set of K-nearest neighbors for oz is the median for the set of Li norms {1lo of l1} for j E (N
A.3 Vision Experiments
We use the NICO+ (Non-I.I.D. Image dataset with Contexts) datasct Zhang et al. 2022 to create multiple spurious datasets (Cow vs. Bird; Dog vs. Lizard) such that the context/ background is spuriously correlated with the larget. NICO+ is a Non-I.L.D image dataset that uses context to differentiate between the test and train distributions. This forms an ideal setup to investigate what spurious correlations the model learns during training. We follow the procodure outlined by Puli et al. (2022) create datascts with spurious correlations (90% prevalence) in the training data. The test data has the relationship belween spurious attributes and the true labels flipped. This is similar to the Chex-MIMIC dataset illustrated in section We test our hypothesis using ResNet-18 and VGG16. We train models for 30 epochs using Adam optimizer and a base learning rate of 0.01. We choose the best checkpoint using early stopping.
NICO (Cows vs Birds; Sparious Backgronn)
Prediction Depth (ResNet-18)
402
350
300
200
150
100
Training Data (hoc-77.%)) gazee; Birch wales)
100
16
38
Test Data (lc=3999% waller; Firds grass)
Layer
GradCAM(l.ger-)
(A)
(F3)
Figure 11: Cow vs. Birds classification on NICO++ dataset. (A) Training data contains images of cows on grass and birds on water (correlation strength=0.9). The model achieves 97.2% training accuracy. (B) PD plot for ResNet-18 reveals spurious peak at layer-1, indicating the model's heavy reliance on very simple (potentially spurious) features. (C) GradCAM plots for layer 1 reveal that the model mainly relics on the spurious background to make its predictions. 59.9% on test data where the spurious correlation is flipped (i.e., cows (birds) are found on water (grass)).
(D) Consequently, the model achieves a lest accuracy of only
Figures 11 13 show PD plots and train/test accuracies for models that learn the spurious background feature present in the NICO+ datasct. While all models achieve > 85% training accuracy, they have poor accuracies ( (50%) on the test data where the spurious correlation is flipped. This can be seen simply by observing the PD plots for the model on the training data. The plots are skewed towards the initial layers indicating that the model relics heavily on very simple (potentially spurious) features for the task. GradCAM maps also confirm that the model often focuses on the background context rather than the foreground object of interest.
18

---

Published in Transactions on Machine Learning Research (10/2023)
NICO (Cows vs Bird; Balanced Dataset)
Prediction Depth (ResNet-18)
200
150
100
50
25
Balanced BalancedTraanin Training (Acc(..99)
Datt (1c
Layer
(Cows onwller Birch ongrass)
Figure 12: Balanced dataset for Cow vs. Birds classification task on NICO+ dataset. (A) The training dataset contains a balanced distribution of cows and birds found on water and grass (each group has an equal number of images). (B) The balanced datasel shifts the PD plot lowards the later layers (compared to Fig I1B, indicating that the model relies lesser on spurious features.
(C) This consequently results in an improved test accuracy of 80.2% (as compared to 59.9% in Fig 110 for the spurious datasct).
NICO (Dogvs Lizard; Spurious Background)
Prediction Depth (ResNet-18)
350
300
250
Images
200
150
NO
100
50
Spantous Training (Aco-6722%) (Outcbor Dogs Loarth Rock)
0
0
2
4
10 Layer
8
12
14
16
18
Test Data (Acc=(339%)
(Dggon Rockq Outckor Lizarc))
(A)
Figure 13: Dog vs. Lizard classification with spurious background feature on NICO++ dataset. (A) Training data contains images of outdoor dogs and lizards on rock (correlation strength=0.9). The spurious background color / texture reveals the foreground object. The model achieves 87.2% training accuracy. (B) PD plot for ResNet-18 reveals a spurious peak at layer-1, indicating the model's reliance on simple (poten- tially spurious) features. (C) The low test accuracy confirms this (63.9%). The test data has the spurious correlation flipped (i.e., images contain dogs on rock and lizards found outdoors.)
We further observe in Fig. 12 that balancing the training data (to remove the spurious correlation) results model with improved test accuracy (80.2%) as expected. This is also reflected in the PD plot 12)). where we see that the distribution of the peaks, as well the mean PD, shift proportionately towards the later layers, indicating that the model now relies lesser the spurious features.
19

---

Published in Transactions on Machine Learning Research (10/2023)
By monitoring PD plots during training and using suitable visualization techniques, we show that obtain useful insights about the spurious correlations that the model may be learning. This can also help the user make an educated guess about the generalization behavior of the model during deployment.
A.4 Empirical Relationship: PD vs V-information
In Scction-4.2 we explore the relationship between PD and V-information. To empirically confirm these results, we further investigate this relationship on four additional datasets: KMNIST, FMNIST, SVHN, and CIFARIO. We train a VGG16 model on these datasets for ten epochs using an Adam optimizer and a base learning rate of 0.01. We use a bar plot to show the correlation between PD and V-entropy. We group PD into intervals of size four and compute the mean V-entropy for samples lying in this PD interval.
FMNIST (VGG16)
KMNIST (VGG16)
SVHN (VGG16)
CIFARIO (VGG16)
0.3
Conditional V entropy
Conditional V-entropy
2.5
Conditional V-eebropyy
0-4
233
0.4
0.8
0.2
50
1241
[5-]]
(912)
50
(912) Prediction Depth
[5-8]
[22-2]]
(14)
(540)
16-21 (1348)
134)
15-3
13-21
(1266]
Prediction Depth
Prediction Depth
Prediction Depth
Figure 14: The bar plots show positive correlation between PD and Conditional V-entropy- Samples with higher PD also have higher V-entropy resulting in lower usable information for models like VGG16.
In Section- 4.4 we find that PD is positively correlated with V-information, and the results shown in Fig 14 further confirm this observation. Instance difficulty increases with PD, and the usable information decreases with an increase V-entropy. is, therefore, clear from Fig that samples with a higher difficulty (PD value) have lower usable information, which is not only intuitive but also provides empirical support to Proposition-1 in Appendix
A.5 Chest Drain Annotations for NIH Dataset
al. 2020), need chest drain annotations for the NIH dataset (Wang et al. 2017a), which is not natively provided. To do this, use the MIMIC-CXR dataset (Johnson et al. 2019), which has rich meta-data information in radiology reports. We collaborate with radiologists to identify terms related to Pneumothorax from the MIMIC-CXB reports. These include pigtail catheters, pleural tubes, chest tubes, thoracostomy tubes, etc. We collect chest drain annotations for MIMIC-CXR by parsing the reports for these terms using the RadGraph NLP pipeline (Jain et al. 2021 Using these annotations, we Irain a DenseNet 121 model detect chest drains relevant. to Pneumothorax. Finally, run this trained model on the NIH dataset obtain the needed chest drain annotations. We use these annotations get the results shown in Fig 8 DD, which closcly reproduces the results obtained by Rayner et al. (2200).
To reproduce the results by Oakden-Rayner
A.6 Notion of Undefined Prediction Depth
Section&3 shows how compute PD in experiments. While fully trained models give valid PD values, application requires working with arbitrary deep-learning models that are not necessarily fully trained. We, therefore, introduce the notion of undefined PD by treating k-NN predictions close to 0.5 (for a binary classification set sing) as invalid. We define a 8 such that [snn (x) - 0.5| < 5 implies an invalid k-NN output. We use 0.1 and & = 29 in our experiments. any k-NN predictions for the last three layers are invalid, we treat the PD of the input image be undefined. To work with high-resolution images (like 512 x 512), we downsample the spatial resolution of all training embeddings to 8 before using the k-NN classifiers the intermediate layers. We empirically see that results insensitive k in the range (5, 30).
20

---

Published in Transactions on Machine Learning Research (10/2023)
A.7 A PD Perspective for Feature Learning
Table 1 shows that the core-only accuracy stays high (>98%) for datasets where the spurious feature is harder to learn than the core feature. When the spurious feature is easier than the core, the model learns to leverage them, and hence the corc-only accuracy drops to nearly random chance (~50%). Interostingly, the KMNpatch-MMN results have a much higher core-only accuracy (~69%) and more significant standard deviation. This is because the choice of features that the model chooses to learn depends on the PD distributions of the core and spurious features. We provide three different perspectives on why K.Mpaatc-- MN runs have better results.
PD Distribution Perspective: The KMNpatch-MNN domino dataset has a smaller difference in the core- spurious mean PDs (2.2 - 1.1 = 1); as compared to other datasets (for e.g., MN-KMN has a difference of - 2.2 2.8 in their mean PDs). The closer the PD distributions of the core and spurious features are, the more the model treats them equivalently. Therefore, in the case of the KMNpatch-Mrl we empirically observe that different initializations (random seeds) lead different choices the model makes in terms of or spurious features. This is why the standard deviation of KMNpatch-MN is high (20.03) compared to the other experiments.
Theoretical Perspective (Proposition-l): This is not surprising and, in fact, corroborates quite well with Proposition-1 in Appendix The Prediction Depth Separation Assumption suggests that without sufficient gap in the mean PDs of the core and spurious features, one cannot concretely assert anything about their ordinal relationship in terms of their usable information. In other words, spurious features will have higher usable information (for given model) than the core feat only if the spurious features have a sufficiently lower mean PD as compared to the core features. On the other hand, as the core and spurious features become comparable in terms of their difficulty, the model begins to treat them equivalently.
Loss Landscape Perspective: (this is conjecture; we do not have empérical evidence) The loss land- scape is function of the model and the dataset. The solutions in the landscape that are reachable by the model depend on the optimizer and the training hyperparameters. Given model and set of training hyper- parameters, we conjecture that the diversity (in terms of the features that the model learns during training) of the solutions in the landscape increase as the distance (difference in mean PD) between the core and spurious features decreases. This diversity manifests as the model's choice of using core vs. spurious features and could potentially result in higher standard deviation of core-only accuracy across initializations.
A.8 Code Reproducibility
The code for this project is publicly available https:://gittub.cmm http://hhwa..aaaaaaaaaaaaaaaaa// _anr:3d__ymmmcccc Courious_Feature
A.9 PD Plot Consistency
We perform several random for the experiments shown in Fig-G and compute the probability density function of the PD plots in Fig.G using kernel density estimation (KDE). Fig-15 shows the resulting PD envelopes (computed using KDE), and also the original histograms of different random runs.
We can see the consistency of PD plots for any given dataset across runs involving different random seeds. The overall ordering of the datasets according to difficulty computed by mean PD remains the same. This shows that. PD can be used as a reliable measure to estimate dalaset difficulty and also detect spurious feature.
A.10 Ensemble Uncertainty Measure of Instance Difficulty
The main idea of our paper can be extended to other instance difficulty metrics and visualization techniques and is not limited to PD grad-CAM (see Sec-B 3) We provide a proof of concept in this section by using
21

---

Published in Transactions on Machine Learning Research (10/2023)
(A) MNIST
(B) FMNIST
(C) CIFAR 10
1.0
0.50
Density
Density
Density
0.5
0.25
0.0
0.00
8
12
Layer
Layer
Layer
Low
High
Axis of Difficulty
Density
Density
0.25
0
8 12
16
0.00
Layer
Layer
(D) KMNIST w/ patch
(E) KMNIST w/o patch
Figure 15: Consistency of PD plots across four random runs. The overall ordering of dataset difficulty based on mcan PD remains the same as in Fig-O
techniques different from PD and grad- to detect spurious features. However, there are several added benefits of using sophisticated metrics like PD and we highlight them here.
A.10.1 Toy Dataset (Synthetic Spurious Feature)
We re-run the experiments shown in 6 with new set of metries. In order compute instance difficulty, train ensemble of lincar models over the various datascts shown in Fig. and compute the uncertainty over softmax outputs for various images. The uncertainty of the ensemble can be used proxy for instance difficulty. We compute the uncertainty by taking expectation over softmax entropy (as common in the literature of uncertainty quantification (Lakshminarayanan et al. 2017 Mukhoti et al. 2023 Van Amersfoort et al. 2020g) of the ensemble models, and we use this metric instead of PD. We order the datasets based on mean entropy, and use SHAP (Lundberg Lee 2017 for visualizing the early peaks order to detect spurious fcatures.
Results shown in Fig 16 We find that the overall order of the dataset difficulty is the same in Fig. and the entropy plots also look similar to the PD plots computed previously. The spurious patch in the KMNIST dataset significantly decreases the entropy of the ensemble models expected. We visualize these images using SHAP to find that the model heavily relies on the spurious patch. Upon removing the patch from the KMNIST dataset (see I6E, the mean entropy of the dataset significantly increases as the model now has rely mainly on the core features. We plot the KMNIST images (with spurious patch) along with their SHAP visualizations for various classes in Fig I7 to show how the model significantly relies on the spurious patch for all the classes.
22

---

Published in Transactions on Machine Learning Research (10/2023)
(A) MNIST
(B) FMNIST
(C) CIFAR 10
300
/
3
No. of Imagee
No. of images
2 5
50
50
50
Extray
Entropy
Encragy
Low
High
Axis of Difficulty
130
$7
e
100
$7 7
122
50
Estrar
Entropy
(D) KMNIST w/ patch
(E) KMNIST w/o patch
SHAP
Eusemble uncertainty as a measure of dataset difficulty and SHAP for visualizing spurious features. Top row shows three reference datasets and their corresponding entropy plots. The datasets are ordered based on their difficulty (measured using mean entropy shown by dotted vertical lines): KMN w/ patch(0.107) < (N41T(0..491) < (MNIST((.5277) < KMN w/o patch(1.04) < CIFARI0(1.593). The bottom row shows the effect of the spurious patch on the KMNIST dataset. The yellow region on the axis indicates the expected difficulty of classifying KMNIST. While the original KMNIST lies in the yellow region, the spurious patch significantly reduces the task difficulty. The SHAP plots show that the model focuses on the spurious patch.
Figure 16:
A.10.2 Medical Dataset (Real Spurious Feature)
We now compare the ensemble entropy method with PD on the real medical dataset (NIH) with real spurious features (like chest drain). The setup is exactly the same as in Sec (Real Spurious Feature in Medical Datasct). Additionally, we try to detect the simple artifacts and chest drains (as shown in Fig- using the ensemble entropy method. We increase the model capacity of the ensemble by adding a convolutional layer and a ReL.U layer before the linear classification models detect more complex spurious features (as compared to simple 1-layer linear models as in the previous (ecctionAA.10.1] but also leads to smoother and better grad-CAM plots, which helps us better debug the spurious features the model is using. The setup for PD experiments, however, remains the same as in
Linerr) This not only helps the ensemble
Sec 4.2
Fig 18 shows that both the PD and the ensemble entropy method can detect the simple spurions features in the NTH dataset. However, Fig. shows that the PD method can additionally detect more complex spurious features like chest drains the NIH datasct, whereas the ensemble entropy method is not able to do so as it comprises simple convolutional neural networks that have low model-capacity and can therefore only detect simple spurious artifacts.
To further validate if the simple convolutional model used above can learn chest drains, we set up a simple classification experiment to try to classify if the X-ray image in the MIMIC-CRR dataset. (Johnson el al., 2019) has chest drains/tubes or not using this simple baseline model. We collect chest drain annotations for the MIMIC-CXR datasct (Johnson et al.) 2019) by parsing through radiology reports using the RadGraph NLP pipeline (Jain et al. (2021). We collaborate with radiologists to figure out terms related to Pneumothorax (like pigtail catheters, pleural tubes, chest tubes, thoracostomy tubes, etc.) Using these annotations, we train
23

---

Published in Transactions on Machine Learning Research (10/2023)
Class-1 1
Class-2
Class-3
Class-4
32.
3/2
3
YS²
$
12
ee
52.
Class-5
Class-6
Class-7
Class-8
3
Figure 17: SHAP plots for visualizing what linear models learn on the KMNIST dataset with a spurious patch. The figure shows how the model is significantly relies on the spurious patch for each of the classes shown.
Entropy +
PD GradCAM
Entropy GradCAM
PD GradCAM
GradCAM
Figure 18: Simple spurious artifacts NIH dataset. GradCAM plots show that both methods, entropy and PD, can detoct simple spurious artifacts in the NIH dataset.
the simple convolutional model for 40 epochs, and the model only achieves an AUC of 0.58 (random guessing gives AUC of 0.5). This experiment demonstrates that simple convolutional model is not capable
24

---

Published in Transactions on Machine Learning Research (10/2023)
Entropy + GradCAM
PD GradCAM
Entropy + GradCAM
PD +
GradCAM
Figure 19: Challenging spurious features (like chest drains) in NIH dataset. Grad-CAM plots for the PD method reveal the spurious chest drain in each of the NIH chest X-ray images. Whereas the entropy method, which comprises an ensemble of simple baseline convolutional architectures, cannot detect more complex spurious features like chest drains. This experiment justifies the use of PD metric over ensemble uncertainty of simple basclinc models having low model capacity.
of detecting complex spurious features like chest drains. Therefore, the gradCAM plots for the ensemble entropy method fail to detect chest drains in the NIH dataset (as shown in Fig 19
This section shows that our work is not limited to PD or GradCAM. It is defined for any example difficulty metric (see Sec-3). We use PD as a proof of concept, but our hypothesis simply suggests that by monitoring the example difficulty of training data points and by visualizing the examples (particularly those with low difficulty), we can detect spurious features that hurt model generalization. The better the example difficulty metric and visualization technique, the better we can detect such spurious features. However, there are several additional benefits of using more sophisticated methods like PD, which clearly illustrated in this section (see Fig 19). PD measures sample difficulty in model-specific manner. It takes the model architecture into account, and we show why this is important in Fig.1 The ensemble of lincar models approach shown here not model-speeific. It comprises simple linear baseline models and hence cannot detect more challenging spurious features like chest drains. Using sophisticated techniques like PD, one can also attempt to broadly identify the layers responsible for learning spurious features. This can further help develop suitable intervention schemes that remove the spurious feature representations from those layers. This promising future direction to address the issue of unlearning spurious features. One may ensemble of multi-layer neural networks to compute uncertainty over samples, but training ensemble of larger models involves significant computational overhead and is time-consuning, Additionally, we develop concepts like usable information (see
useful theoretical connections between PD and information-theoree Appendix-A.11 to explain the empirical success we obtain in our experiments. The above benefits justify our use of the PD metric in this paper.
25


Reviews:
[{"review_id": "7dkZY2I1Hc", "reviewer": "Reviewer_4fZW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper is very well written, polished and an enjoyable read.\n\n- The figures are of high quality, and highlight the results well.\n\n- The difficulty of various datasets, mostly synthetic, with one real-world\n  experiment, is demonstrated very well.\n\n- Given the multiple recent works claiming all spurious correlations are\n  supposedly *shortcuts*, this work challenges those over-hasty assumptions,\n  which is an important contribution to the field.\n\n- The work motivates the important habit of checking for *shortcuts*, which is\n  not reflected in the test accuracy alone.\n\n### Weaknesses\n\n- (critical) The work does not report error bars or number of trials. Reporting\n  these is critical to ensure the statistical significance of the empirical\n  experiments.\n\n- (critical) The work verifies the hypothesis using the (somewhat complex)\n  prediction depth. In the experiments, the main indicator for *shortcuts* is\n  the number of samples with a PD of 1, i.e., the number of samples that can be\n  classified with kNN after a single linear transformation (plus non-linear\n  activation?). The empirical results would be stronger with a\n  weaker, linear baseline to measure the instance difficulty, somewhat similar\n  to He et. al (2019), which will most likely also find the static artifact\n  shortcuts in the synthetic toy datasets.\n\n- (non-critical) The Prediction Depth assumes that the data points are\n  transformed in feature space such that they lie close when they will\n  ultimately be classified as the same class. For the first (linear) layer and\n  simple static artifacts, this may be (intuitively) true. There is no\n  guarantee that this is true for higher layers. This is arguably an issue of\n  PD, which is used as a proof-of-concept approach to measure instance\n  difficulty. Given that this is the only approach used (so far), it would be\n  helpful if this would be discussed, maybe in the Appendix A.1.\n\n- (non-critical) Definition 3 defines spurious correlations as shortcuts if\n  feature $s$ is strictly *easier* than the class-feature. This definition\n  becomes somewhat challenging when assuming that classes may be activated by\n  features of various difficulties, where the spurious feature lies within the\n  least and most difficult class-defining features. This is somewhat going into\n  the direction of the transposed statement \"Not all shortcuts are spurious\n  correlations\" (this is obviously not true given the definition of shortcuts\n  in this work).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates *shortcuts*, which describe a particular kind of spurious\ncorrelation which dominates the prediction for models over class-relevant features.\nBased on the statement of related work that \"all spurious correlations are\nshortcuts\", the work shows that spurious correlations do not qualify as\n*shortcuts* when they do not provide a pattern which is easier to learn than\nthe class-relevant features.\nA method from previous work (Prediction Depth) is used as a proof-of-concept to\nempirically quantify the instance difficulty over datasets.\nUsing PD, the work empirically shows that \"easy\" features are learned within\nthe first two epochs of the training.\nA formal connection between PD and v-usable information from another previous\nwork is demonstrated, backed with some empirical results.\nThis work highlights the importance to monitor the instance difficulty over datasets to detect shortcuts.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (critical) The work must report the number of trials and error bars to\n  support the claim of statistical significance.\n\n- (critical) A linear baseline (similar to He et al. 2019) to quantify the\n  instance difficulty should be used to provide an additional, even simpler\n  instance difficulty metric (and justify the use of the more complex\n  PD-metric).\n\n- (non-critical) The class-relevant locality required for the assumption of kNN\n  in higher layer feature space would be helpful if discussed.\n\n- (non-critical) A discussion on the case where class-relevant features of\n  various difficulties may be an interesting addition.\n\n### Misc. (non-critical):\n\n- While I am happy the paper comes with code, for the sake of reproducibility I\n  would urge the authors to structure their code as a Python module using\n  `setuptools`, without relying too heavily on jupyter notebooks. Adhering to\n  some code styling (PEP8, e.g. via black) and keeping to some docstring-style\n  (googledoc, numpydoc, etc.) does wonders for readability and will thus boost\n  reproducibility and accelerate subsequent research.\n\n- (figure-4) (sections-4.2,A.3) looks somewhat odd\n\n- Def2: Let ... indicate~~s~~\n\n- The text captions of the PD figures is a little too small in all figures after Fig. 2\n", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IauCldEvRP", "reviewer": "Reviewer_vHSj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Overall the experiments are quite interesting and have a reasonable mix of datasets. \n- The figures present the papers result reasonably well and are easier to follow than the text.\n- The general point that it is important to consider the learnability of spurious features is valid.\n\n**Weaknesses**\n- **Lack of clarity in definitions and analysis.** The authors make the following definition of shortcuts: spurious features that are easy to learn. Unfortunately this definition rests crucial on the definition of easiness to learn and is highly model dependent.\n- **PD as definition of difficulty.** First, this definition has no dependence on the labels, which seems odd. Even trivial models could return a PD of 1. For example, imagine that the first embedding layer is just a constant function. This would be true regardless of how intuitively “difficult” a task is. The inline equation is also confusing. There is a minimization and maximization, but only one variable (n) is used. This should be rewritten, since you want the minimum n that maximizes the function.\n- **Model dependence.** PD depends intricately on the architecture of the neural network. Indeed, it seems highly likely that changing the architecture could lead to contradictory results, where two dataset have a different ordering on their mean PD for two different architectures.\n- **Poorly written.** The analysis is quite hard to follow and contains many circular claims. For example, the line “This experiment demonstrates that not all spurious correlations are shortcuts, but only those spurious features that are easier than the core features are potential shortcuts.” Under the authors own terms, this is just the definition of a shortcut and not something that should be empirically tested.\n- **Definition 2.** This is not a definition, it simply introduces notation. In particular, “difficulty of predicting” is not defined and neither is “model.” What does “model” refer to here? Is this a model class, a trained model, etc?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops a distinction between shortcuts and spurious correlations in general, by emphasizing the dependence of shortcuts on learnability. The authors present empirical and theoretical analyses to support these claims. To formalize learnability, the authors start by introducing the notion of Prediction Depth (PD), which states an example is easier if it can be classified by the earlier embedding layers in a neural network. From here, the authors state that a dataset or task is easier if its mean PD is lower. Obviously, these definitions depend intricately on the model class. \n\nThe connection back to features from PD is given by examining the histogram of PDs over the dataset. The authors argue that a peak in this histogram at earlier layers suggests the existence of shortcut features. By examining these histograms over the course of training, the authors claim these shortcut features are learned early in training since the peaks in the histogram at early layers remains constant. Finally a theoretical connection between PD and V-usable information is made.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I feel the paper needs significant rewriting. As discussed above in the Strengths and Weaknesses section, there are major difficulties with the definitions used. These lead to potentially circular claims and an analysis of results that I found difficult to follow.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z9PhV4Mr9B", "reviewer": "Reviewer_BeHk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper's topic holds interest and relevance across multiple domains within machine learning, such as adversarial learning and semi-supervised learning.\n\n2. To validate their hypothesis, the authors conduct experiments on extensive datasets.\n\nWeaknesses:\n\n1. Given that they are termed 'shortcuts', it seems intuitive that spurious features would be less challenging than core features. The inherent suggestion in their names already implies this meaning, so the first claim might not provide particularly meaningful insights.\n\n2. The definition-1 is different from \"Shortcuts are spurious features that perform well on standard benchmarks … real-world setting\". Standard benchmarks should include test set. Should the distribution deviate substantially from the test set, the results would evidently indicate the necessity to investigate these spurious features further. I think the challenge lies in identifying spurious features that exist in both training and test datasets.\n\n3. The aforementioned issue brings up further questions that challenge the second and third claims made by the authors. Are all spurious features relatively simple patterns, like 'block' used in the experiments? Could there exist spurious features with more complex patterns that remain easier to learn than the core features? If so, these features should be learned in the later layers and during later phases of learning. Therefore, while Prediction Depth (PD) might be useful for identifying evident spurious features, its utility for detecting such features in real-world problems may be limited. For instance, is PD capable of identifying any spurious features in databases such as ImageNet or COCO?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this study, the authors discovered that spurious features that are harder than the core features are shortcuts. They also found that Prediction Depth serves as an effective metric for detecting spurious features. Moreover, their findings revealed that shortcuts are learnt by early layers and in the early learning phrase. Lastly, the authors underscore the importance of monitoring early training training with Prediction Depth.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. fonts in some figures are too small to see.\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers found the initial manuscript, though handling interesting problem, unclear and lacking important experiments to support the claims of the paper. Authors in discussion with the reviewers updated the manuscript including important additional experiments and rewriting the paper to make it more understandable. In words of one of the reviewer from discussion, "Given the updated manuscript with a clearer definition of the material, as well as the added analysis with the alternative, simple model-based instance difficulty and the multiple trials for the PD-plots, I feel confident that this manuscript is in a good shape and sufficiently supports its claims. It motivates the importance of checking at least for obvious spurious correlations picked up by the model, which the test accuracy does not reflect, and thus provides a sufficient contribution to the field."

I am happy to recommend acceptance of the current draft with minor revision to add more references as mentioned during the discussion phase.
Decision: Accept (poster)