Shortcut Learning Through the Lens of Early Training Dynamics
Authors: Nihal Murali
Aahlad Manas Puli
Ke Yu
Rajesh Ranganath
kayhan Batmanghelich
Venue: Submitted to ICLR 2023 (2023)
Keywords: shortcut learning
spurious correlations
convolutional neural networks
deep learning
machine learning
computer vision
training dynamics

Abstract: Deep Neural Networks (DNNs) are prone to learn shortcut patterns that damage the generalization of the DNN during deployment. Shortcut learning is concerning, particularly when the DNNs are applied to safety-critical domains. This paper aims to better understand shortcut learning through the lens of the learning dynamics of the internal neurons during the training process. More specifically, we make the following observations: (1) While previous works treat shortcuts as synonymous with spurious correlations, we emphasize that not all spurious correlations are shortcuts. We show that shortcuts are only those spurious features that are ``easier'' than the core features. (2) We build upon this premise and use instance difficulty methods (like Prediction Depth) to quantify ``easy'' and to identify this behavior during the training phase. (3) We empirically show that shortcut learning can be detected by observing the learning dynamics of the DNN's early layers, irrespective of the network architecture used. In other words, easy features learned by the initial layers of a DNN early during the training are potential shortcuts. We verify our claims on simulated and real medical imaging data and justify the empirical success of our hypothesis by showing the theoretical connections between Prediction Depth and information-theoretic concepts like $\mathcal{V}$-usable information. Lastly, our experiments show the insufficiency of monitoring only accuracy plots during training (as is common in machine learning pipelines), and we highlight the need for monitoring early training dynamics using example difficulty metrics.

Under review as a conference paper at ICLR 2023
SHORTCUT LEARNING THROUGH THE LENS OF EARLY TRAINING DYNAMICS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Neural Networks (DNNs) are prone learn shortcut patterns that damage the generalization of the DNN during deployment. Shortcut Learning is concern- ing, particularly when the DNNs are applied to safety-critical domains. This paper aims to better understand shortcut learning through the lens of the learning dynam- ics of the internal neurons during the training process. More specifically, we make the following observations: (1) While previous works treat shortcuts as synony- mous with spurious correlations, we emphasize that not all spurious correlations are shortcuts. We show that shortcuts "easier" than the core features. (2) We build upon this premise and use instance difficulty methods (like Prediction Depth (Baldock et al. (02Th) to quantify "easy" and to identify this behavior during the training phase. (3) We empirically show that shortcut learning can be detected by observing the learning dynamics of the DNN's early layers, irrespective of the network architecture used. In other words. easy features learned by the initial layers of a DNN early during the training are potential shortcuts. We verify our claims on simulated and real medical imaging data and justify the empirical success of our hypothesis by showing the theoreti- cal connections between Prediction Depth and infformation-theoree
only those spurious features that are
concepts like V-usable information (Ethayarajh et al. 2021). Lastly, our experiments show the insufficiency of monitoring only accuracy plots during training (as is common in machine learning pipelines), and we highlight the need for monitoring early train- ing dynamics using example difficulty metrics.
INTRODUCTION
Shortcuts are spurious features that perform well on standard benchmarks but fail to generalize to real-world settings (Geirhos et al.) 2020). Deep neural networks (DNNs) tend to rely on short- cuts even in the presence of core features that generalize well, which poses serious problems when deploying them safety-critical applications such as finance, healthcare, and autonomous driv- ing (Geirhos et al.) 2020; Makden-Rayner et al.) 20200] DeGrave et al., 2021). Previous works view shortcut learning as a distribution shift problem (Kirichenko et al., 2022: Wiles et al.. 2022 Bellamy al. 2022 Adnan al., 2022]. However, show that all spurious correlations are shortcuts. Models suffer from shorteut learning only when the spurious features are much easier to learn than signals that generalize well. We show how monitoring example difficulty metrics like Prediction Depth (PD) (Baldock et al.] 2021) can reveal valuable insights into shortcut learning quite early during the training process. Early detection of shortcut learning is useful it can help develop intervention schemes to fix the shortcut early. To the best of our knowledge, we are the first to detect shortcut learning by monitoring the training dynamics of the model.
Geirhos et al. (2200) define shortcuts as spurious correlations that exist in standard benchmarks but fail to hold in more challenging test conditions, like real- wal-world settings. The emphasis on shortcuts being synonymous with spurious correlations has led to widespread adoption of viewing shortcut learning as a distribution shift problem (Bellamy et al. 2022 Wiles et al. 202 Adnan et al. 2022 Kirichenko et al.) 2022). While the distribution shift explains part of the story, we emphasize that what is equally important for shortcut learning is the difficulty of the spurious features themselves (see Fig.Ip. Previous works like Shah et al. (2020); Scimeea et al. (2021) hint at this. But take this line of thought further by viewing shortcut learning as a phenomenon that impacts the dataset difficulty, which can be captured by monitoring early training dynamics.

---

Under review as a conference paper at ICLR 2023
Tranting Data Shatted
Todeon Date wel Starlesd
13
7
8
Sconarl Dificit Fecture
2atne sundar
I
75
3
T
Cormron
Scoriro Easy Sarlous Festare
IA5M patc3)
(A) An illustration of how the causal view of shortcut learning is insufficient. In the causal view, training and testing are different graphical models between input (æ), output (y), and the spurious feature (s). If x can predict and y is not causally related to s on the test data, then is viewed as a shorteut. (A) The figure shows two scenarios for even-odd classification. Scenario-1 shows a dataset where all even numbers have spurious composite number (located the top-left), and odd numbers have prime number. Scenario-2 shows a dataset where all odd numbers have a spurious white patch. The spurious white patch is easy feature, so the model uses it as shortcut. Whereas classifying prime numbers, as shown in scenario I, is challenging. So the model ignores such spurious features. This shows not all spurious correlations are shortcuts.
(B)
Figure 1:
The premises that support our hypothesis are as follows: (PI) Shortcuts are only those spurious features that are "easier" " to learn than the core features (see Fig (P2) Initial layers of a DNN tend to Iearn easy features, whereas the later layers tend to learn the harder ones (Zeiler & Fergus, 2014 Baldock et al.) (021). (P3) Easy features are learned much earlier than the harder ones during training (Mangalam & Prabhu 2019 Rahaman al. 2019). Premises (P1-3) lead to conjecture that: "Easy features learned by the initial layers of a DNN early during the training are potential shortcuts."
We empirically show that our hypothesis works well on simulated and real medical imaging data (4.ction-4.2) regardless of the DNN architecture used. We justify this empirical success by theoretically connecting prediction depth with information-theorrtic concepts like V-usable infor- mation (Ethayarajh al. 202I) (section-3 and appendix(A.Il) Lastly, our experiments highlight that monitoring only accuracy during training, as is common in machine learning pipelines, is in- sufficient. Rather we need to monitor the learning dynamics of the model using suitable metries to detect shortcut learning (section.4.3 4.3). This could potentially save lot of time and computational costs and help develop reliable models that do not rely on spurious features.
2 RELATED WORK
Not all spurious correlations are shortcuts: Geirhos et al. (2020) define shorteuts as spurious correlations that exist in standard benchmarks but fail to hold in more challenging test conditions. Wiles et al. (202I) view shortcut learning distribution shift problem where two more at- tributes are correlated during training but are independent in the test data. Bellamy et al. (2022) use causal diagrams to explain shorteuts as spurious correlations that hold during training but not during deployment. All these papers characterize shortcuts purely as a consequence of distribution shift;
methods exist build models robust such shifts (Arjovsky et al. 2019 Krueger al. 2021 Puli et al. 2022). In contrast, we stress that not all spurious correlations are shortcuts. Rather only those spurious features that are easier than the core features are potential shorteuts (see Fig D Previous works like Shah et al.] ; /Scineca et al. (22TT) hint at this by saying that DNNs are biased towards simple solutions, and Dagaev et al. (2021) use the "too "too-good-te good-to-obbbe good-to-be-true" prior to emphasize that simple solutions are unlikely to be valid across contexts. Veitch et al. (202T) distinguish various model features using tools from causality and stress test the models for counterfactual invariance. Other works in natural language inference, visual question answering, and action recognition, also assume that simple solutions could be potential shortcuts (Sanh et al. 2020 Li & Vasconcelos 2019) Clark al.) 2019 Cadene et al. 2019 He et al. 2019). We take this line of thought further by view- ing shortcuts as simple solutions or, more explicitly, easy features, which affect the early training dynamics of the model. We suggest using suitable example difficulty metrics measure this effect.
2

---

Under review as a conference paper at ICLR 2023
Estimating Example Difficulty: There are different metrics in the literature for measuring instance-specific difficulty (Agarwal et al.) 2022 Hooker et al., 2019; Lalor et al. 2018). Daang et al. (2020) train many models on data subsets of varying sizes to estimate consistency score that captures the probability of predicting the true label for a particular example. (Tonvva et al. (2018)
define example difficulty the minimum number of iterations needed for a particular example be predicted correctly in all subsequent iterations. AAaawwll et al. (2022) propose a VoG (variance-of-- gradients) score which captures example difficulty by averaging the pre-softmax activation gradients across training checkpoints and image pixels. Feldman & Zhang (2200) use a statistical viewpoint of measuring example difficulty and develop influence functions estimate the actual leave-one-our influences for various examples. [Ethayarajh et al. (2021) use an dnformation-thhoreli approach to propose a metric called pointwise V-usable information (PVI) to compute example difficulty. [Bal-] dock et al. (202T) define prediction depth (PD) as the minimum number of layers required by the DNN to classify given input and use this to compute instance difficulty. our experiments, we use the PD metric provide proof of concept for our hypothesis.
Monitoring Training Dynamics: Other works that monitor training dynamics have a significantly different goal than ours. While we monitor training dynamics detect shortcut learning, Rabanser et al. 2022) use neural network training dynamics for selective classification. They use the disagree- ment between the ground truth label and the intermediate model predictions to reject examples and obtain favorable accuracy/coverage trade-off. Feng & Tu (2021) use statistical physics-based approach to study the training dynamics of stochastic gradient descent (SGD). While they study the effect of mislabeled data SGD training dynamics, we study the effect of shorteut learning on the early-time learning dynamics of DNNs. Hu et al. (2020) use the early training dynamics of neural networks to show that simple linear model can often mimic the learning of two-layer fully connected neural network. Adnan et al.] (2022) have a similar goal as ours and use mutual information monitor shorteut learning. However, computing mutual information is intractable for high-dimensional data, and hence their work is only limited to infinite-width neural networks that offer tractable bounds for this computation. Our work, on the contrary, more general and holds for different neural network architectures and datasets.
3 BACKGROUND AND METHODOLOGY
In this section, we formalize the notion of spurious features, shortcuts, and task difficulty. Let Per and Pte be the training and test distributions defined over the random variables X (input), y (label), and (latent spurious feature).
Definition-1 (Spurious Feature s): A latent feature s is called spurious if it is correlated with label y in the training data but not in the test data. Specifically, the joint probability distributions Ptr and Pte can be factorized as follows.
Po y. s) P
Pie Pic y
(2)
The variable s appears to be causally related to but is not. This is shown in Fig We also need a notion of task difficulty. The difficulty of task depends on the model and data distribution (X,y).
Definition-2 (Task Difficulty V): Let V + y) indicates the difficulty of predicting X + y for a model M,, where X,y ~ P. Consider a joint distribution y, ~ P for two tasks, t, and y. Then, V (X + y) > L (X- + t) indicates that the task X - y is harder than X + t for given model M.
D (Shortcut): The spurious feature is potential shortcut for model M iff v Pa (X + y) > Vi (X - s). In other words, given the input X, predicting spurious feature is easier for M than predicting the true label y.
Note that the term "potential shortcut" because, in the presence of multiple spurious features, Scimeca et al. 202T empirically show that the model selectively favors the casiest spurious feature.
3

---

Under review as a conference paper at ICLR 2023
We now explain two metrics (Prediction Depth and V-Usable Information) to measure V My- Baldock (et al. (2021) proposed the notion of Prediction Depth (PD) to estimate example difficulty. The PD of imput defined as the minimum number of layers required by the model to classify the imput. Our work relies on monitoring early training dynamics using example difficulty metrics, and we use PD) estimation for this purpose. The PD metric is also well-suited for our work as it is defined for easy inputs even when the DNN is not fully trained. We use a binary classification setting to explain the concepts used in this section.
Notion of Prediction Depth: The PID is defined by building k-NN classifiers on the embedding layers of the model. The PD is simply the earliest layer after which all subsequent A-NN predictions remain the same (0 or 1):
N II fanm (6') + (I (11--kawll fkan( (00')
PD min arg max
(3)
n
Sann is a k-NN classifier that outputs O or based on given threshold (see Appendix for details), o' is the feature embedding for the given input at layer-i, and N is the index of the final layer of the model. The lower the PD of input, the easier it is to classify. We also use the notion of undefined PD to work with models that are not fully trained. We treat k-NN predictions close to 0.5 (for binary classification setting) as invalid. If the K-NN predictions for the last three layers (for given input to the model) are invalid, we treat the PD of the input as undefined (see Appendix AapendixAA.4.. A.4 Figure- illustrates iilustrass how to read the PD plots used in experiment.
112
Tremiin Progaecx
Figure 2: Examples of PD plots (for DenseNet-1211 121) at different stages of the training process. The red bar indicates samples with undefined PD, and the dotted vertical line indicates the mean PD of the plot. Notice that the undefined samples (shown in red) slowly accumulate in layer 88 as training progresses. This is because the model needs more time learn the challenging samples which accumulate at higher prediction depth, i.e., later layers.
Notion of y-Usable Information: The Mutual Information between input and output, T(X; Y). is invariant with respect to lossless encryption of the input, i.e., T(r(X); = Y). Such a definition assumes unbounded computation and is counter-intuitive to define task difficulty as heavy encryption of X does not change the task difficulty. The notion of "Usable Information" introduced by Xu et al. (2020) assumes bounded computation based on the model family under consideration. Usable information is measured under framework called predictive V V-information (Xu et al., 2020). Ethayarajh et al. (202I) introduce pointwise V-information (PVI) for measuring example difficulty.
PvI(x + y) = log, g (v) + log, g' (x),
S.I.
g.g' E
(4)
The function g is trained on (é) v) input-label pairs, where & is null input that provides no infor- mation about the label y. g' is trained on (x, y) pairs from the training data. Lower PVI instances are harder for V and vice-versa. Since the first term in correpoosnnn to g is independent of the input x. we only consider the second term having g' in our experiments. In what follows, we relate the notions of PD and V-usable information. We use Venn (of finite depth and width) in our proof as our experiments mainly use CNN architectures.
Proposition 1: (Informal) Consider two datasets: D, ~ Per xX. with shortcuts and D; ~ Pio Pio(X, y) without shortcuts. For some mild assumptions on PD (see Appendix A.p.ndix(A.Ip.. if the mean
4

---

Under review as a conference paper at ICLR 2023
PD of Dg is less than the mean PD of Di, then the Venn -usable-infomaationn for D, is larger than the fean-usabe--ifoomaee
for D;: IP cen (X -) ID. (X Y).
See proof in Appendix-AA.I The proposition intuitively implics that a sufficient gap between the mean PD of shortcuts and core features than core features. In such a scenario, the model will be more inclined to learn shortcuts over core
result in shorteuts having more Vsanle information
features. This proposition justifies using the PD metric detect shortcut learning, as demonstrated in the following experiments.
4 EXPERIMENTS
We set up four experiments to evaluate our hypothesis. First, we consider two kinds of datasets, one where the spurious feature is easier than the core feature and another where the spurious feature is harder. We train a classifier on each dataset and observe that the model can learn the easy spurious feature but not the harder one. This experiment demonstrates that not all spurious correlations shortcuts, but only those spurious features that are easier than the core features are potential shortcuts. Second, we use toy and multiple medical datasets to demonstrate how monitoring the learning dynamies of the initial layers can reveal suspicious shortcut learning activity. We show that an early peak in the PD plot indicates potential shortcuts, and visualization techniques like grad-CAM can provide intuition about the feature used at that layer. Third, we show how shortcuts can often be detected relatively early during training. This is because initial layers which learn the shortcuts converge very early during the training. We observe this by monitoring PD plots across training epochs. In all of our experiments, the shortcut is revealed by the PD plot within two epochs of training. Fourth, we show that datasets with easy spurious features have more "usable information" (Ethayarajh et al. (021) compared to their counterparts without such features. Due to higher usable information, the model requires fewer layers to classify the images with spurious features. We use this experiment to empirically justify Proposition-1 outlined in Appendix A.
4.1 NOT ALL SPURIOUS CORRELATIONS ARE SHORTCUTS
We use the Dominoes binary classification dataset (formed by concatenating two datasets vertically; see 1ig similar the setup of Kirichenko et al. 2022)) The bottom (top) image acts as the core (spurious) feature. Images are of size 64 x 32. We construct three pairs of domino datasets such that each pair has both a hard and an easy spurious feature with respect the common core feature (see Table-1). We use classes {0.1} for MNIST and SVHN, (coat,dress] } for FMNIST, and (airplane, automobile) for CIFARIO. We also include two classes from Kuzushiji-MN -NIIT (or KMNIST) and construct a modification of this dataset called KMNpatch, which has spurious patch feature (5x5 white patch on the top-left corner) for one of the two classes of KMNIST. The spurious features are perfectly correlated with the target. The order of dataset difficulty based on the mean- PD is as follows: KMNpatch(1..) < M21ST(2.2) < FMNIST(339) < KMNIST(5) < SVHN(5.9) < CIFARR0(6.8).. We use ResNet18 model and measure the test and core-only accuracies. The test accuracy is measured on a held-out dataset sampled from the same distribution. For the core- only accuracy, we blank out the spurious feature (top-half image) by replacing it with zeroes (same as KKirichenoo et al. (2022)). The higher the core-only accuracy, the lesser the model's reliance on the spurious feature.
CIF- PMN
MN. PMN
Dataset
Is spurious hurder Uhan cure?
Test
Cure-only Accurwcy
(Sparima-Core Core)
Ascurwcy
Spatons
CIF(6.) PMIN399
yes
09.22%
MN12.2) FMN(3.9)
no
99.95
2.66
CIRIS KMN(S)
yes
98.91 200.1.6.6
08.30:2 18%
MN/2.2) KMN15)
n0
90,97--00055
5044-22/41
Core
CIP668) MIN2.2)
yes
99.742..00%%
99, 99.522.1.00%
KMNpotch MN(2.2)
99.97=20.04%
68: 16.78:2200034
(Hard Spurtoca)
(Raay Spurtona)
Com-only Image
Figm 3: Dominoes Dalaset
Table 1: Results for the Dominoes experiment averaged acmes 4 runs Numbers in brucket show msan: PD (dutaset difficulty). Comv-only accuracy indicates the model's relianze coze featurs. Modals achieve high core-enly only accuracy when spurioss features are harder than core features.
5

---

Under review as a conference paper at ICLR 2023
A MINIST
UR PMINSIT
1C)CIFAR D3
/ 2
25
LDM
1igh
Dificclty
GraCcnn OE
( KNNISTT Staree
Lape
Figure 4: Top row shows three reference datasets at three different levels difficulty and their cor- responding prediction depth (PD) plots. The datasets are ordered based on their difficulty (measured using mean PD shown by dotted vertical lines). The bottom row shows the effect of the shortcut on the KMNIST dataset. The yellow region the axis indicates the expected difficulty of classifying KMNIST. While the original KMNIST lies in the yellow region, the shortcut significantly reduces the task difficulty. The Grad-CAM shows that the model focuses on the spurious patch.
From Table 1, we observe that all datasets achieve high test accuracy expected. The core-only accuracy stays high (> 98%) for datasets where the spurious feature is harder learn than the core feature, indicating the model's high reliance core features. When the spurious feature is easier than the core, the model learns to leverage them, and hence the core-only accuracy drops to nearly random chance 50%). This experiment demonstrates that spurious feature that is harder than the core feature fails to act as a shortcut that the model can leverage to "cheat".
4.2 MONITORING INITIAL LAYERS CAN REVEAL SUSPICIOUS SHORTCUT LEARNING ACTIVITY
Synthetic Shortcut on Toy Dataset: To provide a proof of concept, we demonstrate our method on the KuzushijiMMNIST (KMNIST) (Clanuwat al. 2018) dataset comprising Japanese Hiragana characters. The dataset has ten classes and images of size 28 x 28, similar to MNIST. We insert white patch (shortcut) a particular location for each of the ten classes. The location of the patch is class-specific. We train two VGG16 models, one on the KMNIST with patch shortcut (Mar) and another on the original KMNIST without the patch (Morig).
Fig shows the prediction depth (PD) plots for this experiment. The vertical dashed lines show the mean PD for each plot. Intuitively KMNIST should be harder than MNIST but easier than CIFAR 10. But introducing the white patch makes KMNIST easier than even MNIST for Msh (see Fig A &
4D). The white patch very easy feature, and hence the model only needs a single layer to detect it. The Grad-CAM maps for the layer-1 show that Mah focuses mainly on the patch (see Fig 4 PD, and hence the test accuracy on the original KMNIST images very low ( (%%). The PD plot for MMrrig (see Fig Fig/4B) 4 is not skewed toward lower depth as the plot for Mah- This is expected as Morig is not looking at the shorteut and therefore utilizes more layers to make the prediction. The mean PD for Morig suggests that the original KMNIST is harder than Mashion-MNIST but easier than CI,AR10, which is intuitive. Morig also achieves a higher test accuracy 98%).
This experiment demonstrates how models that learn shorteuts (Mar) exhibit PD plots that suspiciously skewed towards the initial layers. If the dataset is sufficiently difficult, the skewed PD plot should raise concerns. Visualization techniques like Grad-CAM can further aid our intuition about what shortcut the model is utilizing at any given layer.
Semi-Synthetic Shortcut on Medical Datasets: We follow the procedure by DeGrave et al. 2021 to create the ChestX-ray 14//itHHb--COTTT dataset. This dataset comprises 1 pos- itive images from Github Covid repositories and negative images from ChestX-ray 14 dataset (Wang et al. 20176). In addition, we also create the Chex-MIMIC dataset following the procedure by Puli
al. 2022). This dataset comprises 90% images of Pneumonia from Chexpert (Irvin et al.
2019

---

Under review as a conference paper at ICLR 2023
and 90% healthy images from SIMIC-CXE JJonsso et al. 2019). We train two DenseNet121 mod- els, Meovid on the ChestX-ray -ray L4/GitHub-COO/ dataset, and Meher on the CheX-MIMIC dataset. We use DenseNet121, common and standard architecture for medical image analysis. Images are resized to 512 x 512.
AURI
Dagi clodkceei4caaaoo
Figure 5: PD plots for two DenseNet-121 121 models trained CheX-MIMIC and ChestX- ray 1/GGHHHbbDCO datasets are shown in the figure, along with their corresponding Grad-CAM visualizations. Both PD plots exhibit very high peak in the initial layers (1 to 4), indicating that the models use very easy features to make the predictions.
5 shows the PD plots for Meher and Mozvid. Both the plots are highly skewed towards initial layers, similar to the KMNIST with patch shortcut in Fig-AD. This again indicates that the models are using very easy features to make the predictions, which is counterintuitive as the two tasks (pneumonia and covid19 detection) are hard tasks even for humans. Examining the Grad-CAM maps at layer-4 reveals that these models focus on irrelevant spurious features outside the lung region. This raises concern because both diseases are known affect mainly the lungs. The reason for this suspicious behavior is that, in both these datasets, the healthy and diseased samples have been acquired from two different sources. This creates shortcut because source-specific attributes or tokens are predictive of the disease and can be easily learned, as pointed out by DeGrave et al. 2221.
Real Shortcut on Medical Dataset: For this experiment, we use the NIH dataset WWagg et al,, 2017a) which has the popular chest drain shortcut (for Pneumothorax detection) (Oakden-Raynn al. 2020). Chest drains are used to treat positive Pneumothorax cases. The presence of a chest drain in the lung is, therefore, positively correlated with the presence Pneumothorax and can be used by the deep learning model (Oakden-Raa) Rayner al. 2020). Appendix -..dddx--A33um outlines the procedure we use to obtain chest drain annotations for the NIH dataset. We train a DenseNet121 model (Myst)) for Pneumothorax detection NIH images of size 128 128.
XBI
Podiction Depch (DstonNet111)
Cral-CAM Layand
Cae-CA X Layar-4
Layes-ll Stmples
Sbottratcchaance on Pefoemance
Busiitile Coriaae
10:0
(A)
[B)
IC1
(D0
(E)
Figure 6: Shorteut learning NIH dataset. (A) PD plot for DenseNet-121 trained on NIH shows three prominent peaks in layers-4,40,88. (B, C) Grad-CAM reveals that layer-4 uses irrelevant artifacts as shorteuts, and layer-40 uses the chest drain shortcut for classification. (D) Plotting chest drain (shortcut) probability for layer-40 samples reveals that the model segregates healthy patients from diseased ones based on the shorteut: most diseased patients have chest drain, whereas most healthy patients do not. (E) The chest drain shortcut affects the AUC performance of the model. The X-axis (Y-axis) shows the false positive (true positive) rate.
Fig shows the PD plot for Mash- We observe that the distribution is not as skewed as the plots in the previous experiments. This is because all the images come from a single dataset (NIH). But we see two suspicious peaks at layers - 4 & 40. Pucumothorax classification is challenging even for radiologists, and hence peaks initial layers raise suspicion. The Grad-CAM maps in Figs 6 &
7

---

Under review as a conference paper at ICLR 2023
NEB Traning
2.7
2.74
Tay
Epoch
(9)
Progesg
Figure 7: Evolution of PD plot across epochs shows the training dynamics of the DNN on the NIH dataset. The initial peaks (layers-4&40) are relatively stable throughout training, whereas the later peaks (layer-88) change with time. The initial layers learn the easy shortcuts, which can be detected early during the training. Samples with undefined PD (shown in red) take more time to converge and eventually accumulate in the later layers (layer 88 in this case).
GC reveal that layer-4 looks irrelevant artifacts in the chest X-ray image, whereas layer-40 looks at chest drain in the image. The chest drain shortcut is much harder and is therefore detected at layer 40. To verify that layer-40 is looking for chest drains, we ran tube detector on the images at layer-40 and found that most diseased patients had chest drain inserted into their lungs, while most healthy patients did not have the chest drain (see Fig. This provides evidence that layer-40 categorizes patients based on the chest drain shortcut. Appendix A.3 provides details how train a tube detector. oo shows that the AUC performance is 0.94 when the diseased patients have a chest drain and 0.74 when they don't. In both cases, the set of healthy patients remains the same. This observation is consistent with the findings of "aaadenn-aanne et al. (2020) and indicates that the model looks chest drains to classify positive Pneumothorax cases.
The above experiments demonstrate how peak located in the initial layers of the PD plot should raise suspicion, especially when the classification task is challenging. Visualization techniques like Grad-CAM can further aid our intuition and help identify the shortcuts being learned by the model. This approach works well even for realistic scenarios and challenging shortcuts (like chest drain for Pneumothorax classification), as shown above.
4.3 DETECTING SHORTCUTS EARLY
Fig-7 7 shows the evolution of the PD plot across epochs for Muth (which is the model used in Fig-o). This visualization helps us observe the training dynamics of the various layers. The red bar in the PD plots shows the samples with undefined prediction depths.
These plots reveal several useful insights into the learning dynamics of the model. Firstly, we see three prominent peaks in epoch-1 layers-4,40,88 (see Fig. The magnitude of the initial peaks (like layers-4&40)) remains nearly constant throughout the training. The peaks at layers-4&40 corre- spond to shortcuts, as discussed in the previous section. This indicates that easy shortcuts can often be identified early (epoch-1 in this case). Fig shows the PD plots at epoch-2 2 for other datasets with shortcuts. It clear from Fig that the suspiciously high peak at the initial layer visible in the second epoch itself. The Grad-CAM maps reveal that this layer looks at irrelevant artifacts in the dataset. This behavior seen in all datasets shown Fig-8.
Secondly, we also see that accuracy or AUC plots do not reveal shortcut learning patterns, and we need to monitor the training dynamics using suitable metries (like PD) to detect this behavior. Thirdly, the red peak (undefined samples) decreases in magnitude with time, and we propor- tional increase in the layer-88 peak. This corroborates well with the observation that later layers take more time to converge (Rahaman et al. 2019 Mangalam & Prabhul 2019 Zeiler & Fergus) 2014). Therefore, samples with higher PD are initially undefined and do not appear in the PD plot. Nonetheless, samples with lower PD show up very early during the training, and this helps us in the
8

---

Under review as a conference paper at ICLR 2023
Sherral Ond Caalaaer
Cheff Ch (1
Podul: Drenk RiCi
Dusl Sewkkit
Figure 8: Epoch-2 PD plots for various datasets with shortcuts. The high spurious peak in the initial layer is visible in all the datasets indicating that shorteuts can be detected early during the training.
early detection of shorteuts. Early detection can consequently help develop intervention schemes that fix the shortcut early.
4.4 PREDICTION DEPTH RN V-USABLE INFORMATION
Table measures the influence of shortcuts on NIH and KMNIST using PD and PVI metrics. All diseased patients in the "NIH w/ Shortcut" dataset have chest drain, whereas all diseased patients in the "NIH w/o Shortcut" dataset have no chest drain. The set of healthy patients is common for the two datasets. The KMNIST datasets are the same as those used in Section-4.2. We use VGG16 for KMNIST and DenseNet121 for NIH. Other training details remain the same as in Section
Dataset
mean
HVenn (Y X)
NTH Shortcut
53.43
00.1711
NIH w/o Shortcut
75.33
-0.2321
KMNIST Shortcut
1.06
-0.0024
KMNIST w/o Shortcut
5.25
-0.0585
Table 2: Effect of Shortcuts on Prediction Depth and the negative conditional V-entropy (- Hy..n (Y X)). The label marginal distributions for NIH and KMNIST are the same with or without shorteut, and thus the negative conditional V-entropy is proportional to V-information.
Table D shows that datasets with shorteuts (D.) have smaller mean PD values than their counterparts without shortcuts (D;). Proposition-1 (see Section 3 Appendix 1I. shows that a sufficient gap between the mean PDs of D, and D; causes the V-Information of D, to be greater than D;. Table 2 confirms this in medical-imaging dataset with real shortcut, and we see that the mean "usable information" increases when there a shortcut. This implies that the model learns shortcut features as they have more usable information than the core features. We investigate this relationship between PD and V-information in more detail in Appendix/AA. AA.8 Ethayarajh et al. 202I) also show that V-information is positively correlated with test accuracy. This explains the significant change in AUC observed in Fig.BE. Proposition 1 bridges the gap between the notions of PD and V-usable information. This connection between V-information and PD indicates that monitoring early training dynamics using PD not only helps detect shortcut learning but also bears insights into the dataset's difficulty (in information-theoreds terms) for a given model class.
CONCLUSION
In this paper, study shortcut learning by monitoring the early training dynamics of DNNs. We emphasize that not all spurious correlations are shortcuts and empirically show that models suffer from shortcut learning only when the spurious features are easier than the core features. We hy- pothesize that "Potential shortcuts can be found by monitoring the easy features learned by the initial layers of a DNN early during the training." " and validate this hypothesis on real medical imaging data. We empirically demonstrate that shortcuts are learned quite early during the training and that one can detect shortcut learning by monitoring the early training of DNNs using suitable instance difficulty metrics like Prediction Depth (PD). Further, we show theoretical connection between PD and V-information to support our empirical results. Datasets with shortcuts have more V y-information causing the model to leam the shorteut. To conclude, relying only accuracy plots is insufficient, and we recommend monitoring the DNN training dynamics using additional metrics like PD for the early detection of shorteuts.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohammed Adnan, Yani Ioannou, Chuan-Yuus Yung Tsai, Angus Galloway, HR Tizhoosh, and Gra- ham W Taylor. Monitoring shortcut learning using mutual information. arXiv preprint arXiv:2206. 13034, 2022.
Chirag Agarwal, Daniel D' souza, and Sara Hooker. Estimating example diffiiculyy using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10368-10323 2022.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint wrXiv:190..02893 2019.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876-10889, 2021.
David Bellamy, Miguel A Hernán, and Andrew Beam. A structural characterization of shortcut features for prediction. European Journal of Epidemiology, 3766)5563-568, 2022.
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems, 32, 2019.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:/8/2.017/1,, 2018.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint 1909v:1909.03683 2019.
Nikolay Dagaev, Brett D Roads, Xiaoliang Luo. Daniel N Barry, Kaustubh R Patil, and Bradley C Love. A too-good-to-be- prior to reduce shortcut reliance. arXiv preprint arXiv:2102. 06406, 2021.
Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610-611) 2021.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Information- measures of dataset difficulty. arXiv preprint arXiv:2110.08420, 2021.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Yu Feng and Yuhai Tu. Phases of learning dynamics in artificial neural networks in the absence or presence of mislabeled data. Machine Learning: Science and Technology. 2(4):043001, 2021.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673 2020.
He He, Sheng Zha, and Haohan Wang. Unleam dataset bias in natural language inference by fitting the residual. arXiv preprint arXiv: 1908. 10763, 2019.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget? arXiv preprint arXiv:/91 /0tiv////1.0222/00 1.05248,, 2019.
Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early- time learning dynamics of neural networks. Advances in Neural Information Processing Systems, 33:17116-17181 171288 2020.
Jeremy Irvin, Pranay Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ileus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-599,99999 2019.
10

---

Under review as a conference paper at ICLR 2023
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang. Matthew Lungren, Andrew Y Ng. et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint 0tth::2006.14400 2021.
Ziheng Jiang. Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regu- larities of labeled data in overparameterizeed models. arXiv preprint arXiv:2002.03206, 2020.
Alistair EW Johnson, Tom Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng. Yifan Peng, Zhiyong Lu, Roger G Mark, Seth Berkowitz, and Steven Horng. Mimic-cxr-jpg. large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042, 2019.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re- re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937, 2022.
David Krueger, Ethan Caballero, Joern-Henrie Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapo- lation (rex). In International Conference on Machine Learning, pp. 5815 5815-5826. PMLR, 2021.
John P Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu. Understanding deep learning perfor- mance through an examination of test set difficulty: A psychometric case study. In Proceedings of the Conference Empirical Methods in Natural Language Processing. Conference Empirical Methods in Natural Language Processing, volume 2018, pp. 4711. NIH Public Access, 2018.
Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 9572- 2581,2019
Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? 2019.
Luke Oakden- Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, pp. 151 2020.
Aahlad Manas Puli, Lily H Zhang, Eric Karl Oermann, and Rajesh Ranganath. Out-of-distribution spurious correlations. In International Con-
generalization in the presence of ference Learning Representations, 2022.
Stephan Rabanser, Anvith Thudi, Kimia Hamidich, Adam Dziedzic, and Nicolas Papernot. Selective classification via neural network training dynamics. arXiv preprint 23055 2022.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, pp. 5301-5310. PMLR, 2019.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. Learning from others' mistakes: Avoiding dataset biases without modeling them. arXiv preprint arXi::2012.01300, 2020.
Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which short- cut cues will dnns choose? study from the parameter-space perspective. arXiv preprint 10.00:21 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan. Prateek Jain, and Pranecth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
11

---

Under review as a conference paper at ICLR 2023
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXtv:1812.051599 2018.
Victor Veitch, Alexander D' Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invari- ance to spurious correlations: Why and how to pass stress tests. arXiv preprint arXiv:2106.00545, 2021.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest X-ray database and benchmarks weakly-supervised classification and localization common thorax diseases. In Pruceedings of the IEEE conference computer vision and pattern recognition, pp. 2097-2106. 2017a.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106. 2017b.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre ReiseeReeufff, Ira Ktena, Taylan Cemgil, et al. A fine-grained analysis on distribution shift. arXiv preprint arXiv:2110.1 1328, 2021.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint arX1v:2002.10689, 2020.
Matthew D) Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In
European conference on computer vision, pp. 818-833. Springer, 2014.
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Nico++: Towards better benchmarking for domain generalization. arXiv preprint arXiv:2204 08040, 2022.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.I PROOF OF PROPOSITION-1
Proposition 1. Given two datasets, Ds with shortculs and D; without shortculs, we assume the following:
1. (Well-Traine Trained Model Assumption) The part of the network from any representation to the label one of the functions that compute V-information.
2. (Function Class Complexity Assumption) Assume that there exists K E (1,N) such that Venn of depth N K deep enough be a strictly larger function class than Vien with fixed neighbor size (29 in this paper). Assume that this Vienn is larger function class than a linear function.
3. (Controlled Confidence Growth Assumption) For both datasets D {D,, assume that the for all (1,-.. ,N}.
T S I Deen (Øx) - IBenn (0k-1) S
4. (Prediction Depth Separation Assumption) Let L be an integer such that, K and L K + (N K) - + max, (- log = y))
Let there exist a gap in prediction depths of samples in D2 and Dj : 1/ E (0, 0.5) such that - vb fraction of D has prediction depth S L and 1 v/ fraction of Dj has prediction depth K.
Then, for model class of N-layer CNNs, we show that the Vean-informmtion for D2 is greater than V Venn conn-informatinn for Dz:
De (X+
2 DP Venn
(X -1)
Proof. We proceed in two parts: first, we lower bound Venn Veen--nfonnnnnn information for D,, and then we upper bound Venn for D..
Assumpition implics:
B Σ ==A
(B-A)-
S r S
IP Venn (ox) - IPrm (Øx-1) & (B A)e
PD PVI connection. Note that by definition, when the prediction depth is l for sample X, then
PV Jnn (X)) 2 8 but PV (0x- (X)) < 8. This follows from how we compute PD (see Section in main paper, and Appendix A.4).
Lower bounding D-
Venn
= =I ofden (Φk)
{ [Assumption-1]
2 ID (0x)
{ (Assumption-2)
IDD Dunn (0r.
Σ ID (0x) IP (0k-1)
(Teesccoping Sum)
N=L+1
> Vuan (0r) + (K - L)T
{ [Assumption-3)
2 # min PV (X + V)
X.yeD,pu>=l.
+ (1 - 11
min x.V.D,pde.L
PV (X + Y) + (K L)r (Prediction Depth Separation]
20 20 + (1 1) + (K L.)T
(Prediction Depth Separation)
13

---

Under review as a conference paper at ICLR 2023
Upper bounding ID
Venn
Venn S M (ØN) Menn
[Assumption-2]
N
-IPmn (0x)+ +
Σ
IDem (Øx) - IIeen (#k-1)
Telescoping Sum}
k=K+1
I Dana (Øk) (N
k)
{Assumption-3)
$ (N - K)e+0 +1 D.
max
PV PVI
Vann (Xx + Y)
+ (1 1- +))
max
PVI Pane (#x (X) + Y)
(Prediction Depth Separation)
XVED; X)>c
$ <N - K)& + max ( log = u))
+ (1 1b)
max
PVID Venn (Øx (X) + Y)
(PVI v - -logp(Y = b)
xxYEDipd(X)>K
$ (N - K)E+ + max (- p(Y = y)) (1 v/0
PD-PVI connection for pd > K}
The proof follows by comparing the lower bound on V and the upper bound I.p... Intuitivelly what this means that when there a sufficiently large gap in the mean PD between D2 and D;,
then the V-information of D, exceeds the V-information of D,, which is why the model prefers learning the shortcut rather than using the core features for the task.
A.2 GRAD-CAM VISUALIZATION
PD plots help us understand the layers of the model which are actively used for classifying different images. To further aid our intuition, we visualize the Grad-CAM outputs for an arbitrary layer of the model by attaching soft-KNN head. Let ykan denote the soft and differentiable version of k-NN. We compute Iknn as follows:
IsENio exp 16 -0/1/-
oxp
Iknn ( ( oh (I..L.....))) ΣSEN(oh exp 106 -05
This function makes the KNN differentiable and can be used to compute Grad-CAM Selvaraju et al. 2017). We use the L1 norm for all distance computations. de corresponds to feature at layer-k for query image Eig. Let o* (E..1..... be the training data for KNN. W denote the neighborhood function. :) returns the indices of K-nearest neighbors for o NN( 1) returns indices of images with positive label (y 1) from the set of K-nearest neighbors for db. is the median for the set of L, norms {llØ + |} for j N(
A.3 CHEST DRAIN ANNOTATIONS FOR NIH DATASET
To reproduce the results by Dakden-Raynet et al. 2020), we need chest drain annotations for the NIH dataset (Wang al. 7a) which
natively provided. To do this, we use the MIMIC- CXR dataset (Johnson et 2019). which has rich meta-data information available in the form of radiology reports. We collaborate with radiologists identify terms related to Pneumothorax from the MIMIC-CXR reports. These include pigtail catheters, pleural tubes, chest tubes, thoracostomy tubes, etc. We collect chest drain annotations for MIMIC-CXR by parsing the reports for these terms using the RadGraph NLP pipeline (Jain et al. 2021). We train DenseNet121 model to detect chest drains relevant to Pneumothorax using these annotations. Finally, we run this trained model on the NIH dataset obtain the needed chest drain annotations. We use these annotations to obtain the results shown in Figs 6D and 6E closely reproduces the results obtained by
et al. (2020).
14

---

Under review as a conference paper at ICLR 2023
A.4 NOTION OF UNDEFINED PREDICTION DEPTH
Section Shows how we compute PD in our experiments. While fully trained models give valid PD values, our application requires working with arbitrary deep-learning models that are not necessarily fully trained. We, therefore, introduce the notion of undefined PD by treating k-NN predictions close to 0.5 (for binary classification setting) as invalid. We define such that [/nnn (œ) 0.5 < implies invalid k-NN output. We use å 0.1 and k 29 in experiments. If any k-NN predictions for the last three layers are invalid. we treat the PD of the input image to undefined. To work with high-resolution images (like 512 x 512), we downsample the spatial resolution of all training embeddings to x 8 before using the k-NN classifiers on the intermediate layers. We empirically see that results insensitive to & in the range [5, 30].
A.5
PD PLOTS AT EPOCH-0
Epoch
Epoch
1800
1800
1600
1600
1400
1400
1200
1000
1000
800
800
600
600
400
400
200
60
80
100
120
40
80
100
120
140
Layer
Layer
Figure 9: Pneumothorax Detection in NIH
Figure 10: Age Detection in NIH
We show epoch-0 PD plots for various tasks on the NIH dataset (see Figures most samples to be undefined at epoch O (shown by the red bar), and we see some additional noisy patterns in the plot. The model is not trained during initialization, and hence the weights and features are random vectors carrying little information. Most of the instance difficulty metrics are ill-defined until the model fully trained, and therefore we find this uninterpretable behavior in PD plots at epoch-0. Although most of the instance difficulty metrics fail to give meaningful information at the start of training Ethayarajh al. (202 Agarwal al. (2022) we empirically observe that PD plots can capture the distribution of the relatively easy samples in the training dataset even during early stages of training. This favors our hypothesis, which relies on monitoring easy samples early
TO TO We expect
layers detect shortcut learning. We, therefore, PD over other metrics analyze the early
training dynamics of the model.
A.6
CODE REPRODUCIBILIITY
Please find the code here:
http://www...a//ooocooooo//oaa/a Short Learning Through traininggDyramn
A.7 VISION EXPERIMENTS
We use the NICO+ (Non-LL.D. Image dataset with Contexts) dataset Zhang et al. 2022 to create multiple spurious datasets (Cow vs. Bird; Dog vs. Lizard: Flower vs. Lizard) such that the con- text/background is spuriously correlated with the target. NICO+ is a Non-L.ID image dataset that uses context to differentiate between the test and train distributions. This forms an ideal setup to investigate what spurious correlations the model learns during training. We follow the procedure outlined by Puli al. 2002 create datasets with spurious correlatinns (90% prevalence) in the
15

---

Under review as a conference paper at ICLR 2023
training data, and the test data has the relationship between spurious attributes and the true labels flipped. This is similar to the Chex-MIMIC dataset illustrated in section 4.2 We test our hypothesis using ResNet- 18 and VGG16. We train our models for 30 epochs using Adam optimizer and base learning rate of 0.01. We choose the best checkpoint using early stopping.
NICO (Cows vs Bircky Sparious Backeyour))
Prediction Death (ResNet
300
10
Intininni (1c:
Lata (Ace=(RASS))
UaI guince valts))
Layar
ConEE-A[
(Coos wadkeng gpass))
Figure II: Cow vs. Birds classification on NICO+ dataset. (A) Training data contains images of cows on grass and birds on water (correlation strength=0.9). The model achieves 97.2% training accuracy. (B) PD plot for ResNet-18 reveals spurious peak at layer-1, indicating the model's I's heavy reliance on very simple (potentially spurious) features. (C) GradCAM plots for layer 1 reveal that (D) Consequently. the model achieves a test accuracy of only 59.9% on test data where the spurious correlation is flipped (i.c., cows (birds) are found on water (grass)).
the model mainly relies on the spurious background to make its predictions.
NICO (Cows vs Birds, Balanced Dataset)
Prediction Depth (ResNet-18)
225
125
100
25
Data (Acc-802%)
32
(Cons/Bircls GinssWWises
(Conss onvaatery Birch onggree)
(A)
(R)
Balanced dataset for Cow vs. Birds classification task on NICO+ dataset. (A) The training dataset contains balanced distribution of cows and birds found on water and grass (each group has an equal number of images). (B) The balanced dataset shifts the PD plot towards the later layers (compared to Fig- 1Tg indicating that the model relies on both simple and complex features for the prediction task. This consequently results an improved test accuracy of 80.2% (as compared to 59.9% in Fig for the spurious dataset).
Figure 12:
Figures 1 show PD plots and train/test accuracies for models that learn the spurious back- ground feature present in the NICO+ dataset. While all models achieve >> 85% training accuracy, they have poor accuracies ( (5%) on the test data where the spurious correlation is flipped. This can be seen simply by observing the PD plots for the model the training data. The plots skewed towards the initial layers indicating that the model relies heavily on very simple (potentially spurious) features for the task. GradCAM maps also confirm that the model often focuses the background context rather than the foreground object of interest.
We further observe in Fig that balancing the training data (to remove the spurious correlation) results in a model with improved test accuracy (80.2%) as expected. This is also reflected in the PD plot (Fig 123), where see that the distribution of the peaks, as well the mean PD, shift
16

---

Under review as a conference paper at ICLR 2023
NICO (Dogvs (DogvsLiard) Lizad;; Spurious Background))
Prediction Depth (ResNet-18)
300
250
No. of Images
200
150
100
50
Sparious (aaringtttttakxe-raaaa
(Axt=633%)
(Dutciio Dueye Lioarh un Ruck))
Layer
(Doggon Bodg Lizanb))
Figure 13: Dog vs. Lizard classification with a spurious background feature on NICO+ dataset. (A) Training data contains images of outdoor dogs and lizards rock (correlation strength= The spurious background color/texture reveals the foreground object. The model achieves 87.2% training accuracy.
(B) PD plot for ResNet-18 reveals a spurious peak layer-1, indicating the model's 's reliance on simple (potentially spurious) features. (C) The low test accuracy confirms this (63.9%) The test data has the spurious correlation flipped (i.e., images contain dogs rock and lizards found outdoors.)
NICO (Flwer Libatt Sparious Backgrunn))
Prediction Depth (VGG16)
350
Spartion Dusiningaaa (Ac-4P27%)) GlowesssiCroi LiaadlosRRcc)
Layer
GodEAM (layoe-
Boowns Raclig and
(A)
(9
Figure 14: Flower vs. Lizard classification with spurious background feature on NICO+ dataset.
(A) Training data contains images of flowers on grass and lizards on rock (correlation strength=0.9). The spurious background reveals the target class. The model achieves 87.87 87% training accuracy. (B) PD plot for ResNet-18 reveals spurious peak at layer-1. (C) GradCAM plots for layer confirm that the model uses the rock texture the background to detect lizards, whereas it uses simple (non- spurious) color features detect the flowers in the image. (D) Due to partial reliance spurious features, the model obtains low test accuracy (45.16% on the test data with the spurious correlation flipped (i.e., images contain flowers rock and lizards grass.)
proportionately towards the later layers, indicating that the model is now using combination of both easy and complex features for the prediction task.
By monitoring PD plots during training and using suitable visualization techniques, we show that one can obtain useful insights about the spurious correlations that the model may be learning, This can also help the user make an educated guess about the generalization behavior of the model on various test datasets with different distributions.
A.8 EMPIRICAL RELATIONSHIP: PD VS V-INFORMATION
In Section we explore the relationship between PD and V-information. To empirically confirm these results, further investigate this relationship four additional datasets: KMNIST, FMNIST,
17

---

Under review as a conference paper at ICLR 2023
SVHN, and CIFARIO. We train VGG16 model these datasets for ten epochs using an Adam optimizer and base learning rate 0.01. We use bar plot to show the correlation between PD and V-entropy. We group PD into intervals of size four and compute the mean V-entropy for samples lying in this PD interval.
FMNIST (VGG16)
KMNIST (VGG16)
SVHN (VG616)
CIARRO (VGG16)
Coneitiona
Consiitoral
Condittora
11
Predittion Dupth
Prodiction Depth
Pradiciion Depth
Prediction Depth
Figure 15: The bar plots show positive correlation between PD and Conditional V-entropy. Sam-
ples with higher PD also have a higher V-entropy resulting in lower usable information for models like VGG16.
find that PD is positively correlated with V-information, and the results shown in Fig. 15 further confirm this observation. Instance difficulty increases with PD, and the usable information decreases with increase in V-entropy. is, therefore, clear from Fig-[5 samples with a higher difficulty (PD value) have lower usable information, which not only intuitive but also provides empirical support to Proposition- Appendix A.1
In Section
18


Reviews:
[{"review_id": "kJSeq4HPUvw", "reviewer": "Reviewer_RFyP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the relationship between spurious features, shortcuts and learning dynamics in a novel way. By effectively using the prediction depth (PD, Baldock et al.) as its main tool, the paper concludes that only spurious features that are “easier” than core features (instead of all spurious features) can be shortcuts, and the learning of such shortcuts can be identified early during training via PD and GradCAM visualization. The proposed workflow is validated on both synthetic and real (chest X-ray) datasets and shown effective in spotting the shortcuts. Additionally, the paper theoretically connects PD and usable information (PVI, Ethayarajh et al.) to explain its empirical success.", "summary_of_the_review": "The paper in its current form has multiple great strengths (clarity, novelty, significance) but also some unignorable weaknesses (empirical validation). In my opinion, since the paper’s strengths outweigh its weaknesses, I’m learning towards recommending its acceptance to the conference.", "strength_and_weaknesses": "Strengths\n+ (Clarity) The paper is well organized and clearly written. Fig 1 (prime number vs white patch) is a great example explaining the paper’s main finding.\n+ (Novelty) The paper’s main finding, effective tool usage (PD) and theoretical analysis (connection between PD and PVI) are all novel contributions as far as I know.\n+ (Significance) Given the importance of the topic, the novelty of the contributions, and the overall effectiveness of the proposed workflow (although can be better validated, see below), I believe this paper can be a strong, potentially impactful baseline for future researchers & practitioners to use and/or improve upon.\n\nWeaknesses\n- (Quality) While the paper’s technical details are all correct as far as I can tell, the empirical validation still can be improved in the following aspects.\n1. Although medical imaging indeed is a safety-critical domain and deserves more studies regarding spurious correlations, having only medical (chest X-ray) datasets with real (non-synthetic) shortcuts in the experiments is unfortunately non-ideal. Please consider including other commonly used datasets e.g. CelebA (gender vs hair color), Waterbirds (bird type vs background) [1] and/or NICO (object vs context) [2, 3] to diversify and strengthen the empirical evaluation.\n2. Please consider strengthening the claim that only “easier” spurious features can be shortcuts with larger-scale experiments, such as a) randomly combining (as Sec 4.1) more datasets with different PD difficulties or b) creating & using synthetic datasets with controllable shortcut difficulties like e.g. [4], in order to quantify the correlation between feature difficulties and utilization (via e.g. GradCAM analysis).\n3. Similarly, it would be a plus to showcase the correlation between PD and PVI in Sec 4.4 with larger-scale experiments to empirically support Proposition 1 more solidly.\n\n[1] Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, ICLR, 2020.\\\n[2] Towards non-iid image classification: A dataset and baselines, Pattern Recognition, 2021.\\\n[3] NICO++: Towards Better Benchmarking for Domain Generalization, https://arxiv.org/abs/2204.08040 \\\n[4] A Fine-Grained Analysis on Distribution Shift, ICLR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g3dxbjMPJQE", "reviewer": "Reviewer_3xDy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Through an empirical study on vision benchmarks and medical images, this paper explores the concepts of spurious and shortcut features and their links to prediction depth and V-information. They show empirically that shortcuts can be detected early in training. They also show a link between prediction depth and V-information.", "summary_of_the_review": "I liked the method of using the PD in order to examine training and identify potential spurious correlations. I think that the paper could be improved by clarifying the theory part.", "strength_and_weaknesses": "Strength:\n - The idea of using the PD in order to identify potential spurious correlations is new and interesting. The experiments on toy and medical images provide interesting new insights.\n\nWeaknesses:\n - It feels like the statement \"not all spurious correlations are shortcuts\" is somewhat trivial\n - I was not able to fully grasp the theoretical contribution in section 3. Arguably, this could be because I am missing some background on the subject, but I think that this also means that the presentation of your proposition 1 could be improved, even if still informal.\n\nOther suggestions:\nFigure 6: I suggest also looking at epoch 0 or in other words at initialization before any training.\nSection 4: \"shorcut is harder\" why is it a shortcut then ?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IrtiVsh0os", "reviewer": "Reviewer_eVSL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is concerned with clarifying the distinction between shortcut features and spurious correlations.  The authors note that while spurious correlations are rightly acknowledged as a source of shortcuts, and thus for shortcut learning to be identified with distribution shifts, there is a distinct source of shortcuts that is based on the difficulty of shortcut features.  They investigate the learning of shortcut features by interrogating training dynamics and example difficulty, positing that easy features learned by the initial layers of a DNN  early in training are potential shortcuts.  They provide empirical evidence for this hypothesis with synthetic and real medical imaging data, and they show that  prediction depth and $\\nu$-usable information are correlated.\n\nIn summary, the authors establish that:\n1. Shortcut features are spurious correlations, but not all spurious correlations are shortcuts\n2. The presence of shortcut features may sometimes be detected by looking at the distribution of prediction depths of a validation set (or test set)\n3. The readily computable prediction depth is positively correlated with $\\nu$-usable information, a refinement of mutual information that takes computability constraints into account.\n", "summary_of_the_review": "The authors succeed in convincing the reader that spurious correlations are not always shortcuts, though I would argue that they omit connecting their work to the growing literature on spurious correlations.  Perhaps this is intentional, to limit the scope of their claims, but I feel it could be a missed opportunity.  For example, recent work by [Veitch, D'Amour, Yadlowsky and Eisenstein](https://arxiv.org/abs/2106.00545) considers spurious correlation detection using tools from causality, where they distinguish different model features as being independent to changes in any latent factor that may introduce causal (or anti-causal) links to the labels. The authors herein may want to follow up on that line of work to characterize the 'easy' features they detect directly via prediction depth.\n\nWhile the claims of novely may be modest, it is clearly presented and may open up further work to understand shortcut learning through spurious correlations.  \n\n", "strength_and_weaknesses": "### Strengths\n- The experiments are well-designed.  Reading sections 4.1 through 4.4, I could not help but think of follow-on problems to examine for each, probing the fidelity of prediction depth as a function of number of parameter updates, as well as of the relative abundance or scarcity of the data.  \n- The figures (2-7 certainly) are very clearly designed, and informative\n- The direct manner in which the authors lay out their premies in section (1) makes the claims of the paper simple to evaluate in light of the evidence they present. \n\n\n### Weaknesses\n- The authors do not provide any precise definition for easy tasks versus hard tasks, leaving the reader to intuit why certain tasks are ordinally related in difficulty (e.g the claim in section 4 that MNIST is simpler than Fashion MNIST, which in turn is simpler than KMNIST).  Spending a bit more time to agree on a way to measure degrees of difficulty would help not only this paper, but would help to broaden the applicability of their method to other domains.  Perhaps some measure of generalization?\n\n(section 1) Panel (B) of Figure 1 has no caption text; is this intentional or is it an oversight?  \n(section 4.1) A few suggestions for strengthening this experiment:\n1. The distribution (or first two moments) over several re-trainings should be reported in table 1.  It's always better to report distributions rather than point estimates, especially when the point estimate is not precisely defined (i.e was only one experiment performed?  Or does table 1 contain the max / min disparity?)\n2. The results in Table 1 are conditioned on the difficulty of the binary classification tasks, so the authors should either report the difficulty of each of the pairwise core-feature tasks, or try multiple core-feature versus spurious feature combinations to better characterize the shortcut effect.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "96m23UnXTZ", "reviewer": "Reviewer_5RsT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"SHORTCUT LEARNING THROUGH THE LENS OF EARLY TRAINING DYNAMICS\" introduces a distinction between spurious correlations and shortcuts, where shortcuts are defined as \"easy to learn\" by some model. Through a few toy experiments as well as real-world datasets (e.g. chest X-ray), the authors then show that models preferably learn easy features (as identified via prediction depth, an existing technique that trains linear classifiers at each layer to identify the earliest layer after which an example is consistently correctly classified by all subsequent layers). The authors connect their findings to the existing concept of V-usable information and argue that looking at accuracy alone is insufficient to identify shortcuts. The main finding is summarized in this sentence from the paper: \"experiments demonstrate how a peak located in the initial layers of the PD plot [prediction depth] should raise suspicion, especially when the classification task is challenging. Visualization techniques like Grad-CAM can further aid our intuition and help identify the shortcuts being learned by the model\".\n", "summary_of_the_review": "The paper introduces a new definition of shortcuts which is tied to the notion of \"easy to learn\", without providing convincing arguments / a clear model-independent definition of easy to learn features. The authors show that the existing method of \"prediction depth\" can be useful in identifying potential shortcuts. While the paper is mostly clearly written, there are major problems with the definition of shortcuts (inconsistency), circular argumentation, poor reproducibility and limited novelty.", "strength_and_weaknesses": "**Strenghts:**\n- paper connects a few different ideas - prediction depth / example difficulty, early training dynamics, V-information\n- the question of which features models learn, and whether those are intended or shortcut features, is an important one\n\n**Weaknesses:**\n- inconsistent and problematic shortcut definition\n- circular argumentation\n- poor reproducibility\n- limited novelty: paper never provides evidence of finding shortcuts that were not known in advance; theoretical proof - if my understanding is correct - is not surprising/unexpected at all (if one introduces more information, then a model can use more information)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject