Year: 2022

Abstract:
Deep neural networks (DNNs) often rely on easy–to–learn discriminatory features, or cues, that are not necessarily essential to the problem at hand. For example, ducks in an image may be recognized based on their typical background scenery, such as lakes or streams. This phenomenon, also known as shortcut learning, is emerging as a key limitation of the current generation of machine learning models. In this work, we introduce a set of experiments to deepen our understanding of shortcut learning and its implications. We design a training setup with several shortcut cues, named WCST-ML, where each cue is equally conducive to the visual recognition problem at hand. Even under equal opportunities, we observe that (1) certain cues are preferred to others, (2) solutions biased to the easy–to–learn cues tend to converge to relatively flat minima on the loss surface, and (3) the solutions focusing on those preferred cues are far more abundant in the parameter space. We explain the abundance of certain cues via their Kolmogorov (descriptional) complexity: solutions corresponding to Kolmogorov-simple cues are abundant in the parameter space and are thus preferred by DNNs. Our studies are based on the synthetic dataset DSprites and the face dataset UTKFace. In our WCST-ML, we observe that the bias of models leans toward simple cues, such as color and ethnicity. Our findings emphasize the importance of active human intervention to remove the model biases that may cause negative societal impacts.

heading: 1 INTRODUCTION
text: Emerging studies on the inner mechanisms of deep neural networks (DNNs) have revealed that many models have shortcut biases (Cadene et al., 2019; Weinzaepfel & Rogez, 2021; Bahng et al., 2020; Geirhos et al., 2020). DNNs often pick up simple, non-essential cues, which are nonetheless effective within a particular dataset. For example, a DNN trained for the task of animal recognition may recognize ducks while attending on water backgrounds, given the strong correlations between such background cues and the target label (Choe et al., 2020). These shortcut biases often result in a striking qualitative difference between human and machine recognition systems; for example, convolutional neural networks (CNNs) trained on ImageNet extensively rely on texture features, while humans would preferentially look at the global shape of objects (Geirhos et al., 2019). In other cases, the shortcut bias arises in models that suppress certain streams of inputs: visual question answering (VQA) models often neglect the entire image cues, for one does not require images to answer questions like “what color is the banana in the image?” (Cadene et al., 2019)
Since DNNs have successfully outperformed humans on many tasks (Silver et al., 2016; Rajpurkar et al., 2017), such a phenomenon may look benign at face value. However, the shortcut biases become problematic when it comes to the generalization to more challenging test-time conditions, where the shortcuts are no longer valid (Cadene et al., 2019; Weinzaepfel & Rogez, 2021; Bahng et al., 2020; Geirhos et al., 2020). These biases also cause ethical concerns when the shortcut features adopted by a model are sensitive like gender or skin color (Wang et al., 2019; Xu et al., 2020).
Instead of proposing a method or solution, this work focuses on deepening our understanding of the shortcut bias phenomena. In particular, we design a dataset where multiple cues are equally valid
∗First two authors contributed equally.
for solving a particular task and observe which cues tend to be preferentially adopted by DNNs. The experimental setup is inspired by the Wisconsin Card Sorting Test (WCST, Banno et al. (2012)) in Cognitive Neuroscience1. See Figure 1 for an illustration of the setup. It consists of a training set with multiple highly correlated cues (e.g. color, shape, and scale) that offer equally plausible pathways to the successful target prediction (Y ∈ {1, 2, 3}). We call this a diagonal training set, highlighting the spatial arrangement of such samples in the product space of all combinations. A model f trained on such a dataset will adopt or neglect certain cues. We analyse the cues adopted by a model by observing its predictions on off-diagonal samples. With regards to Figure 1, for example, consider the f ’s prediction for a small, blue triangle as an off-diagonal sample. The prediction values f( ) ∈ {1, 2, 3} tell us which cue the model is biased towards to, e.g. if f( ) = 1, then f is biased towards scale; if f( ) = 2, towards shape; and if f( ) = 3 towards color. All three scenarios are plausible and only testing on off-diagonal samples will reveal model bias.
We make important observations on the nature of shortcut bias under WCST-ML. We discover that, despite the equal amounts of correlations with the target label, there tends to be a preferential ordering of cues. The preference is largely shared across different DNN architectures, such as feedforward networks, ResNets (He et al., 2015), Vision Transformers (Dosovitskiy et al., 2021), and multiple initial parameters. From the parameter-space perspective, we further observe that the set of solutions Θp biased to the preferred cues takes a far greater volume than those corresponding to the averted cues Θa. The loss landscape also tends to be flatter around Θp than around Θa.
Why are certain cues preferred to others by general DNNs? We provide an explanation based on the Kolmogorov complexity of cues, which measures the minimal description length for representing cues (Kolmogorov, 1963). Prior studies have shown that in the parameter space of generic DNNs, there are exponentially more Kolmogorov-simple functions than Kolmogorov-complex ones (VallePerez et al., 2019; De Palma et al., 2019). Based on these theoretical results, we argue that DNNs are naturally drawn to Kolmogorov-simple cues. We empirically verify that the preferences for cues correlate well with their Kolmogorov complexity estimates.
What are the consequences of the inborn preference for simple cues? Firstly, this may hinder the generalization of DNNs to challenging test scenarios where the simple shortcut cues are no longer valid (Geirhos et al., 2020). Secondly, we expose the possibility that certain protected attributes correspond to the simple shortcut cue for the task at hand, endangering the fairness of DNNs (Barocas et al., 2017). In such a case, human intervention on the learning procedure may be necessary to enforce fairness, for the dataset and DNNs can be naturally drawn to exploit protected attributes.
The primary goal of this manuscript is to shed light on the nature of shortcut biases and the underlying mechanisms behind the scenes. Our contributions are summarized as follows: an experimental setup for studying the shortcut bias in-depth (WCST-ML) (§2); novel observations on the nature of shortcut biases, such as the existence of preferential ordering of cues and its connections to the geometry of the loss landscape in the parameter space (§3); an explanation based on the descriptional
1Original WCST gauges the subjects’ cognitive ability to flexibly shift their underlying rules (adopted cues) for categorizing samples. Inability to do so may indicate dysfunctional frontal lobe activities.
complexity of cues (§4); and a discussion on the implications on generalization and fairness (§5), such as the preferential use of ethnical features for face recognition on the UTKFace dataset.

heading: 2 SETUP
text: We introduce the setup that will provide the basis for the analysis in this paper. We describe the procedure for building a dataset with multiple equally valid cues for recognition (§2.1). The procedure is applied to DSprites and UTKFace datasets in §2.2. In §2.3, we introduce terminologies for the analysis of the parameter space and make theoretical connections with our data framework.
2.1 DATA FRAMEWORK: WCST-ML
Many factors affect the preference of models to certain cues. The existence of dominant classes is an example; it encourages models to favor cues conducive to a good performance on the dominant classes (Barocas et al., 2017; Hashimoto et al., 2018). In other cases, some cues have higher degrees of correlation with the target label (Geirhos et al., 2020). In this work, we test whether bias is still present under fair conditions, i.e.: when a training dataset contains a set of valid cues, each of which equally correlates with the targets, will DNNs still have a preference for certain cues? If so, why?
To study this, we introduce a data construction framework called Wisconsin Card Sorting Test for Machine Learners (WCST-ML), named after a clinical test in cognitive neuroscience (Banno et al., 2012). See Figure 1 for an overview. As a running example, we assume a dataset where each image can be described by varying two latent variables, object shape and object color. Let X and Y denote image and label, respectively. We write Xij for the image with color i and shape j, where i, j ∈ {1, · · · , L}. When we want to considerK > 2 varying factors, we may writeXi1,··· ,iK for the image random variable with kth factor chosen to be ik ∈ {1, · · · , L}. Importantly, we fix the number of categories for each factor to L to enforce similar conditions for all cues. Similar learning setups have appeared in prior papers: “Cross-bias generalisation” (Bahng et al., 2020), “What if multiple features are predictive?” (Hermann & Lampinen, 2020), and “Zero generalization opportunities” (Eulig et al., 2021). While we fully acknowledge the conceptual similarities, we stress that our work presents the first dedicated study into the cue selection problem and the underlying mechanisms.
The same set of images {Xij | 1 ≤ i, j ≤ L} admits two possible tasks: color and shape classification. The task is determined by the labels Y . Denoting Yij as the label for image Xij , setting Yij = i leads to the color classification, and setting Yij = j leads to the shape classification tasks. We may then build the data distribution for the task at hand via
Dcolor := ⋃
1≤i,j≤L (Xij , Yij = i) Dshape := ⋃ 1≤i,j≤L (Xij , Yij = j) (1)
for color and shape recognition tasks, respectively. More generally, we may write Dk := ⋃
1≤i1,··· ,iK≤L (Xi1,··· ,iK , Yi1,··· ,iK = ik) (2)
for the data distribution where the task is to recognize the kth cue. We define the union of random variables as the balanced mixture: ⋃L i=1 Zi := ZI where I ∼ Unif {1, · · · , L}.
We now introduce the notion of a diagonal dataset, where every cue (e.g. color and shape) contains all the needed information to predict the true label Y . That is, a perfect prediction for either color or shape attribute leads to a 100% accuracy for the task at hand. This can be done by letting the factors always vary together i = j in the dataset (and thus the name). We write
Ddiag := ⋃
1≤i≤L (Xii, Yii = i). (3)
Such a dataset completely leaves it to the model to choose the cue for recognition. Given a model f trained on Ddiag, we analyse the recognition cue adopted by f by measuring its unbiased accuracy on all the cells (See Figure 1). There are K different unbiased accuracies for each task, depending on how the off-diagonal cells are labelled: e.g. Dcolor and Dshape in equation 1. For a general setting with K cues, the unbiased accuracy for kth cue is defined as
acck(f) := 1
Lk
∑
i1,··· ,iK Pr [f(Xi1,··· ,iK ) = ik] . (4)
Proposition 1. For k ∈ {1, · · · ,K}, acck(f) = 1 if and only if f(Xi1,··· ,iK ) = ik almost surely for all 1 ≤ i1, · · · , iK ≤ L. Moreover, if the condition above holds (i.e. f is perfectly biased to cue k), then accm(f) = 1L for all m 6= k.
The proposition implies that the unbiased accuracy is capable of detecting the bias in a model f : acck(f) = 1 implies that f ’s prediction is solely based on the cue k. It also emphasizes that it is impossible for a model to be perfectly biased to multiple cues. Finally, we remark that the WCST-ML analysis does not require the cues to be orthogonal or interpretable to humans. The only requirement is the availability of the labelled samples (Xi1,··· ,iK , Yi1,··· ,iK ) for the cue of interest.

heading: 2.2 DATASETS FOR ANALYSIS
text: DSprites (Matthey et al., 2017) is an image dataset of symbolic objects like triangles, squares, and ellipses on black background. It consists of images with all possible combinations of five factors: shape, scale, orientation, and X-Y position. The total number of combinations is 3× 6× 40× 32× 32 = 737, 280. We augment the dataset with another axis of variation: 4 colors (white, red, green, and blue). This results in 737, 280×4 = 2, 949, 120 images in the augmented dataset. Each image is of resolution 64× 64. We have identified degenerate orientation labels due to rotational symmetries for certain shapes. We process the data further by collapsing the symmetries to a unified orientation label. For example, we assign the same orientation label for squares at rotations 0, π2 , π, and 3π 2 .
UTKFace (Zhang et al., 2017) is a face dataset with 33, 488 images with exhaustive annotations of age, gender, and ethnicity per image. We utilize the Aligned and Cropped Face version to focus the variation in the facial features only. Ages range from 0 to 116; gender is either male or female; and ethnicity is one of White, Black, Asian, Indian, and Others (e.g. Hispanic, Latino, Middle Eastern) following Zhang et al. (2017). Images are resized from 224× 224 to 64× 64.
Applying the data framework. For each analysis, we select a subset of features S and build the training set Ddiag and test sets Dk (§2.1) anew based on S. Figure 2 shows examples from DSprites with S = {shape, color} and UTKFace with S = {age, ethnicity}, respectively. To enforce the same number of classes L per feature, we set l as the minimal number of classes among the selected features S. For features with #classes > L, we sub-sample the classes to match #classes = L. For continuous features like age and rotation, we build categories based on intervals, defined such that the L classes are balanced. We randomly vary cues /∈ S in the constructed datasets Ddiag and Dk. Figure 2 shows the example datasets of Ddiag and Dk for each dataset.

heading: 2.3 PARAMETER SPACE AND SOLUTIONS
text: We define the hypothesis class for DNNs through their weights and biases, collectively written as θ ∈ Θ (e.g. all possible weight and bias values of the ResNet50 architecture (He et al., 2016)). We
will use a non-negative loss L ≥ 0 to characterize the fit of the model fθ on data D. 0-1 loss (and cross-entropy loss) is such an example:
L(θ;D) = Pr(X,Y )∼D[fθ(X) 6= Y ]. (5)
We define the solution set as Θ? := {θ ∈ Θ : L(θ;D) = 0}; in practice we define it with a small threshold L(θ;D) ≤ . We now make a connection between the datasets and solution sets. Assume that we have a dataset of form (Xij , Yij) where Yij = i (the color task; see §2.1). The corresponding solution set is defined as ΘYijij := {θ : L(θ; (Xij , Yij = i)) = 0}. Define it similarly for the shape task (Yij = j). Proposition 2. If D = ⋃ (i,j)∈I(Xij , Yij) for some index set I , then the corresponding solution set
is the intersection of the solutions sets ⋂
(i,j)∈I Θ ?Yij ij .
Proof. If L never attains 0, the solution set is always empty and the statement is vacuously true. Otherwise, we may decompose L(θ) into the average 1|I| ∑ (i,j)∈I Lij(θ). Due to the non-negativity of the loss function, L(θ) = 0 if and only if Lij(θ) = 0 for all (i, j) ∈ I .
The proposition leads to a neat summary of the relations among solution sets like Θcolor := {θ : L(θ; (Xij , i)) = 0} and Θshape defined similarly. Corollary 3. Θcolor ⊂ Θdiag and Θshape ⊂ Θdiag. Moreover, Θcolor ⋂ Θshape = ∅.
Proof. The first statement immediately follows from the proposition above and the fact that Ddiag = Dcolor ⋂ Dshape. For the second, observe that there is no function that satisfies fθ(Xij) = i and fθ(Xij) = j at the same time for i 6= j.
Θ1 Θ2
Θ3
Θdiag ΘK−1 ΘK We may easily extend the result to datasets with K cues. Given the solutions sets Θk for each cue 1 ≤ k ≤ K, we have Θk ⊂ Θdiag for all k and {Θk}k are pairwise disjoint (diagram on the right). This follows from Proposition 1 that a single model cannot achieve low error for multiple cues (Dk) at the same time.

heading: 3 OBSERVATIONS
text: Based on the proposed WCST-ML, we study the natural tendencies of deep neural networks (DNNs) to favor certain cues over the others. We also present a parameter-space view that hints to the underlying mechanisms for the cue preference.

heading: 3.1 PREFERENTIAL ORDERING OF CUES
text: In the first set of experiments, we observe which cues are preferentially used by the network when training on a dataset composed of an equally represented set of cues: the diagonal dataset Ddiag in §2.1. We train three qualitatively different types of DNNs: (1) Feed Forward neural network (FFnet), (2) ResNet with depth 20 (ResNet20), and (3) Vision Transformer (ViT). Details of the architectures and training setups can be found in Appendix §A. The results are summarized in Figure 3.
Models adopt cues with uneven likelihood. Figure 3 shows the unbiased accuracies of the models when trained on the diagonal sets for each dataset (DSprites and UTKFace) with the involved cues denoted as S. We report the diagonal accuracy and the K unbiased accuracies for each cue in S on the held-out test set. The number of labels L is 3 for DSprites and 2 for UTKFace. On DSprites, we observe that for all architectures, color is by far the most preferred cue (unbiased accuracy near 100%). For FFnet and ViT, the extreme color bias forces the accuracy on other cues to be 1 L = 33.3% as predicted by Proposition 1. ResNet20 behaves less extremely and picks up scale, shape, and orientation, in the order of preference. On UTKFace, there is no extreme dominant cue, but there is a general preference to use ethnicity cues for making predictions, followed by gender and age cues. The trend is clear for all models, and while the FFnet model shows a slightly more variable performance across runs, the ranking of cues is preserved throughout. The preference of
For each cue, we show 3-D plots of the same loss surface (side and top views). X and Y axes are randomly chosen directions in the parameter space; Z axis shows the loss value at each (X,Y )
measured against the diagonal data Ddiag. All the surfaces are shown at the same scale.
ethnicity echoes the DNN’s preference of color as a strong cue in DSprites; DNNs are drawn to utilizing skin color. Another curious phenomenon is that none of the three cues sufficiently explains the 100% diagonal accuracy, suggesting the existence of an unlabeled shortcut cue other than the three considered. Finally, when the most dominant cue is left out, other cues activate in its place: scale for DSprites and gender for UTKFace (Appendix §B).
Consistency of preferences. We have observed a consistent preferential ordering of cues shown by qualitatively different types of architectures. The results suggest the existence of a common denominator that depends solely on the nature of the cues, rather than the architectures. We finally remark that the experiments were run 10 times with different random initialization; error bars in Figure 3 show ±σ standard deviation. The small variability observed across most training shows the consistency of the preferential ranking with respect to initialization. For a pair of cues, we refer to the one that is more likely to be chosen by a model trained on Ddiag as the preferred cue and the other as the averted cue.

heading: 3.2 THE ABUNDANCE OF PREFERRED-CUE SOLUTIONS
text: Why are certain cues preferred to others? We look for the answers in the parameter space. More specifically, we study the geometric properties of the loss function and the corresponding solution sets in the parameter space. The section is based on §2.3. We write the solution set corresponding to the preferred cues as preferred solutions Θp and averted cues as averted solutions Θa.
How to find the averted solutions θ ∈ Θa? Naively training a DNN on Ddiag will most likely result in a preferred solution Θp. We thus use the full dataset Dk corresponding to the averted cue k. (§2.1) as the training set to find an averted solution θ ∈ Θa. To ensure a fair comparison in terms of the amount of training data, we also use the full dataset Dk for the preferred cues.
Qualitative view on the loss surface. We examine the geometric properties of the loss surface for ResNet20 around different solutions types (preferred and averted). We use the 2D visualization tool for loss landscape by Li et al. (2018). The X-Y grid is spanned by two random vectors in RD where D is the number of parameters. In Figure 4, we observe that the preferred solutions (e.g. Θcolor) are characterized with a flatter and wider surface in the vicinity. In contrast, local surface around averted (e.g. Θorientation) solutions exhibit relatively sharper and narrower valleys. A similar trend is observed for the UTKFace cues at a milder degree: Θethnicity is a slightly flatter solution than Θage.
Θp Θa
basin for basin forΘp
ΘaSize of the basin of attraction. We compare the size of the preferred Θp and averted Θa solution sets using the notion of the basin of attraction (BA) for a cue k: a subset N in the parameter space such that initializing the training with a point θ0 ∈ N leads to a k-biased solution θ? ∈ Θk. To measure the size of the BA for Θk, we start by obtaining a solution θk ∈ Θk biased to the cue k. We then check when the final models trained from perturbed points θk + v (where ‖v‖2 is small) stops being biased to the cue k. We treat the norm of the bias-shifting critical perturbation ‖v‖2 as the size of the BA for Θk.
Preferred solutions have wider BA. Figure 5 shows the size of BA measured according to the method above. The upper row shows initialization around solutions biased to cues in DSprites. For the color cue, we observe that initializing far away from the color-biased solution still results in a color-biased solution. For shape, scale, and orientation cues, we observe that initializing away from the respective solutions gradually results in color-biased solutions with a reduction in the bias towards the respective biases in the unperturbed initial parameters. The sizes of basins of attraction are ordered according to color>shape>scale>orientation, largely following the preferential ordering observed in §3.1. The lower row of Figure 5 shows the results for the three cues in UTKFace. As opposed to DSprites, the solutions for ethnicity, age, and gender do not re-converge to a clear cue after retraining for only 50 epochs. The sizes of basins of attraction are ordered as ethnicity>gender>age, following the preferential ordering of cues observed in §3.1.
Θdiag
ΘaΘp
path Path connectivity. Recent works have proposed methods to find near zeroloss paths between two solutions (Garipov et al., 2018; Draxler et al., 2018; Benton et al., 2021). We extend the technique to find the zero-loss (w.r.t. Ddiag) path between preferred and averted solutions. We locate the flipping moment for the bias along the path; there must be a bias-shift boundary where the bias to the preferred cue shifts to the averted cue along the path (by an intermediatevalue-theorem-like argument). By comparing the length of the segment up to this boundary, we may estimate the relative abundance of the preferred and averted solutions in the parameter space.
Paths connecting preferred and averted solutions and the bias-shift boundary. Figure 6 visualizes the paths between the preferred and averted solutions as well as the corresponding shifts in biases along the path. We observe that the paths generally show a single shift in bias. The shift is also usually quick; for the most part of the path, the solutions tend to be heavily biased towards either of the cues. For the path from Θcolor to Θorientation, we observe a tendency for Θcolor to dominate the path until near Θorientation, suggesting the relative abundance of color-biased solutions Θcolor in the set of diagonal solutions Θdiag compared to the orientation Θorientation. The path from Θethnicity to Θgender is composed of a longer segment for the Θethnicity part. The relative abundance of cues in the diagonal solutions Θdiag resonates the preferential ordering of cues seen in §3.1.

heading: 4 EXPLANATIONS
text: We have observed intriguing properties of cues and shortcut biases. Even under the equal opportunities for the cues to be adopted, models have a preference for a few cues over the others. The
preference is stable across architectures and weight initialization. We have also found a strong correlation between the preference for a cue and the abundance of solutions biased to that cue. In this section, we explain the observations based on the complexity of cues.
Kolmogorov complexity of cues (KCC). We formally define cue as the condition distribution pY |X for a given input distribution pX ; in a sense, it concurs with the definition of task. Kolmogorov complexity (KC, Kolmogorov (1963)) measures complexity of binary strings based on the minimal length among programs that generate the string. Achille et al. (2018) have defined the Kolmogorov complexity of a task (or a cue) pY |X as:
K(pY |X) = minL(f ;X,Y )<δ K(f) (6)
for some sufficiently small scalar δ > 0. f denotes a discriminative model mapping X to Y and L is a suitable loss function. K(f) is the KC for model f . Intuitively, K(pY |X) measures the minimal complexity of the function f required to memorize the labeling pY |X on the training set (i.e. L < δ).
Measuring KCC. We estimate the KC of four cues (color, scale, shape, and orientation) considered in DSprites based on equation 6. Since it is impractical to optimise K(f) over an open-world of all possible models f , we constrain the search to a homogeneous family of models f ∈ F . We use ResNet20 with variable channel sizes (Zagoruyko & Komodakis, 2016) for F . Given that KC is uncomputable, we use the number of parameters in f as an approximation to K(f) in practice
(Valle-Perez et al., 2019). We set the memorization criterion L(f ;X,Y ) < δ as “training-set error for f is< 1%”. Under this setup, we have verified that the complexities of cues are 1.2K parameters for color, 4.6K for scale, 17.6K for shape, and > 273K for orientation. We were not able to fully memorize the orientation labels under the network scales we considered. The complexity ranking strongly concurs with the preferential ordering found in §3.1.
The simplicity bias in DNN parameter space. How are the cue complexity and shortcut bias connected? We explain the underlying mechanism based on a related theory in a different context: explaining the generalizability of DNNs with the simplicity bias (Valle-Perez et al., 2019; Shah et al., 2020). It has been argued that, despite a large number of parameters, DNNs still generalize due to the inborn tendency to encode simple functions. This phenomenon has filled the theoretical gap left by the classical learning theory. The argument is that a (uniform) random selection of parameters for a DNN is likely to result in a function f that is simple in the sense of KC. More precisely, the likelihood of a function f being encoded by a DNN is governed by the inequality p(f) . 2−aK̃(f)+b, where K(f) is the KC of f and a > 0, b are constants independent of f (Valle-Perez et al., 2019). This inequality stems from the Algorithmic Information Theory and has been validated to hold for a large variety of systems that arise naturally, such as RNA sequences and solutions to a large class of differential equations (Dingle et al., 2018). The argument concludes that the natural abundance of simple functions in the parameter space makes it more likely for the solutions to be simpler (Wu et al., 2017), leading to the simplicity bias and thus helping DNNs generalize well.
The simplicity bias leads to shortcuts. We have seen that cues have different KCs, necessitating DNNs with different complexities to fully represent them. The simplicity bias in the parameter space explains the abundance of solutions biased to simple cues like color Θcolor rather than complex cues like orientation Θorientation (§3.2). The difference in the abundance in turn makes it more likely for a model trained on the diagonal dataset Ddiag to be biased towards simple cues (§3.1).

heading: 5 DISCUSSION AND CONCLUSION
text: We have delved into the shortcut learning phenomenon in deep neural networks (DNNs). We have devised a training setup WCST-ML where multiple cues are equally valid for solving the task at hand and have verified that, despite the equal validity, DNNs tend to choose cues in a certain order (e.g. color is preferred to orientation). We have shown that the simple cues (in the sense of Kolmogorov) are far more frequently represented in the parameter space and thus are far more likely to be adopted by DNNs. We conclude the paper with two final remarks on the implications of our studies.
Simplicity bias and generalization. The simplicity bias theory (Valle-Perez et al., 2019) has been developed to explain the unreasonably good generalizability of DNNs. Our paper shows that it may interfere with the generalization. This apparent paradox stems from the difference in the setup. The simplicity bias is often safe to be exploited in the standard iid setup, albeit with caveats argued by Shah et al. (2020). On the other hand, it may lead to less effective generalization across distribution shifts as in our learning setup (Geirhos et al., 2020), if the simple cues are no longer generalizable. Thus, for the generalization across distribution shifts, machine learners may need to be given additional guidance to overcome the inborn tendencies to pick up simple cues for recognition.
Societal implications. An issue with the simplicity bias and shortcut learning is that relying on simple cues is sometimes unethical. We have seen in the UTKFace experiments that ethnicity tends to be a more convenient shortcut than age, for example, if they equally correlate with the target label at hand. This is an alarming phenomenon. It emphasizes the importance and inevitability of active human intervention to undo certain naturally-arising biases to socially harmful cues in learned models. It also confirms that we are on the right track to actively regularize the models to behave according to the social norms (Barocas et al., 2017; Mehrabi et al., 2021).
Toolbox contribution. Along the way, we contribute interesting novel tools that future research can utilise, such as the computation of Kolmogorov complexity (KC) for generic cues (equation 6) and the method for finding a heterogeneous zero-loss curve connecting two solutions biased to different cues (§3.2; Garipov et al. (2018)).
REPRODUCIBILITY
We understand and resonate with the importance of reproducible research practices in machine learning. The reproducibility of this paper hinges on our use of open-source data, models, training techniques, and deep learning frameworks. We have specified a detailed experimental setup in §2.2 and §A. For the consistency of experimental observations, we have run the experiments multiple times with different random seeds, whenever applicable. For those cases, we have reported the mean and standard deviation for each data point.
ETHICS STATEMENT
We delve into the potential ethical issues with DNNs. Our work excavates the potential ethical harms caused by deep neural networks’ natural tendencies to prefer simple cues, where using the simple cues may not conform well to the social norm (e.g. ethnicity cue). We have thus advocated an active human involvement to regularize such DNNs to adopt more socially acceptable cues. As for the experiments, part of our studies is based on a face dataset (UTKFace). We confirm that the UTKFace dataset has been released with an open-source license with the non-commercial clause2. The copyright for the images still belongs to their respective owners. Inheriting the policy of UTKFace, we will erase the face samples in Figure 2 upon request by the owners of the respective images.

heading: A MODEL ARCHITECTURES
text: FFnet. The FFnet architecture tested is a fully connected feedforward neural network with 5 hidden layers with respectively 4096, 2048, 600, 300 and 100 hidden units. ReLU activations are used throughout. We train it using the Adam optimizer with the default parameters given by PyTorch 1.8.0.
ResNet20. The ResNet20 architecture is adapted from He et al. (2016), with a depth of 20. We train it using the Adam optimizer with the default parameters given by PyTorch 1.8.0.
ViT. The Visual Transformer architecture is adapted from (Touvron et al., 2021). Given the dimensions of the images in ColorDSprites and consequently UTKFace, we implement a ViT model with patch size 8, embedding dimension 192, depth 12, and 3 attention heads. We train the model with the SGD optimizer with learning rate 5× 10−3, momentum 0.9, and weight decay 1× 10−4.

heading: B PREFERENTIAL ORDERING AFTER REMOVING THE DOMINANT CUE
text: For both DSprites and UTKFace, removing the dominant cue (color and ethnicity respectively) from S does not change the ranking of the remaining cues; e.g. scale and gender then become the dominant features in respective datasets. See Figure B.1 that extends the main paper Figure 3.

heading: C LOSS CURVATURE AROUND SOLUTIONS
text: We supplement the visualization of loss landscape around solutions in Figure 4 of main paper with quantitative estimation of the respective curvatures. The mean curvature of a surface defined by function f is defined as:
κ(f) := 1
D ∆f :=
1
D
( ∂2f
∂x21 + · · ·+ ∂
2f
∂x2D
) (C.1)
To make it computationally tractable for large D (e.g. DNN parameters), we estimate the curvature with a Monte-Carlo estimation as done in Izmailov et al. (2018). For a point x ∈ RD, a sufficiently small scalar > 0, and a unit sphere SD−1 ⊂ RD, the curvature is computed by:
κ(f, x, ) ≈ 1 2 [Ef(x+ r)− f(x)] where r ∼ Unif(SD−1). (C.2)
How does this approximate the mean curvature? We provide the following proposition.
Proposition 4. For a point x ∈ RD, a scalar > 0, a twice-differentiable function f , and a unit sphere SD−1 ⊂ RD, the following quantity
1
2 [Ef(x+ r)− f(x)] where r ∼ Unif(SD) (C.3)
approximates the Laplacian ∆f up to a constant multiplier.
Proof. We begin with a Taylor expansion on f(x+ r):
f(x+ r) = f(x) + rT∇f(x) + 1 2 2rT∇2f(x)r + h( , r) (C.4)
where h( ,r) 2 → 0 as ↓ 0 for any r. Thus,
1 2 [Ef(x+ r)− f(x)] = 1 2 E [ rT∇2f(x)r + h( , r) 2 ] (C.5)
= 1
2 E[rT∇2f(x)r] + o( ) (C.6)
by Er = 0 and the Dominated Convergence Theorem for Lebesgue integrals (also noting that r is defined on a compact set SD−1).
We now apply a change of basis such that the Hessian ∇2f(x) is diagonalised: ∇2f(x) = QTΛQ. This is possible because the Hessian is a real, symmetric matrix. Since the change of basis do not alter the distribution for r, we compute
E[rT∇2f(x)r] = E[rTΛr] = Var(r1)λ1 + · · ·+ Var(rD)λD (C.7) = Var(r1)(λ1 + · · ·+ λD) = Var(r1)tr(∇2f(x)) = Var(r1)∆f(x) (C.8)
using the isotropicity of r and the preservation of the trace against changes of basis. We thus conclude
1
2 [Ef(x+ r)− f(x)] = C∆f(x) + o( ) (C.9)
where C is a constant independent of f and x.
Results. Figure C.1 shows the approximated curvature at varying radius values around different solutions for ResNet20. We have used 100 Monte-Carlo samples. We observe that for DSprites, the curvature of the orientation loss surface tends to overshoot quickly as one walks away from the solution point. Surfaces for other cues tend to be smoother. On UTKFace, gender tends to provide a steeper surface than the other cues like age and ethnicity.

Review summaries:
The authors propose a framework for studying the tendency of deep neural networks to preferentially adopt "cues". Specifically, they focus on settings where multiple cues are equally likely, though not all of them are equally exploited. To set up such a scenario, they introduce the WCST-ML task, in which the prevalence of cues can be parametrically controlled. They also conduct empirical studies on the more naturalistic UTKFace dataset. The authors introduce a set of metrics, such as path connectivity, attractor basin properties, etc. to analyze cue preferences from a loss landscape perspective. The authors also explain these observations based on the "complexity" of cues.   

The paper discusses biases in inductive learning in deep neural networks that stem from pathologically sampled data. The authors pose a problem set where a trainer only sees samples where two or more latent values can only be observed in a fully correlated fashion (e.g. scale, color, shape). They then design various criteria and protocols to gain insight in the behavior of the learned model when the correlation does not hold any more (i.e. samples that have not been seen in the training data). They conclude that in this case of generalization to unseen data, the trained model has an implicit bias towards choosing (1) mostly single cues to determine the predicted label (e.g. color only), (2) that more preferred cues are simpler than less preferred cues and (3) that the underlying solution space of possible parameters has more solutions that prefer the simple cues. They demonstrate empirical results on variations of existing datasets (DSprites, UTKFace).   

The paper conducts a study of which visual cues are preferred by current vision models. The paper designs a training setup with several cues where each cue is equally correlated with the image label. The paper shows that visual cues like color are much easier to be learned by a vision model, than other cues such as orientation and shape. The paper also provides evidence that easy-to-learn cues tend to converge to relatively flat minima and models that prefer these cues are more abundant in parameter space.