Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective
Authors: Luca Scimeca
Seong Joon Oh
Sanghyuk Chun
Michael Poli
Sangdoo Yun
Venue: ICLR 2022 Poster (2022)
Keywords: shortcut learning
shortcut bias
loss geometry
simplicity bias
flat minima
generalization
wisconsin card sorting test

Abstract: Deep neural networks (DNNs) often rely on easy–to–learn discriminatory features, or cues, that are not necessarily essential to the problem at hand. For example, ducks in an image may be recognized based on their typical background scenery, such as lakes or streams. This phenomenon, also known as shortcut learning, is emerging as a key limitation of the current generation of machine learning models. In this work, we introduce a set of experiments to deepen our understanding of shortcut learning and its implications. We design a training setup with several shortcut cues, named WCST-ML, where each cue is equally conducive to the visual recognition problem at hand. Even under equal opportunities, we observe that (1) certain cues are preferred to others, (2) solutions biased to the easy–to–learn cues tend to converge to relatively flat minima on the loss surface, and (3) the solutions focusing on those preferred cues are far more abundant in the parameter space. We explain the abundance of certain cues via their Kolmogorov (descriptional) complexity: solutions corresponding to Kolmogorov-simple cues are abundant in the parameter space and are thus preferred by DNNs. Our studies are based on the synthetic dataset DSprites and the face dataset UTKFace. In our WCST-ML, we observe that the inborn bias of models leans toward simple cues, such as color and ethnicity. Our findings emphasize the importance of active human intervention to remove the inborn model biases that may cause negative societal impacts.

WHICH SHORTCUT CUES WILL DNNS CHOOSE? A STUDY FROM THE PARAMETER-SPACE PERSPECTIVE
Luca Scimeca', Seong Joon Oh", Sanghyuk Chun, Michael Poli & Sangdoo Yun NAVER Al Lab, NAVER Corp Seoul, 13638, Republic of Korea luca. sc wwm.ccmmccc.... hww.aaa..eadareoo edu
ABSTRACT
Deep neural networks (DNNs) often rely on easy-to soay-to-leaad learn discriminatory fea- tures, or cues, that are not necessarily essential to the problem at hand. For ex- ample, ducks in an image may be recognized based on their typical background scenery, such as lakes or streams. This phenomenon, also known as shortcul learning, is emerging as key limitation of the current generation of machine learning models. In this work, introduce set of experiments to deepen our understanding of shortcut learning and its implications. We design a training setup with several shorteut cues, named WCST-ML, where each cue is equally conducive to the visual recognition problem at hand. Even under equal opportunities, we observe that (I) certain cues are preferred to others, (2) solutions biased the easy-to-learn cues tend to converge to relatively flat minima on the loss surface, and (3) the solutions focusing on those preferred cues are far more abundant in the parameter space. We explain the abundance of certain cues via their Kolmogorov (descriptional) complexity: solutions corresponding to Kolmogorov-simple cues are abundant in the parameter space and are thus preferred by DNNs. Our stud- ies are based on the synthetic dataset DSprites and the face dataset UTKFaae. our -ML, we observe that the bias of models leans toward simple cues, such as color and ethnicity. Our findings emphasize the importance of active human intervention remove the model biases that may cause negative societal impacts.
INTRODUCTION
Emerging studies on the inner mechanisms of deep neural networks (DNNs) have revealed that many models have shortcut biases (Cadene al. 2019 Weinzaepfel & Rogez, 2021 Bahng al. 2020 Geirhos et al., 2020). DNNs olten pick up simple, non-essential cues, which are nonetheless effective within a particular dataset. For example, a DNN trained for the task of animal recognition may recognize ducks while attending on water backgrounds, given the strong correlations between such background cues and the target label (Choe et al. (020)). These shortcut biases often result in striking qualitative difference between human and machine recognition systems; for example, convolutional neural networks (CNNs) trained on ImageNet extensively rely on texture features, while humans would prefcrentially look the global shape of objects (Geirhos et al.) 2011)). In other cases, the shortcut bias arises in models that suppress certain streams of inputs: visual question answering (VQA) models often neglect the entire image cues, for one does not require images to answer questions like "what color is the banana in the image?" (Cadene et al., 2019)
Since DNNs have successfully outperformed humans on many tasks (Silver al. 2016 Rajpurkar let al. 2017), such a phenomenon may look benign at face value. However, the shortcut biases more challenging test-time conditions, where the shortcuts are no longer valid (Cadene et al. [019 Weinzaepfel & Rogez [201]] Bahng al. 2020 Geirhos et al. 2020). These biases also cause ethical concerns when the shortcut features adopted by model are sensitive like gender or skin color (Wang et al. 2019 Xu et al. 2020).
become problematic when it comes to the generalization
Instead of proposing method or solution, this work focuses on deepening our understanding of the shortcut bias phenomena. In particular, we design dataset where multiple cues are equally valid
First authors contributed equally.

---

Diagonal Training Data
Unbiased Testing Data
Scale
Scale
Train fon the diagonal dataset.
o
Predictions on fnt-diagonal us infer the que adopted by:-
Shape
Shape
A
Rted
edlapted cue
ccale
shape
X
Color
Color
Figure 1: Wisconsin Card Sorting Test for Machine Learners (WCST-ML). The training dataset (left) poses equally plausible cues to the DNN: color, shape, and scale. Which cue will a model ff choose to use, when given such training data? By examining f's output on unbiased off-diagonal samples at test time (middle), one may discover the cue has adopted. In the example: 1-scale, 2-shape, and 3-color.
for solving a particular task and observe which cues tend to be preferentially adopted by DNNs. The experimental setup is inspired by the Wisconsin Card Sorting Test (WCST,Banno et al. (012)) in Cognitive Neuroscience" See Figure for an illustration of the setup. It consists of a training set with multiple highly correlated cues (e.g. color, shape, and scale) that offer equally plausible pathways to the successful target prediction (Y' E (1,2,3)) We call this diagonal training set, highlighting the spatial arrangement of such samples in the product space of all combinations. A model f trained on such a dataset will adopt or neglect certain cues. We analyse e the cues adopted by model by observing its predictions on off-diagonal samples. With regards Figure for example, consider the f's prediction for a small, blue triangle as an off-diagonal sample. The prediction values ((A E 21,2,3) tell us which cue the model is biased towards to, e.g. if f (()) = 1, then f is biased towards scale; if f(A = 2, towards shape; and if f (A) 3 towards color. All three scenarios are plausible and only testing on off-diagonal samples will reveal model bias.
We make important observations the nature of shortcut bias under WCST-ML. We discover that, despite the equal amounts of correlations with the target label, there tends to preferential order- ing of cues. The preference is largely shared across different DNN architectures, such as feedforward networks, ResNets (He et al.) 2015), Vision Transformers (Dosovitskiy et al., 2021), and multiple initial parameters. From the parameter-space perspective, further observe that the set of solutions
O" biased to the preferred cues takes a far greater volume than those corresponding to the averted cues Oa. The loss landscape also tends to be flatter around @ than around O
Why are certain cues preferred to others by general DNNs? We provide an explanation based on the Kolmogorov complexity of cues, which measures the minimal description length for representing cues (Kolmogorov) 1963). Prior studies have shown that in the parameter space of generic DNNs. there are exponentially more Kolmogorov-simpl. functions than Kolmogorov-complex ones (Valle- Perez et al. (2019; De Palma et al.) 2211)). Based on these theoretical results, we argue that DNNs are naturally drawn to Kolmogorov-simpl cues. We empirically verify that the preferences for cues correlate well with their Kolmogorov complexity estimates.
What are the consequences of the inborn preference for simple cues? Firstly, this may hinder the generalization of DNNs to challenging test scenarios where the simple shortcut cues are no longer valid (Geirhos et al. (202)). Secondly, we expose the possibility that certain protected attributes cor- respond to the simple shortcut cue for the task at hand, endangering the faimess of DNNs (Barocas) ctt al. 2017). such a case, human intervention on the learning procedure may be necessary to enforce fairness, for the dataset and DNNs can be naturally drawn to exploit protected attributes.
The primary goal of this manuscript is to shed light on the nature of shortcut biases and the underly- ing mechanisms behind the scenes. Our contributions are summarized as follows: an experimental setup for studying the shortcut bias in-depth (WCST-ML) ($2); novel observations on the nature of shorteut biases, such the existence of preferential ordering of cues and its connections the ge- ometry of the loss landscape in the parameter space (:3); an explanation based on the descriptional
'Original WCST gauges the subjects' cognitive ability to flexibly shift their underlying rules (adopted cues) for categorizing samples. Inability to do so may indicate dysfunctional frontal lobe activities.
2

---

complexity of cues ($4); and a discussion the implications generalization and faimnsss ($5) such as the preferential use ethnical features for face recognition on the UTKFace dataset.
SETUP
We introduce the setup that will provide the basis for the analysis in this paper. We describe the pro- cedure for building dataset with multiple equally valid cues for recognition ($2.I) The procedure applied to DSprites and UTKFace datasets $2.2 In 2.. we introduce terminologies for the analysis of the parameter space and make theoretical connections with our data framework.
2.1 DATA FRAMEWORK: WCST-ML
Many factors affect the preference of models to certain cues. The existence of dominant classes is an example; it encourages models favor cues conducive to a good performance on the dominant classes (Barocas et al. 2017 Hashimoto et al., 2018)) In other cases, some cues have higher degrees of correlation with the target label (Geirhos et al. 2020). In this work, we test whether bias still present under fair conditions, i.e.: when a training dataset contains a set of valid cues, each of which equally correlates with the targets, will DNNs still have a preference for certain cues? If so, why?
To study this, we introduce data construction framework called Wisconsin Card Sorting Test for Machine Learners (RCST-ML), named after a clinical test in cognitive neuroscience (Banno et al. 2012). See Figure T for overview. As running example, we assume dataset where each image can be described by varying two latent variables, object shape and object color. Let and Y denote image and label, respectively. We write Xsj for the image with color and shape j. where i, E (1. LJ. When we want to consider K > 2 varying factors, we may write Xin..,.is .áx for the image random variable with hthh factor chosen to be ig E {1,.... I.J. Importantly, we fix the number of categories for each factor to T. enforce similar conditions for all cues. Similar learning setups have appeared in prior papers: "Cross- bias generalisation" (Bahng et al. (020). "What if multiple features are predictive?" (Hermann Lampinen (020), and "Zero generalization opportunities" (Eulig et al. 202T). While we fully acknowledge the conceptual similarities, we stress that our work presents the first dedicated study into the cue selection problem and the underlying mechanisms.
The same set of images {X.y 1 < i,j < }} admits two possible tasks: color and shape classifi- cation. The task determined by the labels Y. Denoting Yú, as the label for image X,j. setting Yij i leads to the color classification, and setting Yij j leads to the shape classification tasks. We may then build the data distribution for the task at hand via
D color
(Xiy, -.y= =i)
Pihape
U (Xi,, Yop -)
ISUJEL
IEUST
for color and shape recognition tasks, respectively. More generally, we may write
Dr =
U
(Xin Vin.... = ix)
(2)
18t AL
for the data distribution where the task is to recognize the tthh cuc. We define the union of random variables as the balanced mixture: U 7, == Zi where 1 ~ Unif (1, L.).
We now introduce the notion of diagonal dataset, where every cue (e.g. color and shape) contains all the needed information to predict the true label Y. That is, a perfect prediction for either color or shape attribute leads 100% accuracy for the task hand. This can be done by letting the factors always vary together = j in the dataset (and thus the name). We write
Ddiag = U (Xu, Yu i).
(3)
1SISL
Such a dataset completely leaves it to the model to choose the cue for recognition. Given a model f trained on Daiag analyse the recognition cue adopted by f by measuring its unbiased accuracy on all the cells (See Figure I]. There are K different unbiased accuracies for each task, depending on how the off-diagonal cells are labelled: e.g. Deolor and Dahape in equation For a general setting with K cues, the unbiased accuracy for thth cue is defined
3

---

Y-itfnt
Y-chidren Y-young adul:
Y=1 Y=2
Y=3
T-WWhet
Y-1
Y-indan
Y=2
Y-AAson
Y-Block
Figure 2: Dataset samples. The same set of images for each dataset accommodates tasks, depending on the labeling scheme (row-wise or column-wiss The diagonal datasct Daug (indicated with borders of color is identical regardless of the row- or column-wise labeling. Left: DSprites. Right: UTKFace.
accx(() = I
Σ 11,""
[ .K) =ig].
Proposition For k E {1,... K), accç (fcx(f)= = if and only if (( (x) in almost surely for all 1 S i.. ig S. L. Moreover, if the condition above holds (i.e. f is perfectly biased to cue k), then accm.f) T. for all + k.
The proposition implies that the unbiased accuracy capable of detecting the bias in a model f: (f) implies that f's prediction is solely based on the cue k. also emphasizes that is impossible for a model to be perfectly biased to multiple cues. Finally, we remark that the WCST-M. MML analysis does not require the cues to be orthogonal or interpretable to humans. The only requirement is the availability of the labelled samples (Xi,,.. .ix Yir... for the cue of interest.
2.2 DATASETS FOR ANALYSIS
DSprites (Matthey et al. 2017) is an image dataset of symbolic objects like triangles, squares, and ellipses black background. It consists of images with all possible combinations of five factors: shape, scale, orientation, and X-Y position. The total number of combinations 3 x x 40 x 32 x 32 = 737, 280. We augment the dataset with another axis of variation: 4 colors (white, red, green, and blue). This results in 737, 280 4 = 2, 949, 120 images in the augmented dataset. Each image is of resolution 64 64. We have identified degenerate orientation labels due to rotational symmetries for certain shapes. We process the data further by collapsing the symmetries to a unified orientation label. For example, we assign the same orientation label for squares at rotations 0, 2. म, and 3T
UTKFace (Zhang et al. 2017) is a face dataset with 33, 488 images with exhaustive annotations of age, gender, and ethnicity per image. We utilize the Aligned and Cropped Face version focus the variation in the facial features only. Ages range from to 116; gender is either male or female; and ethnicity one White, Black, Asian, Indian, and Others (e.g. Hispanic, Latino, Middle Eastern) following Zhang et al. (2017). Images are resized from 224 x 224 to 64 x 64.
Applying the data framework. For each analysis, we select subset of features and build the training set D ding and test sets Dr ($2.1) anew based on S. Figure 2 shows examples from DSprites with S = (shape,
and UTKeace with S fage, ethnicity), ). respectively. To enforce the same number of classes L per feature, we set 1 as the minimal number of classes among the selected features S. For features with #classes L, we sub-sample the classes to match #classes = For continuous features like age and rotation, we build categories based on intervals, defined such that the classes are balanced. We randomly vary cues & S in the constructed datasets Daing and Dz- Figure shows the example datasets of Daing and Dj for each dataset.
2.3 PARAMETER SPACE AND SOLUTIONS
We define the hypothesis class for DNNs through their weights and biases, collectively written as 0 E O (e.g. all possible weight and bias values of the ResNet50 architecture (He et al. (016)). We
4

---

will use non-negative loss L 2 0 to characterize the fit of the model fo on data D. 0-1 loss (and cross-entropy loss) is such an example:
[(;; D) Prxx.))=olfa(() # Yl.
(5)
We define the solution set as 0 = {0 E e L(0; = 0}: in practice we define it with a small threshold [(0; D) S
We now make a connection between the datasets and solution sets. Assume that we have a dataset of form (Xej, Yij) where Yij (the color task; see 2. The corresponding solution set is defined
as θ 1 = {0: ((; (Xij. Yu = =]) = =0. Define it similarly for the shape task (Yij =).
Proposition 2. Df (E (X1j. Yiz) for some index set I, then the corresponding solution set the intersection of the solutions sets N(JeE 0
Proof. If L never attains O, the solution set is always empty and the statement is vacuously true. Otherwise, we may decompose [(0) into the average [ ((.)) (6) Due to the non non-ne non-negativity of the loss function, [()) = if and only if Liz (0) = 0 for all (i,j) E.
The proposition leads to a neat summary of the relations among solution sets like Ocolor == {0- C(0; (Xij, i)) 0} and shaas defined similarly. Corollary 3. Ocotor daay and O shape 2 diag Moreover, Ocalor Shshpe Ø.
Proof. The first statement immediately follows from the proposition above and the fact that Daing
Deotor n Dshupe- For the second, observe that there no function that satisfies (X.)) = i and (x(X j at the same time for i a
We may easily extend the result datasets with K cues. Given the solutions sets θ₂ for each cue < l < K, we have θ₂ C Oing for all k and {+x) are pairwise disjoint (diagram on the right). This follows from Proposition I that single model cannot achieve low error for multiple cues (D₂) at the same time.
0x
e
Bais
3 OBSERVATIONS
Based on the proposed WCST-MM MLL, we study the natural tendencies of deep neural networks (DNNs) to favor certain cues over the others. We also present parameter-space view that hints to underlying mechanisms for the preference.
3.1
PREFERENTIAL ORDERING OF CUES
In the first set of experiments, we observe which cues are preferentially used by the network when training on a dataset composed of an equally represented set of cues: the diagonal datasel Daing in
$2.1 We train three qualitatively different types of DNNs: (1) Feed Forward neural network (FFnet),
(2) ResNet with depth 20 (ResNet20), and (3) Vision Transformer (VIT). Details of the architectures
and training setups can be found in Appendix A The results are summarized in Figure 3
Models adopt cues with uneven likelihood. FigureJshowe the unbiased accuracies of the models when trained on the diagonal sets for each dataset (DSprites and UTKFace) with the involved cues denoted as S. We report the diagonal accuracy and the K unbiased accuracies for each cue in S on the held-out test set. The number of labels is for DSprites and for UTKFace. On DSprites, we observe that for all architectures, color is by far the most preferred cue (unbiased accuracy near 100%). For FFnet and ViT, the extreme color bias forces the accuracy on other cues to be T, = 33.3% as predicted by Proposition ] ResNet20 behaves less extremely and picks up scale, shape, and orientation, in the order of preference. On UTKFace, there is no extreme dominant cue, but there is general preference to ethnicity cues for making predictions, followed by gender and age cues. The trend is clear for all models, and while the FFnet model shows slightly more variable performance across runs, the ranking of cues is preserved throughout. The preference of
5

---

3agor
Ban
DSprites
oonenlation color
ralo
color
shape scale
80
80
Accurscy
Accurscy
Accurscy
60
2taps
trale
stab
60
shage
20
20
Hape MS 100
60
80
100
120
20
40
80
100
120
40
60
80
100
120
Epoth
Epach
Epoch
100
100
100
Face
80
atheicity
Stinlct
gender, age
Accuracy
Accuracy
Accuracy
etuncity
40
40
20
100 120
20 40
3
Epoch
FFnet
ResNet12
ViT
Figure 3: Preferential ordering cues. Diagonal and unbiased accuracies of models trained on the diagonal
training set Daing composed the cues defined by S and tested on the off-diagonal sets Dj where k
Color Solution
Scale Solution
Shape Solution
Orientation Solution
Ethnicity Solution
Age Solution
Gender Solution
Figure 4: visualisation of loss surface around different solution types e*, For each cue, we show 3-D plots of the same loss surface (side and top views). X and Y axes are randomly chosen directions in the parameter space; axis shows the loss value each (X,Y) measured against the diagonal data Daiag. All the surfaces are shown at the same scale.
ethnicity echoes the DNN's preference of color strong cue in DSprites; DNNs are drawn to utilizing skin color. Another curious phenomenon is that none of the three cues sufficiently explains the 100% diagonal accuracy, suggesting the existence of an unlabeled shorteut cue other than the three considered. Finally, when the most dominant cue is left out, other cues activate in its place: scale for DSprites and gender for Face (Appendix B
Consistency of preferences. We have observed a consistent preferential ordering of cues shown by qualitatively different types of architectures. The results suggest the existence of common denominator that depends solely on the nature of the cues, rather than the architectures. We finally remark that the experiments were run 10 times with different random initialization; error bars in Figure 3 show to standard deviation. The small variability observed across most training shows the consistency of the preferential ranking with respect initialization. For pair of cues, we refer to the one that is more likely to be chosen by model trained on Daiag the preferred cue and the other as the averted cue.
3.2
ABUNDANCE PREFERRED-CUE SOLUTIONS
Why are certain cues preferred to others? We look for the answers in the parameter space. More specifically, study the geometric properties of the loss function and the corresponding solution sets the parameter space. The section based on 22.) We write the solution set corresponding the preferred cues as preferred solutions θ and averted cues averted solutions O..

---

How to find the averted solutions 0 E O Naively training a DNN Dang will most likely result in a preferred solution p.. We thus use the full dataset Dx corresponding to the averted cue k. ($2.T) as the training set to find averted solution 0 θ,, To ensure fair comparison in terms of the amount of training data, we also use the full dataset Dj for the preferred cues.
Qualitative view on the loss surface. We examine the geometric properties of the loss surface for ResNet20 around different solutions types (preferred and averted). We use the 2DD visualization tool for loss landscape by Li et al. (2018). The X-Y grid is spanned by two random vectors in IR where D is the number of parameters. In Figure 4 we observe that the preferred solutions (e.g. color) are characterized with a flatter and wider surface in the vicinity. In contrast, local surface around averted (e.g. urientation) solutions exhibit relatively sharper and narrower valleys. A similar trend is observed for the UTKKFace cues at a milder degree: eutinicty is a slightly flatter solution than θ age.
Size of the basin of attraction. We compare the size of the preferred Q and averted O solution sets using the notion of the basin of attraction (BA) for cue k: subset N in the parameter space such that initializing the training with a point Oo E leads to a k-biased solution 0* E O₁. To measure the size of the BA for θ₁, start by obtaining solution 0g E 0₂ biased to the cue k. We then check when the final models trained from perturbed points 0% + v (where |//||2 is small) stops being biased to the cue k. We treat the norm of the bias-shifting critical perturbation ||u||2 as the size of the BA for O
boen for @
8
Preferred solutions have wider BA. Figure 5 shows the size of BA measured according to the method above. The upper row shows initialization around solutions biased to cues in DSprites. For the color cue, we observe that initializing far away from the color-biased solution still results in color-biased solution. For shape, scale, and orientation cues, we observe that initializing away from the respective solutions gradually results in color-biased solutions with reduction in the bias towards the respective biases in the umperturbed initial parameters. The sizes of basins of attrac- seataape>scaleorrriiia
largely following the preferential ordering observed in $3.1 The lower row of 3 shows the results for the three cues UTK- Face. As opposed to DSprites, the solutions for ethnicity, age, and gender do not re-converge to a clear cue after retraining for only 50 epochs. The sizes of basins of attraction are ordered as cchnictt>-gender>aaee following the preferential ordering of cues observed in $3.1
tion are ordered according to
Path connectivity. Recent works have proposed methods find near zero- loss paths between two solutions (Garipov et al.) 2018 Draxler et al. 2018 Benton et al. 202T). We extend the technique to find the zero-loss (w.r.t. Daing path between preferred and averted solutions. We locate the flipping moment for the bias along the path; there must be bias-shift boundary where the bias to the preferred cue shifts to the averted cue along the path (by an intermediate- value-theorem-like argument). By comparing the length of the segment up this boundary, we may estimate the relative abundance of the preferred and averted solutions in the parameter space.
Odiox
Paths connecting preferred and averted solutions and the bias-shift boundary. Figure 6 visu- alizes the paths between the preferred and averted solutions as well as the corresponding shifts in biases along the path. We observe that the paths generally show a single shift in bias. The shift is also usually quick; for the most part the path, the solutions tend to be heavily biased towards ei- ther of the cues. For the path from Ocalor to 2 Opcienation we observe tendency for Ocokar to dominate the path until near Oocientation: suggesting the relative abundance of color-biased solutions Ocolor in the set of diagonal solutions Oding compared to the orientation Oorisnaaionn The path from @stnnicity to ④ ggender is composed of a longer segment for the Qettmicity part. The relative abundance cues in the diagonal solutions Odiag resonates the preferential ordering of cues seen in $3.1
4 EXPLANATIONS
We have observed intriguing properties of cues and shortcut biases. Even under the equal oppor- tunities for the cues to be adopted, models have preference for few cues over the others. The
7

---

132
102-
100
Accuracy
Accuracy
Accuracy
Accuraxy
60
Shapp
R
53
ocale
Sc3le
Shape
14
20
zhape
50 25 10e
150
75 18t
perturbation I|vla
perturbation Ilvil:
perturbation [Ivils (c) Scale
perturbation lilll:
Color
(b) Shape
(d) Orientation
60
Accuracy
Accuracy
Accuracy
stnnigto
Bendia
25 53 75 366 125 15o 1/s 260
25 50 IS 103 125 350 176 263
25 S0 75 36 156 17 200
Size of perturbation (iVla
Size of perturbation [Mll
Size of perturbation (MVl2
(c) Ethnicity
(f) Gender
(g) Age
Figure 5: Basin of attraction. We initialize ResNet20 around solutions biased to each of the four in DSprites (upper row)
cues UTKFace (lower row). report the averaged unbiased accuracies for models initialized with perturbed solutions 8" + The x-axes are the perturbation sizes |||||2 From perturbed initial parameters, each model is trained for 50 epnchs.
Accuracies on path
Diagonal loss
Preferred-cue loss
Averted-cue loss
orientation
25
Ethnicity Color gender
Calo
Dageral
675
625
orfentation
oTennation
Fath-Strp
6822
63s
27
64
Gurder
288
2014
perterr
TethStop
Figure 6: Paths connecting O and O We compute paths between preferred and averted solutions along which the diagonal-datasee (Daug) loss is zero. On the leftmost column, we show the unbiased accuracies for the preferred and averted cues along the path; columns visualize the loss values around the zcro-loss paths (solid curves), with respect the losses computed on diagonal Daig- preferred cue D,. and averted cue De datasets. The zero-loss paths are quadratic Bezier curves, parametrised by three dots shown in the plots: the two end points (preferred and averted solutions) and a third point that regulate the curve shape by "pulling" it. Inspired by Garipov al. (2018)
observe bias shift in the middle in cach case. The three rightmost
preference is stable across architectures and weight initialization. We have also found strong cor- relation between the preference for cue and the abundance of solutions biased to that cue. In this section, we explain the observations based the complexity of cues.
Kolmogorov complexity of cues (KCC). We formally define cue as the condition distribution Pyix for given input distribution px: in sense, concurs with the definition of task. Kolmogorov complexity (KC, Kolmogorov (1963)) measures complexity of binary strings based on the minimal length among programs that generate the string. |Achille al. (2018) have defined the Kolmogorov complexity of task (or a cue) Py[x
K K(prix)=
min K K()) -8
(6)
for some sufficiently small scalar 8 > f denotes a discriminative model mapping to and is a suitable loss function. K K(J) is the KC for model f. Intuitively, K (pYlx) measures the minimal complexity of the function f required memorize the labeling PY]x on the training set (i.e. 8).
Measuring KCC. We estimate the KC of four cues (color, scale, shape, and orientation) consid- ered in DSprites based on equation 6 Since is impractical to optimise K(f)) over an open-world of all possible models use ResNet20 with variable channel sizes (Zagoruyko & Komodakis 2016) for F. Given that KC uncomputable, the number of parameters in f approximation to K(J) in practice
constram the search homogeneous family of models f E F. We
8

---

Valle-Peree et al. 2019 We set the memorization criterion Y) < å as "training set error for f is < 1% " Under this setup, we have verified that the complexities of cues are 1.2K parameters for color, 4.6K for scale, 17.6K for shape, and 273K for orientation. We were not able to fully memorize the orientation labels under the network scales we considered. The complexity ranking strongly concurs with the preferential ordering found in $3.1
The simplicity bias in DNN parameter space. How the cue complexity and shortcut bias connected? We explain the underlying mechanism based on a related theory a different context: explaining the generalizability of DNNs with the simplicity bias Perez- et al. 2019 (Sahh et al. 2020). It has been argued that, despite a large number of parameters, DNNs still generalize due to the inborn tendency to encode simple functions. This phenomenon has filled the theoretical gap left by the classical learning theory. The argument is that a (uniform) random selection of parameters for a DNN likely to result in a function f that is simple in the sense of KC. More precisely, the likelihood of a function f being encoded by a DNN is governed by the inequality p(f) S 2 2-oR(
where (f)) the KC of f and
O, b are constants independent of (Valle-Perez et al. 2019)
This inequality stems from the Algorithmic Information Theory and has been validated to hold for large class
a large variety of systems that arise naturally, such as RNA sequences and solutions of differential equations (Dingle et al. 2018). The argument concludes that the natural abundance of simple functions in the parameter space makes it more likely for the solutions to be simpler (Wu) et al. 2017). leading to the simplicity bius and thus helping DNNs generalize well.
The simplicity bias leads to shortcuts. We have seen that cues have different KCs, necessitating DNNs with different complexities fully represent them. The simplicity bias in the parameter space explains the abundance of solutions biased to simple cues like color color rather than complex cues like orientation Ovveenation (3.2). The difference the abundance in turn makes it more likely for a model trained on the diagonal dataset Danng to be biased towards simple cues ($3.1) T.
DISCUSSION AND CONCLUSION
We have delved into the shorteut learning phenomenon in deep neural networks (DNNs). We have devised training setup WCST-M MLL where multiple cues are equally valid for solving the task at hand and have verified that, despite the equal validity, DNNs tend to choose cues in a certain order (e.g. color is preferred to orientation). We have shown that the simple cues (in the sense of Kolmogorov) are far more frequently represented the parameter space and thus are far more likely to be adopted by DNNs. We conclude the paper with two final remarks on the implications of our studies.
Simplicity bias and generalization. The simplicity bias theory (Valle-Perez et al. 2019) has been developed explain the unreasonably good generalizability of DNNs. Our paper shows that it may interfere with the generalization. This apparent paradox stems from the difference in the setup. The simplicity bias often safe to be exploited in the standard iid setup, albeit with caveats argued by Shah et al. (2020). On the other hand, it may lead to less effective generalization across distribution shifts as in our learning setup (Geirhos et al. 2020), if the simple cues are no longer generaliz- able. Thus, for the generalization across distribution shifts, machine learners may need to be given additional guidance to overcome the inborn tendencies to pick up simple cues for recognition.
Societal implications. An issue with the simplicity bias and shorteut learning is that relying on simple cues is sometimes unethical. We have seen in the UTKFace experiments that ethnicity tends to be a more convenient shortcut than age, for example, they equally correlate with the target label at hand. This is an alarming phenomenon. It emphasizes the importance and inevitability of active human intervention to undo certain naturally-ane ariisig biases to socially harmful cues in learned models. It also confirms that we are the right track to actively regularize the models to behave according to the social norms (Barocas et al. 2017 Mehrabi et al. 2021).
Toolbox contribution. Along the way. we contribute interesting novel tools that future research can utilise, such the computation of Kolmogorov complexity (KC) for generic cues (equation [) and the method for finding a heterogeneous zero-loss curve connecting two solutions biased to dif- ferent cues 333.2 Garipov et al. (2018))

---

REPRODUCIBILITY
We understand and resonate with the importance of reproducible research practices in machine learn- ing. The reproducibility of this paper hinges on our use of open-source data, models, training tech- niques, and deep learning frameworks. We have specified a detailed experimental setup in $2.2 and { For the consistency of experimental observations, we have run the experiments multiple times with different random seeds, whenever applicable. For those cases, we have reported the mean and standard deviation for each data point.
ETHICS STATEMENT
We delve into the potential ethical issues with DNNs. Our work excavates the potential ethical harms caused by deep neural networks' natural tendencies to prefer simple cues, where using the simple cues may not conform well to the social norm (e.g. ethnicity cue). We have thus advocated an active human involvement regularize such DNNs adopt more socially acceptable cues. As for the experiments, part of our studies is based on a face dataset (UTKFace)). We confirm that the UTK- Face dataset has been released with open-source license with the non-commercial clause The copyright for the images still belongs to their respective owners. Inheriting the policy of UTKFace, we will erase the face samples Figure upon request by the owners of the respective images.
REFERENCES
Alessandro Achille, Glen Mbeng, and Stefano Soatto. Dynamics and reachability of learning tasks. arXiv preprint arXiv:1810.024400 2018
Hyojin Bahng. Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning (ICML), 2020.
Masahiro Banno, Takayoshi Koide, Branko Aleksic, Takashi Okada, Tsutomu Kikuchi, Kunihiro Kohmura, Yasunori Adachi, Naoko Kawano, Tetsuya lidaka, and Norio Ozaki. Wisconsin card sorting test scores and clinical and sociodemographic correlates in schizophrenia: mul-
tiple logistic regression analysis. BMJ Open, 2(6), 2012. ISSN 2044-6055. doi: 10.1136/ 20ijopen-2012-000
URL
bmj. com/ /ttmm//oonnn/////00000
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness in machine learning. Conference on Neural Information Processing Systems (NeurIPS) Tutorial, 1:2017, 2017.
Gregory W Benton, Wesley J Maddox, Sanae Lotfi, and Andrew Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. arXiv preprint arXiv:2 10f.::/02213444: 13042, 2021.
Remi Cadene, Corentin Dancette, Matthicu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. In Advances in Neural Information Processing Systems, pp. 839-850, 2019.
Junsuk Choe, Seong Joon Oh, Seungho Lee, Sanghyuk Chun, Zeynep Akata, and Hyunjung Shim. Evaluating weakly supervised object localization methods right. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Giacomo De Palma, Bobak Kiani, and Seth Lloyd. Random deep neural networks are biased towards simple functions. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu -uc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL httpps:::ppooceeddrggee/oo Feab05aa9 00201005577885 005555aa999185555812252852 1555778888011111155333335-- Papess pdf
newr ieeurrps. ips. cc/paper /2019//////9//ee
Kamaludin Dingle, Chico Q Camargo. and Ard A Louis. Input-output maps strongly biased towards simple outputs. Nature communications, 91):1-7,
10

---

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations (ICLR), 2021.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309- 1318. PMLR. 2018.
Elias Bulig. Piyapat Saranritticha,, Chaithanya Kumar Mummadi, Kilian Rambach, William Beluch, Xiahan Shi, and Volker Fischer. Diagvib-6: A diagnostic benchmark suite for vision models in the presence of shorteut and generalization opportunities. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), October 2021.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Proceedings of the 32nd Inter- national Conference on Neural Information Processing Systems, pp. 5803-8812, 2018
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trainee cnns are biased towards texture: increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020.
Tatsunori Hashimoto, Megha Srivastava, Hongscok Namkoong, and Percy Liang. Fairness with- out demographics in repeated loss minimization. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceed- ings of Machine Learning Research, pp. 1929-1938. PMLR, 10-15 Jul 2018. URL https:
//proceedings. .ml..mmr..... www.m.r....s......mmmmmmmmmmmm.e /vress/v80/A https//////aaaaooolloael1o html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9995-10006. Curran Associates, Inc., 2020. URL http::///ooceeeeeeeeeeooe .eeuuers www.rip........... cc/
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
Andrei N Kolmogorov. On tables of random numbers. Sankhyā: The Indian Journal of Statistics, Series A, pp. 369-376, 1963.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018 URL /tpppp/psp///rrrceeiissssss newrpesss....................... http://cccc/pppp/1/////LL pdff
6
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset.
2017.

---

Ninareh Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, al. Chexnet: Radiologist-lvvel pneumo- nia detection on chest x-rays with deep learning. arXiv preprint arXiv: 2017.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli, The pitfalls of simplicity bias in neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9573-9585. Curran Associates, Inc., 2020. URL http::tt::////tt U hew.ooceeeddnnng..............pp neurips. cc/ htt.://///1.....00020aa220222200
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-48,, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes be- cause the parameter-function map is biased towards simple functions. In International Confer- ence on Learning Representations, 2019. URL /ett::://openrewwe..eee. net. forum?id Tye4q3AqFm
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
Philippe Weinzaepfel and Grégory Rogez. Mimetics: Towards understanding human actions out of context. International Journal of Computer Vision, 12965):665--1690, 2021.. 2021.
Lei Wu, Zhanxing Zhu, al. Towards understanding generalization of deep learning: Perspective of loss landscapes. arXiv preprint urXiv:/706 /0239, 2017.
Tian Xu, Jennifer White, Sinan Kalkan, and Hatice Gunes. Investigating bias and fairness in facial expression recognition. In European Conference on Computer Vision, pp. 506-523. Springer, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In The British Machine Vision Conference (BMVC), 2016.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regressoon by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5810-5818, 2017.
12

---

A MODEL ARCHITECTURES
The FFnett architecture tested a fully connected feedforward neural network with hid- den layers with respectively 4096, 2048, 600, 300 and 100 hidden units. ReLU activations are used throughout. We train using the Adam optimizer with the default parameters given by PyTorch 1.8.0.
FFnet.
ResNet20. The ResNer20 architecture adapted from He et al. (2016), with a depth of 20. We train it using the Adam optimizer with the default parameters given by PyTorch 1.8.0.
VIT. The Visual Transformer architecture is adapted from (Touvron al. 202T). Given the di- mensions of the images in ColorDSprites and consequently UTKFaae, we implement VIT model with patch size 8, embedding dimension 192, depth 12, and 3 attention heads. We train model with the SGD optimizer with learning rate x 10 momentum 0.9, and weight decay 1 10-4.
B PREFERENTIAL ORDERING AFTER REMOVING THE DOMINANT CUE
For both DSprites and UTKFace, removing the dominant cue (color and ethnicity respectively) from S does not change the ranking of the remaining cues; e.g. scale and gender then become the dominant features in respective datasets. See Figure B. that extends the main paper Figure
deggnal
DSprites
shape, scale orientation
scale
Accuracy
Accuracy
Accuracy
40
orsentation
Of ontination
40
20
20
20
60 Epach
00
100
120
60 Epach
00
100
120
60 Epach
00
100
120
$
100
dlggonal
100
diagona
100
UTKFace
Accuracy
gender
Accuracy
gender
Accuracy
gender
200
20C
200
gender
age
4nmmC ANE IECEO
22
Epoch
Epoch
Epoch
FFnet
ResNet20
ViT
Figure B.1: Preferential ordering of cues. Diagonal and unbiased accuracies of models trained the diagonal training set Daing composed of the cues defined by & and tested on off-diagonal sets D3 where k E.S.
C LOSS CURVATURE AROUND SOLUTIONS
We supplement the visualization of loss landscape around solutions in Figure of main paper with quantitative estimation of the respective curvatures. The mean curvature of surface defined by function f is defined as:
x()) = - = 1 888
1 Af
8f
(C.1)
D
dxx dx²
dx
To make it computationally tractable for large D (e.g. DNN parameters), estimate the curvature with a Monte-Carlo estimation as done in Izmailov et al. (2018). For point æ E RD, a sufficiently small scalar > and unit sphere S RD, the curvature is computed by:
x(f 1
z [E + er) f(x)] where r~ Unif(S
(C.2)
How does this approximate the mean curvature? We provide the following proposition.
13

---

Proposition 4. For point E RD, scalar 0, twicc-differentiable function and unit sphere SD C RR,, the following quantity
[f +r) -fa)) where T~ Unif(SD)
(C.3)
approximates the Laplacian A up constant multiplier.
Proof. We begin with Taylor expansion on f(æ + t
[ + er) = f(x) + er +rr 1 (()) (1 ((e,))
(C.4)
where blerr) + as e 0 for any r. Thus,
k(( 2 Ef(x +r) - f(x)) = r T (z)r +
(C.5)
2 1 B|r [())] o(t)
(C.6)
by 0 and the Dominated Convergence Theorem Lebesgue integrals (also noting that defined on compact set SD-1).
We now apply change of basis such that the Hessian V (xx(x) is diagonalised: (=) AQ.
This is possible because the Hessian is real, symmetric matrix. Since the change of basis do not alter the distribution for r, we compute
V ((x)r) = "E[""Arr = Var(r: ) + +... + Var(rD)Ap
(C.7)
Var((r) = + Ap) Var(ri)tr(V Var(r:)4f(z)
(C.8)
using the isotropicity of r and the preservation the trace against changes basis. We thus con- clude
1
(C.9)
2
where C' is constant independent of f and
Results. Figure C.I shows the approximated curvature at varying radius values around different solutions for ResNet22.. We have used 100 Monte-Carie samples. We observe that for DSprites, the curvature of the orientation loss surface tends to overshoot quickly as one walks away from the solution point. Surfaces for other cues tend be smoother. On UTKFace, gender tends provide steeper surface than the other cues like age and ethnicity.
0.010
Mean curvature (X)
0.008
Color
Mean (e)
Age
0.008
Ethnicity
0.006
Scale
Gender
Shape
0.006
0.004
0.004
0.002
0.002
-0.002
0.000
140
50
60
Weight offset radius (6)
Welght offset radius (e)
Figure C.I: Mean curvature around different solutions. Left: DSprites. Right: UTKFace.
14


Reviews:
[{"review_id": "gXTSZocs01", "reviewer": "Reviewer_QJQp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is written well, the claims are laid out clearly, and the method is largely well described. The approach seems methodically sound given the problem statement  (caveat in W2 below). The path of investigation is fundamentally valuable in the sense that models of biases and generalization of networks are helpful tools.\n\nWeaknesses:\n[W1] Some parts of the exposition need minor clarification:\n(a) p6: \"The trend is clear for ResNet20 and ViT, while the ethnicity and gender preferences are within the error bars for FFnet.\" This refers to Figure 3, and I cannot understand this from Figure 3, in the least it is confusingly stated. The ethnicity and gender graphs for FFNet in figure 3 seem separated by several standard deviations according to the error bars.\n(b) Figure 6, three right columns: What does the dot and the black arch signify? I was not able to gain this from text or caption. I assume that the zero loss path is the dashed line?\n\n[W2] A somewhat methodical weakness of the paper seems to be the problem statement. For instance in the introductory example of section one/figure 1, the authors imply that the model is biased towards scale (then shape, then color). I do agree that this is consistent with the model preferring the property of scale in making the decision. Is this bias, though? In order to evidence bias, I'd need a truth/label assignment on the off diagonals (e.g. indicating that blue small triangle is not supposed to be class 1). Without this truth assignment I do not see the bias of the model.\n\n[W3] The main weakness of the paper seems to me that I am asking myself: What have I learned after reading the paper? I feel that the experiment setup is sound in the stated sense: what does the network learn if it only ever sees pathological samples (i.e. where all latent variables are exactly correlated). However, the presented answers that networks tend to learn simpler cues, where simpler can be measured by Kolmogorov complexity are not overly surprising or counter-intuitive. Perhaps the authors can rephrase the contributions more clearly towards the benefits? For instance: With the tools, analysis and exposure of this paper, what can I now see different, do different, etc. What are possible next steps where stronger impact is on the horizon?", "summary_of_the_paper": "The paper discusses biases in inductive learning in deep neural networks that stem from pathologically sampled data. The authors pose a problem set where a trainer only sees samples where two or more latent values can only be observed in a fully correlated fashion (e.g. scale, color, shape). They then design various criteria and protocols to gain insight in the behavior of the learned model when the correlation does not hold any more (i.e. samples that have not been seen in the training data). They conclude that in this case of generalization to unseen data, the trained model has an implicit bias towards choosing (1) mostly single cues to determine the predicted label (e.g. color only), (2) that more preferred cues are simpler than less preferred cues and (3) that the underlying solution space of possible parameters has more solutions that prefer the simple cues. They demonstrate empirical results on variations of existing datasets (DSprites, UTKFace).   ", "summary_of_the_review": "The paper addresses an important topic, but I feel that the overall impact, measured by depth of presented contributions is too shallow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XPOiPxLU6FG", "reviewer": "Reviewer_tkjD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nOverall this is a well-written paper, clearly motivated and carefully constructed. The finding that the number of solutions (i.e. parameter configurations) that rely on preferred cues are also abundant is a good corroboratory result.\n\nThe motivations for this study are also exactly what the field needs at this point, given the abundant use of deep convolutional networks for a variety of applications with minimal understanding of its exact decision-making mechanisms.\n\nThough the synthetic task (Wisconsin card sorting) has been widely adopted in the cognitive neuroscience community, their formulation for mainstream ML is nifty and allows for systematic empirical evaluation.\n\nWeaknesses:\n\nHaving said that, this paper fails to deliver on its promise of intricate analysis.\n(i) There is an inherent question that authors fail to sufficiently address. The selected \"cues\" for analysis have intrinsically different extraction demands from the stimuli, thus making the argument about them being \"equally prevalent\" moot. For example, accessing color is more direct (pixel-level information is directly available) than accessing, say, shape (for which a network needs to build sufficiently large receptive fields to understand the global geometry of a scene).\n\n(ii) The experiments on the UTKFace dataset are not convincing. The authors themselves acknowledge that there might be other \"shortcut cues\" outside of the selected ethnicity and age cues. This brings up a subtle (yet important) question. In naturalistic datasets, how can one determine or interpret a basis set of cues that are truly orthogonal dimensions? The experiments on WCST-ML work because the shape, color, scale, and orientation are by definition orthogonal. For all we know, the \"cues\" in naturalistic datasets could be abstract and not human-interpretable too.\n\n(iii) Treating the number of model parameters as a proxy for the Kolmogorov complexity of the input-output mapping seems ill-advised. If one trains a parameter-shared recurrent neural network it can perform the same effective computations as a ResNet, but with much fewer parameters (though the task per se has not changed, and by extension the cues). I would appreciate it if the authors can offer some insight into this.\n\nMinor:\n(i) Figures need to be made more legible. Axis labels and text insets are barely visible without zooming in.\n(ii) Particularly, Figure 4 could be refined further. The caption isn't very descriptive, and the section on \"Qualitative view on the loss surface\" is a bit misleading. The ethnicity solution doesn't seem to be \"characterized with a flatter and wider surface\" as compared to age and gender solutions in Fig. 4. If there are more obvious examples, the authors should use them instead.", "summary_of_the_paper": "The authors propose a framework for studying the tendency of deep neural networks to preferentially adopt \"cues\". Specifically, they focus on settings where multiple cues are equally likely, though not all of them are equally exploited. To set up such a scenario, they introduce the WCST-ML task, in which the prevalence of cues can be parametrically controlled. They also conduct empirical studies on the more naturalistic UTKFace dataset. The authors introduce a set of metrics, such as path connectivity, attractor basin properties, etc. to analyze cue preferences from a loss landscape perspective. The authors also explain these observations based on the \"complexity\" of cues.   ", "summary_of_the_review": "This is a well-written paper trying to address an important problem. However, as I've expressed in the main review, some of the claims are unsupported and hence my initial assessment. I am willing to update my score if the authors are able to provide a convincing response!", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NRueFjy40Z_", "reviewer": "Reviewer_s3vA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is a very interesting read. It provides an interesting analysis of model preferences for various visual cues. The analysis shows the preference of visual models towards low complexity visual cues, such as color and ethnicity. \n\nMain comments:\n\n- Averted solution or averted cues are not introduced, before being used in section 3.2. What is the difference between averted and preferred? Section 3.2 seems to imply that preferred solutions are computed by optimizing the model using D_diag and averted solutions computed using D_i for a specific property i that is not deemed preferred. Am I correct? If this is the case, it seems unreasonable to be comparing the two solutions directly (eg in Fig 4) since they are computed using datasets of different sizes.\n- In Fig 4, it also is not clear what the loss landscape represents. Section 3.2 mentions that it's the loss around averted and preferred solutions, but Figure 4 does not mention which ones are averted and which ones are preferred. Also, Figure 4 says that the two directions of parameter variation were chosen at random. Given the high dimensionality of DNN parameters, it would be much more informative to depict several of these plots per solution, to cover more directions of variation.\n- Section 4 provides evidence to the idea that certain cues have higher complexity than others. This is measured by searching for the smallest model that can memorized the training set. Color could be memorized by a model with only 1.2K parameters, and orientations needed 273K parameters. A natural question is then whether large models present less biases towards simpler cues. As the parameter count increases, the solution set for the orientation task must also increase. It would be interesting to see if the model preference for more complex cues would also increase, or if solution sets for simpler cues would still dominate for very large models.\n- Although the diagonal dataset construct is appropriate to see which cues are preferred by a model, in practice this construct almost never applies. Real data may present correlations similar to those found in the diagonal construct, but will also contain off-diagonal samples. Thus, an interesting analysis would be to measure how much you'd need to deviate from D_diag to obtain an \"averted solution\". In other words, if we train a model with (100-p)% diagonal samples and p% off-diagonal samples labeled according to cue i (e.g. labeled for orientation), how likely is the model to correctly predict cue i for increasing values of p?", "summary_of_the_paper": "The paper conducts a study of which visual cues are preferred by current vision models. The paper designs a training setup with several cues where each cue is equally correlated with the image label. The paper shows that visual cues like color are much easier to be learned by a vision model, than other cues such as orientation and shape. The paper also provides evidence that easy-to-learn cues tend to converge to relatively flat minima and models that prefer these cues are more abundant in parameter space.", "summary_of_the_review": "The paper is a very interesting read. It provides an interesting and insightful analysis of model preferences for various visual cues. The analysis shows the preference of visual models towards low complexity visual cues, such as color and ethnicity. Given the potential impact of the analysis, I recommend the paper to be accepted. I would nevertheless strongly encourage the authors to address my main comments/concerns.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were split, with one of them leaning towards rejection, primarily due the (perceived) limited impact of the study. I tend to agree with the other reviewers that this paper provides an interesting and original framework for analysis of learning models, and while there are substantial shortcomings, they are outweighed by the positives (including the promise this approach may hold for analysis of learning in more realistic scenarios). I therefore recommend acceptance, if space in the proceedings allows.
Decision: Accept (Poster)