Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective

1 INTRODUCTION . Emerging studies on the inner mechanisms of deep neural networks ( DNNs ) have revealed that many models have shortcut biases ( Cadene et al. , 2019 ; Weinzaepfel & Rogez , 2021 ; Bahng et al. , 2020 ; Geirhos et al. , 2020 ) . DNNs often pick up simple , non-essential cues , which are nonetheless effective within a particular dataset . For example , a DNN trained for the task of animal recognition may recognize ducks while attending on water backgrounds , given the strong correlations between such background cues and the target label ( Choe et al. , 2020 ) . These shortcut biases often result in a striking qualitative difference between human and machine recognition systems ; for example , convolutional neural networks ( CNNs ) trained on ImageNet extensively rely on texture features , while humans would preferentially look at the global shape of objects ( Geirhos et al. , 2019 ) . In other cases , the shortcut bias arises in models that suppress certain streams of inputs : visual question answering ( VQA ) models often neglect the entire image cues , for one does not require images to answer questions like “ what color is the banana in the image ? ” ( Cadene et al. , 2019 ) Since DNNs have successfully outperformed humans on many tasks ( Silver et al. , 2016 ; Rajpurkar et al. , 2017 ) , such a phenomenon may look benign at face value . However , the shortcut biases become problematic when it comes to the generalization to more challenging test-time conditions , where the shortcuts are no longer valid ( Cadene et al. , 2019 ; Weinzaepfel & Rogez , 2021 ; Bahng et al. , 2020 ; Geirhos et al. , 2020 ) . These biases also cause ethical concerns when the shortcut features adopted by a model are sensitive like gender or skin color ( Wang et al. , 2019 ; Xu et al. , 2020 ) . Instead of proposing a method or solution , this work focuses on deepening our understanding of the shortcut bias phenomena . In particular , we design a dataset where multiple cues are equally valid ∗First two authors contributed equally . for solving a particular task and observe which cues tend to be preferentially adopted by DNNs . The experimental setup is inspired by the Wisconsin Card Sorting Test ( WCST , Banno et al . ( 2012 ) ) in Cognitive Neuroscience1 . See Figure 1 for an illustration of the setup . It consists of a training set with multiple highly correlated cues ( e.g . color , shape , and scale ) that offer equally plausible pathways to the successful target prediction ( Y ∈ { 1 , 2 , 3 } ) . We call this a diagonal training set , highlighting the spatial arrangement of such samples in the product space of all combinations . A model f trained on such a dataset will adopt or neglect certain cues . We analyse the cues adopted by a model by observing its predictions on off-diagonal samples . With regards to Figure 1 , for example , consider the f ’ s prediction for a small , blue triangle as an off-diagonal sample . The prediction values f ( ) ∈ { 1 , 2 , 3 } tell us which cue the model is biased towards to , e.g . if f ( ) = 1 , then f is biased towards scale ; if f ( ) = 2 , towards shape ; and if f ( ) = 3 towards color . All three scenarios are plausible and only testing on off-diagonal samples will reveal model bias . We make important observations on the nature of shortcut bias under WCST-ML . We discover that , despite the equal amounts of correlations with the target label , there tends to be a preferential ordering of cues . The preference is largely shared across different DNN architectures , such as feedforward networks , ResNets ( He et al. , 2015 ) , Vision Transformers ( Dosovitskiy et al. , 2021 ) , and multiple initial parameters . From the parameter-space perspective , we further observe that the set of solutions Θp biased to the preferred cues takes a far greater volume than those corresponding to the averted cues Θa . The loss landscape also tends to be flatter around Θp than around Θa . Why are certain cues preferred to others by general DNNs ? We provide an explanation based on the Kolmogorov complexity of cues , which measures the minimal description length for representing cues ( Kolmogorov , 1963 ) . Prior studies have shown that in the parameter space of generic DNNs , there are exponentially more Kolmogorov-simple functions than Kolmogorov-complex ones ( VallePerez et al. , 2019 ; De Palma et al. , 2019 ) . Based on these theoretical results , we argue that DNNs are naturally drawn to Kolmogorov-simple cues . We empirically verify that the preferences for cues correlate well with their Kolmogorov complexity estimates . What are the consequences of the inborn preference for simple cues ? Firstly , this may hinder the generalization of DNNs to challenging test scenarios where the simple shortcut cues are no longer valid ( Geirhos et al. , 2020 ) . Secondly , we expose the possibility that certain protected attributes correspond to the simple shortcut cue for the task at hand , endangering the fairness of DNNs ( Barocas et al. , 2017 ) . In such a case , human intervention on the learning procedure may be necessary to enforce fairness , for the dataset and DNNs can be naturally drawn to exploit protected attributes . The primary goal of this manuscript is to shed light on the nature of shortcut biases and the underlying mechanisms behind the scenes . Our contributions are summarized as follows : an experimental setup for studying the shortcut bias in-depth ( WCST-ML ) ( §2 ) ; novel observations on the nature of shortcut biases , such as the existence of preferential ordering of cues and its connections to the geometry of the loss landscape in the parameter space ( §3 ) ; an explanation based on the descriptional 1Original WCST gauges the subjects ’ cognitive ability to flexibly shift their underlying rules ( adopted cues ) for categorizing samples . Inability to do so may indicate dysfunctional frontal lobe activities . complexity of cues ( §4 ) ; and a discussion on the implications on generalization and fairness ( §5 ) , such as the preferential use of ethnical features for face recognition on the UTKFace dataset . 2 SETUP . We introduce the setup that will provide the basis for the analysis in this paper . We describe the procedure for building a dataset with multiple equally valid cues for recognition ( §2.1 ) . The procedure is applied to DSprites and UTKFace datasets in §2.2 . In §2.3 , we introduce terminologies for the analysis of the parameter space and make theoretical connections with our data framework . 2.1 DATA FRAMEWORK : WCST-ML Many factors affect the preference of models to certain cues . The existence of dominant classes is an example ; it encourages models to favor cues conducive to a good performance on the dominant classes ( Barocas et al. , 2017 ; Hashimoto et al. , 2018 ) . In other cases , some cues have higher degrees of correlation with the target label ( Geirhos et al. , 2020 ) . In this work , we test whether bias is still present under fair conditions , i.e . : when a training dataset contains a set of valid cues , each of which equally correlates with the targets , will DNNs still have a preference for certain cues ? If so , why ? To study this , we introduce a data construction framework called Wisconsin Card Sorting Test for Machine Learners ( WCST-ML ) , named after a clinical test in cognitive neuroscience ( Banno et al. , 2012 ) . See Figure 1 for an overview . As a running example , we assume a dataset where each image can be described by varying two latent variables , object shape and object color . Let X and Y denote image and label , respectively . We write Xij for the image with color i and shape j , where i , j ∈ { 1 , · · · , L } . When we want to considerK > 2 varying factors , we may writeXi1 , ··· , iK for the image random variable with kth factor chosen to be ik ∈ { 1 , · · · , L } . Importantly , we fix the number of categories for each factor to L to enforce similar conditions for all cues . Similar learning setups have appeared in prior papers : “ Cross-bias generalisation ” ( Bahng et al. , 2020 ) , “ What if multiple features are predictive ? ” ( Hermann & Lampinen , 2020 ) , and “ Zero generalization opportunities ” ( Eulig et al. , 2021 ) . While we fully acknowledge the conceptual similarities , we stress that our work presents the first dedicated study into the cue selection problem and the underlying mechanisms . The same set of images { Xij | 1 ≤ i , j ≤ L } admits two possible tasks : color and shape classification . The task is determined by the labels Y . Denoting Yij as the label for image Xij , setting Yij = i leads to the color classification , and setting Yij = j leads to the shape classification tasks . We may then build the data distribution for the task at hand via Dcolor : = ⋃ 1≤i , j≤L ( Xij , Yij = i ) Dshape : = ⋃ 1≤i , j≤L ( Xij , Yij = j ) ( 1 ) for color and shape recognition tasks , respectively . More generally , we may write Dk : = ⋃ 1≤i1 , ··· , iK≤L ( Xi1 , ··· , iK , Yi1 , ··· , iK = ik ) ( 2 ) for the data distribution where the task is to recognize the kth cue . We define the union of random variables as the balanced mixture : ⋃L i=1 Zi : = ZI where I ∼ Unif { 1 , · · · , L } . We now introduce the notion of a diagonal dataset , where every cue ( e.g . color and shape ) contains all the needed information to predict the true label Y . That is , a perfect prediction for either color or shape attribute leads to a 100 % accuracy for the task at hand . This can be done by letting the factors always vary together i = j in the dataset ( and thus the name ) . We write Ddiag : = ⋃ 1≤i≤L ( Xii , Yii = i ) . ( 3 ) Such a dataset completely leaves it to the model to choose the cue for recognition . Given a model f trained on Ddiag , we analyse the recognition cue adopted by f by measuring its unbiased accuracy on all the cells ( See Figure 1 ) . There are K different unbiased accuracies for each task , depending on how the off-diagonal cells are labelled : e.g . Dcolor and Dshape in equation 1 . For a general setting with K cues , the unbiased accuracy for kth cue is defined as acck ( f ) : = 1 Lk ∑ i1 , ··· , iK Pr [ f ( Xi1 , ··· , iK ) = ik ] . ( 4 ) Proposition 1 . For k ∈ { 1 , · · · , K } , acck ( f ) = 1 if and only if f ( Xi1 , ··· , iK ) = ik almost surely for all 1 ≤ i1 , · · · , iK ≤ L. Moreover , if the condition above holds ( i.e . f is perfectly biased to cue k ) , then accm ( f ) = 1L for all m 6= k. The proposition implies that the unbiased accuracy is capable of detecting the bias in a model f : acck ( f ) = 1 implies that f ’ s prediction is solely based on the cue k. It also emphasizes that it is impossible for a model to be perfectly biased to multiple cues . Finally , we remark that the WCST-ML analysis does not require the cues to be orthogonal or interpretable to humans . The only requirement is the availability of the labelled samples ( Xi1 , ··· , iK , Yi1 , ··· , iK ) for the cue of interest .

Summary: The paper discusses biases in inductive learning in deep neural networks that stem from pathologically sampled data. The authors pose a problem set where a trainer only sees samples where two or more latent values can only be observed in a fully correlated fashion (e.g. scale, color, shape). They then design various criteria and protocols to gain insight in the behavior of the learned model when the correlation does not hold any more (i.e. samples that have not been seen in the training data). They conclude that in this case of generalization to unseen data, the trained model has an implicit bias towards choosing (1) mostly single cues to determine the predicted label (e.g. color only), (2) that more preferred cues are simpler than less preferred cues and (3) that the underlying solution space of possible parameters has more solutions that prefer the simple cues. They demonstrate empirical results on variations of existing datasets (DSprites, UTKFace).