Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective

1 INTRODUCTION . Emerging studies on the inner mechanisms of deep neural networks ( DNNs ) have revealed that many models have shortcut biases ( Cadene et al. , 2019 ; Weinzaepfel & Rogez , 2021 ; Bahng et al. , 2020 ; Geirhos et al. , 2020 ) . DNNs often pick up simple , non-essential cues , which are nonetheless effective within a particular dataset . For example , a DNN trained for the task of animal recognition may recognize ducks while attending on water backgrounds , given the strong correlations between such background cues and the target label ( Choe et al. , 2020 ) . These shortcut biases often result in a striking qualitative difference between human and machine recognition systems ; for example , convolutional neural networks ( CNNs ) trained on ImageNet extensively rely on texture features , while humans would preferentially look at the global shape of objects ( Geirhos et al. , 2019 ) . In other cases , the shortcut bias arises in models that suppress certain streams of inputs : visual question answering ( VQA ) models often neglect the entire image cues , for one does not require images to answer questions like “ what color is the banana in the image ? ” ( Cadene et al. , 2019 ) Since DNNs have successfully outperformed humans on many tasks ( Silver et al. , 2016 ; Rajpurkar et al. , 2017 ) , such a phenomenon may look benign at face value . However , the shortcut biases become problematic when it comes to the generalization to more challenging test-time conditions , where the shortcuts are no longer valid ( Cadene et al. , 2019 ; Weinzaepfel & Rogez , 2021 ; Bahng et al. , 2020 ; Geirhos et al. , 2020 ) . These biases also cause ethical concerns when the shortcut features adopted by a model are sensitive like gender or skin color ( Wang et al. , 2019 ; Xu et al. , 2020 ) . Instead of proposing a method or solution , this work focuses on deepening our understanding of the shortcut bias phenomena . In particular , we design a dataset where multiple cues are equally valid ∗First two authors contributed equally . for solving a particular task and observe which cues tend to be preferentially adopted by DNNs . The experimental setup is inspired by the Wisconsin Card Sorting Test ( WCST , Banno et al . ( 2012 ) ) in Cognitive Neuroscience1 . See Figure 1 for an illustration of the setup . It consists of a training set with multiple highly correlated cues ( e.g . color , shape , and scale ) that offer equally plausible pathways to the successful target prediction ( Y ∈ { 1 , 2 , 3 } ) . We call this a diagonal training set , highlighting the spatial arrangement of such samples in the product space of all combinations . A model f trained on such a dataset will adopt or neglect certain cues . We analyse the cues adopted by a model by observing its predictions on off-diagonal samples . With regards to Figure 1 , for example , consider the f ’ s prediction for a small , blue triangle as an off-diagonal sample . The prediction values f ( ) ∈ { 1 , 2 , 3 } tell us which cue the model is biased towards to , e.g . if f ( ) = 1 , then f is biased towards scale ; if f ( ) = 2 , towards shape ; and if f ( ) = 3 towards color . All three scenarios are plausible and only testing on off-diagonal samples will reveal model bias . We make important observations on the nature of shortcut bias under WCST-ML . We discover that , despite the equal amounts of correlations with the target label , there tends to be a preferential ordering of cues . The preference is largely shared across different DNN architectures , such as feedforward networks , ResNets ( He et al. , 2015 ) , Vision Transformers ( Dosovitskiy et al. , 2021 ) , and multiple initial parameters . From the parameter-space perspective , we further observe that the set of solutions Θp biased to the preferred cues takes a far greater volume than those corresponding to the averted cues Θa . The loss landscape also tends to be flatter around Θp than around Θa . Why are certain cues preferred to others by general DNNs ? We provide an explanation based on the Kolmogorov complexity of cues , which measures the minimal description length for representing cues ( Kolmogorov , 1963 ) . Prior studies have shown that in the parameter space of generic DNNs , there are exponentially more Kolmogorov-simple functions than Kolmogorov-complex ones ( VallePerez et al. , 2019 ; De Palma et al. , 2019 ) . Based on these theoretical results , we argue that DNNs are naturally drawn to Kolmogorov-simple cues . We empirically verify that the preferences for cues correlate well with their Kolmogorov complexity estimates . What are the consequences of the inborn preference for simple cues ? Firstly , this may hinder the generalization of DNNs to challenging test scenarios where the simple shortcut cues are no longer valid ( Geirhos et al. , 2020 ) . Secondly , we expose the possibility that certain protected attributes correspond to the simple shortcut cue for the task at hand , endangering the fairness of DNNs ( Barocas et al. , 2017 ) . In such a case , human intervention on the learning procedure may be necessary to enforce fairness , for the dataset and DNNs can be naturally drawn to exploit protected attributes . The primary goal of this manuscript is to shed light on the nature of shortcut biases and the underlying mechanisms behind the scenes . Our contributions are summarized as follows : an experimental setup for studying the shortcut bias in-depth ( WCST-ML ) ( §2 ) ; novel observations on the nature of shortcut biases , such as the existence of preferential ordering of cues and its connections to the geometry of the loss landscape in the parameter space ( §3 ) ; an explanation based on the descriptional 1Original WCST gauges the subjects ’ cognitive ability to flexibly shift their underlying rules ( adopted cues ) for categorizing samples . Inability to do so may indicate dysfunctional frontal lobe activities . complexity of cues ( §4 ) ; and a discussion on the implications on generalization and fairness ( §5 ) , such as the preferential use of ethnical features for face recognition on the UTKFace dataset . 2 SETUP . We introduce the setup that will provide the basis for the analysis in this paper . We describe the procedure for building a dataset with multiple equally valid cues for recognition ( §2.1 ) . The procedure is applied to DSprites and UTKFace datasets in §2.2 . In §2.3 , we introduce terminologies for the analysis of the parameter space and make theoretical connections with our data framework . 2.1 DATA FRAMEWORK : WCST-ML Many factors affect the preference of models to certain cues . The existence of dominant classes is an example ; it encourages models to favor cues conducive to a good performance on the dominant classes ( Barocas et al. , 2017 ; Hashimoto et al. , 2018 ) . In other cases , some cues have higher degrees of correlation with the target label ( Geirhos et al. , 2020 ) . In this work , we test whether bias is still present under fair conditions , i.e . : when a training dataset contains a set of valid cues , each of which equally correlates with the targets , will DNNs still have a preference for certain cues ? If so , why ? To study this , we introduce a data construction framework called Wisconsin Card Sorting Test for Machine Learners ( WCST-ML ) , named after a clinical test in cognitive neuroscience ( Banno et al. , 2012 ) . See Figure 1 for an overview . As a running example , we assume a dataset where each image can be described by varying two latent variables , object shape and object color . Let X and Y denote image and label , respectively . We write Xij for the image with color i and shape j , where i , j ∈ { 1 , · · · , L } . When we want to considerK > 2 varying factors , we may writeXi1 , ··· , iK for the image random variable with kth factor chosen to be ik ∈ { 1 , · · · , L } . Importantly , we fix the number of categories for each factor to L to enforce similar conditions for all cues . Similar learning setups have appeared in prior papers : “ Cross-bias generalisation ” ( Bahng et al. , 2020 ) , “ What if multiple features are predictive ? ” ( Hermann & Lampinen , 2020 ) , and “ Zero generalization opportunities ” ( Eulig et al. , 2021 ) . While we fully acknowledge the conceptual similarities , we stress that our work presents the first dedicated study into the cue selection problem and the underlying mechanisms . The same set of images { Xij | 1 ≤ i , j ≤ L } admits two possible tasks : color and shape classification . The task is determined by the labels Y . Denoting Yij as the label for image Xij , setting Yij = i leads to the color classification , and setting Yij = j leads to the shape classification tasks . We may then build the data distribution for the task at hand via Dcolor : = ⋃ 1≤i , j≤L ( Xij , Yij = i ) Dshape : = ⋃ 1≤i , j≤L ( Xij , Yij = j ) ( 1 ) for color and shape recognition tasks , respectively . More generally , we may write Dk : = ⋃ 1≤i1 , ··· , iK≤L ( Xi1 , ··· , iK , Yi1 , ··· , iK = ik ) ( 2 ) for the data distribution where the task is to recognize the kth cue . We define the union of random variables as the balanced mixture : ⋃L i=1 Zi : = ZI where I ∼ Unif { 1 , · · · , L } . We now introduce the notion of a diagonal dataset , where every cue ( e.g . color and shape ) contains all the needed information to predict the true label Y . That is , a perfect prediction for either color or shape attribute leads to a 100 % accuracy for the task at hand . This can be done by letting the factors always vary together i = j in the dataset ( and thus the name ) . We write Ddiag : = ⋃ 1≤i≤L ( Xii , Yii = i ) . ( 3 ) Such a dataset completely leaves it to the model to choose the cue for recognition . Given a model f trained on Ddiag , we analyse the recognition cue adopted by f by measuring its unbiased accuracy on all the cells ( See Figure 1 ) . There are K different unbiased accuracies for each task , depending on how the off-diagonal cells are labelled : e.g . Dcolor and Dshape in equation 1 . For a general setting with K cues , the unbiased accuracy for kth cue is defined as acck ( f ) : = 1 Lk ∑ i1 , ··· , iK Pr [ f ( Xi1 , ··· , iK ) = ik ] . ( 4 ) Proposition 1 . For k ∈ { 1 , · · · , K } , acck ( f ) = 1 if and only if f ( Xi1 , ··· , iK ) = ik almost surely for all 1 ≤ i1 , · · · , iK ≤ L. Moreover , if the condition above holds ( i.e . f is perfectly biased to cue k ) , then accm ( f ) = 1L for all m 6= k. The proposition implies that the unbiased accuracy is capable of detecting the bias in a model f : acck ( f ) = 1 implies that f ’ s prediction is solely based on the cue k. It also emphasizes that it is impossible for a model to be perfectly biased to multiple cues . Finally , we remark that the WCST-ML analysis does not require the cues to be orthogonal or interpretable to humans . The only requirement is the availability of the labelled samples ( Xi1 , ··· , iK , Yi1 , ··· , iK ) for the cue of interest .

Summary: The paper conducts a study of which visual cues are preferred by current vision models. The paper designs a training setup with several cues where each cue is equally correlated with the image label. The paper shows that visual cues like color are much easier to be learned by a vision model, than other cues such as orientation and shape. The paper also provides evidence that easy-to-learn cues tend to converge to relatively flat minima and models that prefer these cues are more abundant in parameter space.