ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

∗Joint senior authors 1 INTRODUCTION . How are Convolutional Neural Networks ( CNNs ) able to reach impressive performance on complex perceptual tasks such as object recognition ( Krizhevsky et al. , 2012 ) and semantic segmentation ( Long et al. , 2015 ) ? One widely accepted intuition is that CNNs combine low-level features ( e.g . edges ) to increasingly complex shapes ( such as wheels , car windows ) until the object ( e.g . car ) can be readily classified . As Kriegeskorte ( 2015 ) puts it , “ the network acquires complex knowledge about the kinds of shapes associated with each category . [ ... ] High-level units appear to learn representations of shapes occurring in natural images ” ( p. 429 ) . This notion also appears in other explanations , such as in LeCun et al . ( 2015 ) : Intermediate CNN layers recognise “ parts of familiar objects , and subsequent layers [ ... ] detect objects as combinations of these parts ” ( p. 436 ) . We term this explanation the shape hypothesis . This hypothesis is supported by a number of empirical findings . Visualisation techniques like Deconvolutional Networks ( Zeiler & Fergus , 2014 ) often highlight object parts in high-level CNN features.1 Moreover , CNNs have been proposed as computational models of human shape perception by Kubilius et al . ( 2016 ) , who conducted an impressive number of experiments comparing human and CNN shape representations and concluded that CNNs “ implicitly learn representations of shape that reflect human shape perception ” ( p. 15 ) . Ritter et al . ( 2017 ) discovered that CNNs develop a so-called “ shape bias ” just like children , i.e . that object shape is more important than colour for object classification ( although see Hosseini et al . ( 2018 ) for contrary evidence ) . Furthermore , CNNs are currently the most predictive models for human ventral stream object recognition ( e.g . Cadieu et al. , 2014 ; Yamins et al. , 2014 ) ; and it is well-known that object shape is the single most important cue for human object recognition ( Landau et al. , 1988 ) , much more than other cues like size or texture ( which may explain the ease at which humans recognise line drawings or millennia-old cave paintings ) . On the other hand , some rather disconnected findings point to an important role of object textures for CNN object recognition . CNNs can still classify texturised images perfectly well , even if the global shape structure is completely destroyed ( Gatys et al. , 2017 ; Brendel & Bethge , 2019 ) . Conversely , standard CNNs are bad at recognising object sketches where object shapes are preserved yet all texture cues are missing ( Ballester & de Araújo , 2016 ) . Additionally , two studies suggest that local information such as textures may actually be sufficient to “ solve ” ImageNet object recognition : Gatys et al . ( 2015 ) discovered that a linear classifier on top of a CNN ’ s texture representation ( Gram matrix ) achieves hardly any classification performance loss compared to original network performance . More recently , Brendel & Bethge ( 2019 ) demonstrated that CNNs with explicitly constrained receptive field sizes throughout all layers are able to reach surprisingly high accuracies on ImageNet , even though this effectively limits a model to recognising small local patches rather than integrating object parts for shape recognition . Taken together , it seems that local textures indeed provide sufficient information about object classes—ImageNet object recognition could , in principle , be achieved through texture recognition alone . In the light of these findings , we believe that it is time to consider a second explanation , which we term the texture hypothesis : in contrast to the common assumption , object textures are more important than global object shapes for CNN object recognition . Resolving these two contradictory hypotheses is important both for the deep learning community ( to increase our understanding of neural network decisions ) as well as for the human vision and neuroscience communities ( where CNNs are being used as computational models of human object recognition and shape perception ) . In this work we aim to shed light on this debate with a number of carefully designed yet relatively straightforward experiments . Utilising style transfer ( Gatys et al. , 2016 ) , we created images with a texture-shape cue conflict such as the cat shape with elephant texture depicted in Figure 1c . This enables us to quantify texture and shape biases in both humans and CNNs . To this end , we perform nine comprehensive and careful psychophysical experiments comparing humans against CNNs on exactly the same images , totalling 48,560 psychophysical trials across 97 observers . These experiments provide behavioural evidence in favour of the texture hypothesis : A cat with an elephant texture is an elephant to CNNs , and still a cat to humans . Beyond quantifying existing biases , we subsequently present results for our two other main contributions : 1To avoid any confusion caused by different meanings of the term ‘ feature ’ , we consistently use it to refer to properties of CNNs ( learned features ) rather than to object properties ( such as colour ) . When referring to physical objects , we use the term ‘ cue ’ instead . changing biases , and discovering emergent benefits of changed biases . We show that the texture bias in standard CNNs can be overcome and changed towards a shape bias if trained on a suitable data set . Remarkably , networks with a higher shape bias are inherently more robust to many different image distortions ( for some even reaching or surpassing human performance , despite never being trained on any of them ) and reach higher performance on classification and object recognition tasks . 2 METHODS . In this section we outline the core elements of paradigm and procedure . Extensive details to facilitate replication are provided in the Appendix . Data , code and materials are available from this repository : https : //github.com/rgeirhos/texture-vs-shape 2.1 PSYCHOPHYSICAL EXPERIMENTS . All psychophysical experiments were conducted in a well-controlled psychophysical lab setting and follow the paradigm of Geirhos et al . ( 2018 ) , which allows for direct comparisons between human and CNN classification performance on exactly the same images . Briefly , in each trial participants were presented a fixation square for 300 ms , followed by a 300 ms presentation of the stimulus image . After the stimulus image we presented a full-contrast pink noise mask ( 1/f spectral shape ) for 200 ms to minimise feedback processing in the human visual system and to thereby make the comparison to feedforward CNNs as fair as possible . Subsequently , participants had to choose one of 16 entry-level categories by clicking on a response screen shown for 1500 ms. On this screen , icons of all 16 categories were arranged in a 4× 4 grid . Those categories were airplane , bear , bicycle , bird , boat , bottle , car , cat , chair , clock , dog , elephant , keyboard , knife , oven and truck . Those are the so-called “ 16-class-ImageNet ” categories introduced in Geirhos et al . ( 2018 ) . The same images were fed to four CNNs pre-trained on standard ImageNet , namely AlexNet ( Krizhevsky et al. , 2012 ) , GoogLeNet ( Szegedy et al. , 2015 ) , VGG-16 ( Simonyan & Zisserman , 2015 ) and ResNet-50 ( He et al. , 2015 ) . The 1,000 ImageNet class predictions were mapped to the 16 categories using the WordNet hierarchy ( Miller , 1995 ) —e.g . ImageNet category tabby cat would be mapped to cat . In total , the results presented in this study are based on 48,560 psychophysical trials and 97 participants . 2.2 DATA SETS ( PSYCHOPHYSICS ) . In order to assess texture and shape biases , we conducted six major experiments along with three control experiments , which are described in the Appendix . The first five experiments ( samples visualised in Figure 2 ) are simple object recognition tasks with the only difference being the image features available to the participant : Original 160 natural colour images of objects ( 10 per category ) with white background . Greyscale Images from Original data set converted to greyscale using skimage.color.rgb2gray . For CNNs , greyscale images were stacked along the colour channel . Silhouette Images from Original data set converted to silhouette images showing an entirely black object on a white background ( see Appendix A.6 for procedure ) . Edges Images from Original data set converted to an edge-based representation using Canny edge extractor implemented in MATLAB . Texture 48 natural colour images of textures ( 3 per category ) . Typically the textures consist of full-width patches of an animal ( e.g . skin or fur ) or , in particular for man-made objects , of images with many repetitions of the same objects ( e.g . many bottles next to each other , see Figure 7 in the Appendix ) . It is important to note that we only selected object and texture images that were correctly classified by all four networks . This was made to ensure that our results in the sixth experiment on cue conflicts , which is most decisive in terms of the shape vs texture hypothesis , are fully interpretable . In the cue conflict experiment we present images with contradictory features ( see Figure 1 ) but still ask the participant to assign a single class . Note that the instructions to human observers were entirely neutral w.r.t . shape or texture ( “ click on the object category that you see in the presented image ; guess if unsure . There is no right or wrong answer , we are interested in your subjective impression ” ) . Cue conflict Images generated using iterative style transfer ( Gatys et al. , 2016 ) between an image of the Texture data set ( as style ) and an image from the Original data set ( as content ) . We generated a total of 1280 cue conflict images ( 80 per category ) , which allows for presentation to human observers within a single experimental session . We define “ silhouette ” as the bounding contour of an object in 2D ( i.e. , the outline of object segmentation ) . When mentioning “ object shape ” , we use a definition that is broader than just the silhouette of an object : we refer to the set of contours that describe the 3D form of an object , i.e . including those contours that are not part of the silhouette . Following Gatys et al . ( 2017 ) , we define “ texture ” as an image ( region ) with spatially stationary statistics . Note that on a very local level , textures ( according to this definition ) can have non-stationary elements ( such as a local shape ) : e.g . a single bottle clearly has non-stationary statistics , but many bottles next to each other are perceived as a texture : “ things ” become “ stuff ” ( Gatys et al. , 2017 , p. 178 ) . For an example of a “ bottle texture ” see Figure 7 . 2.3 STYLIZED-IMAGENET . Starting from ImageNet we constructed a new data set ( termed Stylized-ImageNet or SIN ) by stripping every single image of its original texture and replacing it with the style of a randomly selected painting through AdaIN style transfer ( Huang & Belongie , 2017 ) ( see examples in Figure 3 ) with a stylization coefficient of α = 1.0 . We used Kaggle ’ s Painter by Numbers data set2 as a style source due to its large style variety and size ( 79,434 paintings ) . We used AdaIN fast style transfer rather than iterative stylization ( e.g . Gatys et al. , 2016 ) for two reasons : Firstly , to ensure that training on SIN and testing on cue conflict stimuli is done using different stylization techniques , such that the results do not rely on a single stylization method . Secondly , to enable stylizing entire ImageNet , which would take prohibitively long with an iterative approach . We provide code to create Stylized-ImageNet here : https : //github.com/rgeirhos/Stylized-ImageNet

Summary: This paper talks about the behavior bias between human and advanced CNN classifier when classifying objects. A clear conclusion is that DNN classifiers lean on texture cues more than human, which is in contrast to empirical evidence. The experimental results are delighting and convincing to some extent. This paper is also inspiring and potentially useful to interpret how CNN works in object classification task.