Shape-Texture Debiased Neural Network Training

1 INTRODUCTION . It is known that both shape and texture serve as essential cues for object recognition . A decade ago , computer vision researchers had explicitly designed a variety of hand-crafted features , either based on shape ( e.g. , shape context ( Belongie et al. , 2002 ) and inner distance shape context ( Ling & Jacobs , 2007 ) ) or texture ( e.g. , textons ( Malik et al. , 2001 ) ) , for object recognition . Moreover , researchers found that properly combining shape and texture can further recognition performance ( Shotton et al. , 2009 ; Zheng et al. , 2007 ) , demonstrating the superiority of possessing both features . Nowadays , as popularized by Convolutional Neural Networks ( CNNs ) ( Krizhevsky et al. , 2012 ) , the features used for object recognition are automatically learned , rather than manually designed . This change not only eases human efforts on feature engineering , but also yields much better performance on a wide range of visual benchmarks ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Girshick et al. , 2014 ; Girshick , 2015 ; Ren et al. , 2015 ; Long et al. , 2015 ; Chen et al. , 2015 ) . But interestingly , as pointed by Geirhos et al . ( 2019 ) , the features learned by CNNs tend to bias toward either shape or texture , depending on the training dataset . We verify that such biased representation learning ( towards either shape or texture ) weakens CNNs ’ performance.1 Nonetheless , surprisingly , we also find ( 1 ) the model with shape-biased representations and the model with texture-biased representations are highly complementary to each other , e.g. , they focus on completely different cues for predictions ( an example is provided in Figure 1 ) ; and ( 2 ) being biased towards either cue may inevitably limit model performance , e.g. , models may not be able to tell the difference between a lemon and an orange without texture information . These observations altogether deliver a promising message—biased models ( e.g. , ImageNet trained ( texturebiased ) CNNs ( Geirhos et al. , 2019 ) or ( shape-biased ) CNNs ( Shi et al. , 2020 ) ) are improvable . 1Biased models are acquired similar to Geirhos et al . ( 2019 ) , see Section 2 for details . To this end , we hereby develop a shape-texture debiased neural network training framework to guide CNNs for learning better representations . Our method is a data-driven approach , which let CNNs automatically figure out how to avoid being biased towards either shape or texture from their training samples . Specifically , we apply style transfer to generate cue conflict images , which breaks the correlation between shape and texture , for augmenting the original training data . The most important recipe of training a successful shape-texture debiased model is that we need to provide supervision from both shape and texture on these generated cue conflict images , otherwise models will remain being biased . Experiments show that our proposed shape-texture debiased neural network training significantly improves recognition models . For example , on the challenging ImageNet dataset ( Russakovsky et al. , 2015 ) , our method helps ResNet-152 gain an absolute improvement of 1.2 % , achieving 79.8 % top-1 accuracy . Additionally , compared to its vanilla counterpart , this debiased ResNet-152 shows better generalization on ImageNet-A ( Hendrycks et al. , 2019 ) ( +5.2 % ) , ImageNet-C ( Hendrycks & Dietterich , 2019 ) ( +8.3 % ) and Stylized ImageNet ( Geirhos et al. , 2019 ) ( +11.1 % ) , and stronger robustness on defending against FGSM adversarial attacker on ImageNet ( +14.4 % ) . Our shape-texture debiased neural network training is orthogonal to other advanced data augmentation strategies , e.g. , it further boosts CutMix-ResNeXt-101 ( Yun et al. , 2019 ) by 0.7 % on ImageNet , achieving 81.2 % top-1 accuracy . 2 SHAPE/TEXTURE BIASED NEURAL NETWORKS . The biased feature representation of CNNs mainly stems from the training dataset , e.g. , Geirhos et al . ( 2019 ) point out that models will be biased towards shape if trained on Stylized-ImageNet dataset . Following Geirhos et al . ( 2019 ) , we hereby present a similar training pipeline to acquire shapebiased models or texture-biased models . By evaluating these two kinds of models , we observe the necessity of possessing both shape and texture representations for CNNs to better recognize objects . 2.1 MODEL ACQUISITION . Data generation . Similar to Geirhos et al . ( 2019 ) , we apply images with conflicting shape and texture information as training samples to obtain shape-biased or texture-biased models . But different from Geirhos et al . ( 2019 ) , an important change in our cue conflict image generation procedure is that we override the original texture information with the informative texture patterns from another … Chimpanzee Lemon … Original Data Chimpanzee Lemon Chimpanzee & Lemon ( a ) Shape-biased Model ( b ) Texture-biased Model ( c ) Debiased Model Training Images Label Assignment & Results … Chimpanzee Lemon … Original Data Chimpanzee Lemon Chimpanzee & Lemon ( a ) Shape-biased Model ( b ) Texture-biased Model ( c ) Debiased Model Training Images Label Assignment & Results Figure 2 : Illustration of the our training pipeline for acquiring ( a ) a shape-biased model , ( b ) a texture-biased model , and ( c ) a shape-texture debiased model . Specifically , these models share the same training samples , i.e . images with conflicting texture and shape information , generated by style transfer between two randomly selected images ; but apply distinct labelling strategies : in ( a ) & ( b ) , labels are determined by the images that provides shape ( or texture ) information in style transfer , for guiding models to learn more shape ( or texture ) representations ; in ( c ) , labels are jointly determined by the pair of images in style transfer , for avoiding bias in representation learning . randomly selected image , rather than with the uninformative style of randomly selected artistic paintings . That being said , to create a new training sample , we need to first select a pair of images from the training set uniformly at random , and then apply style transfer to blend their shape and texture information . Such a generated example is shown in Figure 2 , i.e. , the image of chimpanzee shape but with lemon texture . Label assignment . The way of assigning labels to cue conflict images controls the bias of learned models . Without loss of generality , we show the case of learning a texture-biased model . To guide the model to attend more on texture , the labels assigned to the cue conflict images here will be exclusively based on the texture information , e.g. , the image of chimpanzee shape but with lemon texture will be labelled as lemon , shown in Figure 2 ( b ) . By this way , the texture information is highly related to the “ ground-truth ” while the shape information only serves as a nuisance factor during learning . Similarly , to learn a shape-biased model , the label assignment of cue conflict images will be based on shape only , e.g. , the image of chimpanzee shape but with lemon texture now will be labelled as chimpanzee , shown in Figure 2 ( a ) . 2.2 EVALUATION AND OBSERVATION . To reduce the computational overhead in this ablation , all models are trained and evaluated on ImageNet-200 , which is a 200 classes subset of the original ImageNet , including 100,000 images ( 500 images per class ) for training and 10,000 images ( 50 images per class ) for validation . Akin to Geirhos et al . ( 2019 ) , we observe that the models with biased feature representations tend to have inferior accuracy than their vanilla counterparts . For example , our shape-biased ResNet-18 only achieves 73.9 % top-5 ImageNet-200 accuracy , which is much lower than the vanilla ResNet-18 with 88.2 % top-5 ImageNet-200 accuracy . Though biased representations weaken the overall classification accuracy , surprisingly , we find they are highly complementary to each other . We first visualize the attended image regions of biased models , via Class Activation Mapping ( Zhou et al. , 2016 ) , in Figure 3 . As we can see here , the shape-biased model and the texture-biased model concentrate on different cues for predictions . For Egyptian Cat ✕ Tabby Cat ✓ Cl as sif ie d As Orange ✕ Lemon ✓ Lion ✓ Tabby Cat ✕ Alsatian ✓ Chihuahua ✕ Shape-biased ModelMore Accurate Less Accurate Obelisk ATM Bucket , pail Fur CoatPotpieCliff , Drop … Brain Coral Expresso Wooden Spoon TrolleybusTriumphal ArchBarn instance , on the leftmost tabby cat image , the shape-biased model mainly focuses on the cat head , while the texture-biased model mainly focuses on the lower body and the front legs of the cat . Such attention mechanisms are correlated to their learned representations—the shape-biased model extracts the shape of the cat head as an important signal for predictions , while the texture-biased model relies on the texture information of cat fur for predictions . As distinct cues are picked by shape-biased/texture-biased models , a more concrete observation is they are good/bad at classifying quite different object categories . As showed in Figure 4 , the shapebiased model is good at recognizing objects with representative shape structure like obelisk , but is bad at recognizing objects whose shape is uninformative or almost indistinguishable from others like fur coat . Similarly , the texture-biased model can effectively recognize objects with unique texture patterns like brain coral but may fail to recognize objects with unpredictable texture like trolleybus ( as its side body can be painted with different advertisements ) . Besides , biased models may inevitably perform poorly on certain categories as insufficient cues are applied . For examples , it is challenging to distinguish between a lemon and an orange if texture information can not be utilized , or to distinguish between an lion and a tabby cat without shape information . Given the analysis above , we can conclude that biased representations limit models ’ recognition ability . But meanwhile , our ablation delivers a promising message—the features learned by biased models are highly complementary to each other . This observation indicates the current training framework is improvable ( as the resulted models are biased towards texture ( Geirhos et al. , 2019 ) or shape ( Shi et al. , 2020 ) ) , and offers a potential direction for building a stronger one—we should train models to properly acquire both shape and texture feature representations . We will introduce a simple method for doing so next . 3 SHAPE-TEXTURE DEBIASED NEURAL NETWORK TRAINING . Recall that when obtaining a biased model , the strategy of label assignment is pivot—when the labels are exclusively determined by the images that provide shape ( or texture ) information in style transfer , we will obtain a shape-biased ( or texture-biased ) model . Therefore , to guide models for leveraging both shape and texture for predictions , we hereby propose a simple way , which is inspired by Mixup ( Zhang et al. , 2018 ) , to softly construct labels during training . In other words , given the one-hot label of the shape-source image ys and the one-hot label of the texture-source image yt , the new label that we assigned to the cue conflict image is ỹ = γ ∗ ys + ( 1− γ ) ∗ yt , ( 1 ) where γ ∈ [ 0 , 1 ] is a manually selected hyperparameter to control the relative importance between shape and texture . By ranging the shape-texture coefficient γ from 0 to 1 , we obtain a path to evolve the model from being a texture-biased one ( i.e. , γ = 0 ) to being a shape-biased one ( i.e. , γ = 1 ) . Although the two extreme ends lead to biased models with inferior performance , we empirically show that there exist a sweet point along this interpolation path , i.e. , the learned models can properly acquires both shape and texture feature representations and achieve superior performance on a wide range of image recognition benchmarks . We name this simple method as shape-texture debiased neural network training , and illustrate the training pipeline in Figure 2 ( c ) . It is worth to mention that , although Figure 2 only shows the procedure of applying our method to the image classification task , this training framework is general and has the potential to be extended to other computer vision tasks , e.g. , a simple showcase on semantic segmentation is presented in Section 4.4 .

Summary: The authors propose a method to mitigate the bias towards either texture or shape, in convolutional network training. The method follows the idea from Geirhos et al (2019), but use images randomly sampled from the same dataset, instead of style transfer from paintings. Then, depending on a manually selected hyperparameter, the weights of conflicting labels are blended by weighted average of the one-hot encoding.