Does enhanced shape bias improve neural network robustness to common corruptions?

1 INTRODUCTION . As deep learning is increasingly applied to open-world perception problems in safety-critical domains such as robotics and autonomous driving , its robustness properties become of paramount importance . Generally , a lack of robustness against adversarial examples has been observed ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , making physical-world adversarial attacks on perception systems feasible ( Kurakin et al. , 2017 ; Eykholt et al. , 2018 ; Lee & Kolter , 2019 ) . In this work , we focus on a different kind of robustness : namely , robustness against naturally occurring common image corruptions . Robustness of image classifiers against such corruptions can be evaluated using the ImageNet-C benchmark ( Hendrycks & Dietterich , 2019 ) , in which corruptions such as noise , blur , weather effects , and digital image transformations are simulated . Hendrycks & Dietterich ( 2019 ) observed that recent advances in neural architectures increased performance on undistorted data without significant increase in relative corruption robustness . One hypothesis for the lack of robustness is an over-reliance on non-robust features that generalize well within the distribution used for training but fail to generalize to out-of-distribution data . Ilyas ∗Equal contribution . et al . ( 2019 ) provide evidence for this hypothesis on adversarial examples . Similarly , it has been hypothesized that models which rely strongly on texture information are more vulnerable to common corruptions than models based on features encoding shape information ( Geirhos et al. , 2019 ; Hendrycks & Dietterich , 2019 ) . Alternative methods for increasing corruption robustness not motivated by enhancing shape bias use more ( potentially unlabeled ) training data ( Xie et al. , 2019 ) or use stronger data augmentation ( Lopes et al. , 2019 ; Hendrycks * et al. , 2020 ) . Note that our meaning of “ shape ” & “ texture ” is built on the definitions by Geirhos et al . ( 2019 ) . In this paper , we re-examine the question of whether increasing the shape bias of a model actually helps in terms of corruption robustness . While prior work has found that there are training methods that increase both shape bias and corruption robustness ( Geirhos et al. , 2019 ; Hendrycks & Dietterich , 2019 ) , this only establishes a correlation and not a causal relationship . To increase the shape bias , Geirhos et al . ( 2019 ) “ stylize ” images by imposing the style of a painting onto the image , leaving the shape-related structure of the image mostly unchanged while modifying texture cues so that they get largely uninformative of the class . Note that image stylization can be interpreted as a specific form of data augmentation , providing an alternative hypothesis for increased corruption robustness which would leave the enhanced shape bias as a mostly unrelated byproduct . In this work , we investigate the role of the shape bias for corruption robustness in more detail . We propose two novel methods for increasing the shape bias : • Similar to Geirhos et al . ( 2019 ) , we pre-train the CNN on an auxiliary dataset which encourages learning shape features . In contrast to Geirhos et al . ( 2019 ) that use stylized images , this dataset consists of the edge maps for the training images that are generated using the pre-trained neural network of Liu et al . ( 2017 ) for edge detection . This method maintains global object shapes but removes texture-related information , thereby encouraging learning shape-based representations . • In addition to pre-training on edge maps , we also propose style randomization to further enhance the shape bias . Style randomization is based upon sampling parameters of the affine transformations of normalization layers for each input from a uniform distribution . Our key finding is summarized in Figure 1 . While pre-training on stylized images increases both shape bias and corruption robustness , these two quantities are not necessarily correlated : pre-training on edge maps increases the shape bias without consistently helping in terms of corruption robustness . In order to explain this finding , we conduct a systematic study in which we create inputs based on natural images , explicit edge information , and different ways of stylization ( see Figure 2 for an illustration ) . We find that the shape bias gets maximized when combining edge information with stylization without including any texture information ( Stylized Edges ) . However , for maximal corruption robustness , superimposing the image ( and thus its textures ) on these stylized edges is required . This , however , strongly reduces shape bias . In summary , corruption robustness seems to benefit most from style variation in the vicinity of the image manifold , while shape bias is mostly unrelated . Thus , image stylization is best interpreted as a strong data augmentation technique that encourages robust representations , regardless whether these representations are shape-based or not . Moreover , we present results for a setting where we fine-tune only parameters of the affine transformation of a normalization layer on the target distribution ( stylized or corrupted images , respectively ) for a CNN trained on regular images . Surprisingly , this is already sufficient for increasing the shape bias/corruption robustness considerably . We conclude that CNNs trained on normal images do learn shape-based features and features robust to corruptions but assign little weight to them . It may thus be sufficient to perform augmentation in feature space ( extending Nam et al . ( 2019 ) ; Li et al . ( 2020 ) ) so that higher weights are assigned to features that are robust to relevant domain shifts . 2 RELATED WORK . Texture-vs-Shape Bias Geirhos et al . ( 2019 ) and Baker et al . ( 2018 ) hypothesized that CNNs tend to be biased towards textural cues rather than shape cues . This line of research is further supported by Brendel & Bethge ( 2019 ) , where the authors show that BagNets , Deep Neural Networks ( DNN ) trained and evaluated only on small restricted local image patches , already perform reasonably well on ImageNet . Similarly , Yin et al . ( 2019 ) and Jo & Bengio ( 2017 ) showed using a Fourier space analysis that DNNs rely on surface statistical regularities and high-frequency components . The texture-vs-shape bias can be quantified by evaluating a network either on images with texture-shape cue conflict ( Geirhos et al. , 2019 ) or on images which were patch-wise shuffled ( Luo et al. , 2019 ) . Robustness Against Common Corruptions Common corruptions are potentially stochastic image transformations motivated by real-world effects that can be used for evaluating model robustness . Hendrycks & Dietterich ( 2019 ) proposed the ImageNet-C dataset that contains simulated corruptions such as noise , blur , weather effects and digital image transformations . Geirhos et al . ( 2018 ) showed that humans are more robust to image corruptions than CNNs . Approaches to improve corruption robustness include data augmentation ( Lopes et al. , 2019 ; Yun et al. , 2019 ; Hendrycks * et al. , 2020 ; Cubuk et al. , 2019 ) , self-training with more training data ( Xie et al. , 2019 ) , novel architectures and building blocks ( Zhang , 2019 ; Hu et al. , 2018 ) , and changes in the training procedure ( Hendrycks et al. , 2019 ; Rusak et al. , 2020 ; Wang et al. , 2019 ) . Motivated by the texture-vs-shape hypothesis , Geirhos et al . ( 2019 ) and Michaelis et al . ( 2019 ) train their network on a stylized version of ImageNet . The idea is that style transfer removes textural cues and models trained on stylized data thus have to rely more on shape information . The observed increase in corruption robustness on this stylized data was attributed to the shape bias . In this work , we provide evidence that contradicts this claim . Similar to training on stylized images , Style Blending ( Nam et al. , 2019 ) employs style transfer in latent space by interpolating between feature statistics of different samples in a batch . Li et al . ( 2020 ) extend this idea and use feature space blending along with label interpolation . Hendrycks et al . ( 2019 ) considers self-supervised training with the prediction of image rotations as an auxiliary task . The authors argue that predicting rotation requires shape information and thus improves robustness . Similarly , Shi et al . ( 2020 ) proposes Dropout-like algorithm to reduce the texture bias and thereby increase the shape bias to improve model robustness . However , the authors also discuss that a “ sweet spot ” between shape and texture is needed for the model to be robust for domain generalization . With Patch-wise Adversarial Regularization , Wang et al . ( 2019 ) try to penalize reliance on local predictive representations in early layers and encourage the network to learn global concepts . Other augmentation techniques that aim to improve common corruption robustness are PatchGaussian ( Lopes et al. , 2019 ) , CutMix ( Yun et al. , 2019 ) , AugMix ( Hendrycks * et al. , 2020 ) , and RandAugment ( Cubuk et al. , 2019 ) . At this point , it remains unclear whether the increase in robustness caused by these augmentations is due to learning fundamentally different representations such as more shapebiased ones or to more incremental improvements in feature quality . Edge-based Representations A classical method for extracting edge maps is the Canny edge extractor Canny ( 1986 ) . More recent approaches use DNNs ( Xie & Tu , 2015 ; Liu et al. , 2017 ) ( see Figure A1 ) . Geirhos et al . ( 2019 ) evaluate their shape-biased models on edge maps obtained with a Canny edge detector . ImageNet-Sketch ( Wang et al. , 2019 ) is a newly collected sketch-like dataset matching the ImageNet validation dataset in shape and size . It is used to evaluate generalization to domain shifts . In contrast to these works , we generate the edge-based representations with an edge detector using Richer Convolutional Features ( RCF ) ( Liu et al. , 2017 ) ( see Figure A1 ) and use them explicitly for training . We provide evidence that edge-based representations enhance the shape bias , through an evaluation on images with induced texture-shape cue conflict and patch-shuffled images . 3 LEARNING SHAPE-BASED REPRESENTATIONS . Similar to Geirhos et al . ( 2019 ) , we aim to enhance the shape bias of a network so that it bases its decision more on shape details than on the style of objects encoded in textures . While Geirhos et al . ( 2019 ) augment training data with different styles ( stylization ) , thereby making texture cues less predictive , we extract edge information ( edge maps ) from the training images to maintain explicit shape details and remove texture-related information completely . Here , we consider grayscale intensity edge maps rather than separate edge maps for each color channel . We propose to train CNNs using the edge maps in addition to the standard training data to learn shape-based representations for more effective shape-based decision-making . Besides training on the dataset with explicit shape cues , high capacity networks learn different feature representations when trained jointly on datasets from different distributions . Despite edge maps encouraging CNNs to learn shape-based representations , we observe that the network learns to encode features with texture details when introduced to the standard image data during training . We propose here to further restrain the network from learning texture details on standard image data . We discuss below the extraction of edge details from images to create the edge map dataset and explain the technique to reduce the texture bias of the CNN . Edge dataset Given a standard image dataset , we construct a new dataset with edge maps ( named the Edge dataset ) by extracting the edge details of each image . The edge details are extracted by the CNN-based edge detector using richer convolutional features ( RCF ) proposed in Liu et al . ( 2017 ) . RCF network produces a single-channel edge map that contains the pixel values between [ 0 , 255 ] . We convert the non-binary edge map into a binary map with values in { 0 , 255 } using a threshold of 128 and transform it into a 3-channel RGB edge map by duplicating the channels , so we can use the edge maps as a direct input to train the CNNs . The edge maps from the Edge dataset are used as input and can be independently used to train or evaluate CNNs without necessarily being combined with the standard image data . Please refer to Section A.1 for the details of RCF network . Style Randomization ( SR ) While using a dataset with explicit shape cues enhances shape-based representations , we propose to further reduce the texture bias of the network when training on standard images . It is shown in the literature of style transfer ( Dumoulin et al. , 2016 ; Huang & Belongie , 2017 ) that the statistics of feature maps ( e.g. , mean and standard deviation ) of a CNN effectively capture the style of an image and changing these statistics would correspond to a change in the style of an image . SIN dataset is generated using such style transfer technique and shown to reduce the texture bias of the networks . Inspired by this observation , we propose a simple technique to effectively reduce the texture bias using the feature statistics when being trained on standard training data . We modify the style of an image in the feature space so that the network becomes style-invariant . In particular , we randomize the style details , i.e . feature statistics , of an image during training such that the network can not rely on the texture cues . A similar approach named Style Blending ( SB ) is proposed in Nam et al . ( 2019 ) which randomizes the style information by interpolating the feature statistics between different samples in a mini-batch . We propose here a slightly different approach to make the network invariant to style information . Instead of interpolating the statistics of similar distribution of data i.e , training samples , we completely randomize the feature statistics ( mean and standard deviation ) by randomly sampling them from an uniform distribution . Considering Xi as the ith feature map of an intermediate layer in CNN , and µi & σi as the feature statistics of Xi , the style randomized feature map X̂i is defined as : X̂i : = σ̂i ∗ ( Xi − µi σi ) + µ̂i ( 1 ) where σ̂i ∼ Uniform ( 0.1 , 1 ) and µ̂i ∼ Uniform ( −1 , 1 ) . These specific choices of sampling for σ̂i and µ̂i were found to perform best on our evaluations . The style transfer technique described in Huang & Belongie ( 2017 ) replaces the feature statistics of content image with the statistics of a desired style image to change the style . Similarly , we replace the statistics of content image with random statistics to change the style information . Training the network with SR reduces the texture bias and improves shape-based decision making . An advantage of SR over SB is that the feature statistics are sampled from a different distribution than the training data , that encourages learning representations to generalize better to out-of-distribution data . We show in Section 5 that SR outperforms SB and aids the network to induce stronger shape-based representations .

Summary: This paper delves deeper into understanding shape-based representation of CNNs in an empirical way. Based on the stylized images, it proposes to use edge maps to more explicitly feed shape information to learning models. Besides, the common way to let models learn the shape-based representation is to train on the dataset contained the shape information while the texture information is severely distorted. The paper takes the point of changing the statistics of feature maps would result in style changes and proposed style-randomization to help CNNs better focus on shape information. Also, it connects the biasing degree on shape information of models with the defensive performance against common corruptions, like Gaussian additive noise, blur, and etc. An intuitive conclusion was drawn that there is no clear correlation between shape bias and robustness against common corruptions, and justified by extensive experiments.