Shape-Texture Debiased Neural Network Training
Authors: Yingwei Li
Qihang Yu
Mingxing Tan
Jieru Mei
Peng Tang
Wei Shen
Alan Yuille
cihang xie
Venue: ICLR 2021 Poster (2021)
Keywords: data augmentation
representation learning
debiased training

Abstract: Shape and texture are two prominent and complementary cues for recognizing objects. Nonetheless, Convolutional Neural Networks are often biased towards either texture or shape, depending on the training dataset. Our ablation shows that such bias degenerates model performance. Motivated by this observation, we develop a simple algorithm for shape-texture debiased learning. To prevent models from exclusively attending on a single cue in representation learning, we augment training data with images with conflicting shape and texture information (eg, an image of chimpanzee shape but with lemon texture) and, most importantly, provide the corresponding supervisions from shape and texture simultaneously. 

Experiments show that our method successfully improves model performance on several image recognition benchmarks and adversarial robustness. For example, by training on ImageNet, it helps ResNet-152 achieve substantial improvements on ImageNet (+1.2%), ImageNet-A  (+5.2%), ImageNet-C (+8.3%) and Stylized-ImageNet (+11.1%), and on defending against FGSM adversarial attacker on ImageNet (+14.4%). Our method also claims to be compatible with other advanced data augmentation strategies, eg, Mixup, and CutMix. The code is available here: https://github.com/LiYingwei/ShapeTextureDebiasedTraining.

Published as a conference paper at ICLR 2021
SHAPE-TEXTURE DEBIASED TRAINING
NEURAL NETWORK
Yingwei Li', Qihang Yu', Mingxing Tan Jieru Mei', Peng Tang', Wei Shen
Alan Yuille' & Cihang Xie
JJohns Hopkins University 2Google Brain *University of California, Santa Cruz
SShangaai Jiaotong University
ABSTRACT
Shape and texture are two prominent and complementary cues for recognizing objects. Nonetheless, Convolutional Neural Networks are often biased towards either texture or shape, depending on the training dataset. Our ablation shows that such bias degenerates model performance. Motivated by this observation, we de- velop simple algorithm for shape-texture debiased learning. To prevent models from exclusively attending on a single cue in representation learning, we augment training data with images with conflicting shape and texture information (e.g.. image of chimpanzee shape but with lemon texture) and, most importantly, pro- vide the corresponding supervisions from shape and texture simultaneously.
Experiments show that our method successfully improves model performance on several image recognition benchmarks and adversarial robustness, For ex- ample, by training on ImageNet, it helps ResNet-152 achieve substantial im- provements on ImageNet (+1.2%), ImageNet-A (+5.2%). ImageNet-C (+8.3%) and Stylized-ImageNet (+11.1%), and on defending against FGSM adversar- ial attacker ImageNet (+14.4%). Our method also claims to be com- patible to other advanced data augmentation strategies, e,g., Mixup and Cut- Mix. The code is available here: hittps://gitniee com/ LiYingwei/ shaaeeeeettreeeeesssseee......a
INTRODUCTION
It is known that both shape and texture serve as essential cues for object recognition. A decade ago, computer vision researchers had explicitly designed a variety of hand-crafted features, either based on shape (e.g., shape context (Belongie et al. 22022) and inner distance shape context (Ling & Jacobs 2007 or texture (e.g., textons (Malik et al. 200T)), for object recognition. Moreover, researchers found that properly combining shape and texture can further recognition performance (Shotton et al., 2009 Zheng et al. 2007). demonstrating the superiority of possessing both features.
Nowadays, as popularized by Convolutional Neural Networks (CNNs) (Krizhevsky et al. 2012). the features used for object recognition are automatically learned, rather than manually designed. This change not only eases human efforts on feature engineering, but also yields much better performance on a wide range of visual benchmarks (Simonyan & Zisserman| [0015 He et al.) Girshick et al.) 2014 Girshick] 20155 Ren et al. 2015 Long al. [2015] Chen al. 2015). But interestingly, as pointed by Geirhos et al. (2019). the features learned by CNNs tend to bias toward either shape or texture, depending on the training dataset.
We verify that such biased representation learning (towards either shape or texture) weakens CNNs' performance!! Nonetheless, surprisingly, we also find (1) the model with shape-biased representa- tions and the model with texture-biased representations are highly complementary to each other, e.g., they focus on completely different cues for predictions (an example provided in Figure [D: and (2) being biased towards either cue may inevitably limit model performance, e.g., models may not be able to tell the difference between a lemon and an orange without texture information. These ob- servations altogether deliver promising message- biased models (e.g., ImageNet trained (texture- biased) CNNs (Geirhos et al. (019) or (iape-bbased) CNNs (Shi et al. (0200) are improvable.
BBased models are acquired similar Geirhos et al. (0019) Section 2 for details.

---

Published as a conference paper at ICLR 2021
Shape-biased Model
Texture- birsed Model
Debiased Model (ours)
Models pay attention to...
Test Image Label: Fur Coat
/Shape
XTexture
XShape
Texture
/Shape
Texttre
Models prediction
Poncho X
Egyptian cat X
Fur Coat /
Figure 1: Both shape and texture essential cues for object recognition, and biasing towards either one degenerates model performance. As shown above, when classifying this fur coat image, the shape-biased model is confounded by the cloth-like shape therefore predict it a poncho, and the texture-biased model confuses it as an Egyptian cat because the misleading texture. Nonetheless, our debiased model can successfully recognize it as a afur coat by leveraging both shape and texture.
To this end, we hereby develop a shape-texture debiased neural network training framework to guide CNNs for learning better representations. Our method is data-driven approach, which let CNNs automatically figure out how to avoid being biased towards either shape or texture from their training samples. Specifically, we apply style transfer to generate cue conflict images, which breaks the correlation between shape and texture, for augmenting the original training data. The most important recipe of training successful shape-texture debiased model is that we need to provide supervision from both shape and texture on these generated cue conflict images, otherwise models will remain being biased.
Experiments show that our proposed shape-texture debiased neural network training significantly improves recognition models. For example, on the challenging ImageNet dataset (Russakovsky al. 2015). method helps ResNet-152 gain absolute improvement of 1.2%, achieving 79.8% top-1 accuracy. Additionally, compared to its vanilla counterpart, this debiased ResNet-152 shows better generalization ImageNet-A (Hendrycks et al. 2019) (+5 2%%. ImageNet-C (Hendrycks & Dietterich] 2019) (+8.3%) and Stylized ImageNet (Geirhos et al. 2019) (+11.1%). and stronger ro- bustness defending against FGSM adversarial attacker ImageNet (+14.4%) Our shape-texture debiased neural network training is orthogonal to other advanced data augmentation strategies, e,g., it further boosts NesMix-ResNeXt-1O0 (Yun al. 2019 by 0.7% ImageNet, achieving 81.2% top-1 accuracy.
2 SHAPE/TEXTURE BIASED NEURAL NETWORKS
The biased feature representation of CNNs mainly stems from the training dataset, e.g.. Geirhos al. (2019) point out that models will be biased towards shape if trained on Stylized-ImageNe dataset. Following Geirhos et al. 22099, we hereby present a similar training pipeline to acquire shape- biased models or texture- biased models. By evaluating these two kinds of models, we observe the necessity of possessing both shape and texture representations for CNNs better recognize objects.
2.1 MODEL ACQUISITION
Data generation. Similar to Geirhos et al. 2019). we apply images with conflicting shape and tex- ture information training samples obtain shape-biase biased texture-biased models. But different from Geirhos et al. (2019), an important change in our cue conflict image generation procedure is that we override the original texture information with the informative texture patterns from another
2

---

Published as a conference paper at ICLR 2021
Original Data
Training Images
Label Assignment & Results
Chimpanzee
(a) Shape-biased Model
Chimpanzee
Lemon
(b) Texture-biased Model
Chimpanzee
(c) Debiased Model
Lemon
Lemon
Figure 2: Illustration of the our training pipeline for acquiring (a) shape-biased model, (b) texture-biased model, and (c) a shape-texture debiased model. Specifically, these models share the same training samples, ie. images with conflicting texture and shape information, generated by style transfer between two randomly selected images; but apply distinct labelling strategies: in (a) & (b), labels are determined by the images that provides shape (or texture) information in style transfer, for guiding models to learn more shape (or texture) representations: (c), labels are jointly determined by the pair of images in style transfer, for avoiding bias in representation learning.
randomly selected image, rather than with the uninformative style of randomly selected artistic paintings. That being said, to create a new training sample, we need to first select pair of images from the training set uniformly random, and then apply style transfer blend their shape and texture information. Such generated example is shown in Figure ie.. the image of chimpanzee shape but with lemon texture.
Label assignment. The way of assigning labels to cue conflict images controls the bias of learned models. Without loss generality, we show the case of learning the model to attend more on texture, the labels assigned to the cue conflict images here will be exclusively based on the texture information, e.g., the image of chimpanzee shape but with lemon texture will be labelled as lemon, shown in Figure By this way. the texture information is highly related the "ground-truth" while the shape information only serves as nuisance factor during learning. Similarly, to learn shape-biased model, the label assignment of cue conflict images will be based on shape only, e.g., the image of chimpanzee shape but with lemon texture now will be labelled as chimpanzee, shown in Figure 2 a).
-batee model. To guide
2.2 EVALUATION AND OBSERVATION
To reduce the computational overhead in this ablation, all models are trained and evaluated on 1m0g which is 200 classes subset of the original ImageNet, including 100,000 images (500 images per class) for training and 10,000 images (50 images per class) for validation. Akin to Geirhos et al. (2019), we observe that the models with biased feature representations tend to have inferior accuracy than their vanilla counterparts. For example, shape-biased ResNet-18 only achieves 73.9% top-5 ImageNet-200 accuracy, which is much lower than the vanilla ResNet-18 with 88.2% top-5 ImageNet-209 accuracy.
Though biased representations weaken the overall classification accuracy, surprisingly, we find they highly complementary to each other. We first visualize the attended image regions of biased models, via Class Activation Mapping (Zhou et al. (2166), in Figure 3 As we can see here, the shape-biased model and the texture-biased model concentrate different cues for predictions. For
3

---

Published as a conference paper at ICLR 2021
Label: Tabby Cat
Label: Lemon
Label: Lion
Test Images
Shape Model
Texture Model
Shape Model
Texture Model
Shape Model
Texture Model
Attention Map
Figure 3: The shape- biased model and the texture-biased model attend on complementary cues for predictions. We use Class Activation Mapping to visualize which image regions are attended by models. Redder regions indicates more attentions are paid by models.
Obelisk
ATM
Bucket, pail
Cliff, Drop
Potpie
Fur Coat
More Accurate
Shape-biased Model
Less Accurate
Brain Coral
Expresso
Wooden Spoon
Triumphal Arch
Trolleybus
Less Accurate Figure 4: The shape-biased model and the bexture-biased model are good/bad at classifying different object categories. We these object categories according to the model's corresponding top-1 accuracy, where the righter one indicates lower accuracy achieved by the model.
More Accurate
Texture-biased Model
instance, on the leftmost tabby cat image, the shape-biased model mainly focuses on the cat head, while the texture-biased model mainly focuses on the lower body and the front legs of the cat. Such attention mechanisms extracts the shape of the cat head as an important signal for predictions, while the texture-biasse model relies the texture information of cat fur for predictions.
correlated to their learned representations-tthe shape-biased model
As distinct cues are picked by suape-biased/textrrm biased models, a more concrete observation is they good/bad classifying quite different object categories. As showed in Figure the shape- biased model is good at recognizing objects with representative shape structure like obelisk, but is bad recognizing objects whose shape is uninformative or almost indistinguishable from others like fur coat. Similarly, the texture-biased model can effectively recognize objects with unique texture patterns like brain coral but may fail to recognize objects with unpredietable texture like trolleybus (as its side body can be painted with different advertisements). Besides, biased models may inevitably perform poorly on certain categories as insufficient cues are applied. For examples, it is challenging to distinguish between lemon and an orange if texture information cannot be utilized, or distinguish between an lion and a tabby cat without shape information.
Given the analysis above, we can conclude that biased representations limit models' recognition ability. But meanwhile, our ablation delivers a promising message-the features learned by biased models are highly complementary to each other. This observation indicates the current training framework improvable (as the resulted models are biased towards texture (Geirhos et al., 2019) shape (Shi et al. 2020)), and offers potential direction for building stronger train models to properly acquire both shape and texture feature representations. We will introduce a simple method for doing next.
should
4

---

Published as a conference paper at ICLR 2021
3 SHAPE-TEXTURE DEBIASED NEURAL NETWORK TRAINING
Recall that when obtaining a biased model, the strategy of label assignment is pivot-when the labels are exclusively determined by the images that provide shape (or texture) information in style transfer, we will obtain shape-biased (or texture-biased) model. Therefore, to guide models for leveraging both shape and texture for predictions, we hereby propose a simple way, which is inspired by Mixup Zhang et al., 2018)) to softly construct labels during training. In other words, given the one-hot label of the shape-source image Y/s and the one-hot label of the texture-source image yt, the new label that we assigned to the cue conflict image is
V = + (1-7)- yt:
(I)
where y E [0, 1] is a manually selected hyperparameter to control the relative importance between shape and texture. By ranging the shape-texture coefficient from 0 to 1, we obtain a path to evolve the model from being texture-biased one (i.e., = 0) to being shape-biased one (i.e., 7 = 1). Although the two extreme ends lead to biased models with inferior performance, we empirically show that there exist a sweet point along this interpolation path, i.e., the learned models can properly acquires both shape and texture feature representations and achieve superior performance on wide range of image recognition benchmarks.
We name this simple method as shape-texture debiased neural network training, and illustrate the training pipeline in Figure c)). It is worth to mention that, although Figure 2 only shows the procedure of applying our method to the image classification task, this training framework is general and has the potential to be extended to other computer vision tasks, egg., a simple showcase on semantic segmentation is presented in Section 4.4
4 EXPERIMENTS
4.1 EXPERIMENTS SETUP
Datasets. We evaluate models on ImageNet classification and PASCAL VOC semantic segmenta- tion. ImageNet dataset (Russakovsky et al., 2013) consists of 1.2 million images for training, and 50,000 for validation, from 1,000 classes. PASCAL VOC 2012 segmentation dataset (Everingham et al. 2012) with extra annotated images from (Hariharan et al.) 2011) involves 20 foreground object classes and one background class, including 10,582 training images and 1,449 validation images.
Going beyond the standard benchmarks, we further evaluate models' generalization on ImageNet- A, ImageNet-C and Stylized-ImageNee and robustness by defending against FGSM adversarial attacker on ImageNet. ImageNet-C (Hendrycks & Dietterich) 2019) is a benckmark dataset that measures models' corruption robustness. It is constructed by applying 75 common visual corrup- tions to the ImageNet validation set. ImageNet-A (Hendrycks et al. 2019) includes 7,500 natural adversarial examples that successfully attacks unseen classifers. These examples are much harder than original ImageNet validation images due to scene complications encountered in the long tail of scene configurations and by exploiting classifier blind spots (Hendrycks et al. (019). Stylized- ImageNet (Geirhos et al.) (2019) is stylized version of ImageNet that constructed by re-rendering the original images by AdaIN stylizer (Huang & Belongic 2017). The generated images keep the original global shape information but removes the local texture information. FGSM (Goodfellow jet al. 2013) is a widely used adversarial attacker to evaluate model robustness. We set the maxi- mum perturbation change per pixel 16/255 55 for FGSM.
Implementation details. We choose ResNet (He et al. 2016) as the default architecture. For image classification tasks, our implementation is based on the publicly available framework in PyTorcl To generate cue conflict images, we follow follow/Geirhos et al. (2019) to use Adaptive Instance Normaliza- tion (Huang& & Belongie, (017) in style transfer, and set stylization coefficient to increase the diversity of training samples, we generate these cue conflict images on-the-fly fly during training. We choose the shape-texture coefficient 7 = 0.8 when assigning labels.
0.5. Importantly,
When training shape- texture- biased and our shape-texture debiased models, we always apply
the auxiliary batch normalization (BN) design (Xie et al. 2020 Xie & Yuille 2020 Chen et al.
assi fication
5

---

Published as a conference paper at ICLR 2021
VANILLA
2xEPOCHS
S-BIASED
T-BIASED
DEBIASED
ResNet-50
76,4
76.4 (+0.0)
76.2 (-0.2)
75.3 (-1.1)
76.9 (+0.5)
ResNet-101
78.0
78.0 (+0.0)
78.0 (-0.0)
77.4 (-0.6)
78.9 (+0.9)
ResNet-152
78.6
79.1 (+0.5)
78.6 (-0.0)
78.1 (-0.5)
79.8 (+1.2)
Table 1: The performance of the vanilla training, the shape-biased (S-biased) training, the texture-
biased (T-biased) training, and our shape-texture debiased training on ImageNet. For all ResNet models, our debiased training shows the best performance among others.
IN-A Acc. T
IN-C mCE +
S-IN Acc.
FGSM Acc. 1
Re
2.0 3.5 (+1.5)
75.0 67.5 (-7.5)
7.4 17.4 (+10.0)
17.1 27.4 (+10.3)
+ Debiased
ResNet-101
5.6 9.1 (+3.5)
69.8 62.2 (-7.6)
9.9 22.0 (+12.1)
23.1 34.4 (+11.3)
+ Dehiased
ResNet-112
7.4
67.2
11.3
25.2
+ Debiased
12.6 (+5.2)
58.9 (-8.3)
22.4 (+11.))
39.6 (+14.4)
Table 2: The model robustness on ImageNet-A A (IN-A), ImageNet-C (IN-C), Imaaeeeet (S- IN), and on defending against FGSM adversarial attacker on ImageNet. Our shape-texture debiased neural network training significantly boosts the model robustness over the vanilla training baseline.
2021 to bridge the domain gap between the original data and the augmented data, i.e., the main BN is exclusively running on original ImageNet images and the auxiliary BN is exclusively running on cue conflict images. We follow Xie et al. (2020) to always apply the main BN for performance evaluation. Besides, since our biased models and debiased models are all trained with both the original data and the augmented data (i.e., 2x data are used in training), we also consider a stronger baseline (i.e., 2x epochs training) which doubles the schedule of the vanilla training baseline, for the purpose of matching the total training cost.
4.2 RESULTS
Model accuracy. Table [ shows the results ImageNet. For all ResNet models, the proposed shape-texture debiased neural network training consistently outperforms the vanilla training base- line. For example, it helps ResNet-50 achieve 76.9% top-1 accuracy, beating its vanilla counterpart by 0.5%. Our method works better for larger models, e.g., it further improves the vanilla ResNet-152 by 1.2%, achieving 79.89 79.8% top-1 accuracy.
We then compare our shape-texture debiased training to the 2x epochs training baseline. We find that simply doubling the schedule of the vanilla training baseline cannot effectively lead to improve- ments like ours. For examples, compared to the vanilla ResNet-101 101, this 2x epochs training fails to provide additional improvements, while ours furthers the top-1 accuracy by 1.0%. This result suggests that it is non-trivial to improve performance even if more computational budgets are given.
Lastly, we compare ours to the biased training methods. Though the only difference between our method and the biased training methods is the strategy of label assignment (as shown in Figure 2). it imperatively affects model performance. For example, compared to the vanilla baseline, both the shape-biased biased training and the texture-biased training fail improve (sometimes even slightly hurt) the model accuracy, while our shape-texture debiased neural network training successfully leads to consistent and substantial accuracy improvements.
Model robustness. Next, we evaluate models' generalization ImageNet-A, ImageNet-C and Stylized-ImageNer and robustness on defending against FGSM on ImageNet. We note these tasks are much more challenging than the original ImageNet classification, e.g., the ImageNet trained ResNet-50 only achieves 2.0% accuracy on ImageNet-A, 75.0% mCE on ImageNet-C, 7.4% accu- racy on Beylized-Imageniy As shown in Table our shape-texture debiased neural network training beats the vanilla training baseline by a large margin on all tasks for all ResNet models. For example, it substantially boosts ResNet- 152's performance on ImageNet-A (+5.2% from 7.4% to 12.6%), ImageNet-C (-8.3% from 67.2% to 58.9%, the lower the better) and Stylized-ImageNN and defending against FGSM on ImageNet (+14.4% from 25.2% to 39.6%). These results alto- gether suggest that our shape-texture debiased neural network training is an effective way to mitigate the issue of shortcut learning (Geirhos et al. 2020).
and 17.1% accuracy on defending against FGSM adversarial attacker.
++....1 1%, from 11.3% to 22.4%)

---

Published as a conference paper at ICLR 2021
IN Acc.
IN-A
IN-C
S-IN
FGSM
Acc.
mCE
Acc.
Acc. 1
ResNet-50
76.4
2.0
75.0
7.4
17.1
CutMix MoEx (Li et 2021)
79.0
8.0
74.8
5.0
41.0
DeepAugment AugMIX Henddrykk et al. 2020)
75.8
3.9
53.6
21.2
18.8
SIN (Geirhos al. 2019)
60.2
2.4
77.3
56.2
5.6
Shape-Texture Debiased Training (ours)
76.9
3.5
67.5
17.4
27.4
Table 3: Compare with state-of-the-art methods using ResNet-50 on ImageNet (IN), ImageNet-A
(IN-A), ImageNet-C (IN-C), Stylized--mageNNtt (S-IN), and on defending against FGSM on Ima- geNet. We use green to denote sigmificant improvement, red to denote performance drop, and gray to denote similar performance. We observe our thape-texture debiased training is the only method that successfully leads to improvements over the vanilla baseline on all benchmarks.
Datasets
VANILLA
S-BIASED
T-BIASED
DEBIASED
NettehettS
23.8
27.9
24.3
28.4
ImageNet-R
36.2
40.6
36.7
40.8
Kylberg Texture
99.5
99.1
99.6
99.5
Flicker Material 74.6
73.3
79.2
75.8
Table 4: The performance comparison between Vanilla, Shape-biased, biased, Texture-biased, and Shape- Texture Debiased models on ImageNet-Sketch, ImageNet-R, Kylberg Texture, and Flicker Mate- rial datasets. We note the shape-biased and the shape-texture debiased models perform better on shape datasets (ImageNet-Skec Sketch and ImageNet-RR. R);; the cexture-biased and the shape-texture debi- ased models perform better on texture datasets (Kylberg Texture and Flicker Material).
Comparing to SoTAs. We further compare our shape-texture debiased model with the SoTA on ImageNet and ImageNet-A (CutMix + MoEx (Li et al.] (02I)), the SoTA on ImageNet-C (Deep- Augment + AugMix (Hendrycks et al., 2020 and the SoTA on Stylized-ImageNe et al. 2019)). Interestingly, we note the improvements of all these SoTAs are not consistent across different benchmarks. For example, as shown in Table SIN significantly improves the results on Italize,-ImageNet, but at the cost of huge performance drop on ImageNet (-16.2%) and ImageNet- C (-2.3%). Our shape-texture debiased training stands as the only method that can improve the vanilla training baseline holistically.
(SIN (Geirhos
4.3 ABLATIONS
Comparing model ensembles. An alternative but naive way for obtaining the model with both shape and texture information is to ensemble a shape-biased model and a texture-biased model. We note this ensemble strategy yields a model of on-par performance with our shape-texture debiased model on ImageNet (77.2% vs. 76.9%). Nonetheless, interestingly, when measuring model robust- ness, such model ensemble strategy is inferior than ours. For example, compared to our proposed debiased training, this ensemble strategy is 1.5% worse on ImageNet-A (2.0% vs. 3.5%), 1.1% worse on ImageNer-C (68.6 mCE vs. 67.5 mCH). % worse on Stylized-ImageNet (16.3% vs. 17.4%), and 7. 7.0% worse on defending against FGSM (20.4% vs. 27.4%). Moreover, due to model ensem- ble, this strategy is 2x expensive at the inference stage. These evidences clearly demonstrate the effectiveness and efficiency of the proposed shape-texture debiased training.
Does our method help models to learn debiased shape-texture representations? Here we take a close look at whether our method indeed prevents models from being biased toward shape or texture during learning. We evaluate models Section 4.2 2 on two kinds of datasets: (1) ImageNet-
Sketch dataset (Wang et a 2019) and ImageNet-R (Hendrycks et al.) 2020) for examining how well models can capture shape; and (2) Kylberg Texture dataset (Kylberg) 201T) and Flicker Material dataset (Sharan et al. 2014) for examining how well models can capture texture. Specifically, since object categories from two texture datasets are not compatible to that from ImageNet dataset, we retrain the last fe-layer (while keeping all other layers untouched) of all models on Kylberg Texture dataset or Flicker Material dataset for epochs. The results are shown in Table
We first analyze results ImageNet-Skette dataset. We observe shape- texture debiased models are as good as the shape- biased models, and significantly outperforms the texture- biased models and the vanilla training models. For instance, using ResNet-50, our shape-texture debiased training and
7

---

Published as a conference paper at ICLR 2021
Texture Source
Texture Source Object Shape Source Image & Label
Generated Image & Label
Image & Label
Figure 5: Illustration of the data preparation pipeline of our shape-texture debiased neural network training on the semantic segmentation task.
Seg
shape-biased training achieve 28.4% top-1 accuracy and 27.9% top-1 accuracy, while texture-biased training and vanilla training only get 24.3% top-1 accuracy and 23.8% top-1 accuracy. A similar observation can be seen from ImageNet-R. These results support that our method helps models acquire stronger shape representations than the vanilla training.
We next analyze results on Kylberg Texture dataset. Similarly, we observe that our debiased model are comparable to the texture-biased model and the vanilla training model, and get better perfor- mance than the shape-biased model. On Flicker Material dataset, we observe that our debiased models are better than the vanilla training model and the shape-biased model. This phenomenon suggests texture information is effectively caught by our shape-texture debiased training. As side note, is expected that vanilla training are better than shape- biased training on these texture datasets, as Geirhos et al. (2019) point out that ImageNet trained models (i.e., vanilla training) also tend to be biased towards texture.
With the analysis above, conclude that, compared to vanilla training, our shape- texture debiased training successfully helps networks effectively acquire both shape and texture representations.
Combining with other data augmentation methods. Our shape-texture debiased neural network training be viewed as data augmentation method, which trains models cue conflict images. Nonetheless, our method specifically guides the model to learn debiased shape and texture repre- sentations, which could potentially serve as a complementary feature to other data augmentation methods. To validate this argument, we train models using combination of our method and an existing data augmentation method (i.e., Mixup (Zhang al. 2018) or CutMix (Yun et al. 2019)).
We choose ResNeXt-100 101 (Xie et al., (017) as the backbone network, which reports the best top-1 ImageNet accuracy in both the Mixup paper, i.e., 79.9%, and the CutMix paper, i.e., 80.5%. Though building upon very strong baselines, our shape-texture debiased neural network training still leads to substantial improvements, e.g., it furthers ResNeett-O1-Mixxups accuracy to 80.5% (+0.6%), and ResNeXt-100 101-CutMix's accuracy to 81.2% (+0.7%). Mcanwhile, models' generalization also get greatly improved. For example, by combining CutMix and our method. ResNeXt-101 gets additional improvements ImageNet-A (+1.4%), ImageNet-C (-5.9%, the lower the better) and Stylized ImageNet (+7.5%). These results support that our shape-texture debiased neural network training is compatible existing data augmentation methods.
Shape-texture coefficient 7. We set 7 0.8 in our shape-texture debiased training. This value is found via the grid search over ImageNet-200 using ResNet-18. We now ablate its sensitivity on ImageNet using ResNet-50, where 7 is linearly interpolated between 0.0 and 1.0. By increasing the value of y, we observe that the corresponding accuracy on ImageNet first monotonically goes up, and then monotonically goes down. The sweet point can be reached by setting 7 0.7, where ResNet-50 achieves 77.0% top-1 ImageNet accuracy. Besides, we note that by setting 7 E [0.5. 0.9) can always lead to performance improvements over the vanilla baseline. These results demonstrate the robustness of our shape-texture debiased neural network training w.r.t. the coefficient 7
4.4 SEMANTIC SEGMENTATION RESULTS
We extend our shape-texture debiased neural network training to the segmentation task. We select Deepl.bbv3 3 (Chen al. 2017) backbone. To better incorporate method
8

---

Published as a conference paper at ICLR 2021
with the segmentation task, the following changes are made when generating cue conflict images: (1) unlike in the classification task where the whole image is used as the texture source, we use a specific object (which can cropped from the background using the segmentation ground-truth) to provide texture information in style transfer; (2) when composing the soft label for the cue conflict image, we set the label mask from texture source the full image (since the pattern from the texture source will fill the whole image after style transfer); and (3) we set stylization coefficient = 0.2 and shape-texture coefficient 7 0.95 to prevent object boundaries from being overly blurred in style transfer. Figure 3 an illustration of our data preparation pipeline.
Results. Our shape-texture debiased training can also effectively improve segmentation models. For example, our method helps Beepl.abv3-ResNet-ID achieve 77.6% mIOU, significantly heating its vanilla counterpart by 1.1%. Our method still shows advantages when compared to the 2x epochs training baseline. Doubling the learning schedule of the vanilla training can only lead to an improvement of 0.2%, which is still 0.9% worse than our shape-texture debiased training. These results demonstrate the potential of our methods helping recognition tasks in general.
RELATED WORK
Data augmentation. Data augmentation is essential for the success of deep learning (LeCun et al.
1998
Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; Zhong et al.) 2020 Cubuk et al.
2019 Lim et al.) 2019; Cubuk et al. 2020). Our shape-texture debiased neural network training is related to specific family data augmentation, called Mixup (Zhang et al. 2018), which blends pairs of images and their labels in a convex manner, either at pixel-level (Zhang et al., 2018 Yun jet al.) 2019) or feature-level (Verma et al. 2019 Li et al., 202I). Our method can be interpreted special instantiation of Mixup which blends pairs of images at the abstraction level- -images' texture information and shape information are mixed, Our method successfully guides CNNs to learn better shape and texture representations, which is an important but missing piece in existing data argumentation methods.
Style transfer.
Style transfer, closely related to texture synthesis and transfer, means generating stylized image by combining shape-source image and a texture-source image (Efros & Leung
1999 Eiros & Freeman 2001; Elad & Milanfar 2017). The seminal work (Gatys et al. 2016)
demonstrate impressive style transfer results by matching feature statisties in convolutional layers
of a CNN. Later follow-ups further improve the generation quality and speed (Huang & Belongie
2017 Chen & Schmidi [2016) Ghiasi al. 2017 Li al.) 2017). In this work, follow Geirhos jet al. (2019) to use AdaIN (Huang & Belongie, 2017) to generate stylized images. Nonetheless, instead of applying style transfer between an image and an artistic paintings as in Geirhos et al. (2019), we directly apply style transfer on a pair of images to generate cue conflict images. This change vital as enables us to provide supervisions from both shape and texture during training.
6 CONCLUSION
There is a long-time debate about which cue dominates the object recognition. By carefully ablate the shape- biased model and the texture-biased model, we found though biased feature representa- tions lead to performance degradation, they are complementary to each other and are both necessary for image recognition. To this end, we propose shape-texture debiased neural network training for guiding CNNs to learn better feature representations. The key in our method is that we should not only augment training set with cue conflict images, but also provide supervisions from both shape and texture. We empirically demonstrate the advantages of our shape-texture debiased neural net- work training on boosting both accuracy and robustness. Our method is conceptually simple and is generalizable to different image recognition tasks. We hope our work will shed light on understand- ing and improving convolutional neural networks.
ACKNOWLEDGEMENT
This project is partially supported by ONR 900014-18-1-2110 and ONR N00014-20-1-220 Ci- hang Xie is supported by the Facebook PhD Fellowship and a gift grant from Open Philanthropy. Yingwei Li thanks Zhiwen Wang for suggestions on figures.

---

Published as a conference paper at ICLR 2021
REFERENCES
Serge Belongie, Jitendra Malik, and Jan Puzicha. Shape matching and object recognition using shape contexts. 7PAMI, 2002.
Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Se- mantic image segmentation with deep convolutional nets and fully connected CREs, In ICLR, 2015.
Liang- Chigg Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 2017.
Tian Qi Chen and Mark Schmidt. Fast patch-based style transfer of arbitrary style. NeurIPS Work- shop, 2016.
Xiangning Chen, Cihang Xie, Mingxing Tan, Li Zhang, Cho-Jui Hsieh, and Boqing Gong. Robust and accurate object detection via adversarial learning. In CVPR, 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In CVPR, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 2001.
Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In ICCV, 1999.
Michael Elad and Peyman Milanfar. Style transfer via texture synthesis. TIP, 2017.
M. Everingham. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.paccal 2012.
newort..ggcaalenees/O/Cvo222 (eCCCv/o20iwwotsssoonnnneee....
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet- trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. ICLR, 2019.
Robert Geirhos, förn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.0077 077880 2020.
Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens. Explor- ing the structure of real-time, arbitrary neural artistic stylization network. In BMVC, 2017.
Ross Girshick. Fast R-CNN, In ICCV, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. In CVPR, 2014.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Bharath Hariharan, Pablo Arbelacz, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Seman- tic contours from inverse detectors. ICCV, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
10

---

Published as a conference paper at ICLR 2021
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In ICLR,, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint orXiv:1907.07172 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of distribbuii generalization. arXiv preprint arXiv:2006. /6241, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer real-time with adaptive instance normal- ization. In ICCV, 2017.
Alex Krizhevsky, Iya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
Gustaf Kylberg. Kylberg Texture Dataset v. 1.0. Centre for Image Analysis, Swedish University of Agricultural Sciences and Uppsala University, 2011.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
Boyi Li, Felix Wu, Ser-Nam Lim, Serge Belongie, and Kilian Q Weinberger. On feature normaliza- tion and data augmentation. In CVPR, 2021.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang. Xin Lu, and Ming- Hsuan Yang. Universal style transfer via feature transforms. In NeurIPS, 2017.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In NeurIPS, 2019.
Haibin Ling and David Jacobs. Shape classification using the inner-distance. TPAMI, 2007.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Jitendra Malik, Serge J. Belongie, Thomas K. Leung, and Jianbo Shi. Contour and texture analysis for image segmentation. IJCV, 2001.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJC,, 2015.
Lavanya Sharan, Ruth Rosenholtz, and Edward H. Adelson. Accuracy and speed of material cate- gorization in real-world images. Journal of Vision, 14(10), 2014.
Baifeng Shi, Dinghuai Zhang, Qi Dai, Zhanxing Zhu, Yadong Mu, and Jingdong Wang. Informative dropout for robust representation learning: A shape-bias perspective. In /CML, 2020.
Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost for image under- standing: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. IJCV, 2009.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David I Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In NeurlPS, 2019.
11

---

Published as a conference paper at ICLR 2021
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In ICLR, 2020.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoe V Le. Adversarial examples improve image recognition. In CVPR, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In ICLR, 2018.
Songfeng Zheng, Zhuowen Tu, and Alan L. Yuille. Detecting object boundaries using low-, mid- and high-level information. In CVPR, 2007.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In AAAI, 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
12


Reviews:
[{"review_id": "pq0HNGLN2z1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of an existing bias in the classification networks, which makes them focus on a particular set of features (either local, \"textures\", or global, \"shapes\"). While it is clear how to train shape- and texture-biased networks (by preparing datasets with items easier to recognize using global or local features respectively), the authors claim that a combined training on both of these datasets would improve the performance of the network.\n\nPros.\n\n1. The proposed method tackles an important problem and improves over some baselines (when compared to plain training or other similar regularizations, like Mixup and CutMix) on ImageNet-based datasets. Also, it can be applied to different problems like segmentation and improve the results, which shows the generality of the approach.\n\n2. The authors try to explain the behavior of the method by providing extensive motivation and some experimental confirmations (unfortunately, they are quite limited).\n\n3. The paper is extremely well-written and concise.\n\nCons.\n\n1. The authors claim that whether or not the trained model will have bias depends on the statistics of items in the dataset: the more items easier to recognize using shape features, the more shape-biased the model will be. A simple test for that hypothesis would be to mix shape- and texture-biased datasets and train a model or their concatenation. If the authors' hypothesis is correct, this would aid in the final performance of the model, but I did not find such an experiment or any discussion of it in the experiments section. It seems to me that a major part of the performance of this method could simply come from data augmentation (basically, the same principle as described in the Mixup paper).\n\n2. Following on the previous point, it would be really helpful to see attention visualizations for the proposed model, similar to Figure 4. Right now it is not possible to discern whether or not it has any noticeable differences from the baseline shape model in terms of attention maps, and it could reinforce the proposed explanation of the method.\n\n3. Another important baseline that is missing is evaluating an ensemble of two models: one texture- and one shape-biased. It is a useful baseline, since it is hyperparameter-free, while the proposed method requires a grid-search to find an optimal parameter for blending the labels. Importantly, while the authors state that there is a \"sweet spot\" in this continuous hyperparameter, they do not propose a way to efficiently search for it other than grid-search. While they show that for the ImageNet dataset the network is quite robust to this hyperparameter, the same result may not be true for other datasets. Searching for this hyperparameter may prove to be significantly more costly than training an ensemble of two models.\n\nIn general, the proposed method itself is quite simple and has a lot of familiarities to the previous approaches, which the authors note. The main distinguishing feature of the proposed approach is the mechanism by which it works, suggested by the authors. And, in my opinion, there is not enough numerical and qualitative justification that this approach works the way it is explained.\n\nYet, assuming that some additional clarifications for the mechanisms behind the proposed approach will be provided in the rebuttal, I place my initial rating as \"accept\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "blW4lRCluWM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose a method to mitigate the bias towards either texture or shape, in convolutional network training. The method follows the idea from Geirhos et al (2019), but use images randomly sampled from the same dataset, instead of style transfer from paintings. Then, depending on a manually selected hyperparameter, the weights of conflicting labels are blended by weighted average of the one-hot encoding.\n\n###################\n\nReasons for score\n\nThe method proposed is simple, yet effective; however, parts of the method can be more principled, for example, the sampling of textures used for style transfer and debiasing - instead of using random pairs, random sampling from certain classes or external texture or material dataset would help understand better how debiasing works.\nSince the model claims that the method is debiasing the model (or can bias it towards shape or texture), it would be interesting to validate this claim on a texture database (e.g. Materials in Context, Flickr Material Database (FMD) or Describable Textures Dataset (DTD) ). The expected outcome would be - shape biased model underperforms; texture biased model, improves texture recognition. The last two datasets are small (1K images; 6K images), so it would not increase the computational overhead. \n\n###################\n\nStrong points of the paper:\n\n- visualization of the cues of the two models (texture and shape), using CAM (Class Activation Mapping);\n- showing the orthogonality of shape and texture-biased models (Fig. 4), with examples of tasks where each model is more accurate or less acurate.\n- significant improvement on Stylized ImageNet \n- usecase of the method to mitigate an adversarial attacker (FGSM).\n\n###################\n\nWeaker aspects:\n\n- method is a  bit too simple (however, it is effective).\n The whole paper can be summarized as: pick a random (texture) image, style transfer and blend the labels via weighted average.\nThe contribution in addition to Geirhos et al seems to consist in changing the domain of the texture source, with same dataset images. \n\n- What is the difference between the \"Vanilla\" training and shape biased training? My understanding from Fig. 2 and equation (1) - the two seem the same -- however they have separate columns in Table1 -- with similar results.\nPlease clarify the distinction.\n\n- the method would benefit from a more principled way of choosing textures.\nThe proposed method consists of  (Sec 2.1., Data Generation: \"... first select a pair of images from the training set uniformly at random, and then apply style transfer to blend their shape and texture\".  How the algorithm decides which of the images in the pair is a texture and which one contains shape information? Can the authors make public a list of such fixed pairs, for other researchers to evaluate the methods on?\nJust using style transfer (please also cite the method used for style transfer -- or if it is novel, please describe it) is very similar to the approach in Geirhos et al (2019), and it is not clear what the contribution is apart from changing the domain of the conflicting texture information.\nIt would be interesting to use a texture or material dataset (e.g. Materials in Context, Flickr Material Database, Describable Textures Dataset) as a source of texture for style transfer, or use class-specific textures (e.g. fur-like texture, to bias non-animal classes towards animal classes, e.g. cat, dog). \n\n- lack of evaluation of impact on other tasks, such as transfer learning:\nHow does debiasing affect the features on transfer learning tasks? For example, using the features from the debiased model (pretrained) on some external dataset, e.g. with a linear classifier on top of the features.\n\n- style transfer seems to be performed on entire image -- e.g. Figure 2 - could the authors please clarify this?\nThe example in Fig. 1 shows using the shape of the object, while Fig. 2  There are more recent datasets, such as MS-COCO or LVIS which provide image segmentation which could be used for training and evaluation - and also for guiding the style transfer, e.g. restricting the texture style transfer or retexturing only to the object mask.\nThere is an evaluation on Semantic segmentation in Sec. 4.4; however, it is not clear from Fig 5 why the background is replaced with the texture source.\n\n\n- not too much information on how / why the proposed method improves against FGSM.\n\n- Please add the benchmarks (state-of-the-art and methods compared with, e.g. Geirhos et al in Table 2). It's difficult to chase the numbers in the paper to compare the method with existing work.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QzTMgnaPtFU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Note that I don't follow this research area closely, this is thus more an educated impression of the paper.\n\nThe paper is extremely simple. It builds on the idea of Geirhos et al. 2019 to use style transfer to augment IN training data by mixing shape and texture information but when using a training image resulting from mixing the shape of an image with label A and the texture of an image with label B, they use 0.8*A + 0.2*B as label (which is similar to mixup/cutmix, except the merging is done using style transfer). This alone seem to provide a clear boost on IN and different variants.\n\nMy main worry/question, since Geirhos paper has had a lot of attention (as well as mixup/cutmix), is why nobody has done this yet? The paper is very clear and there don't seem to be any hidden difficulty (except maybe using an auxiliary BN?) If indeed there are no similar work, I think the clear performance boost on IN with a simple and clearly explained method justifies publication.\n\nI would however complain the analysis is not very strong. In particular, a natural alternative to the proposed approach would be to jointly train the convolutional layers but use three different final fc layers to classify synthetic images based on shape, synthetic images based on texture, or natural images. This experiment might help to clarify why the method works.\n\nOther comments:\n- I am not sure the \"debiaised\" term is really appropriate\n- I would prefer to see the results of 4.3 in tables (some could actually be in table 1), they would be more readable, more exhaustive and allow more effective analysis (with the current presentation I have a hard time believing the current approach peforms better than cutmix) + I would like to actually see the curve corresponding to the sensitivity to gamma\n- for the segmentation results, the results are really not sufficient: what are the results with gamma=1?\n- end of p.7 \"which can BE cropped\"\n- personally, I think it's a shame one has to go to p.5 to understand what the method is actually doing when it is so simple. In general, a lot could be shortened in the first 4 pages, and dedicated to better analysis and more exhaustive evaluation", "extra_scores": {}, "extra_text": {}}, {"review_id": "LtPy2_SWrql", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Hypothesis: Convolutional Neural Networks are biased towards either texture or shape according to the dataset used for training. If so, can we develop an approach that can do shape-texture debiased learning?\n\nThis hypothesis is in line with Geihros et al., 2019 who observed that ImageNet trained models are texture-biased. \n\nVerification of the hypothesis: The paper qualitatively and quantitatively shows the influence of a shape-biased and a texture-biased model.\n\nHow to solve the problem?: The authors make two observations - (1). the model with shape-biased representations and the models with texture-biased representations are highly complementary to each other; and (2) being biased towards either shape cues or texture cues may hurt the performance but there exists a sweet-spot along the interpolation path of shape-texture debiased model that allows acquiring both shape and texture representations and achieve superior performance than vanilla models. \n\nThe rest of the paper consists of experiments to demonstrate this property and show analysis for different aspects of the model. \n\nQuestion: One thing that I did not understand is how shape-texture debiasing can help against adversarial attacks? The authors present some quantitative analysis demonstrating it but I still don't get the rationale behind it.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the rebuttal stage, three of four reviewers recommend acceptance, and one gives a borderline score but argues they lean positive. Concerns seem well addressed; the method is simple yet effective.
Decision: Accept (Poster)