The shape and simplicity biases of adversarially robust ImageNet-trained CNNs
Authors: Peijie Chen
Chirag Agarwal
Anh Totti Nguyen
Venue:  (2021)
Keywords: shape bias
texture bias
interpretability
smoothness
visualization

Abstract: Adversarial training has been the topic of dozens of studies and a leading method for defending against adversarial attacks.
Yet, it remains largely unknown (a) how adversarially-robust ImageNet classifiers (R classifiers) generalize to out-of-distribution examples; and (b) how their generalization capability relates to their hidden representations. In this paper, we perform a thorough, systematic study to answer these two questions across AlexNet, GoogLeNet, and ResNet-50 architectures. We found that while standard ImageNet classifiers have a strong texture bias, their R counterparts rely heavily on shapes. Remarkably, adversarial training induces three simplicity biases into hidden neurons in the process of ``"robustifying" networks. That is, each convolutional neuron in R networks often changes to detecting (1) pixel-wise smoother patterns i.e. a mechanism that blocks high-frequency noise from passing through the network; (2) more lower-level features i.e. textures and colors (instead of objects); and (3) fewer types of inputs. Our findings reveal the interesting mechanisms that made networks more adversarially robust and also explain some recent findings e.g. why R networks benefit from much larger capacity and can act as a strong image prior in image synthesis. 

Under review as a conference paper at ICLR 2021
THE SHAPE AND SIMPLICITY BIASES OF ADVERSARI- ALLY ROBUST IMAGENET-TRAINED CNNs
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial training has been the topic of dozens of studies and leading method for defending against adversarial attacks. Yet, remains largely unknown (a) how adversarially-robur ImageNet classifiers (R classifiers) generalize to out-of- distribution examples; and (b) how their generalization capability relates to their hidden representations. In this paper, perform thorough, systematic study to answer these two questions across AlexNet, GoogLeNet, and ResNet-50 archi- tectures. We found that while standard ImageNet classifiers have strong texture bias, their R counterparts rely heavily on shapes. Remarkably, adversarial training induces three simplicity biases into hidden neurons in the process of "robustify- ing" the network. That is, each convolutional neuron in R networks often changes detecting (1) pixel-wise smoother patterns i.e. mechanism that blocks high- frequency noise from passing through the network; (2) more lower-level features i.e. textures and colors (instead of objects); and (3) fewer types of inputs. Our findings reveal the interesting mechanisms that made networks more adversarially robust and also explain some recent findings c.g. why R networks benefit from much larger capacity (Xie & Yuille 2020) and can act as strong image prior in image synthesis (Santurkar et al. 2019).
INTRODUCTION
Given excellent test-set performance, deep neural networks often fail to generalize to out-of- distribution (OOD) examples (Nguyen al.) (015)) including "adversarial examples", i.c. modi- fied inputs that are imperceptibly different from the real data but change predicted labels entirely (Szegedy al. 2014). Importantly, adversarial examples can transfer between models and cause unseen, all machine learning (ML) models to misbehave (Papernot et al. (017), threatening the se- curity and reliability of ML applications (Akhtar & Mian (0018). Adversarial training- classifier to correctly label adversarial examples (instead of real data)-has been a leading method in defending against adversarial attacks and the most effective defense in ICLR 2018 (Athalye al. 2018). Besides improved performance on adversarial examples, test-set accuracy can also be im- proved, for some architectures. when real images are properly incorporated into adversarial training (Xie et al. 2020). It is therefore important to study how the standard adversarial training (by Madry et al. 2018) changes the hidden representations and generalization capabilities of neural networks.
On smaller datasets, Zhang & Zhu (2019) found that adversarially networks (hereafter, R networks) rely heavily on shapes (instead of textures) to classify images. Intuitively, training on pixel-wise noisy images would encourage R networks to focus less on local statistics (e.g. textures) and instead harness global features (c.g. shapes) more. However, an important, open question is:
Q1: On ImageNet, do R networks still prefer shapes over textures?
remains unknown whether such shape preference carries over to the large-scale ImageNet Ruus- sakovsky et al. 2015), which often induces a large texture bias into networks (Geirhos et al.) 2019)
e.g. to separate 150 lour-legged species in ImageNet. Also, this shape- bias hypothesis suggested
by Zhang& & Zhu 2019) seems contradict the recent findings that R networks ImageNet act
as a strong texture priori.e they can be successfully used for many image translation tasks without any extra image prior (Santurkar al. 2019). The above discussion leads follow-up question:

---

Under review as a conference paper at ICLR 2021
Q2: If an R network has a stronger preference for shapes than standard ImageNet networks (here- after, networks), will it perform better on OOD distorted images?
Networks trained to be more shape-biased can generalize better to many unseen ImageNet-C (Hendrycks & Dietterich 2019) image corruptions than S networks, which have a strong texture
bias (Brendel & Bethge| 2019). In contrast, there also evidence that classifiers trained on one type of images often do not generalize well to others (Geirhos et al.] 2018 Nguyen et al.] [015] Kang al., 2019). Importantly, R networks often underperform S networks on original test sets TTiipaa et al. 20199 perhaps due to an inherent trade-off (Madry et al. 2018), a mismatch between
real vs. adversarial distributions (Xie al. 2020), or a limitation in architectures Pdvpooo AdvProp helps improving performance of EfficientNets but not ResNets (Xie et al. (2020).
Most previous work aimed at understanding the behaviors of R classifiers as a function but little is known about the internal characteristics of R networks and, furthermore, their connections to the shape bias and generalization performance. Here, we ask:
Q3: How did adversarial training change the hidden neural representations to make classifiers more shupe-biud biused and udversarially robust?
In this paper, we harness the common benchmarks in MI interpretability and neuroscience conflict (Geirhos et al.) 2019). NetDissect (Bau al. (0177) and ImageNet-C- answer the three
questions above via systematic study across three different convolutional architectures AlexNet
Krizhevsky et al. 2012), GoogLeNet (Szegedy et al. 2015), and ResNet-50 (He et al.) 2016) trained to pertorm image classification on the large-scale ImageNet dataset (Russakovsky et al. 2015). Our main findings include
R classifiers trained on ImageNet prefer shapes over textures ~67% of the time (Sec. a stark contrast to the S classifiers, which use shapes at only ~25%.
2. Consistent with the strong shape bias, R classifiers interestingly outperform S counterparts texture-less, distorted images (stylized and silhouetted images) (Sec. 3.2.2
3. Adversarial training makes R networks more robust by (1) blocking pixel-wise input noise via smooth filters (Sec. 3.3.1): (2) narrowing the input range that highly activates neurons to simpler patterns, effectively reducing the space of adversarial inputs (Sec. 3.3.2).
4. Units that detect texture patterns (according to NetDissect) are not only useful to texture- based recognition as expected but can be also highly useful to shape-based recognition (Sec. B.4p. By aligning NetDissect and cue-conflict frameworks, we found that hidden neurons R networks are surprisingly neither strongly shape-biased nor texture-biased, but instead generalists that detect low-level features (Sec. 3.4).
2 NETWORKS AND DATASETS
Networks To understand the effects of adversarial training across a wide range of architectures, we compare each pair of S and R models while keeping their network architectures constant. That is, we conduct all experiments on two groups of classifiers: (a) standard AlexNet, GoogLeNet, & ResNet-50 (hereafter, ResNet) models pre-trained on the 1000-class 2012 ImageNet dataset; and (b) three adversarially-robus. counterparts i.e. AlexNet-R, GeogLeNet-R, & ResNet-R which were trained via adversarial training (see below) (Madry et al. 2018).
Training A standard classifier with parameters 0 was trained to minimize the cross-entropy loss T, over pairs of (training example x. ground-truth label y) drawn from the ImageNet training set D:
arg min ( (a.)) I(0, x. v)
On the other hand, we trained each R classifier via Madry et al. (2018) adversarial training frame- work where each real example is changed by perturbation A:
arg min E. (E max L(0, + A v)
(2)
0
LAEP
1 'lll code and data will be available on github upon publication.
2

---

Under review as a conference paper at ICLR 2021
where P is the perturbation range (Madry et al. 2018). here, within Lg norm.
Hyperparameters The S models were downloaded from Py Torch model zoo (Py 2019). We trained all R models using the robustness library (Engstrom al. 2019), using the same hyperpa- rameters in Engstrom et al. (020)); Santurkar et al. (2019); Bansal et al. (2020). That is, adverarial examples were generated using Projected Gradient Descent (PGD) (aadr et al. 2018) with L2 norm constraint с of 3, a step size of 0.5, and 7 PGD-attack steps. models were trained using an SGD optimizer for 90 epochs with momentum of 0.9, initial learning rate of 0.1 (which is reduced 10 times every 30 epochs), a weight decay of 10 10-4, and batch size of 256 on 4 Tesla- V100 GPU's.
Compared to the standard counterparts, R models have substantially higher adversarial accuracy but lower ImageNet validation-set accuracy (Table D To compute adversarial accuracy, we perturbed validation-set images with the same PGD attack settings as used in training.
Table I: Top-1 accuracy (%) on 50K-image ImageNet validation-set and PGD adversarial examples.
AlexNet
AlexNet-R
GoogLeNet
GoogLeNett
ResNet
ResNet-R
ImageNet
56.52
39.83
69.78
43.57
76.13
57.90
Adversarial
0.18
22.27
0.08
31.23
0.35
36.11
Correctly-labeled image subsets: ImageNet-CLL Following Bansal et al. (2020). to compare the behaviors of two networks of identical architectures the same inputs, we tested them on the largest ImageNet validation subset (hereafter, ImageNet-CI.) where both models have 100% accuracy. The sizes of the three subsets for three urchtectures-AlexNee GoogLeNet, and ResNet-are respec- tively: 17,693, 24,581, and 27,343. On modified ImageNet images (c.g. ImageNet-C), we only tested each pair of networks on the modified images whose original versions exist in ImageNet-CL. That is, we wish to gain deeper insights into how networks behave on correctly-classife images, and then how their behaviors change when some input feature (e.g. textures or shapes) modified.
EXPERIMENT AND RESULTS
3.1 DO IMAGENET ADVERSARIALLY ROBUST NETWORKS PREFER SHAPES OR TEXTURES?
is important to know which type of feature classifier uses when making decisions. While standard ImageNet networks often carry a strong texture bias (Geirhos et al., 2019). it is unknown whether their addersarially-robuss counterparts would be heavily texture- shape-biased. Here, we test this hypothesis by comparing S and R models on the well-known cue-conflict dataset (Geirhos et al.) 2019). That is, we feed "stylized" images provided by |Geirhos al. (2019) that contain contradict- ing texture and shape cues (e.g. elephant skin on a cat silhouette) and count the times a model uses textures or shapes (i.e. outputting elephant or cat) when makes a correct prediction.
Experiment Our procedure follows Geirhos et al. (2019). First, we excluded 80 images that do not have conflicting cues (e.g. cat textures on cat shapes) from their 1,280-image dataset. Each texture shape cue belongs to one of 16 MS COCO (Caesar et al. 2018) coarse labels (e.g. cat or elephant). Second, we ran the networks on these images and converted their 1000-class probability vector outputs into 16-class probability vectors by taking the average over the probabilities of the fine-grained classes that are under the same COCO label. Third, we took only the images that each network correctly labels (i.e. into the texture or shape class), which ranges from 669 to 877 images (out of 1,200) for networks and computed the texture and shape accuracies over 16 classes.
Results On average, over three architectures, R classifiers rely shapes > 67.08% of the time ie. ~2.7 x higher than 24.56% of the S models (Table 2). In other words, by replacing the real examples with adversarial examples, adversarial training causes the heavy texture bias of ImageNet classifiers (Geirhos et al. 2019 Brendel & Bethge 2019) to drop substantially (~2.7xx).
3.2 DO ROBUST NETWORKS GENERALIZE TO UNSEEN TYPES OF DISTORTED IMAGES?
We have found that changing from standard training to adversarial training changes ImageNet clas- sifiers entirely from texture-biased into shape-biased (Sec. 3.1) Furthermore, Geirhos et al. (019)
3

---

Under review as a conference paper at ICLR 2021
Table While standard classifiers rely heavily on textures, R classifiers rely heavily on shapes. The top-1 accuracy scores (%) are computed on the cue-conflict dataset by Geirhos et al. (019).
AlexNet
AlexNet-R
GoogLeNet
GoogLeNet-R
ResNet
ResNet-R
Texture
73.61
34.67
74.91
34.43
77.79
29.63
Shape
26.39
65.32
25.08
65.56
22.20
70.36
found that some training regimes that encourage classifiers to focus more shape improve their performance on unseen image distortions. Therefore, interesting to test whether models a type of shape-biased classifiers would generalize well to any OOD image types.
We compare S and R networks on the ImageNet- dataset which was designed to test model robustness 15 common types of image corruptions (Fig. [Tp,, where several shape-biased classifiers were known to outperform S classifiers (Geirhos et al., 2019). Here, we tested each pair of S and models the ImageNer-C distorted images whose original versions correctly labeled by both (i.e. in ImageNet-CI. sets; Sec.
ImageNet-C
Results R models show no generalization boost on ImageNet-C i.c. they performed on-par or worse than the S counterparts (Table This is consistent with the findings in Table 4 in Geirhos al. 2019) that stronger shape bias does not necessarily imply better generalizability.
To further understand the generalization capability of R models, we tested them on two controlled image types where either shape or texture cues are removed from the original, correctly- ImageNet images. Note that when both shape and texture cues are present e.g. in cue-conflict images, classifiers consistently prefer shape over texture i.e. shape bias. However, this bias orthogonal to the performance when only either texture or shape cues are present.
Table 3: R models often do not generalize well to common distorted images (c-d), which are outside their training distribution (b), but interestingly outperform models on texture-less images (e-f). Here, we report top-1 accuracy scores (in %) on the transformed images whose original real versions were correctly labeled (a) by both S and R models. "ImageNet-C"" column (c) shows the mean accuracy scores over all 15 distortion types (Hendrycks & Dietterich] (019). "Scrambled" column (d) shows the mean accuracy scores over three patch-scramble types (dctails in Fig.Al AT
(a)
(b)
Shape-less
Texture-less
Network
Real
Adv.
(c)
(d)
(e)
(f)
g
ImageNet-C
Scrambled
Stylized
Contour
Silhouette
AlexNet
100
0.18
21.10
34.59
6.31
20.08
7.72
AlexNet-R
100
22.27
20.19
16.92
9.11
35.25
9.30
GoogLe .eNet
100
0.08
37.90
49.74
13.74
43.48
10.17
GeogLeNet-R
100
31.23
26.23
31.15
12.54
44.55
24.12
ResNet
100
0.35
39.27
58.04
10.68
16.96
3.95
ResNet-R
100
36.11
31.13
34.46
15.62
53.89
22.30
(a) Real
(b) Adversarial (c) ImageNet-C (d) Scrambled
(e) Stylized
B&W
(g) Silhouette
Figure 1: Example distorted images (b-g). We show an example Gaussian noise-added image (c)
of all ImageNet-C types. See Fig. A4 for more examples.

---

Under review as a conference paper at ICLR 2021
3.2.1 PERFORMANCE ON SHAPE-LESS, TEXTURE-PRESEEE PRESERVING IMAGES
We created shape-less images by dividing each ImageNet-CL image into grid of px p even patches where p E {2, 4, 8 and re-combining them randomly into a new "scrambled" version (Fig. (t) On average, over three grid types, we observed a larger accuracy drop R models compared to models, ranging from 1.6x to 2.04x lower accuracy (Table (d). That is, R model performance drops substantially when object shapes are removed-anotie Compare predictions of ResNet vs. ResNet-R for scrambled images in Fig. A6 Remarkably, ResNet accuracy only drops from 100% to 94.77% on the 2 x scrambled images (Fig. AT).
evidence for their reliance on shapes.
3.2.2 PERFORMANCE ON TEXTURE-LESS, SHAPE- PRESERVING IMAGES
Following Geirhos et al. 2019), we tested R models three types of texture-less images where the texture is increasingly removed: (1) stylized ImageNet images where textures are randomly modified; (2) binary, black-and- white, i.e. B&W, images (Fig. Th: and (3) silhouette images where the texture information is completely removed (Fig. Te. g).
took all IeageNet-CI. images (Sec.2) ( and changed their textures via a stylization procedure in [Geirhos] al. 2019), which harnesses the style transfer technique (Gatys al. 2016) apply random style to each ImageNet "content" image.
Stylized ImageNet To construct set of stylized ImageNet images (see Fig. [e),
B&W images For all ImageNettCL images, we used the same process described in Geirhos et al. (219 to generate silhouettes, but we did not manually select and modify the images. We used the ImageMagick command-lin line tool (ImageMagick) to binarize ImageNet images into B&W images via the following steps:
1. convert image.jpeg image.bmp
2. potrace - -svg image.bmp -o image.svg
3. rsvg-convert image.svg > image.jpeg
Silhouette For all ImageNet-CC CL images, obtained their segmentation maps via PyTorch DeepLab-v2 model (Chen et al.) 2017) pre-trained on MS COCO-Stuff. We used the ImageNet-CL images that belong airplane, etc.). When evaluating classifiers, an image is considered correctly labeled if its ImageNet predicted label is a subclass of the correct class among the 16 COCO classes (Fig. Tt: mapping sandpiper + bird).
of 16 COCO coarse classes in Geirhos et al. (2019) (e.g. bird, bicycle,
Results On all three texture-less sets, R models consistently outperformed their counterparts (Table 3 g) remarkable generalization capability, especially on B&W and silhouette images where all texture information is mostly removed.
3.3 How DOES ADVERSARIAL TRAINING MAKE NETWORKS MORE ROBUST?
What internal mechanisms help R networks become more robust? Here, we shed light into this question by analyzing networks at the weight (Sec. 3 and neuron (Sec. 3.3.2) levels.
3.3. WEIGHT LEVEL: SMOOTH FILTERS TO BLOCK PIXEL-WISE NOISE
Consistent with Yin et al. (2019): Gilmer et al. (2019), we observed that AlexNet-R substantially outperforms AlexNet not only on adversarial examples but also several types of high-frequency image types (e.g. additive noise) in ImageNet-C (Table[A]]
Smoother filters explain this phenomenon, we visualized the weights of all convl filters [11x1x3]] in both AlexNet and AlexNet-R, RGB images. We compare each AlexNet convl filter with its nearest convl filter (via Spearman rank correlation) in AlexNet-R. Remarkably, R filters appear qualitatively much smoother than their counterparts (Fig. 2ah. The R filter bank is also less diverse e.g. edge detectors are often black-and and-white in contrast to the colorful AlexNet edges (Fig. 2b). A similar contrast was also seen for the GoogLeNet and ResNet models (Fig. A3).
We also quantify the smoothness, total variation (TV), of the filters of all 6 models (Table, 4 and found that, on average, the filters in networks are much smoother. For example, the mean TV of
5

---

Under review as a conference paper at ICLR 2021
r:0.93
r:0.88
f:0.77
r:0.71
r:0.70
r:0.67
r:0.66
AlexNet
AlexNet-R
107
(a) Standard filters (top) matching filters (bottom)
(b) 40 convl filters in AlexNet and AlexNet-R
Figure 2: Left: For each AlexNet convl filter (top row), we show the highest-correlate filter in AlexNet-R R (bottom row), their Spearman rank correlation (e.g. r: 0.93) and the Total Variation (TV) difference (e.g. 22) between the top kernel and the bottom. Here, the TV differences are all positive i.e. AlexNet filters have higher TV.. Right: convl filters of AlexNet-R are smoother and less diverse than the counterparts. Similar two plots for all convl filters are Figs. A2 A3
AlexNet-R is about 2 times smaller than AlexNet. Also, in lower layers, the filters in R classifiers are consistently 2 to 3 times smoother (Fig. A27).
Blocking pixel-wise noise We hypothesize that the smoothness of filters makes R classifiers more robust against noisy images. To test this hypothesis, we computed the total variation (TV) (Rudin et al. 1992) of the channels across 5 conv layers when feeding ImageNet-CI images and their noisy versions (Fig. De: ImageNet-C Level additive noise ~ (0, 0.08)) to S and models.
At conv1, the smoothness of R activation maps remains almost unchanged before and after noise addition (Fig. (a yellow circles are on the diagonal line). In contrast, the convl filters in standard AlexNet allow Gaussian noise to pass through, yielding larger-TV channels (Fig. 3a blue circles are mostly above the diagonal). That is, the smooth filters in R models indeed can filter out pixel-wise Gaussian noise despite that R models were not explicitly trained on this image type! Inter- estingly, Ford et al.] (2019) finding that the reverse engineering also works: training with Gaussian noise can improve ad adversarial robustness.
In higher layers, it is intuitive that the pixel-wise noise added to the input image might not necessarily cause activation maps, in both S and R networks, to be noisy because higher-layered units detect more abstract concepts. However, interestingly, we still found that R channels to have consistently less mean TV (Fig. 3b c). Our result suggests that most of the de-noising effects take place at lower layers (which contain generic features) instead of higher layers.
Table 4: Mean total variance of conv layers of 6 different models. Our observation shows that the total variance in early layer of R models are consistently lower than S model. See Fig. A27 for layer-wsie analysis.
AlexNet 110.20
AlexNet-R
GoogLeNet 36.53
GoogLeNet-R 22.79
ResNet 18.35
ResNet-R
Mean TV
63.59
19.96
3.3.2 NEURON LEVEL: ROBUST NEURONS PREFER LOWER-LEVES AND FEWER INPUTS
Here, via NetDissect framework, we wish to characterize how adversarial training changed the hid- den neurons in R networks to make R classifiers more adversarially robust.
Network Dissection (hereafter, NetDissect) is a common framework for quantifying the functions of neuron by computing the Intersection over Union (IoU) between each activation map (i.e. chan- nels) and the numan-annotall given an IoU score per human-defined concept (e.g. dog or zigzagged) indicating its accuracy in detecting images of that concept. A channel is tested for its accuracy on all 1,,400 concepts, which span across six coarse categories: object, part, scene, texture, color, and material (Bau et al. 2017) (c.f. Fig. All for example NetDissect images in texture and color concepts). Following Bau et al. (2017), we assign each channel C' a main functional label i.e. the concept that has the highest loU with. In both S and R models, we NetDissect all 1 1152, 5808, and 3904 channels from,
segmentation maps for the same input images. That is, each channel

---

Under review as a conference paper at ICLR 2021
TV of channels for Reloy im imagee
TV of channels for Notiyy Imagges
5050-
200
4000
500
3000
og
400-
2000
Channets
200
100
1000
200
50-
Alzrhec corsi
300
Aeshet canus
Alurhec comel
Acchhet & cons5
0 1000
2030 3000
4000
5000
202
400
600
5b
100
150
200
TV of channels for Clean images
TV of channels for Clean Images
TV of channels for Clean Images
(a) 64 convl channels
(b) 384 conv3 channels
(c) 256 conv5 channels
Figure 3: In each subpanel, one point shows the mean Total Variation (TV) of one channel when running clean ImageNet-CL. images and their noisy versions through AlexNet (teal) or AlexNet-R (yellow). R channels have similar TV before and after adding noise, suggesting that conv1 kernels filter out the added noise. In higher layers (conv3 and conv5), R channels are consistently more invariant to the input noise than S channels (yellow circles are clustered around the diagonal line while teal circles have higher variance). See Fig. A5 for the same scatter plot (a) for all five layers.
Charge in Percentage
AlexhAi GoogLoNst Reshet
1203
Shape Score (MexNet)
Texture Score (Alekket) e of Charnels (Alewhieti Shape Score (Auottu:)))
Stappe/Tocture
100
10.
Texture (Alexhet
Channels (Alextict
20-
400
200
uddor
part
mulinnl
Color
Tedore
(a) Total channel increases (%) in R models
(b) Shape & Texture score of AlexNet & AlexNet-R
Figure 4: Left: For all three architectures, the numbers of color and texture detectors in R models increase, e.g. by 117% and 7%, respectively, for AlexNet, while the number of object units decreases by 28%. See Fig. A9 for layer wise plots for detectors in other category. Right: The average Shape all channels in the entire network ("All") or in a NetDissect category ("Object", "Color", and "Texture"). While AlexNet-R has more color and texture channels (Il above these R channels are not heavily shape- texture-biased. In contrast, the corresponding channels in AlexNet are heavily texture-biased (A is almost 2 of @).
(0) and Texture (A) scores
respectively, 12, and 5 main convolutional layers (post-Rel.U) of the AlexNet, GoogLeNNtt, and ResNet-50 architectures (c.f. Sec. A for more details of layers used).
Shift to detecting more low-level features i.e. colors and textures We found consistent trend adversarial training resulted in substantially more filters that detect colors and textures (i.e. in models) in exchange for fewer object and part detectors. For example, throughout the same GoogLeNet architecture, we observed a 102% and 34% increase of color and texture detectors, respectively, the model, but 20% and a 26% fewer object and part detectors. compared the S model (c.f. Fig. 4a). After adversarial training. ~ ~ 1%, 15%, and 10% of all hidden neurons (in the tested layers) AlexNet, Googleeb eNet, and ResNet, respectively, shift their roles detecting lower- level features (i.e. textures and colors) instead of higher-level features (Fig. A12). Across three architectures, the increases in texture and color channels are often larger in higher layers. While lower-layered units often learn more generic features, higher-layered units more task-specific (Nguyen et al. 20166), hence the largest functional shifts in higher layers.
We also compare the shape-biased ResNet-R with ResNet-SIN i.e. a ResNet-50 trained exclusively on stylized images (Geirhos et al. 2019), which also has a strong shape bias of 81.37%. ? Interest- ingly, similar to ResNet-R, ResNet-SIN also have more low-level feature detectors (colors and tex- tures) and fewer high-level feature detectors (objects and parts) than the vanilla ResNet (Fig. A28).
Shift to detecting simpler objects Analyzing the concepts in the object category where ob- served largest changes in channel count, we found evidence that neurons change from detecting
model A in http:///w./////ttttt.c/////////r shape/

---

Under review as a conference paper at ICLR 2021
complex simpler objects. That is, for each NetDissect concept, computed the difference in the numbers of channels between the and model. In the same object category, AlexNet-R model has substantially fewer channels detecting complex concepts e.g. -30 dog. - 13 cat. and -11 person detectors (Fig. [88)) rightmost columns), compared to the standard network. In contrast, the R model has more channels detecting simpler concepts, e.g. +-0 sky and +12 ceiling channels (Fig. A8b leftmost columns). The top-49 images that highest-activated R units across five conv layers also show their strong preference for simpler backgrounds and objects (Figs. (15
Shift to detecting fewer unique concepts The previous sections have revealed that neurons in models often prefer images that are pixel-wise smoother (Sec. [33.1] and of lower-level features (Sec. 3.3.2), compared to S neurons. Another important property of the complexity of the function computed at each neuron is the diversity of types of imputs detected by the neuron (Nguyen et al. 2016b 2019). Here, we compare the diversity score of NetDissect concepts detected by units in and R networks. For each channel C, we calculated a diversity score i.e. the number of unique concepts that C' detects with an loU score 2 0.01.
Interestingly, on average, an unit fires for 1.16 times fewer unique concepts than an unit (22.43 vs. 26.07; c.f. Fig. ATOa). Similar trends were observed in ResNet (Fig. A1Ob). Qualitatively com- paring the highest-activation training-sel images by the highest-loU channels in both networks, for the same most-frequent concepts (e.g. striped), often confirms a striking difference: R units prefer a less diverse set of inputs (Fig. A12). As R hidden units fire for fewer concepts, i.c. significantly fewer inputs, the space for adversarial inputs to cause R models to misbehave is strictly smaller.
3.4 WHICH NEURONS ARE IMPORTANT FOR SHAPE- OR TEXTURE- BASED RECOGNITION?
To understand how the changes in R hidden neurons (Sec. 3.3) relate to the shape bias of R classifiers (Sec. [.I], here, we zero out every channel, one at a time, in S and R networks and measure the performance drop in recognizing shape and texture from cue-conflict images.
dlec bayce1
cleck-cai
clsck--cent
clock apllaan
Sccld
desd srcysea
docid-intt2
(a) Shope
LAUCE
chal
Tlpphart
washons
Inste7 Bris
chal
krtebi Sched sycled
boxt doddd
(b) Testure
Figure 5: Left: Top-25 highest-activation images of the AlexNet unit conv419, which has a NetDis- sect label of spiralled under texture category. The unit prefer circular patterns e.g. car wheels and clock. Right: Example cue-conflict images originally labeled as shape (top) or texture (bottom) but that were given a different label after the unit is ablated. Qualitatively, the unit helps AlexNet detect clocks and cars using shapes (top row) and reddish pink cars, birds, chairs, and bicycles using textures (bottom row). The unit has Shape and Texture scores of 18 and 22. "clock2-bice" --iccclll the image has the shape of clock and texture of bicycle. See Fig. A20 for full version.
Shape & Texture scores For each channel, we computed a Shape score i.e. the number of images originally correctly labeled into the shape class by the network but that, after the ablation, are labeled differently (examples in Fig bb).). Similarly, we computed Texture score per channel. The Shape and Texture scores quantify the importance of a channel in classification using shapes textures.
First, we found that the channels labeled texture by NetDissect are not only important texture- but also shape- based recognition. That is, average, zero-ing out these channels caused non-zero Texture and Shape scores (Fig. 4b] Texture O and A are above 0). See Fig, Fig-5lfor for an example of texture channels with high Shape and Texture scores This result sheds light into the fact that R networks consistently have more texture units (Fig. 4a) but are shape-biased (Sec. 3.1).
Similar visualizations of some other neurons from both S and networks in Appendix Fig. A21
8

---

Under review as a conference paper at ICLR 2021
Second, the texture units are, expected, highly texture- biased in AlexNet (Fig. 4b Texture; á is almost 2x of O). However, surprisingly, those texture units in AlexNet-R are neither strongly shape-biased nor texture-biased (Fig. 4bj Texture @ 2 A). That is, across all three groups of the object, color, and texture, R neurons appear mostly to be generalist, low-level feature detectors. This generalist property might be reason for why R networks than S networks (Salman et al. 2020).
effective in transfer learning
Finally, the contrast above between the texture bias of S and R channels (Fig. 4b) reminds researchers that the single NetDissect label assigned to each neuron is not describing a full picture of what the neuron does and how it helps in downstream tasks. To the best of our knowledge, this is the first work to align the NetDissect and cue-conflict frameworks to study how individual neurons contribute to the generalizability and shape bias of the entire network.
DISCUSSION AND RELATED WORK
Deep neural networks tend to prioritize learning simple patterns that are common across the training set (Arpit al. 2017). Furthermore, deep ReLU networks often prefer learning simple functions VVale-Perez et al. 2019 De Palma et al. 2019). specifically low-frequency functions (Rahaman al. 2019). which are more robust to random parameter perturbations. Along this direction, here, we have shown that R networks (1) have smoother weights (Sec. B.3.I. (2) prefer even simpler and fewer inputs (Sec. (.3.2) than standard deep networks -ie. networks represent even simpler functions. Such simplicity biases are consistent with the fact that gradient images of R networks are much smoother (Tsipras al. 2019) and that R classifiers act as a strong image prior for image synthesis (Santurkar et al. 2019).
Each R neuron computing more restricted function than an S neuron (Sec. 3.3.2 implies that
models would require more neurons mimic complex S network. This is consistent with recent findings that adversarial training requires a larger model capacity (Xie & Yuille 2020).
While AdvProp did not yet show benefits on ResNet (Xie et al. 2020), it might be interesting future work to find out whether EfficientNets trained via AdvProp also have shape and simplicity biases. Furthermore, simplicity biases may be incorporated regularizers into future training algorithms to improve model robustness. For example, encouraging filters to be smoother might improve ro- bustness high-frequency noise. Also aligned with findings, |Rzsa & Boult (2019) found that explicitly narrowing down the non-zero input regions of ReLUs can improve adversarial robustness.
We found that R networks heavily rely on shape cues in contrast to S networks. One may fuse an S network and R network (two channels, one uses texture and one uses shape) into a single, more robust, interpretable ML model. That such model may (1) have better generalization on OOD data than S or R network alone and (2) enable an explanation to users on what features a network uses to label a given image.
Our study on how individual hidden neurons contribute to the R network shape preference (Sec. 3.4 revealed that texture-detector units are equally important to the texture-based and shape-based recog- nition. This is in contrast to a common hypothesis that texture detectors should be exclusively only useful to texture-biased recognition. Our surprising finding suggests that the categories of stimuli in the well-known Network Dissection (Bau et al. 2017) need to be re-labeled and also extended with low-frequency patterns e.g. single lines or silhouettes in order to more accurately quantify hidden representations.
5 CONCLUSION
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 6: 6:14410-14430, 2018.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer

---

Under review as a conference paper at ICLR 2021
look memorization in deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 242. JMLR. org. 2017.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, pp. 274-283, 2018. URL
bttp::/proceecings /tt..////////////aaaaaaa........
Naman Bansal, Chirag Agarwal, and Anh Nguyen. Sam: The sensitivity of attribution methods to hyperparameters. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2020.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 6541-6541 2017.
Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-features models works surprisingly well on imagenet. In International Conference Learning Representations, 2019.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1209- 1218, 2018.
Liang-Chie Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Giacomo De Palma, Bobak Kiani, and Seth Lloyd. Random deep neural networks are biased towards simple functions. In Advances in Neural Information Processing Systems, pp. 1962-1974, 2019.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, and Dimitris Tsipras. Robustness (python li- brary), 2019. URL http://hwhwubooooo/aaaaaaaaaaaap
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations, 2020. URL https://
openreview. wttp://wwwwee/foruruuro http:///ooooummdddoyooyo.tt
Nicolas Ford, Justin Gilmer, and Ekin Cubuk. Adversarial examples are natural consequence of test error in noise, 2019. URL htt.::tttt//t httpss:///pperrreeee .ne.//ee/ee fod
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and patter recog- nition, pp. 2414-2423, 2016.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In Advances in Neural Informa- fion Processing Systems, pp. 7538-7550, 2018
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-traime trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019. URL /wRLLtttpss//ppeeeeeew..ooo foormm htteee//foommmddd///deeteea 1d-Byghh-00KXX
Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples are a natural consequence of test error in noise. In International Conference on Machine Learning, pp. 2280- 2289, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2021
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conference on Learning Representations, 2019. URL
/ttpp:://ceeneevee..net/ htt.:/neee/.ouumm.dd--tttttt--..
ImageMagick. Imagemagick. URL httpp:://mmaagggagiiiii/oo////
Daniel Kang, Yi Sun, Tom Brown, Dan Hendrycks, and Jacob Steinhardt. Transfer of adversarial robustness between perturbation types. arXiv preprint arXiv: (a0:::905.00344 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu, To- wards deep learning models resistant to adversarial attacks, In International Conference on Learn- ing Representations, 2018. URL https: /ttp::::///peeevieee net/ Formmoiddiiaai 2
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks easily fooled: High fidence predictions for unrecognizable images. In Computer Vision and Pattern Recognition (CVPR), 2015.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in neural information processing systems, pp. 3387-3395, 2016a.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. In Visualization for Deep Learning Workshop, ICML conference, 2016b.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding neural networks via feature visualiza- tion: A survey. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp- 35-76.56 Springer, 2019.
Nicolas Papernot. Patrick McDaniel, Ian Goodfellow. Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM Asia conference computer and communications security, pp. 506-519, 2017.
PyTorch. torchvision.models pytorch master documentation. https: : httpp//ytttoorrrre bbcss/stable/tooccviii.oo wwwpe//ooes....ee html] 2019. (Accessed on 09/21/2019).
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Dräxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron C. Courville. On the spectral bias of neural networks. In ICML, pp. 5301-5310, 2019.
Andras Rozsa and Terrance E Boult. Improved adversarial robustness by reducing open space risk via tent activations. arXiv preprint 100.00390 2019.
Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-285 1992.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision. 115(3):211-25) 2015.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33, 2020.
Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. In Advances in Neural Information Processing Systems, pp. 1260-1271 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. URL htt.tttt:///aiiii htttp://aaxii..oor/a/////113111
wrg/abs /1312 61..122..999 6199

---

Under review as a conference paper at ICLR 2021
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representa- tions, 2019. URL /ttp://ee////e net/
Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes be- cause the parameter-function map is biased towards simple functions. In International Confer- ence on Leurning Representations, 2019. URL h Kye4g3AqFm
net / forum?
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In International
Conference on Learning Reprexentations, 2020.
Hd=HyXJCCEFDS
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 819-828, 2020.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. fourier perspective model robustness in computer vision. Advances in Neural Information Processing Systems, 04113276-12288 2019.
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural net- works. In International Conference in Machine Learning, 2019.
12

---

Under review as a conference paper at ICLR 2021
A CONVOLUTIONAL LAYERS USED IN NETWORK DISSECTION ANALYSIS
For both standard and robust models, we ran NetDissect on convolutional layers in AlexNet (Krizhevsky al., 2012), 12 in GoogLeNet (Szegedy al. 2015), and 5 in ResNet-50 architec- tures (He et al. 2016). For each layer, we use after-ReLU activations (if ReLU exists).
AlexNet
layers: convl, conv2, conv3, conv4, conv5. Refer to these names in Krizhevsky et
(0122)
GoogLeNet layers: convl, conv2, conv3, inception3a, inception3b, inception4a, inception4b, inception4c, inception4d, inception4e, inception5a, inception5b
Refer to these names in PyTorch code htt.://ww...//ao///to.oooooooooa has.er//ecccaaaaaaa////ooo.///r2
ResNet-50 layers: conv1, layerl, laye3,, layer4
Refer to these names in httpa//wwc..aaaaaa/tt/tototoaaaa
code
Table A1; Top-1 accuracy of 6 models (in %) on all 15 types of image corruptions in ImageNet-C
Hendrycks & Dietterich, 2019). On average over all 15 distortion types, R models underperform their standard counterparts.
AlexNet
AlexNet-R GoogleeNet Googl ResNet ResNet-R
Gaussian
11.36
21.98
33.28
18.71
29.03
24.53
Noise
Shot
10.55
21.35
31.01
17.86
26.97
23.92
Impulse
7.74
19.68
24.54
15.30
23.55
21.07
Defocus
18.01
15.59
28.42
20.72
38.40
26.36
Glass
17.37
17.91
23.91
29.02
26.78
34.29
Blur
Motion
21.40
21.45
31.14
28.29
38.61
33.15
7oom
20.16
21.60
25.57
28.98
35.73
33.83
Snow
13.32
12.25
32.66
21.36
33.19
25.83
Frost
17.34
11.00
36.80
20.31
39.08
27.83
Weather
Fog
18.07
1.83
42.80
3.48
46.17
5.65
Brightness
43.54
27.71
64.46
42.96
68.32
49.71
Contrast
14.68
3.28
43.66
5.90
38.86
8.78
Elastic
35.39
32.29
42.79
41.98
46.16
44.94
Digital
Pixelate
28.22
36.33
54.86
48.11
44.49
52.62
JPEG
39.35
38.65
52.57
50.44
53.80
54.37
mean Accuracy
21.10
20.19
37.90
26.23
39.27
31.13
13

---

Under review as a conference paper at ICLR 2021
1x1
2x2
4x4
8x8
100
100
100
100
100
100
91.18
94.77
33.35
56.76
68.31
61.75
Accuracy %)
50
36.03
30.09
25.96
22.31
15.98
11.02
5,99
4.70
6.31
4.37
4.06
0
AlexNet
AlexNet-R
GoogLeNet
GoogLeNot-R
ResNet
ResNet-R
Figure A1: Standard models substantially outperform R models when tested on scrambled images due to their capability of recognizing images based on textures. See Fig, (6g. examples scram- bled images and their top-5 predictions from ResNet-R and ResNet (which achieves remarkable accuracy of 94.77%).
Here, report top-1 accuracy scores (in %) on the scrambled images whose original versions were correctly labeled by both standard and classifiers (hence, the 100% for x 1 blue bars).
r:0.99
f:0.77
r:0.61
1:0
r:0.28
1:0.22
r:0 26 1:0.17
1.0.12
n.0.31
n:0.18
r:0 1:0.27
n:0.27
n:0.67
1:0.30
r.0.99
r0.62
cO.14
r0.33
m0.00
m0.25
0:
m0.29
r:0.40
r0.34
m0.37
r:0 54
n0.30 30
mo045
n: 30
n:0.42
113
42
164
82
183
76
151
-33
15
117
167
19
93
46
-27
93
110
r.336
r:0.45
r:0.61
r:
r:0.56
0:
r:
r:011
r:0.33
r:0.9
r:0.42
r:
49
r:0 34
r:0.35
r:0.66
r:0.28
-24
163
139
128
128
91
Figure A2: convl filters of AlexNet-R are smoother than the filters in standard AlexNet. In each column, show an AlexNet filter convl filter and their nearest filter (bottom) from the AlexNet-R. Above each pair filters are their Spearman rank correlation score (e.g. 0.36) and their total variation (TV) difference (i.e. smoothness differences). Standard AlexNet filters are mostly noisier than their nearest filter (i.c. positive TV differences).
14

---

Under review as a conference paper at ICLR 2021
AlexNet
11x11x3
AlexNet-R
GoogLeNet
7x7x3
GoogLeNet-R
ResNet
7x7x3
ResNet-R
Figure A3: All 64 convl filters of each standard network (left) and counterpart (right). The filters of R models (right) are smoother and less diverse compared to those in standard models (left). Especially, the edge filters of standard networks are noisier and often contain multiple colors in them.
15

---

Under review as a conference paper at ICLR 2021
(a) Real
(b) Scrambled
(c) Stylized
(d) Contour
(c) Silhouette
PST AuM
OTU
Figure A4: Applying different transformation that remove shape/texture real images. We ran- domly show an example of 7 out of 16 COCO coarser classes. See Table for classification ac- curacy scores different images distortion dataset in 1000 classes(Except for Silhouette). *Note: Silhouette are validate 16 COCO coarse classes.
16

---

Under review as a conference paper at ICLR 2021
TV channels for Noisy Images
TV of channels for Noisy Images
1750
AlexNet conv2 AlexNet-R conv2
5000
4000
1250
3000
1000
750
2000
500
1000
AlexNet convl
AlexNet-R conv1
0
1000
2000
3000
4000
5000
500
1000
1500
TV of channels for Clean Images
TV of channels for Clean Images
(a) convl
(b) conv2
TV of channels for Noisy Images
TV of channels for Noisy Images
700
AlexNet conv3
400-
AlexNet conv4
AlexNet-R conv3
AlexNet-R conv4
500
300
400
200
300
200
100
100
0-
200
400
600
100
200
300
400
TV of channels for Clean Images
TV of channels for Clean Images
(c) conv3
(d) conv4
of channels for Noisy Images
200
150
100
50
AlexNet conv5
AlexNet-R conv5
50
100
150
200
TV of channels for Clean Images
(e) conv5
Figure A5: Each point shows the Total Variation (TV) of the activation maps on clean and noisy images for AlexNet AlexNet-R channel. We observe striking difference in convl: The smoothness R channels remains unchanged before and after noise addition, explaining their supe- rior performance in classifying noisy images. While the channel smoothness differences (between two networks) are gradually smaller in higher layers, we still observe R channels are consistently smoother.
17

---

Under review as a conference paper at ICLR 2021
1x1
2
4 x
8
1x1
2
4
8
ResNet
pool
pool table
pool table
pool
ball
Chvrra
Oone Sheater
bac
panol sharpenar
beaker
endessved
extaserr ckipg
jana duapanee
Tgreagr
pakket
mertmbe
stage
haveyceet
syrlgg
pendl bet
Bane Ald
lelly
ResNet-R
pool
poot
buttle
de
inteasert
eaary
control
libaar
Tri engite
sabacce shop
ocerts
nubber aroser
een Ald
crpamword puntte
grand plane
luncher svag
setectionery
Summnn
packet
eeatt
stere
sane
saket
ResNet
Chibuahuu
Chihuahua
Chilwahuu
Chibuaaua
Safe
safe
safe
safe
Canunnas
auth
sans
LaFt
saFs
terteer
tenter
banon grayhauss
showeer
scare
scate
mevistuee
yan piracher
eesspet
sewving reaciine
photocoplerr
ResNer-R
Chihuahus
Chihuahus
Chiluahus
Chibushhu
safe
safe
Safe
safe
Svoves
Suht
Whytocopler
Shetocoster
Nlatees
eerfectinery
Buttal
ResNet
bowl
bowl
bowl
bout
Indigo bunting
Indigo bunting Indigo bunting
Indigo bunnting
gtp
dep
galliiasle
Carapean galliinsl by
Rgaan paczle
pepresto
bdiag locall
murnapai gadlliste
mnrresse
dier the
verturboard
ResNet-R
inclge
indige
indige
Step
Intiga burring
Intigo buiniigg
pazzle
Ioe eater
cater
Merbeebord
sefety dfe croamoord puzdle
Windsor D4
anvelage
house Sack
Gratic Eight Sattic iige
Figure A6: ResNet-R, on average across the three patch sizes, underperforms the standard ResNet model. Surprisingly, observe that ResNet correctly classifies the image to their ground truth class even when the image is randomly shuffled into 16 patches, e.g., ResNet classifies the 4 x case of rule, safe with 100% confidence. The results are consistent with the strong texture bias of ResNet and shape bias of ResNet-R (described in Sec. 3.2.1
18

---

Under review as a conference paper at ICLR 2021
AlexNet V5 AlexNet-R
628
600
AlexNet AlexNet-R
Number of Channels
400
300
239
226
200
172
152
100
70
color texture object
part
scene material
GoogleNet vs GoogleNet-R
2500
GoogleNet GoogleNet-R
Number of Channels
2000
1893
1500
1517
1206
1165
1000
500
438414
187
139
0
color texture object
part
scene material
ResNet vs ResNet-R
1508
ResNet ResNet-R
1400
Number of Channels
1200
1169
1180
1000
800
705
832
600
502
400
291
127 170
131122
color texture object
part
scene material
Figure A7: For each network, we show the number of channels in each of the NetDissect categories (color, texture, etc) in Bau et al. (2017). Across all three architectures, R models consistently have more color and texture channels while substantially fewer object detectors.
19

---

Under review as a conference paper at ICLR 2021
Increase in number of channels
60
20
11
-2
-40
striped
Bardled
chequered
freciled
warleed
Intorlacend eerieredd
vprred
wnver
lacentl
linee
mesteed
parcuss
sigoaspedd cny atattne
surinsted
stathed
enwared
tumpy
Fibruu
pleaand
wriniteed
Leitted
murferatee
Hon peeyccomte bed
paivlay
pusiia-dee vivetted
aturseed
colvertion Atthertttttn
spiratted
grid
dotted
(a) Differences in texture channels between AlexNet and AlexNet-R
Increase in number of channels
40-
30
20
10
-10
20
-30
cntion
(b) Differences object channels between AlexNet and AlexNet-R
Figure A8: In each bar plot, column shows the difference in the number of channels (between AlexNet-R and AlexNet) for a given concept e.g. striped or banded. That is, yellow bars (i.e. positive numbers) show the count of channels that the model has more than the standard network in the same concept. Vice versa, teal bars represent the concepts that R models have fewer channels. The NetDissect concept names are given in the x-axis.
Top: In the texture category, the R model has a lot more simple texture patterns e.g. striped and banded (see Fig. All for example patterns these concepts).
Bottom: In the object category, AlexNet-R often prefers simpler-object detectors e.g. sky or ceiling (Fig. A8b leftmost) while the standard network has more complex-objest detectors e.g. dog and cat (Fig. A8b rightmost).
20

---

Under review as a conference paper at ICLR 2021
Number of object channels
100
AlexNet AlexNet-R
Number of colar channels
AlexNet AlexNet-RR
30-
46
20
20.
10
78
12
20.
convl
conv2
conv3
conv4
conv5
convl
conv2
conv3
(a) Number of object detectors per AlexNet layer
(b) Number of color detectors per AlexNet layer
Figure A9: In higher layers (here, conv4 and conv5), AlexNet-R have fewer object detectors but more color detector units compared to standard AlexNet. The differences between the two networks increase as we go from lower higher layers. Because both networks share identical architecture, the plots here demonstrate substantial shift in the functionality of the neurons as the result of adversarial
detecting more colors and textures and fewer objects. Similar trends were also observed between standard and R models of GoogLeNet and ResNet-50 architectures.
AlexNet
45
AlexNet-R
40
Mean diversity score
35
Mean diversity
30-
25
20-
ResNet ResNet-R
conv1
conv2
conv3
conv4
conv5
convl
layer1
layer2
layer3
layer4
(a) AlexNet layer-wise mean diversity
(b) ResNet layer-wise mean diversity
Figure A10: each plot, we show the mean diversity scores across channels in each layer. Both AlexNet-R and ResNet-R consistently have channels with lower diversity scores (i.e. detecting fewer unique concepts) than the standard counterparts.
21

---

Under review as a conference paper at ICLR 2021
conv
atriped
choquered
orange
conv2
perforated
striped
choquered
handed
sky
EA COIC
conv3
choquered
perforated
21g7893ed
srpped
handed
conv4
chequered
pertorated
spiralled
zigzageed
pane
conv5
dettedd
chaquarrd
stripad
orange
freckied
Figure A11: The NetDissect images preferred by the channels in the top-5 most important concepts in AlexNet (i.e. highest accuracy drop when zeroed out; see Sec. 3.3.2p. For each concept, we show the highest-loU channels.
22

---

Under review as a conference paper at ICLR 2021
conv5 255 strpes ped 0.18
conv5 111 strped 1..00
ABBA S
COTP 159 strped 0.09
COTT 190 strped 0.13
TONKSS
conv2 84 striped 0.07
corv2 87 striped 0.13
convt striped 1.00
striped 0.01
(a) AlexNet
(b) AlexNet-R
Figure A12: Each 7x7 grid shows the top-49 training-set images that highest activate the center unit in a channel. Each column shows five highest-loU striped concept channels, each from one AlexNet's conv layer in their original resolutions. From top bottom, AlexNet-R (b) consistently preferred striped patterns, i.c., edges (convl), vertical bars (conv2), tools, to grids and zebra (conv5). In contrast, AlexNet striped images (a) are much more diverse, including curly patterns (conv4) and dog faces (conv5).
23

---

Under review as a conference paper at ICLR 2021
conv5 56 chequered 0.10
conv5 184 chequered 0.20
conv4 33 chequered 0.18
comv4 127
chequered 0,14
THE SLEE
conv3 289
chequered | 0.17
conva 22
chequered 0.11
chequered 0.06
conv2 chequered 0.07
6 chequered 0.02
conv1 chequered 0.01
(a) AlexNet
(b) AlexNet-R
Figure A13: Same figure as Fig. A12 for chequered concept. From top to bottom, AlexNet-R (b) preferred consistently simple diagonal edges (convl), chequered patterns (conv2), crosswords, to dots and net patterns (conv5). contrast, AlexNet images (a) are much more diverse, including shelves (conv4) and gila monsters in (conv5). Remarkably, the R channels consistently prefer black- and-white patterns in conv2 and conv3 while the standard channels prefer colorful images.
24

---

Under review as a conference paper at ICLR 2021
conv5110, sky, 0.06
conv5g1, sky, 0.04
conv54s. sky, 0.02
AlexNet
conv5 sky, 0.06
conv582, sky, 0.06
conv5 157. sky, 0.03
AlexNet-R
(a) sky concept
conv5255, striped, 0.18
conv5229, striped, 0.11
conv57s, striped. 0.10
AlexNet
conv5111. striped, 0.20
conv5202- striped, 0.14
conv5g2. striped, 0.14
Tiin
AlexNet-R
(b) striped concept
Figure A14: We show the top-49 highest-activation training-set images for the top 3 channels with highest loU for the sky (a) and striped (b) concepts in the conv5 layer of AlexNet and AlexNet- R. The top-49 images for AlexNet-R nets align with the NetDissect concept labels whereas, for AlexNet, we observe diverse sets of preferred images of animals, such as, gilamonster and dog in sky (top panel) and striped concept (bottom panel) channels, respectively.
25

---

Under review as a conference paper at ICLR 2021
0050
0019
0017
0049
0042
0062
0040
0032
0063
0036
0028
0029
0013
0041
0038
0009
0020
0035
0014
0026
0010
0014
0031
0023
0060
0022
0048
0063
0060
0000
0057
0036
0023
0008
0024
0061
(a) AlexNet convl
AlexNet-R conv1
Figure A15: Each 7 x grid shows the top-49 training-set images that highest activate the center unit channel for AlexNet (a) and AlexNet-R (b). set of 20 channels were randomly sampled from the 64 channels in convl of the networks. The top-49 of AlexNet-R comprises of cleaner edge and color detector channels. a complete set 64 channels, refer to
26

---

Under review as a conference paper at ICLR 2021
LOU
WOOSH TEE
(a) AlexNet conv2
(b) AlexNet-R conv2
Figure A16: Each 7 7 grid shows the top-49 training-set images that highest activate the center unit in a channel for AlexNet (a) and AlexNet-R (b). A set 20 channels were ran- domly sampled from the 192 channels in conv2 of the networks. The top-49 of AlexNet- comprises of more just green color, edge, and chequered pattern detector channels. For complete of 192 channels, refer https:://rriee..ottttt IeNmkzYTInSBO TUT
27

---

Under review as a conference paper at ICLR 2021
(a) AlexNet conv3
(b) AlexNet-R conv3
Figure A17: Each 7 grid shows the top-49 training-set images that highest activate the center unit in a channel for AlexNet (a) and AlexNet-R (b). A set of 20 channels were randomly sampled from the 384 channels conv3 of the networks. The top-49 of AlexNet-R comprises of more simple edge and dotted pattern detector channels. For wewriveefflllo.
complete set of 384 channels, refer httpss///rrive..ooolll BuunaaaeenSso---reeEEEEEEEEEEESS
28

---

Under review as a conference paper at ICLR 2021
(a) AlexNet conv4
(b) AlexNet-R conv4
Figure A18: Each 7 x 7 grid shows the top-49 training-set images that highest activate the center unit channel for AlexNet (a) and AlexNet-R (b). A set of 20 channels were randomly sampled from the 256 channels in conv4 of the networks. The top-49 of AlexNet-R comprises of more simple texture detector channels whereas we observe face detector channels AlexNet (a). For a complete set of 256 channels, refer to //ditttsss//dii.e.eeeee TeNMkxYTInSBO T wwwgle.ooooooooooooooooooooooo.. com/ drive/folders
29

---

Under review as a conference paper at ICLR 2021
(a) AlexNet conv5
(b) AlexNet-R conv5
Figure A19: Each 7 grid shows the top-49 training- set images that highest activate the center unit in channel for AlexNet (a) and AlexNet-R (b). A set of 20 channels were randomly sampled from the 256 channels in conv5 of the networks. The top-49 AlexNet-R comprises less diverse object detector channels whereas some AlexNet channels activate for both daisy and spider. For a complete set of 256 channels refer to hta.tts://driiv..oogt gowgle...e..oooooooooooooo.. com/ com/ hesive//fodeer////
30

---

Under review as a conference paper at ICLR 2021
ME DIE RIDEES EEPEEEEEEE EIOE
Latel ispiear.
preterenneee 18/22
SAUCE
ANEA
Buik
Boysld appicet
Waph!
IM1 arpirat
1M0
Figure A20: AlexNet conv419 with Shape and Texture scores of 18 and 22, respectively. It has a NetDissect label of spiralled (IoU: 0.0568) under texture category. Although this neuron in NetDissect texture category, the misclassified images suggest that this neuron helps in both shape- and texture-based recognition. Top: Top-49 images that highest-activated this channel. Middle: Mis-classified images in shape category (18 images). Bottom: Mis-classified images in texture category (22 images).
31

---

Under review as a conference paper at ICLR 2021
egnrysenll 1oust. 0331
preramcnen Toxero (Toxto 8/T:5/13
ARprar BaO!
With pal
Aar
Expole
Figure A21: AlexNet conv55 with Shape and Texture scores of 5 and 10, respectively. It has NetDissect label of red (IoU: 0.0331) under color category. But this neuron also detects circular shapes. Top: Top-49 images that highest-activated this channel. Middle: Mis-classified images in shape category. Bottom: Mis-classified images in texture category.
32

---

Under review as a conference paper at ICLR 2021
mo_eategorggtie N0 mplisepielll Joush 1989
profercacci 8/Ta9/4
apyor 4040
artandl dock
Hrboodd Docas
Figure A22: AlexNet conv5187 with Shape and Texture scores of 9 and 4, respectively. has NetDissect label of spiralled (IoU: 0.1383) under texture category. A shape-biased neuron that detects objects of circular shapes e.g. clock and wheels. Top: Top-49 images that highest-activated this channel. Middle: Mis-classified images in shape category. Bottom: Mis-classified images in texture category.
33

---

Under review as a conference paper at ICLR 2021
MOLcat Magteggggyt Jabell Ennhwwhhe Tou 0542
pzeterence Texto 3/T:3/17
atac EtUca
acnocass
Figure A23: AlexNet conv5221 with Shape and Texture scores of 3 and 17, respectively. has NetDissect label of cobwebbed (loU: 0.0542) under texture category. This is a heavily texture- biased neuron that helps networks detect animals by their fur textures. Top: Top-49 images that highest-activated this channel. Middle: Mis-classified images in shape category. Bottom: Mis- classified images in texture category.
34

---

Under review as a conference paper at ICLR 2021
So Lutona SeniEaer
Louro 00010
prefeesenne Teextt RUTORTT
Figure A24: AlexNet-R conv4g2 with Shape and Texture scores of 8 and respectively. It has NetDissect label of white (IoU: 0.0143) under color category. Combining the top-49 images with the mis-classified shape images, this neuron seems to detect shapes of bottles. Top: Top-49 images that highest-activated this channel. Middle: Mis-classified images in shape category. Bottom: Mis-classified images texture category.
35

---

Under review as a conference paper at ICLR 2021
NDL migut yr Lox No Labe shended
protarence /Texto 8/T:17/19
dboc A60
coac saaegaan
coakliin wnd
0010 2036
CALS UT31
Figure A25: AlexNer-R conv5 with Shape and Texture scores of 17 and 19, respectively. It has NetDissect label of banded (loU: 0.0409) under texture category. This neuron has almost equal Shape and Texture scores and is useful in detecting both the shape and textures of knives and bottles the same time. Top: Top-49 images that highest-activated this channel. Middle: Mis-classified images in shape category. Bottom: Mis-classified images in texture category.
36

---

Under review as a conference paper at ICLR 2021
Bogine Label twalte
lou:b.011 prererencs Shape 8/T:4/0
NOT Mooee
Figure A26: AlexNet-R conv5136 with Shape and Texture scores of and 0, respectively. It has NetDissect label of white (loU: 0.0119) under color category. Interestingly. this neuron is heavily shape-biased. Top: Top-49 images that highest-activated this channel. Middle: Mis-classified images shape category. Bottom: Mis-classified images in texture category.
37

---

Under review as a conference paper at ICLR 2021
160
AlexNet AlexNet-R
Mean TV
100
80
60
0
conv1
conv2
conv3
conv4
conv5
(a) Mean layer-wise kerne AlexNet and AlexNet-R
GoogLeNet
50
GoogLeNet-R
Mean TV
40
30
20
10
conv1
conv2
conv3
inceptionaa
inceptionab
inception4a
inceptlon4b
Inceptiondc
inception4d
inceptionde
inceptionSa
inceptionSb
(b) Mean layer-wise kernel TV of GoogLeNet and GoogLeNNt-R
ResNet50
ResNet50-R
25
Mean TV
15
conv1
layerl
layer2
layer3
layer4
(c) Mean layer-wise kemnl TV of ResNet and ResNet-R
Figure A27: early convolutional layers, the kernels much smoother robust network. i.e. (b)& the kernel is smoother in first few conv layers. (c) bit special since the layer* is residual blocks, but we can still see that conv1 in robust network has much smoother kernel compared to its' counter part. Note: We compute the mean TV of all conv layers inside each residual(inceptinn blocks.
38

---

Under review as a conference paper at ICLR 2021
Different ResNet50
1500
Number of Channels
1400
ResNet
1200
1169
1249
1163
1180
1122
1272
ResNet-R
ResNet-SIN
1005
980
795
809
IN
718
397
291
127
122334
13112324299
calor
texture
object
material
Figure A28: NetDissect results of ResNet50 trained differently. It obvious that ResNet-R is very special compared to all other variance. An interesting trend is that shape biased model (ResNet-R, ResNet-SIN) tends to have more texture detectors and fewer object detectors, wise verser.
39


Reviews:
[{"review_id": "u5f8UWw8_sp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors show that adversarially robust versions of three popular CNN architectures trained for image classification on ImageNet rely on shape rather than on textures to perform recognition. They also show that adversarially robust networks do not outperform non-robust networks on corrupted data. Finally, they perform some analysis to determine whether intermediate features are more related to shape or texture, finding that these representations to intertwine both types of information.\n \n**Quality:** The paper is well written and the experiments are very interesting. However, the contributions may be somewhat incremental (see below).\n\n**Clarity:** The paper is written very clearly.\n\n**Significance and originality:** The paper is interesting, but the novelty is a bit limited. I am torn about this, because I do think that the experiments are interesting and well explained, and verifying results on different datasets (especially very large datasets like ImageNet) is important.  However,  given that the shape bias of adversarially robust networks was already known previously (as pointed out by the authors) and that there is no methodological contribution, I think that the contributions are somewhat incremental. \n\n**Pros:** Well written, interesting experiments.\n\n**Cons:** Incremental contribution.\n\nAfter reading the author feedback, I would like to thank the authors and I agree with them that it is critical to test hypotheses on large-scale datasets. However, I still think that the contribution is marginally below the acceptance threshold.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WnzTn8MZFmQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe submission concerns an experimental study of the behavior of networks trained with and without adversarial robustness criteria (using Madry et al., 2017). Given a set of such trained networks, a detailed look at the behavior and properties of adversarially robust networks and their non-robust counterparts is taken. This includes evaluation on the cue-conflict dataset (Geirhos et al.), and on scrambled, or texture-less (silhouette) variants of ImageNet images. Furthermore, visualizations of filter banks are analyzed and compared, as well as an analysis on the neuron level is carried out, using the NetDissect framework by Bau et al.\nInsights include that adversarially trained networks are more shape-biased (and reliant) than their counterparts which are known to be texture-biased. Furthermore, three simplicity biases are found that result in smoother filters, increased focus on low-level cues, and decreased diversity of inputs detected by neurons of adversarially trained networks.\n\nReview:\nThe authors have conducted a fairly thorough experimental study w.r.t. the level of detail considered; insights range from experiment to neuron level. The main limitations of the study are set by the types of networks considered (2012-2015 era networks only) and the types of adversarial training considered (only the work by Madry et al.). With regards to the latter, incorporation of newer work (e.g. (1), (2), or (3)) on adversarial training could have made the argument stronger, but this may simply have been subject due to scope and time limitations.\nDespite these limitations, the scope of evaluation is sufficient, and the level of supplied visualizations is exemplary. The work lacks any theoretical insight, but there is potential value in the type of conducted detailed empirical study. A large part of the results seems to agree with prior work (cf. mentions of “consistent with” and similar), such that part of the value of this study is additional confirmation of what prior work may have found or otherwise hypothesized.\nWhat I am missing a bit, however, is a discussion on what the mentioned findings may mean for future developments of adversarial training, adversarial attack design, or other mitigations of adversarial attacks. The discussion of possible consequences is limited to the last paragraph, which is meant to discuss future work.\n\nClarity of writing is generally good; however, I was not really a fan of the chosen nomenclature and the abundant definition of abbreviations which may clash with own notions of their meaning. For example, is it necessary to define “S-networks” and “R-networks”, where R-networks happen to be *s*hape-biased and S-networks *t*exture-biased? My preference would have been to simply spell out “adversarially trained” (“-adv”) vs. “not adversarially trained”. This is just a personal opinion and does not affect the rating. Similarly with the datasets; “ImageNet-C” is mentioned on page 3, but does not seem to be defined beyond “modified ImageNet images” and then shown in Figure 1.\n\nOverall, I believe the scope of experiments and evaluations goes beyond workshop-level work. But originality and significance remain limited, as mentioned above.\n\n—-\n(1) Shafahi et al., “Adversarial Training for Free!”, NeurIPS 2019.\n(2) Cohen et al., “Certified Adversarial Robustness via Randomized Smoothing”, ICML 2019.\n(3) Xie et al., “Smooth Adversarial Training”, arXiv preprint, 2006.14536", "extra_scores": {}, "extra_text": {}}, {"review_id": "HbsheXNzW4c", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors show that adversarially trained models have higher shape and simplicity biases compared to their vanilla counterparts.\nThe paper is well written and easy to follow. I think I understood most parts of the paper. The authors were explicit in stating which methods they used which is good.\nMy major concerns are:\n1.\tLacking novelty/ comparison to previous work: The authors did not compare to previous work properly. In my general comments, I suggest adding and discussing 11 additional publications. One of their major findings that adversarially trained models are invariant to additive high frequency noise has been shown and discussed in [8] which they did not cite. Other cited papers are cited in a wrong way, e.g. Geirhos et al 2018b did not perform experiments on adversarial examples; there are other examples in the General comments section below.\n2.\tNot relevant/ not surprising results: I am not sure how useful and interesting the finding is that adversarially robust models are more shape biased. It seems intuitive that given that adversarial perturbations are high-frequency, becoming invariant to them should make a model more attentive to low-frequency features such as shape.\n3.\tLacking discussion/ analysis of the results: In the first experiments section, the authors found that adversarially robust models are shape biased. With the NetDissect analysis, they showed that there are more filters in adversarially robust models compared to vanilla models that detect textures. These two results seem contradictory to me and should be discussed. If they are not contradictory, this should be explained.\n4.\tThe number of PGD steps used for the adversarial attack is 7 which is very low. I am not sure whether the results in this paper are meaningful in the sense that the models might not be adversarially robust at all if a higher number of PGD steps/random restarts were used. The authors should test bigger numbers of steps and make the step size dependent on the number of steps.\n\n\nGeneral comments:\n\nPage 2 “In contrast, there was also evidence that classifiers trained on one type of adversarial example often do not generalize well to other image types [Geirhos et al, 2018b, Nguyen et al., 2015]”. What is image types here? Both references do not provide any support for the claim made in this sentence. In Gheiros et al, the authors train on different types of image corruptions such as additive uniform noise or using high/loss pass filters. They do not show results for adversarial examples. Nguyen et al was published two years before adversarial training was proposed by Madry et al. There are other references that would be good to cite here though: [1] show that increasing robustness against adversarial attacks does not increase robustness against translations and rotations. [2] show that adversarial robustness does not transfer easily between attack classes. [3] also show that there is a performance trade-off between robustness types.\n\nPage 3: “Adversarial examples were generated using Projected Gradient Descent (PGD) (Madry et al., 2017) with an L2 norm constraint of 3, a step size of 0.5, and 7 PGD-attack steps.” A number of 7 PGD steps seems very very low. A more sensible number might be e.g. 1000 [4] or at least 200-400 steps. In [5], the authors urge to check the convergence of the attack and that white-box attacks generally converge in under 100-1000 steps. Additionally, Carlini et al [5] write that “the number of iterations necessary is generally inversely proportional to step size and proportional to distortion allowed.” -> Here, a step-size of 0.5 is chosen without providing any reasoning and the choice seems arbitrary.\n\nTable 1: On the torchvision website, the top1 accuracies for the S models are given as: AlexNet: 56.55%, GoogleNet: 69.78%, ResNet50: 76.15%. These numbers differ from what the authors show in Table 1. Working with the pretrained torchvision models myself every day, I know that a ResNet50 from torchvision will get a top1 accuracy of 76.13% (same to what is stated on the used “robustness” package repo of Engstrom et al). Where does the difference to the numbers in Table 1 come from?\n\nPage 4: “R models show no generalization boost on ImageNet-C i.e. they performed on-par or worse than the S counterparts (Table 3c). This is consistent with the findings in Table 4 & 5 in Geirhos et al. (2018a) that a stronger shape bias does not necessarily imply better generalizability.” -> This is completely opposite to what we see in Table 5 in Geirhos et al. 2018a. There, the authors attribute the higher robustness to common corruptions to the higher shape bias of their SIN model: “Again, none of these corruption types were explicitly part of the training data, reinforcing that incorporating SIN in the training regime improves model robustness in a very general way.” Due to the contradictory results, it rather seems that shape bias and corruption robustness might be more disconnected than previously thought. Also, the result that adversarially trained networks perform badly on ImageNet-C has been observed in [6] which might be good to cite here. \n\nPage 4: “However, this bias is orthogonal to the performance when only either texture or shape cues are present.” I don’t understand why it would be orthogonal? I would expect a model with a strong shape bias to care less if I remove texture cues from an image compared to a model with a strong texture bias. What exactly is new in this test compared to the cue conflict task?\n\nPage 6: “To confirm this hypothesis”, please write “to test this hypothesis”\n\nPage 6: “That is, the smooth filters in R models indeed can filter out pixel-wise Gaussian noise despite that R models were not explicitly trained on this image type!” The finding here seems very related to [7] and [8]. In [7], the authors show that adversarial training improves robustness to Gaussian noise and in [8], the authors study the Fourier properties of common corruptions and show that adversarial training improves accuracy for corruptions in the higher frequencies.\n\nPage 6: “Our result suggests that most of the de-noising effects take place at lower layers (which contain generic features) instead of higher layers.” I do not understand how this conclusion was made given the results, please explain.\n\nPage 6: “We found a consistent trend—adversarial training resulted in substantially more filters that detect colors and textures (i.e. in R models) in exchange for fewer object and part detectors”. This result seems surprising and not consistent with the finding that adversarially robust models are more shape biased. If shape bias is indeed correlated to the filter responses (which is not necessarily clear), I would expect filters of a shape-biased model to rather detect objects and parts instead of textures and colors. Please discuss this. It would also be nice to test the SIN trained model from Geirhos et al 2018a, since it has a high shape bias.\n\nPage 7: “As R hidden units fire for fewer concepts, i.e. significantly fewer inputs, the space for adversarial inputs to cause R models to misbehave is strictly smaller.” I believe this is meant to be an explanation for why adversarial training works. I don’t understand the argument well, but in [9], the authors show that adversarially robust models are better for transfer learning than their vanilla counterparts. Intuitively, a broad set of (unique) features must have been learned by the adversarially robust model for this task. I would like to see a discussion comparing these results to [9]. I also think that the analysis is not sufficient to claim that the space of adversarial examples is smaller for R models.\n\nPage 8: “For example, our preliminary results showed that encouraging S networks to have smoother kernels in early layers improves CIFAR-10 ResNet-18 network robustness to adversarial and noisy images.” This has not been shown in this paper/ reference missing.\n\nSuggestion: It would make the work stronger if the authors considered more adversarially robust models, e.g. [10], [11].\nPlease fix the references: I noticed that the authors cite the arxiv version for [Geirhos et al 2018a]. This paper has been presented at ICLR 2019 and this should be reflected in the references. Please check and fix all other references as well where this critique applies.\n\nReferences:\n\n[1] Kang et al: “Transfer of adversarial robustness between perturbation types”\n[2] Jordan et al: “Quantifying perceptual distortion of adversarial examples”\n[3] Tramèr et al: “Adversarial training and robustness for multiple perturbations”.\n[4] Tramèr et al: “On Adaptive Attacks to Adversarial Example Defenses”. \n[5] Carlini et al: On Evaluating Adversarial Robustness\n[6] Rusak et al. “A simple way to make neural networks robust against diverse image corruptions”.\n[7] Ford et al: “Adversarial Examples Are a Natural Consequence of Test Error in Noise”\n[8] Yin et al: “A Fourier Perspective on Model Robustness in Computer Vision”\n[9] Salman et al: “Do Adversarially Robust ImageNet Models Transfer Better?”\n[10] Xie et al: “Feature denoising for improving adversarial robustness”\n[11] Shafahi et al: “Adversarial training for free!”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-N7MjzT0d5l", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper takes a step further to understand the relationships between the adversarial trained CNNs (R-CNNs) and shape-based representation, and delve deeper into the R-CNNs via studying the hidden units. First, it justifies that the R-CNNs prefer shape cues based on random-shuffled, Stylized-ImageNet, and silhouette experiments. Then, it tests R-CNNs on ImageNet-C to show the less connection between the shape-biased and the robustness against common corruptions. Finally, it studies the hidden unit via qualitative tools including NetDissect.\n\nThe studied direction is important for both representation learning and robustness communities. The methodology of this paper and all experiments are technically sound. Using Network Dissection is a good point to study here. Testing on three benchmarks, and evaluate three different network architectures can help to verify the conclusions are general. The results are sufficient and would support their arguments, though more analysis would add support to the claims. Overall the paper is easy to follow and the reviewer thinks the experiments are easy to replicate.\n\n---------------\nBelow are some concerns and suggestions:\n\nThe novelty is slightly limited. Many understanding is complementary to the findings in the literature, such as [1] that adversarially trained models are shape-biased, [2] that there're nor significant correlations between shape-biased and robustness against common corruptions, and [3] that low-frequency help the generalization. Many experimental methods are the same as previous work but only perform on adversarially trained models, like the scrambled (Sec. 3.2.1) is the same as the random shuffled in [4],  test on Stylized-ImageNet[2], ImageNet-C, and etc. Also, there're no new techniques proposed. Basically, all the used techniques are from literature, like Network Dissection, and etc. \n\n\nAccording to Table 3, compare to the Shape-Less column, the difference in the Texture-less column is not that significant. The reviewer wonder if the author would perform extra experiments to verify the R models significantly more prefer shape information, such as test on edge maps. The edge map is easy to drive from the silhouette and may be more suitable to be tested here. The value of the edge map and silhouette may be set to binary or greyscale [0, 255].\n\nCurrently, only qualitative results are provided to justify the R models would contain smoother filters than the standard ones. The reviewer wonders if any quantitative criterion can be designed and be reported thereon. \n\nIn the blocking pixel-wise noise section (Page 6), the author claims that R models are more robust against Gaussian additive noise based on Figure 3. However, the ImageNet-C contains such distorted images, and S models outperform than R models. The reviewer wonders if the author can test on Gaussian additive noise distorted images and report the results to further justify the claim.\n\n---------------------------------\nTo sum up, the reviewer would vote 5 currently -- due to its limited novelty but some understanding of the representation of adversarially trained models are proposed.\n\n[1]Interpreting Adversarially Trained Convolutional Neural Network\n\n[2]Imagenet-trained Cnns are Biased towards Texture; Increasing Shape Bias improves Accuracy and Robustness\n\n[3]High-frequency Component Helps Explain the Generalization of Convolutional Neural Networks\n\n[4]Defective Convolutional Layers Learn Robust CNNs\n\n\n-------------------after rebuttal-----------------------\n\nI thank the authors for their rebuttal. Since the authors reply near the discussion phase end, I cannot ask follow-up questions. \n\nThe answers partially address my concerns and thus I would raise my score to 6. \n\nFor novelty , the author answers three points. For the first point, fuse two channels of information are not so convincing on the novelty aspects. Also, there needs an extra cost to collect process the images. For the second point, how would you formulate a regularize?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work investigates the relationship between adversarial robustness and shape bias of neural networks. Reviewers pointed out that one of the primary questions being investigated "(a) how adversarially-robust ImageNet classifiers (R classifiers) generalize to out-of-distribution examples;" has already been a primary focus of several prior works, and that many of the findings are already well established, or expected given known connections between adversarial robustness and corruption robustness. I recommend the authors rework the paper to focus more on building upon these prior results. As a possible example, the work would be strengthened if the authors compared adversarial training to other data augmentation strategies known to directly improve shape bias and corruption robustness, does adversarial training provide any unique ood robustness properties distinct from these other methods?
Decision: Reject