Shape or Texture: Understanding Discriminative Features in CNNs
Authors: Md Amirul Islam
Matthew Kowal
Patrick Esser
Sen Jia
Björn Ommer
Konstantinos G. Derpanis
Neil Bruce
Venue: ICLR 2021 Poster (2021)
Keywords: Shape
Texture
Shape Bias
Texture Bias
Shape Encoding
Mutual Information

Abstract: Contrasting the previous evidence that neurons in the later layers of a Convolutional Neural Network (CNN) respond to complex object shapes, recent studies have shown that CNNs actually exhibit a 'texture bias': given an image with both texture and shape cues (e.g., a stylized image), a CNN is biased towards predicting the category corresponding to the texture. However, these previous studies conduct experiments on the final classification output of the network, and fail to robustly evaluate the bias contained (i) in the latent representations, and (ii) on a per-pixel level. In this paper, we design a series of experiments that overcome these issues. We do this with the goal of better understanding what type of shape information contained in the network is discriminative, where shape information is encoded, as well as when the network learns about object shape during training. We show that a network learns the majority of overall shape information at the first few epochs of training and that this information is largely encoded in the last few layers of a CNN. Finally, we show that the encoding of shape does not imply the encoding of localized per-pixel semantic information. The experimental results and findings provide a more accurate understanding of the behaviour of current CNNs, thus helping to inform future design choices.

Published as a conference paper at ICLR 2021
SHAPE OR TEXTURE: UNDERSTANDING DISCRIMINATIVE FEATURES IN CNNS
Md Amirul Islam Matthew Kowal
Patrick Esser Sen Jia
Björn Ommer" Konstantinos G. Derpanis 15,6 Neil D. B. Bruce
1 DDpartment of Computer Science, Ryerson University, Canada
University of Waterloo, Canada
3 IWR, HCI, Heidelberg University, Germany
"School of Computer Science, University of Guelph, Canada "Samsung AI Centre Toronto, Canada
Vector Institute for AI, Canada
(mdami rull is am,, matthew... kowal, kostak/ryeraaoeannoo.o ca, ca, sen. wan.jllwwwttrrooo... (patrick. baatrickeessen esser, björn. heeeeeeeeee......ieeeeeeeeeeeeee de, bracen@uoguellh..ca
uni uni-teidelberg
ABSTRACT
Contrasting the previous evidence that neurons the later layers of Convolutional Neural Network (CNN) respond to complex object shapes, recent studies have shown that CNNs actually exhibit a 'texture bias': given an image with both texture and shape cues (e.g., stylized image), CNN is biased towards predicting the category corresponding to the texture. However, these previous studies conduct experiments the final classification output of the network, and fail to robustly evaluate the bias contained (i) in the latent representations, and (ii) on per-pixel level. In this paper, we design series of experiments that overcome these issues. We do this with the goal of better understanding what type of shape information contained in the network is discriminative, where shape information is encoded, as well when the network learns about object shape during training. We show that a network learns the majority of overall shape information the first few epochs of training and that this information is largely encoded in the last few layers of a CNN. Finally, we show that the encoding of shape does not imply the encoding localized per-pixel semantic information. The experimental results and findings provide a more accurate understanding of the behaviour of current CNNs, thus helping to inform future design choices.
INTRODUCTION
Convolutional neural networks (CNNs) have achieved unprecedented performance in various com- puter vision tasks, such image classification (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016), object detection (Ren et al., 2015; He et al., 2017) and semantic segmentation (Long et al., 2015; Chen et al., 2017; Islam al., 2017). Despite their black box nature, various studies have shown that early layers in CNNs activate for low-level patterns, like edges and blobs, while deeper layers activate for more complex and high-level patterns (Zeiler Fergus, 2014; Springenberg et al., 2014). The intuition is that this hierarchical learning of latent representations allows CNNs to recognize complex object shapes to correctly classify images (Kriegeskorte, 2015). 5). In contrast, recent works (Brendel & Bethge, 2019; Hermann & Lampinen, 2020) have argued that CNNs trained on ImageNet (IN) (Deng et al., 2009) classify images mainly according to their texture, rather than object shape. These conflicting results have large implications for the field of computer vision as it suggests that CNNs trained for image classification might be making decisions based largely off spurious correlations rather than full understanding of different object categories.
One example of these spurious correlations is how the Inception CNN (Szegedy et al., 2015) rec- ognizes the difference between 'Wolf' and 'Husky', based on whether there is snow in the back- ground (Tulio Ribeiro et al., 2016). Recognizing object shapes is important for the generalization to out-of-domain examples g., few-shot learning), as shape more discriminative than texture when

---

Published as a conference paper at ICLR 2021
texture-affecting phenomena arise, such as lighting, shading, weather, motion blur, or when switching be- tween synthetic and real data. In addition performance, identifying the discriminative features that CNNs use for decision making is critical for the transparency and further im- provements of computer vision mod- els. While the model may achieve good performance for a certain task, cannot communicate to the user about the reasons it made certain predic- tions. In other words, successful mod- els need to be good, and interpretable (Lipton, 2019). This is crucial for many domains where causal mechanisms should play significant role in short long-term decision making such as healthcare (e.g., what in the MRI indicates a patient has cancer?). Additionally, if researchers intend for their algorithms be deployed, there must be certain degree of trust in the decision making algorithm.
Bird
Shrep
Dog
Bird
Stylized image
GT
Shape
Semantic
Figure 1: A shape biased model (trained on Stylized Ima- geNet) makes predictions based on the object's shape, or does it? Extracting binary (3th column) and semantic (4th col.) segmentation maps with one convolutional layer read- out module shows that, while the model classifies the image level shape label correctly as a 'bird', it fails to encode the full object shape (3th col.) as well as fails to categorically assign every object pixel to the 'bird' class (4th col.).
One downside of the increasing abstraction capabilittes of deep CNNs is the lack of interpretability of the latent representations due to hidden layer activations coding semantic concepts in distributed fashion (Fong& & Vedaldi, 2018). It has therefore been difficult to precisely quantify the type of information contained in the latent representations of CNNs. Some methods have looked ways to analyze the latent representations of CNNs on et al., 2017) quantify the number of interpretable neurons for CNN by evaluating the semantic segmentation performance of an individual neuron from an upsampled latent representation. Later work (Fong& & Vedaldi, 2018) then removed the assumption that each neuron encodes a single semantic concept. These works successfully quantify the number of filters that recognize textures or specific objects in a CNN, but do not identify shape information within these representations.
level. For instance, (Bau
The most similar works to ours are those that aim to directly quantify the shape information in CNNs. For example, (Geirhos et al., 2018) analyzed the outputs of CNNs on images with conflicting shape and texture cues. By using image stylization (Huang & Belongie, 2017), they generated the Stylized ImageNet dataset (SIN), where each image has an associated shape and texture label. They then measured the 'shape bias' and 'texture bias' of CNN by calculating the percentage of images a CNN predicts as either the shape or texture label, respectively. They conclude that CNNs are 'texture biased' and make predictions mainly from texture an image. This metric has been used in subsequent work exploring shape and texture bias in CNNs (Hermann & Kornblith, 2019); however, the method only compares the output of a CNN, and fails to robustly quantify the amount of shape information contained in the latent representations (note that they refer to 'shape' as the entire 3D form of an object, including contours that are not part of the silhouette, while in our work, we define 'shape" as the 2D class-agnostic silhouette of an object). Thus, the method from (Hermann & Kornblith, 2019) cannot answer a question of focus in our paper: 'What fraction of the object's s shape is actually encoded in the latent representation?". Further, as their metric for shape relies solely on the semantic class label, it precludes them from evaluating the encoded shape and associated categorical information on per-pixel level. For instance, we show in Fig. that shape biased models (i.e., trained on stylized images) do not classify images based on the entire object shape: even though the CNN correctly classifies the image as bird, only the partial binary mask (i.e., 'shape") can be extracted from the latent representations and it cannot attribute the correct class label to the entire object region (i.e., semantic segmentation mask).
Contributions. To address these issues, we perform an empirical study on the ability of CNNs to encode shape information on neuron-to-neuroy and per-pixel level. To quantify these two aspects, we first approximate the mutual information of latent representations between pairs of semantically related images which allows us to estimate the number of dimensions in the feature space dedicated to encoding shape and texture. We then propose a simple strategy to evaluate the amount of shape information contained in the internal representations of a CNN, on a per-pixel level. The latter technique is utilized to distinguish the quality of different shape encodings, regardless of the number of neurons used in each encoding. After showing the efficacy of the two methods, we reveal a number of meaningful properties of CNNs with respect to their ability to encode shape information,
2

---

Published as a conference paper at ICLR 2021
Image pairs reflecting semantics
E
B
frozen
frozen
E
E
E
Fahape
Read-out
E(I") 2
Input images
Textures
Texture
Shape
Shape
Semantic
Figure 2: Illustration of the techniques used to quantify shape in this paper. (A) Estimating the dimensionality of semantic concepts in latent representations: We stylize each image with five textures to generate image pairs which share the semantic concepts shape (right pair) and texture (left pair). We feed these image pairs (shown is shape) to an encoder, E(-), and calculate the mutual information between the two latent representations, z" and z", estimate the dimensionality, |Zahapel- (B) We quantify the shape information encoded in a convolutional neural network by freezing the weights, and then training small read-out module (i.e., three 3x3 convolutional layers) on the latent representation to predict either a binary or semantic segmentation map.
including the following: (i) Biasing a CNN towards shape predominantly changes the number of shape encoding neurons in the last feature encoding stage. (ii) When CNN is trained on ImageNet, the majority of shape information is learned during the first few epochs. (iii) A significant amount of shape is encoded in the early layers of CNNs, which can be utilized to extract additional shape information from the network, by combining with shape encodings from deeper layers. (iv) Encoding shape and class of an object does not imply the useful encoding of localized per-pixel categorical information. All code will be released reproduce data and results.
2
Do CNNS SPEND MORE LEARNING CAPACITY ON SHAPE OR TEXTURE?
With the goal of revealing the characteristics of where, when, and how much shape information is encoded in CNNs, first aim quantify the number of dimensions which encode shape in CNN's latent representation. This analysis on the latent representations will allow us to determine where the network spends learning capacity on shape, while other methods that focus solely on the network outputs have difficulty measuring the difference in shape information between convolutional layers.
2.1 ESTIMATING SHAPE AND TEXTURE DIMENSIONALITY
Previous works (Bau et al., 2017: Esser et al., 2020) proposed various mechanisms to reveal the semantic concepts encoded in latent representations of CNNs. To quantify the amount of texture and shape information, we follow the approach of (Esser et al., 2020), where the number of neurons that represent certain semantic concept is estimated. Given pretrained CNN encoder, E(!) where z is a latent representation, we aim to estimate the dimensionality the semantic concepts shape and texture within The main idea is that the mutual information between image pairs, Ie and 1d, which are similar in a semantic concept, will be preserved in a neuron 2, only if the neuron encodes that specific semantic concept. Hence, the mutual information between the corresponding neuron pairs, 2 = ETTa) and 2i E()"), can be used to quantify the degree to which a semantic concept represented by the neuron. A simple and efficient estimate for their mutual information MI 2 can be obtained based on the correlation coefficient p;. Indeed, under the assumption that the marginal distribution of the neuron z; is Gaussian, the correlation coefficient p; provides a lower bound on the true mutual information through the following relationship which becomes tight for jointly Gaussian 2/, of (Kraskov al., 2004; Foster Grassberger, 2011).
1 log(1 p?), where P VVar(=;) Var(=?)
Cov (2ff,
MI 2
2
To quantify how well concept l is represented in terms of the number of neurons | that encode the concept, we compute score for each concept and the relative number of neurons is determined
3

---

Published as a conference paper at ICLR 2021
Table 1: Dimensionality estimation of semantic factors | for the stage-S latent representation.
Note that the total dimension of the latent representation, |z]. is 2048 for all networks, and that the remaining dimensions are allocated to the 'residual' factor. (a) ResNet50 compared with BagNets. BagNets have more neurons which encode texture than shape due their restricted receptive field. (b) Networks with varying levels of shape bias. The number of neurons which encode shape correlates with shape bias. (c) Deeper networks contain more shape encoding neurons.
Factor |
Model
Factor
|2p Texture
Factor
Shape
Texture
Training Data
Model
Shape
Shape
Texture
ResNer50
349
692
IN
349
692
ResNer50
349
692
BagNet33
284
825
SIN
536
477
ResNet101
365
667
BagNet17
278
839
(SIN+IN) +IN
376
640
5e
371
661
BagNet9
276
841
with a softmax over these scores and a baseline score. The latter is given by the number of neurons |z]. and shape and texture scores are given by the sum of their respective correlation coefficients Pi
shupe and of which are computed according to Eq. with statistics taken over image pairs that are similar in shape and texture, respectively. Note that & E {1,2] in our case, and the remaining dimensions not captured in any of the two semantic factors are allocated to the residual semantic factor, which by definition captures all other variability in the latent representation, 2.
Stylized PASCAL VOC 2012 Dataset. Our goal is to estimate the dimensionality of two semantic concepts: (i) shape and (ii) texture, and analyze pixel-wise shape information. Therefore we must generate a dataset that we can sample image pairs which share the semantic factors shape or texture, and have per-pixel object annotations. To accomplish this goal, we create the Stylized PASCAL VOC 2012 (SVOC) dataset. Similar to SIN, we use the AdaIN style transfer algorithm (Huang & Belongie, 2017) to generate stylized images from the PASCAL VOC 2012 dataset (Everingham et al., 2010) with the same settings and hyperparameters as in the original paper (Huang & Belongie, 2017). We choose five random textures from the Describable Textures Dataset (Cimpoi et al., 2014) as the styles and we stylize every PASCAL VOC image with all five of these textures. For fair comparison with models trained on ImageNet variants, we take only the images from PASCAL VOC which contain a single object. With the SVOC dataset, can now sample image pairs which are similar in texture, by using two images from different categories but stylized with the same texture (Fig. 2(A) left), or shape, by using the same image stylized with two different textures (Fig. 2(A) right).
2.2 RESULTS
We now evaluate the efficacy of the dimensionality estimation method by comparing two networks which differ significantly in their ability to encode shape information. The first is a standard ResNet50 architecture and the second is the recently proposed BagNet (Brendel & Bethge, 2019). BagNets are a modified version of ResNet50 that restrict the height and width of the effective receptive field of the CNN to be fixed maximum, i.e., either 9, 17, or 33 pixels. This patch-based construction precludes BagNets from classifying images based on extended shape cues. The results of this comparison are presented in Table ((a) where both the ResNet50 and the BagNet variants are trained on IN. Note that "Stage' refers to a residual block in a ResNet (i.e., there are five stages in ResNet) and all experiments in Table use the stage-5 features. As expected, BagNets have more neurons encoding texture than the ResNet50 and there is a clear correlation between the receptive field of the network and the amount of shape encoded. As the receptive field decreases, the number of neurons encoding texture increases even further, while the number of neurons encoding shape decreases.
We now examine whether the 'shape bias' metric (Geirhos et al., 2018) correlates with the number of shape encoding neurons. Table 1(b) compares the estimated dimensionality of ResNet50 trained on ImageNet against networks which are biased towards shape using two different training strategies: (i) training solely on SIN and (ii) training on SIN and IN simultaneously followed by fine-tuning (denoted as (SIN+IN)~ IN, which achieves the best accuracy on ImageNet top- 1 % out of the three vari- ations (Geirhos et al., 2018)). ResNet50 trained on IN has far more neurons dedicated to encoding tex- ture than shape. There is large difference when training a ResNet solely on SIN, where it has less neu- rons which encode texture than shape. When trained and fine-tuned on (SIN+IN) and IN, respectively, there is an increase in the number of neurons which encode shape compared to IN. We consider if there is any pattern in the number of neurons encoding shape or texture as the network depth increases.
IN
4

---

Published as a conference paper at ICLR 2021
As can be seen in Table I(c), networks have more shape and less texture neurons as depth increases. This may be due to the increase in learning capacity of the deeper networks, as more hierarchical representations allow for the network to learn increasingly complex shapes compared to the shallower networks. Further, deeper networks have stronger long
Table 2: Comparing dimensionality estimations for the texture factor between two methods. Both meth- ods show that more texture neurons are found in rep- resentations with smaller receptive fields.
Method
Rex152
Res101
Rex50
Bag33
Bag17
Bag9
Net. Diss.
433
481
499
592
623
537
Dim, est.
661
667
692
825
839
841
range connections due to a larger effective receptive field potentially resulting in additional shape encoding neurons. The increase in shape understanding could be one of the reasons why deeper networks achieve better performance on various tasks, e.g., image classification. Finally, we assess the consistency between the dimensionality estimation technique and network dissection (Bau et al., 2017), another method which estimates the number of neurons representing different concepts (described in Sec. A.2). Since network dissection cannot estimate shape dimensionalities, the comparison is limited to the texture dimensions shown in Table 2. Except for the case of BagNer9 and a difference in the absolute numbers of neurons (discussed in Sec. A.2), both methods agree about the correlation between texture dimensionality and the receptive field, which provides further evidence that dimensionality estimates quantify the relevance of semantic concepts faithfully.
Stage- Wise Analysis of Shape and Texture Dimensionality. We now explore where CNNs en- code shape by applying the dimensionality estimation method with the latent representations from ResNet50 stages one to five with different amounts of shape bias. Due to the different dimensions each of these stages, we present the results as the percentage of di- mensions encoding the particular semantic factor, | l/// where |z] refers to length of the latent representation. Table 3 shows that all stages of the network en- code shape with an increase in the last two stages. Further, bi- asing the models towards shape only changes the percentage of shape encoding in the final two stages. Beginning at the fourth stage, there a significant jump in the number of shape dimensions for all three models with the shape biased models having larger in- crease. At the final stage, latent representations encode even more shape, where SIN in particular has a large increase of 8.5%. This indicates that biasing a model towards shape mainly affects the last two stages of the network, suggesting that future work could focus on improving the shape bias of earlier layers. An increase in shape dimensions is inversely proportional to the amount of texture dimensions. Notably, from stage four to stage five, there is large drop in the amount of texture dimensions for all networks
Table Percentage of neurons (<xl/12)) encoding different semantic concepts, k. for different stages of ResNet50 trained for various levels of shape bias. While a moderate percentage of neurons encode shape in stages fı, ʃz. and fa, the majority of shape neurons are found in stage fs. Networks with shape bias learn additional shape information in stage fs.
IN
SIN
(SIN+IN)
Stage
Factor
2 ||
Factor
2 /|2
Factor 22 /
Shape
Texture
Shape
Texture
Shape
Texture
f1 12.5%
42.2%
12.5%
42.2%
12.5%
42.2%
f2 14.1%
40.2%
14.1%
40.6%
14.1%
40.6%
fa 14.6%
39.5%
14.8%
39.5%
14.8%
39.5%
fa 15.3%
37.9%
17.7%
34.7%
17.7%
34.8%
fs 17.0%
33.8%
26.2%
23.3%
18.4%
31.2%
When Does Shape Become Relevant During Training? To answer the question 'When do models learn to encode shape and texture during training?", we capture the changes of shape and texture occurring over the course of training classifier on ImageNet (IN) and Stylized ImageNet (SIN). We obtain 18 different instances of a ResNet50 model during training on IN and SIN, each representing a checkpoint between epochs 0 and 90 (equally distributed). For each checkpoint, we measure the dimensionality of shape and texture semantic factors and plot the results in Fig. 3. The shape factor in the stage-5 latent representations for both IN (Fig. middle) and SIN (Fig. 3 right) models become increasingly more relevant during the course of training, however the percentage of dimensions grows much larger and faster in the case of the SIN trained model. The texture factor decreases as the training progresses in both cases well. For the stage-4 representation in a model trained on IN (Fig. 3 left), note that the shape encoding neurons increase only marginally over the course of training. This further reveals that large proportion of shape information is encoded at the deepest layer.
5

---

Published as a conference paper at ICLR 2021
Stage-1
Stages5
Stages5
KSuueu 437
Aartur 300
-70
30
42-
Koutue 072
50
Dimirattn too
.30
Drove site PR)
22
Accusro of Caul or
35
-40
20
12
Zouture
-30
Pwasue
18
Redire 457
Arap 152
250
Training
Training Epoch
Tratning Spoch
Figure 3: Analyzing the number of dimensions in ResNet50 which encode shape ([ashape) 1) and texture (Fteetture) over the course of ImageNet (IN, left two) and Stylized ImageNet (SIN, right) training. Dimensions are estimated using stage four, |2(4] = 1024, and stage five, |=(6] = 2048, latent representations. When training begins, is very sensitive to texture but over the course of training learns to focus on the shape instead (faster in SIN case). The vertical lines represent multiplying the learning rate by factor of 0.1. Note that the estimated dimensions differ slightly from Table as we trained the IN and SIN models used in this figure from scratch.
3 How MUCH SHAPE INFORMATION DO CNNS ENCODE?
The previous section measured the dimensionality of shape encodings for various CNNs and settings. We now aim to evaluate the quality of these encodings, and whether more shape encoding neurons implies that more robust shape information can be extracted from these latent representations. We also conduct set of experiments by targeting the shape and texture-specific neurons (see Sec. 3.3 and Sec. A. for results and discussion), revealing an additional link between the two techniques used in our paper (i.e., dimensionality estimation and read-out module). Hermann & Kornblith (2019) measured the quality of shape encodings in CNN's latent representations by training a linear classifier on the CNN's late-stage features to predict the shape label of SIN images. Quantifying shape information by using image level labels does not allow for the per-pixel evaluation of the encoded shape, and its relation to the associated categorical label, two key components for fully evaluating the characteristics of shape information contained in particular encoding.
3.1 QUANTIFYING SHAPE INFORMATION IN CNN LATENT REPRESENTATIONS
To overcome the aforementioned issues, we consider two tasks which require detailed understanding of object shape: binary and semantic segmentation. A 'shape encoding network' (SEN), the network being analyzed, consists of CNN with fixed weights. We then train shallow read-out module that takes latent representation from the SEN, to predict segmentation map (i.e., binary or semantic). If the read-out module can accurately segment objects with binary mask, we conclude the SEN encodes the precise shape of the objects of interest. Further, the read-out modules ability to perform semantic segmentation, measures how much of this encoded shape is successfully localized with per-pixel categorical information. We use ResNet networks of various depths (i.e., 34, 50, and 101) as SENs with readout module containing either one or three convolution layers with 3x3 kernels.
3.2 RESULTS
We use the trainaug and val split of the VOC 2012 dataset to train and test the read-out module, respectively. The binary segmentation ground truth labels are generated by converting all semantic
Table 4: Left: We measure the amount of shape encoded in frozen CNN by training read-out module on either binary (Bin) or semantic segmentation (Sem) under different training settings. 'None': random initialization, 'End-to-End network is not frozen and trained with the read-out module, 'IN': pre-trained on ImageNet. Right: Shape information contained in various shape biased models.
1 Layer Readout
Layers Readout
ResNet50
Training
ResNet34
Net5o
ResNet101
ResNet34
ResNet50
ResNet101 Bin
Training
Bin
Sem
Bin
Sem
Bin
Sem
Bin
Sem
Bin
Sem
Bin
Sem
Sem
79.8 61.6
None
46.5 5.2
48.0
6.1
44.9
5.1
58.0
7.2
58.0
6.0
55.0
4.8
SIN
76.4 53.7
End-to-End
80.2 63.4
80.2
62.7
81.0
65.8
82.1
67.7
82.2
68.1
82.9
71.5
(SIN+IN)
77.8 58.0
IN
66.3 48.1
70.6
50.9
72.1
51.9
78.9
59.1
79.8
61.6
80.4
63.4

---

Published as a conference paper at ICLR 2021
Table 5: Shape encoding results for dif- ferent stages of ResNet networks trained on ImageNet. Combining features from early stages increases shape encoding.
Image
Bin GT
Sem GT
NesNet50
ResNet101
f1
f1
I5
Sa
fb
Bin
Sem
Bin
Sem
Bin
44.7
4.6
42.3
4.7
52.7
6.4
53.0
5.6
59.6
10.9
57.7
9.4
70.8
33,9
73.2
43.6
70.6 50.9
72.1
51.9
Sem
66.0 16.6
63.8
13.5
74.5 42.2
76.8
49.8
77.3
53.7
78.2
56.2
Figure 4: Stage-wise predictions of read-out module
77.3
52.9
78.2
55.2
binary (Bin) and semantic (Sem) segmentation.
categories single 'object' class. Note that the binary segmentation and semantic segmentation experiments are done completely independently of one another.
Table 4 presents the results in terms of mean-Intersectionnoveeee Uver-Unnon (mloU) under different initial- ization settings with; 'IN': SEN trained for ImageNet classification, 'None": a SEN with random weight initialization and without any training. 'End-to-End'; the SEN and readout module trained in an end-to-end manner on either the binary (Bin) or semantic (Sem) segmentation ground truth. The None and End-to- End-to-End networks represent lower and upper bounds for encoding shape, respectively. All read-out modules in this section are trained on the last layer's 's latent representations. Interestingly, three convolutional layers can extract similar amounts of shape information from the IN-SEN as the End-to-End-SEN. For example, training the ResNet101 by merely 2.5% compared to the IN-SEN. ImageNet trained CNNs also contain shape encodings which successfully localize per- nixep categorical information as well, which can be seen when com- paring the performance of the IN-SEN and End-to-Fne SEN, e.g., for ResNet50, the IN-SEN and -nd-S--E achieve 61.6% and 68. 1%, respectively. This is an interesting result considering the difficulty of semantic segmentation and that none of the IN-SEN weights are trained for pixel-wise objectives. Shape information also increases relative to the depth of the network which supports the results presented in Table I(c). As expected, the End-to-Emn -nd-End--- SEN and IN-SEN contain significantly more shape information in their latent representations than the baseline NENe-SEN.
for Bin improves the mloU
We now evaluate the shape information encoded in networks which have different levels of shape bias. We compare the Bin and Sem performance of the read-out module trained on the features of three different SENs trained on IN, SIN, and (SIN+IN)+IIN As the validation is on non-stylized images, SIN-SEN has slightly lower performance for Bin, and significantly less performance on Sem. Such a large difference in performance implies that while the boundary of the object is known, it is difficult for the network to correctly assign per-pixel categorical information, a phenomenon further explored in Sec. 3.2. Interestingly. the -SIN+IN)~IN-SS SEN also has slightly lower performance than the IN-SEN for Bin, but does not suffer in performance as much as the SIN-SEN in the case of Sem.
Where is Shape Information Stored? We now examine if the large amount of shape informa- tion contained in ImageNet pretrained models is equally distributed across different stages of the CNN. In this experiment, we train one layer read-out modules U1, f2, [4, fts, of the SEN to examine which stage of CNN encodes shape information. As shown in Table 5, the read-out module trained on the last stage features, (f4, Js), achieves higher performance compared to the earlier stage features, (/1, J2, Jz), for both Bin and Sem. This is to be expected, as feature maps from later stages have higher channel dimensions and larger effective receptive fields compared to the feature maps extracted from earlier layers. A surprising amount of shape information (i.e., Bin) can extracted from stages fi, and fs: however, these features lack high-level semantics to correlate with this shape information, which can be observed as the corresponding Sem performance is much lower. Figure reveals this phenomenon; the horse and person are outlined even for the early stage binary masks, but are only labelled with correct per-pixel categorical assignments in the later stages. Considering the non-tre trivial amount of shape information contained in the early stages, investigate if aggregating multi-stage features encodes more shape compared to the last stage feature, fs- Table 5 (bottom) shows that training a readout module on multi-stage features significantly improves the Bin and Sem performance, suggesting that tasks
features from different stages,
7

---

Published as a conference paper at ICLR 2021
Table 6: Shape encoding
results for shape biased ResNet50s on stylized VOC12 validation set.
ResNet50
Training Data
Bin. Sem.
IN
45.7
8,5
Sty lized Image
Bin
Sem
Bin
Sem
SIN
60.3
26.4
Figure 6: Binary and semantic segmentation masks extracted from
(SIN+IN)- IN
44,5 9.0
CNNs trained on ImageNet (IN) and Stylized ImageNet (SIN).
requiring shape information may benefit from hypercolumn style architectures (Hariharan et al., 2015). This indicates that some shape information is encoded in earlier layers but not captured in the late stages, which agrees with the dimensionality estimation results in Table 3, as around 12.5% and 14% of neurons encode shape in the first stage and second stage, respectively.
When do CNNs Encode Shape During ImageNet Train- ing? Now quantify the amount of shape encoded in the latent representations over the same ImageNet training snap- shots as in Sec. 2.1. Fig. shows the performance of the read- out module trained on the frozen SEN every five epochs, as
Stapa Encoding mbu 180)
-00
mini>77
mloU=79.4
86
mlou-58.0
mlo0m60.0
Accuracy
well as the ResNet50's validation accuracy on ImageNet clas- sification. Note that we train a separate read-out module for every snapshot. Similar to the findings in Fig. 3, we see that the majority of both binary and semantic shape information is
Strary Seg
Semantic Seg
10
mioU-6.0
learned within the first 10 epochs i.e., 77.1% = 97.1% of the 79.1%
20 40
0
ImageNet Training Epoch
60.0% 88.2% of the final Sem mloU is obtained by the read-out module after only 10 epochs. Con- trasting purely shape related information (i.c., Bin), a small but significant portion of per-pixel categorical information is learned after the initial 10 epochs, when the learning rate decay is employed.
final Bin mloU and 52.9%
Figure 5: Quantifying the shape and semantic information encoded by CNN over the course of ImageNet training. Vertical lines represent the learning rate decay.
3.2.1 DOES KNOWING AN OBJECT'S SHAPE IMPLY KNOWING ITS SEMANTIC CLASS?
We now explore whether CNN encoding an object's shape necessarily implies that it also encodes the correct semantic category on a per-pixel level. In other words, for a frozen CNN, can a read-out module (trained for binary segmentation) successfully extract the binary mask while another read-out module (trained for semantic segmentation) cannot successfully extract the semantic segmentation mask? Previous results (e.g., Table 4, Table 5) show that, for certain layers and networks, the binary segmentation performance of a read-out module is much higher relative to the semantic segmentation performance. This suggests that shape information (i.e., the binary mask) and semantic information can be encoded in mutuallly exclusive manner, i.c., CNN can encode the silhouette of the object without encoding the semantic category of each pixel of the silhouette belongs to.
To this end, we validate various SENs and their read-out modules on stylized VOC12 val images as this ensures the networks must encode per-pixel semantic information based solely on the object's shape (note that stylization removes all texture information, see Sec. 2.2). The difference in performance between the Bin mIoU and Sem mloU can therefore approximate the amount of shape information that is not correlated to its corresponding semantic class. As shown in Table 6 the large difference in performance between Bin and Sem suggests that these SENS capture the shape (i.e., Bin mask) of the object but lack the ability correctly assign per-pixel semantic labels to these objects. Qualitative results are presented in Fig. 6; note how the binary mask (presented as likelihood heatmaps) for the SIN trained model reasonably segments the objects, while the semantic masks fail resemble realistic predictions, i.e., multiple object categories are placed spuriously over the object of interest.
3.3 TARGETING SHAPE AND TEXTURE NEURONS
In Sec. 2, we used a dimensionality estimation technique to estimate the number of dimensions which encode shape and texture in CNNs latent representations. Given these neurons, we now validate
8

---

Published as a conference paper at ICLR 2021
Binary Seprienttionn IN
Semantic Seginemation IN
Binary Segmennction SIN
Seciuntic Segmentarion SIN
mosaa toll (9)
40
60
Shape Texture
Baseline
40
40
Top X% of Neurons (5g)(55)
Pescenaage Neuooss (R)
Pescesrage of Neurus Kape (9)
Peocestage Neurons Keje (5)
Figure 7: Shape encoding results by means of training read-out module on the latent representa- tions of an ImageNet (i.e., texture-biase biased model) (left two) and stylized ImageNet (shape-biase biased (right two) trained ResNet-50 for binary and semantic segmentation when removing all but top X% shape or texture-specific neurons.
if the most shape-specific, or texture-specific neurons can influence the performance of a read-out module when keeping these specific neurons during training. We hypothesize that the network trained on Stylized ImageNet (i.e., a shape biased model) will be more reliant on the shape neurons than a network trained on ImageNet (i.e., a texture biased model) which are known to naturally exhibit a texture bias. See Appendix A. for additional experiments where we remove the targeted neurons instead of keeping them to further validate if the most shape or texture-specific neurons can influence the performance of a read-out module during inference. To asses this hypothesis, we conduct series of read-out module experiment and the same settings as Sec. are imposed. However, during this experiment manipulate the latent representation as image passes through the ResNet-50, before it is fed through the read-out module. More specifically, we rank the neurons by mutual information for both the shape and texture semantic factors, and then identify the top X% of neurons from either the shape or texture neurons. Then, we train read-out modules on the latent representations of two frozen ResNet-50s, one trained on ImageNet (IN) and another model trained on Stylized ImageNet (SIN). Before the latent representation is fed through the read-out module, we remove all other neurons except for the top X% of shape or texture- texture-specifi neurons. This forces the read-out modules to learn to perform binary segmentation and semantic segmentation solely from the top X% of neurons for either semantic factor, and we can identify which neurons are more heavily relied on for each network (i.e., the shape biased or texture biased model).
Results. Figure 7 illustrates the binary and semantic segmentation results in terms of mIoU obtained from training read-out modules on IN (left two) and SIN (Iright two)) trained ResNet50s, respectively, is clear that for the model biased towards texture (IN pretrained), keeping texture neurons while removing all other neurons results in better performance than keeping only the shape neurons. In contrast, Fig. (right two) shows that for shape-biased model (SIN pretrained), keeping shape-specifis neurons achieves better performance than keeping only texture-specific neurons. These results support the hypothesis that the network trained on Stylized ImageNet (i.e., a shape biased model) is not only biased towards making predictions based on object shape, but more reliant on shape-specific neurons than network trained on IN.
4 CONCLUSION
In this paper, we presented a systematic study of the capacity and quality of shape encoded in CNNs latent representations. Approximating the mutual information between stylized PASCAL VOC images allowed to estimate the dimensionality of the semantic concepts shape and texture (Sec. 2.1.. We also designed a simple strategy for determining how much shape information is encoded in these latent representations, by training read-out module on per-pixel binary segmentation ground truth labels. Additionally, we perform semantic segmentation to quantify how much of this shape encoding can be correctly attributed per-pixel categorical information. We showed that a model pre-trained on ImageNet has weights that contain almost all the shape and categorical information needed to perform binary or semantic segmentation from the late stage features. We showed that CNNs encode surprising amount of shape information at all stages of the network, but correctly assigning categorical labels to the corresponding shape only occurs at the last layers of the network. and that removing the image's texture information severely hurts this correspondence. Finally, we showed how removing all but a certain number of targeted shape or texture- rexture-specific neurons affects performance differently depending on the reliance these neurons. These findings reveal important mechanisms which characterize a network's 's ability to encode shape information. We anticipate these findings will be valuable for designing more robust and trustworthy computer vision algorithms.

---

Published as a conference paper at ICLR 2021
REFERENCES
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection:
Quantifying interpretability of deep visual representations. CVPR, 2017.
Wieland Brendel and Matthias Bethge. Approximating CNNs with hag-of-local-featuee models works surprisingly well on imagenet. In ICLR, 2019.
Liang-Chied Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv:170600588, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Patrick Esser, Robin Rombach, and Bjöm Ommer. A disentangling invertible interpretation network for explaining latent representations. In CVPR, 2020.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal 2010.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In CVPR, 2018.
David Foster and Peter Grassberger. Lower bounds on mutual information. Physical review. E, Statistical, nonlinear, and soft malter physics, 2011.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2018.
Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns for object segmentation and fine-grained localization. In CVPR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In 2C17, 2017.
Katherine L Hermann and Simon Korblithh. Exploring the origins and prevalence of texture bias in convolutional neural networks. arXiv preprint (xXivi1911.0901 2019.
Katherine L. Hermann and Andrew K Lampinen. What shapes feature representations? exploring datasets, architectures, and training. In NeurlPS, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Md Amirul Islam, Mrigank Rochan, Neil DB Bruce, and Yang Wang. Gated feedback refinement network for dense image labeling. In CVPR, 2017.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys. Rev. E, 69:066138. 2004.
Nikolaus Kriegeskorte. Deep neural networks: new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolu- tional neural networks. In NIPS, 2012.
ZC Lipton. The mythos of model interpretability. arxiv 2016. arXiv preprint arXiv: 1606.03490 2019.
10

---

Published as a conference paper at ICLR 2021
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In ICAIS, 2020.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. PMLR, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In ICLRW, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In ICLR, 2020.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why Should Trust You?": Explaining the predictions of any classifier. arXiv, pp. arXiv-1602, 2016.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. ECCV, 2014.
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In CVPR, 2020.

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 REMOVING SHAPP AND TEXTURE NEURONS
In Sec. 3.3, we performed an experiment where we kept the top X% shape and texture-specific neurons to compare how much different CNNs relied on these neurons to encode shape. We now perform similar experiment, but instead of keeping the top X neurons, we first identify the top N of neurons from either the shape or texture-specific neurons. We then remove these neurons before passing the latent representation to the read-out module by simply setting the features at other dimensions to zero. This allows us to identify which neurons in each network (i.c., the shape biased texture biased model) are relied more heavily to encode shape and semantic information. Note that for this experiment, no training occurs. The goal is to simply measure the difference in inference performance when removing N shape-specific neurons, N texture-specific neurons. Therefore we simply take the trained models and read-out modules from Sec. 3 to perform inference while masking out the targeted neurons. Note that validation is done on the val split from (non-sty nonnsttllzzed) VOC 2012.
Experimental Details. For this experiment, the dimensions sharing the most mutual information with respect to shape and texture are obtained from the same experiments from Sec. 2. We then rank the dimensions for each semantic factor by mutual information. Training and inference are done with the trainug and val split, respectively, from the noo-stylizedd) PASCAL VOC 2012 (Everingham et al., 2010) dataset.
Table 7: Shape encoding results for ResNet50's trained on ImageNet (IN) and stylized ImageNet (SIN) based read-out modules when the top N shape or texture-specific neurons are removed from the latent representation during inference. Removing the top N shape specific neurons from the SIN-read-out hurts the network's shape-recognition abilities more compared to the IN-read-out model.
Shape
Texture
Residual
N
IN
SIN
IN
SIN
IN
SIN
Bin
Sem
Bin
Sem
Bin
Sem
Bin
Sem
Bin
Sem
Bin
Sem
70.6
50.9
76.4
53.7
70.6
50.9
76.4
53.7
70.6
50.9
76.4
53.7
100
68.8
46.4
64.6
37.1
69.3
44.7
65.7
39.2
68.3
46.7
64.7
37.3
200
67.9
40.0
57.3
31.4
67.9
39.7
62.5
32.7
66.1
44.2
64.9
34.8
300
61.6
38.0
58.6
25.8
66.3
37.6
55.0
27.9
62.9
40.0
62.3
30.1
A.1.1
RESULTS
Table presents the binary and semantic segmentation results in terms of mloU. We report the results under three different settings; (i) top N shape-specific neurons removed, (ii) top N fexture-specific neurons removed, and (ii) top N residual neurons removed. Note that for this experiment, we do not train the read-out module. Instead, we first remove the specified neurons, and then run inference using the pretrained IN and SIN models as well as the already trained read-out modules. Interestingly, we find that gradually removing the shape-specific neurons from SIN pretrained model more significantly hurts performance than the IN pretrained model. For instance, removing 100 shape-specific neurons from SIN achieves 37.1% % sem mloU, while the performance dropped 25.8% sem IoU when the top 300 shape-specific neurons are removed (i.e., an 11.3% drop). When comparing this to the performance drop of the IN trained model, we see that the difference lower, from 46.4% 38.0% (i.c.. an 8.4% drop). This further supports the hypothesis that SIN trained models are more reliant on the individual shape encoding neurons than the texture encoding neurons. In addition, removing shape-specific neurons from SIN pretrained model hurts performance more than removing the texture neurons. For example, when removing 300 shape neurons for the SIN trained model, the performance drops to 25.8%, while removing 300 texture-specific neurons decreases the performance to only 27.9%. Finally, we observe that removing shape or texture specific neurons hurts performance more than removing the residual neurons. This suggests that shape and texture are the two most important semantic factors for a network to encode and that other semantic factors contained in the residual (e.g., color, lighting) not discriminative for the task of image classification.
12

---

Published as a conference paper at ICLR 2021
A.2 CONSISTENCY OF DIMENSIONALITY ESTIMATION
Sec. 2.1 analyzed the consistency between the dimensionality estimate of (Esser et al., 2020) and that of (Bau et al., 2017). order quantify interpretability, the latter evaluates the alignment between neurons and semantic concepts using the Broden dataset, which consists of images with pixel-wise labelings of different semantic concepts. For each neuron 2,, it determines the top quantile level T; such that the activation value of exceeds T; only in 0.5% of all observed cases over the dataset, Le. 2 > 7i) = 0.005. Feature maps are then upsampled to the original image resolution and each neuron is thresholded according its quantile T; to obtain a binary segmentation mask. A neuron is then termed a detector for a concept, if its segmentation mask has the highest intersection over union score (IoU) for this concept, and the IoU exceeds threshold of 0.04.
Because the Broden dataset contains no shape concepts, the comparison in Table is limited to estimates the number of neurons encoding texture, which is determined by the number of detectors for concepts from the categories material, texture and color of the Broden validation dataset.
We observe that both methods predict inverse relationship between the estimated texture dimension- alities and the receptive field, except for the case of BagNet9, where a sudden drop in the number of texture detectors is observed for network dissection. Besides this qualitative agreement, the predicted absolute numbers differ. There are two main sources for the incompatibility in the absolute number of neurons. First, both approaches rely on hyperparameters.uu i.e. the baseline score and the choice of the normalization function in the case of dimensionality estimation, and the chosen quantile and IoU threshold in the case of network dissection. Second, the semantic meaning texture depends on the data, i.e. dimensionality estimation relies on the image pairs of SVOC, whereas network dissection relies on texture images of Broden. This might also explain the drop in detectors for BagNet9 its receptive field is too small for some of the Broden textures. While absolute numbers depend on hyperparameters, results obtained with both methods are comparable across networks.
The dimensionality estimate relies on estimate of mutual information from samples. This remains a challenging problem, and even powerful variational bounds exhibit either high- bias or high-variance and suffer from sensitivity to batch sizes (Poole et al., 2019). Besides statistical limitations the ability to accurately estimate mutual information (Mc lesterr & Stratos, 2020), even estimates which give neither upper lower bounds those which give loose bounds are still useful in practice. For example, (Tschannen et al., 2020) demonstrate that loose bounds can lead to better representations when they are learned by mutual information maximization. For dimensionality estimation, potential biases of estimates will cancel out when comparing them between shape and texture neurons, hence an estimate based on the correlation a suitable and efficient choice for our purposes.
A.3 ESTIMATING SHAPE AND TEXTURE DIMENSIONALITY OF DIFFERENT NETWORKS TRAINED ON STYLIZED IMAGENET
We further estimate the dimensionality of shape and texture semantic concepts of different networks in Table to test the consistency of the results reported in Table on different architectures. We run the dimensionality estimation experiment (see Sec. 2) on AlexNet (Krizhevsky et al., 2012) and VGG-16 (Simonyan & Zisserman, 2015), trained on IN and SIN. Consistent with the findings for ResNet50, Table 8 shows that training SIN increases the number of dimensions encoding shape and concurrently decreases the number of dimensions encoding texture: AlexNet-IN: [Shape=729, Pexture=1299), AlexNet-SIN: [1119, 870]. VGG-16-IN: [710, 1321], VGG-16-SIN: [1090, 879]. The dimensionality estimation is done on the final representation before the last linear layer for all networks.
Table 8: Comparison of shape bias and shape dimensionality for different networks.
IN
SIN
Network
Factor [zg]
Bias
Factor 12
Bias
Shape
Texture
Shape
Texture
Shape
Texture
Shape
Texture
ResNer-50
14.1%
40.2%
22.1%
77.9%
26.2%
23.3%
81.0%
19.0%
AlexNet
18.0%
30.6%
42.9%
57.1%
26.0%
21.5%
75.5%
24.5%
VGG-16
15.3%
37.9%
17.2%
82.8%
26.6%
21.5%
77.4%
22.6%
13

---

Published as a conference paper at ICLR 2021
A.4 ESTIMATING SHAPE AND TEXTURE DIMENSIONALITY OF SELF-ATTENTION NETWORKS
We also experiment with the recently proposed Self-Attention Networks (Zhao et al., 2020), which replace convolutional layers with self-attention layers. Three different depths of Seff-Attention Networks (SANs) were proposed. For fair comparison, we experiment with SAN-19, which the authors claim the most similar size ResNet50, in terms of the number of parameters in the network. Additional, SANs come with two types of layer operations, patch- based and pair-based. The patch-based SAN compares patches of pixels within the attention operations, while the pair-based SAN compares individual pixels and achieves lower performance on ImageNet. Due to the lower effective receptive field of the pair-based SAN compared to the patch-based SAN, we expect to see larger number of neurons encoding shape in the patch-based SAN. The results are shown in Table 9. The patch- based SAN19 has the largest number of shape encoding dimensions and lowest number of texture encoding dimensions, when compared the pair-based SAN19 and ResNet50.
Table 9: Comparing the number of shape encoding neurons and texture encoding neurons for self-attention networks (Zhao et al., 2020).
Factor [x
Factor
|>x|/|>|
Model
Shape
Texture
Shape
Texture
ResNet50
349
692
17.0%
33.8%
SAN-19 19 (patch)
384
610
18.8%
29.8%
SAN-19 (pair)
304
764
14.9%
37.3%
A.5 LAYER-WISG WISE DIMENSIONALITY ESTIMATION ON ALEXNET
We now explore where another CNN encodes shape and texture at each layer of the network. More specifically, we apply the dimensionality estimation technique from Sec. 2.1 on AlexNet (Krizhevsky et al., 2012) on a number of different layers. Due to the different dimensions at each of these stages, we present the results as the percentage of dimensions encoding the particular semantic factor, | /|2l, where |z| refers to length of the latent representation. The results are presented in Table 10 and Fig. Note that the output from the convolutional layers also include the ReLU activation function.
ImageNet
Stylized ImageNet
Shape
Texture
30
40
Dimensions (%)
35
25
30
20
25
15
20
convi
pooll
conv2
pool2
conv3
conv4
conv5
pool3
fc6
fc7
convi
pool|
conv2
pool2
conv3
conv4
conv5
pool3
fc6
fc7
Layer
Layer
Figure 8: Shape (left) and texture (right) encoding dimensions estimated each layer of
AlexNet (Krizhevsky et al., 2012). Shape biased AlexNet trained on Stylized ImageNet (Geirhos
al., 2018) encode more shape at the later layers of the network which is consistent with the findings for ResNets (He et al., 2016).
14

---

Published as a conference paper at ICLR 2021
Table 10: Percentage of neurons (/2x|/12() encoding different semantic concepts, k, for different stages of AlexNet (Krizhevsky et al., 2012) trained for various levels of shape bias.
IN
SIN
Stage
Factor
z /
Factor
2 ///
Shape
Texture
Shape
Texture
conv1
17.0%
35.0%
16.6%
35.3%
pool1
19.0%
31.7%
18.8%
31.8%
conv2
20.7%
27.2%
21.1%
27.7%
pool2 20.7%
26.1%
21.1%
25.9%
conv3
20.6%
27.0%
23.9%
23.2%
conv4
21.2%
25.8%
25.4%
21.7%
conv5
21.3%
24.5%
25.2%
21.0%
pool3
21.7%
23.8%
25.4%
20.9%
fc6
18.8%
28.7%
24.5%
21.9%
fc7
18.0%
30.6%
26.0%
21.5%
15


Reviews:
[{"review_id": "g8Lmgu46Auf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper performs a detailed analysis of shape vs texture tradeoff in deep neural networks. The authors use two methods to analyze the learned shape features in CNNs: one based on estimation of concept dimensionality, one based on reconstructing shape from latent representations.\n\n1. The contributions of this work is limited. First of all, the methods of this work is adopted or based on previous methods. The dimensionality estimation method is based on (Esser et al, 2020). The idea of using “read-out module” to segment images is similar to the idea in (Hermann et al, 2019) where a linear classifier is trained to predict the shape label. \n\n    Moreover, the results are not interesting. The analysis conclusions made in this paper are all expected and not surprising, though i have to acknowledge that this is a more fine-grained study. I would suggest the authors to stress a few key messages that you are trying to make in this paper, rather than the lots of analysis results presented in the current one without a central conclusion.\n\n2. Overall, I am not satisfied with the structure of this paper. It seems that the authors   struggle to put two not so relevant ideas together. I would like the authors to spend some time to discuss the motivation for using the two analysis methods and the dependencies between them. This will make this work coherent and consistent.\n\n3. The design of readout module is not clear to me. Specifically, why did the authors choose the readout module to be “either one or three convolution layers with 3×3 kernels”? The current one seems to be too simple, can their be other designs for the read out module? \n\n4. What hidden layer is analyzed for CNN in Table 1? \n\n5. What exactly is shown in Figure 1, i understand the subfigure “stylized image” and “semantic”, but what are the two figures in the middle “GT” and “Shape”? What are these two images? What does GT means? I didn’t find the authors rigorously define these for figure 1 in the paper.\n\nAfter rebuttal:  \nI appreciate the authors for the response. However, I feel that the conclusions drawn from this paper are not focused, there is no central message that i can get as a clear take-away for understanding CNNs after reading the paper. I do agree with each of the findings the authors make, it is just that the presentations and writings of the paper make me confused about what the authors are trying to convey through this paper. Therefore, i am keeping my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IMaDScsq19A", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pro’s: \n* The question that the authors are trying to understand is interesting and relevant in the field of object recognition, texture/shape bias, and learned representations in deep neural networks.\n* Authors provide a nice set of controlled experiments that suggest some of these effects, and authors present a scientific login in their approach (that although it is not perfect), it is quite relevant for the field. This is not “yet another paper on trying to overcome the texture bias without any intuition what-so-ever, to try to get better numbers (as has unfortunately been done in computer vision these days)”, on the contrary, this paper is about understanding the underlying mechanisms of the texture/shape bias that extend the final stages of computation in the visual hierarchy, and is why I am leaning towards accept.\nNearly all figures in the paper are clear and help convey what the authors are trying to express (although see somes clarification points)\n* The paper is clear and easy to read/understand.\n\nCon’s:\n* Some experimental evaluations/methods are not clear (see below Observeration/Clarifications that if clarified may convince me to raise my score).\n\n----\n\nObservations/Clarifications: \n\nFigure 3 shows an interesting plot. It would have been great to compare the result with that of Stylized ImageNet as well (that presumably will have a greater increase in shape-tuned neurons vs texture-tuned neurons).\n\nWhat is a “stage” in the ResNet? Is this a residual block, or a specific layer? Stages are mentioned and used throughout the paper but are never formally defined. I have an intuition that they are related to the depth in the hierarchy of the network, but not on the layer type or the computation (example: is it a conv layer, is it rectified?) That being said, why is Stage 1 not included at all? (presumably it is the input/image so it is redundant?) Authors should make this more clear.\n\nThe Binary and Semantic Segmentation tasks are not clear. The binary segmentation task’s goal is to purely create a binary mask of any object? Or more than one? Does the mask have to be a closed contour? Further what is confusing about the semantic segmentation task is if the pixel-wise labeling is done GIVEN the ground-truth binary mask or the predicted binary mask. This should be made more clear in section 3.1\n\nSection 3.2.1: Knowing an Object’s shape implies knowing its semantic class: I don’t think the results in Table 6 actually show what the authors claim. If by default the output of the binary masks are different, these effects will naturally carry over into the semantic segmentation procedure. The only way to truly account for the proposed question of “if shapes implies better semantic class”, is if all systems are equalized and given with the SAME binary mask (assume ground truth), and must use this oracle to then perform semantic segmentation. If the disparity is still great, then the before-mentioned claim has stronger computational support. Otherwise, I can see that there the confounding variable is that the nature of one system knowing how to binarize the image better a priori, will lead to differences in the final semantic segmentation score/maps.\n\n----\n\nAltogether the paper is well written and there is a good body of results/evaluations that support some of the claims the authors are doing. The problem they are working on is interesting and relevant to the ICLR and greater CV/ML/Vision Science community. I am leaning towards acceptance, and would appreciate it if authors clarify my doubts above to raise my score.\n\n----\n*Update as of December 5th. I would like to thank the authors for clearing up my concerns and would like to raise my score from 6 to 7. I think this paper would be a good poster that can provide complimentary insights through different experiments to the recent paper of Hermann et al. NeurIPS 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DC9U5_CRqC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Texture vs shape sensitivity is a basic and important question in understanding how deep convolutional nets work. This paper investigates this question by asking how CNNs represent shape versus texture information internally.  Using a (texture-vs-shape) stylized imagenet pioneered by Geirhos (2018), the paper applies a dimensionality estimation method of Esser (2020) and a segmentation-readout method for quantifying and visualizing the encoding of shape information in networks.  Several different network architectures and layers are compared; and results are compared to baselines from previous papers as well as natural lower- and upper-bounds.  The presentation is clear.\n\nThe paper makes several contributions, extending beyond Geirhos 2018 by investigating the internal representations of shape- vs texture rather than just the external performance characteristics.  It extends beyond Esser 2020 by more fully examining shape versus texture representations, and comparing a variety of networks and layers.  It extends beyond Bau 2017 by counting partially correlated neurons and explicitly training segmenters to explicitly read out local shape or class information.  The sharp increase in shape representation dimension in the last stage of resnet is an interesting finding, as is the rapid growth of shape representations early in training. The measurement of texture over shape representations in BagNet is a nice confirmation that the proposed measurements match observed behavior. The use of segmentation readout, tested several ways, provides helpful and new qualitative visualizations.\n\nThe paper could be strengthened if it proposed further testable hypotheses.  The paper does confirm that the measured shape dimensions is lower for BagNet architecture with reduced receptive field, but beyond observing correlations, the paper does not try to directly verify that the identified shape dimensions are responsible for the network’s shape-recognition abilities.  For example, would the shape segmentation readouts be more damaged by removal of the most shape-specific dimensions (vs other neurons)?  Or would the removal of these dimensions affect external shape or texture performance?  If an effect like this were verified, it would strengthen confidence in the proposed correlation-as-dimensionality measure of shape and texture content.\n\n**edit, after revisions**  As mentioned in discussion below, I think the addition of intervention experiments makes the effects very clear and strengthen the paper.  I revised my score from 7 to 8.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1mDdTzM1iYZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The method provides two measures for assessing the degree at which shape is represented in CNNs. The first measure attempts to asses, within a given representation layer, the dimentionality required to encoder shape information. The second, evaluates the per-pixel shape representation, by attempting to generate the input's segmentation mask from the given representation (activation of the image at that layer). \n\n*****\n\nPros:\n\nThe method provides a comprehensive review of the encoded shape information, both on the representation level and pixel level. \nIt answers questions in a variety of settings:\n1. The shape encoded in different stages, or representation layers, for standard networks, and for networks trained to aleviate the shape bias (SIN, SIN + IN)\n2. The degree of shape representation as a function of epochs or time trained. \n3. The per pixel representation of shape, i.e to what degree can the semantic map of the input image be recovered from a given representation layer. \n\nThe paper is also very clear and results are presented well and well explained. \n\nCons:\n\nRegarding the first of the two methods (A in Fig 2), I am not convinced of its correctness. Eq. 1 upper bounds the mutual information using variable p_i, which can be calculated. However, with a lack of a lower bound, this doesn't really mean much. The actual mutual information can be much higher. Why is this bound tight?\n\nRegarding the second of the algorithm (B in Fig 2), existing algorithms in literature, presented for example in [1], learn a classifier, that given the representation of an image, attempts to classify it shape. As such novelty of Alg.2 is very limited: Instead of classifying the shape, the method is trained to produce a segmentation map. \n\nIn addition, the use of a classifier as in [1], could also be used to quantify the use of shape in different representation layers. Why is it better to use the existing formulation instead of a classifier? Can't the use of a classifier also replace the need of Alg.1 in the sense of capturing the amount of shape information used? Does the existing measure reveal something more interesting in Tables 1-4 and Figure 3?\n\n********\n\nOverall, while the paper provides a comprehensive analysis of the use of shape in CNNs, both on pixel level and the representation level, I am not sure of the correctness of Alg.1 and the originality and relevance of Alg.2 given existing work. \n\n[1] The Origins and Prevalence of Texture Bias in Convolutional Neural Networks. Hermann et al., ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to do a fine-grained analysis of how shape and texture play a role in the decisions made by CNNs. Lots of recent evidence suggests that CNNs exhibit a texture bias, and there has been considerable effort in understanding where this comes from and how to overcome it. The paper focuses in particular on understanding: (a) what fraction of the neurons are devoted to shape-vs-texture (roughly speaking), and (b) per-pixel results using a convolutional readout function. The reviewers were divided at the time of submission and remained so at the end of discussion. At the end of discussion, the reviewers were split, with scores ranging from 4 (R1,R4), 7 (R2), and 8 (R3). The AC wants to thank and acknowledge the authors as well as all of the reviewers for their engagement in the discussion.

- R2 and R3 are largely positive, driven by the extent of the experiments and the number of interesting results (e.g., how the fraction of the dimensions used for shape changes as a function of depth in the network). Both had smaller non-critical concerns that were addressed (as far as the AC can tell) in the discussion. 
- R4’s most important concern, in the AC's view, is the question: could these results / different conclusions have been obtained via linear probe methods like Hermann et al.? The authors argue that analyzing the fraction of neurons used and at a per-pixel is more fine-grained than linear probes. This boils down to an intangible question of contribution, on which the AC is inclined to agree with the authors and R2 and R3: analyzing the dimensions contribute provides, at least to the AC, a complementary view to the linear probe and that this will be of interest to the ICLR community (although see final comment). R4 also had a number of smaller concerns that seem to be largely addressed (e.g., about correctness).
- R1 argues primarily that the paper does not have a clear point or methodological contribution, for instance pointing out that readout modules were used in Hermann et al. or (as an example) arguing the readout function design is too simple.  The AC is inclined to agree with the authors’ response that the other reviewers seem to largely agree on the contribution (especially contributions via experiments rather than method) but disagree on how to weigh these contributions. The AC would also add that readout modules are a core idea for understanding neural representations that long predate Hermann and are by design (as the authors note) almost always as simple as possible.

At the end of the day, the AC is agrees with R2 and R3 for the contribution of the work and is inclined to accept. Given the other reviews, the AC does not agree with R1’s arguments, but would suggest that the authors think about how to sharpen their claims further. The AC is sympathetic to the concerns of R4, and urges the authors to think about a more concise and clean argument for R4’s concerns --- many other readers will have similar concerns and as clean of an illustration will be helpful. Overall, the AC believes that the paper’s methods, experiments and analysis are of interest and value and is thus in favor of acceptance.
Decision: Accept (Poster)