Emergence of Shape Bias in Convolutional Neural Networks through Activation Sparsity
Authors: Tianqin Li
Ziqi Wen
Yangfan Li
Tai Sing Lee
Venue: NeurIPS 2023 oral (2023)
Keywords: neuroscience
computer vision
shape & texture bias
TL;DR: This study demonstrates that shape bias can emerge through the principle of efficient coding, paving the way for more human-like visual perception in CNNs.

Abstract: Current deep-learning models for object recognition are known to be heavily biased toward texture. In contrast, human visual systems are known to be biased toward shape and structure. What could be the design principles in human visual systems that led to this difference? How could we introduce more shape bias into the deep learning models? In this paper, we report that sparse coding, a ubiquitous principle in the brain,  can in itself introduce shape bias into the network. We found that enforcing the sparse coding constraint using a non-differential Top-K operation  can lead to the emergence of structural encoding in neurons in convolutional neural networks,  resulting in a smooth decomposition of objects into parts and subparts and endowing the networks with shape bias.  We demonstrated this emergence of shape bias and its functional benefits for different network structures with various datasets. For object recognition convolutional neural networks, the shape bias leads to greater robustness against style and pattern change distraction. For the image synthesis generative adversary networks,  the emerged shape bias leads to more coherent and decomposable structures in the synthesized images. Ablation studies suggest that sparse codes tend to encode structures, whereas the more distributed codes tend to favor texture. Our code is host at the github repository: https://topk-shape-bias.github.io/

Emergence of Shape Bias in Convolutional Neural Networks through Activation Sparsity
Tianqin Li Carnegie Mellon University tianqinl@es. cmu. edu
Ziqi Wen
Carnegie Mellon University ziqiven@cs. cmu edu
Yangfan Li Nortwestern University htwp:aaaweeeeeee...oooooeoeeeeee edu
Tai Sing Lee
Carnegic Mellon University tai@cnbc. cmu. edu
Abstract
Current deep-learning models for object recognition are known to be heavily biased toward texture. In contrast, human visual systems known be biased toward shape and structure. What could be the design principles human visual systems that led this difference? How could we introduce more shape bias into the deep learning models? In this paper, we report that sparse coding, a ubiquitous principle in the brain, can in itself introduce shape bias into the network. We found that enforcing the sparse coding constraint using a non-differential Top-K operation can lead the emergence of structural encoding in neurons in convolutional neural networks, resulting in a smooth decomposition of objects into parts and subparts and endowing the networks with shape bias. We demonstrated this emergence of shape bias and its functional benefits for different network structures with various datasets. For object recognition convolutional neural networks, the shape bias leads to greater robustness against style and pattern change distraction. For the image synthesis generative adversary networks, the emerged shape bias leads more coherent and decomposable structures in the synthesized images. Ablation studies suggest that sparse codes tend to encode structures, whereas the more distributed codes tend to favor texture. Our code is host at the github repository: httpps:/topss shape- bias. gi github.io..
Introduction
Sparse and efficient coding is well-know design principle the sensory systems of the brain Recent neurophysiological findings based on calcium imaging found that neurons in the superficial layer of the macaque primary visual cortex (VI) exhibit even a higher degree of lifetime sparsity and population sparsity in their responses than previously expected. Only 4-6 out of roughly 1000 neurons would respond strongly to any given natural image [37. Conversely, a neuron typically responded strongly to only 0.4% of the randomly selected natural scene images. This high degree of response sparsity commensurated with the observation that many V1 neurons are strongly tuned to more complex local patterns in global context rather than just oriented bars and gratings (36]. the other hand, over 90% of these neurons did exhibit statistically significant orientation tuning, though mostly with much weaker responses. This finding is reminiscent of earlier study that found similarly sparse encoding of multi-modal concepts in the hippocampus
[32]. This leads to the hypothesis that neurons potentially serve both super-sparse specialist code with their strong responses, encoding specific prototypes and concepts, and as a more distributed code, serving as the classical sparse basis functions for encoding images with much weaker responses. The specialist
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

code is related to the idea of prototype code, the usefulness of which has been explored in deep learning for serving as memory priors [24] in image generation. for representing structural visual or for constraining parsimonious networks 226 for object recognition.
concepts 38 39
In computer vision community, recent studies found that Convolutional Neural Networks (CNNs) trained for object recognition rely heavily on texture information [I]]. This texture bias leads to misclassification when objects possess similar textures but different shapes [2). In contrast, human visual systems exhibit a strong 'shape bias' in that we rely primarily on shape and structures over texture for object recognition and categorization (20) For instance, human observer would see spherical object as a ball, regardless of its texture patterns or material make-up [34]. This poses an
interesting question: What is the design feature in the human vision systems that lead to the shape bias in perception?
Fraction of 'shape' decisions
0.9
0.8
0.7
0,6
0.5
0.4 0.3
0.2
0.1
Stapee biss
1.0
0.0
models
Shape categories
AlexNet (+ Top-K w/o Training) (Ours)
VGG16 (+ ToTTp-K w/o Truining) (Ours)
AlexNet
VGG16
VIT-L
CLIP ViT-B(400M)
buman
HHst transformer models
-Bstt CNNs w/ Top-K (Ours)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Fraction 'texture'
Figure 1: Shape bias of our sparse CNNs versus standard CNNs and SOTA transformer-hased networks in comparison to the shape bias of human subjects, evaluated benchmark dataset [10) across 16 classes. The red dotted line shows the frontier of transformer- networks with the best shape bias. The greed dotted line shows that sparse CNNs push the frontier of the shape bias boundary toward humans.
In this paper, we explore whether the constraint of the high degree of strong-response sparsity in biological neural networks can induce shape bias in neural networks. Sparsity, particularly in overcomplete representations, is known to encourage the formation of neurons encoding more specific patterns [30]. Here, hypothesize that these learned specific patterns contain more shape and structure information, thus sparsifying the neuronal activation could induce shape bias in neuronal representation. To test this hypothesis, we impose a sparsity mechanism by keeping the Top K absolute response of neuronal activation at each channel in one or multiple layers of the network, and zeroing out the less significant activation with K is sparsity parameter that we can adjust for systematic evaluation.
We found that this sparsity mechanism can indeed introduce more shape bias in the network. In fact, simply introducing the Top-K operation during inference in the pre-trained CNNs such as AlexNet [19] or VGG16 (33 can already push the frontier of the shape bias benchmark created by [10 (as shown in Figure D. Additional training of these networks with the Top-K operation in place further enhances the shape bias in these object recognition networks. Furthermore, we found that the Top-K mechanism also improves the shape and structural bias in image synthesis networks. In the few-shot image synthesis task, we show that the Top-K operation make objects in the synthesized images more distinct and coherent.
To understand why Top-K K operation can induce these effects, we analyzed the information encoded in Top-K and non-Top-K responses using the texture synthesis paradigm and found that Top-K responses tend to encode structural parts, whereas non-Top-K responses contribute primarily to texture and color encoding, even in the higher layers of the networks. Our finding suggests that sparse coding is important not just for making neural representation more efficient and saving metabolic energy but

---

also for contributing to the explicit encoding of shape and structure information in neurons for image analysis and synthesis, which might allow the system to analyze and understand 3D scenes more structurally oriented part-based manner 4 making object recognition more robust.
2 Related Works
Shape Bias v.s. Texture Bias There has been considerable debate over the intrinsic biases of Convolutional Neural Networks (CNNs). [I]] conducted a pivotal study demonstrating that these models tend to rely heavily on texture information for object recognition, leading to misclassifications when objects have similar textures but distinct shapes. In addition, it has also been shown that using the texture information alone are sufficient to achieve object classification [6]. This texture bias contrasts markedly with human visual perception, which exhibits strong preference for shape over texture - a phenomenon known as 'shape bias' [20]. Humans tend to categorize and recognize objects primarily based on their shape, a factor that remains consistent across various viewing conditions and despite changes in texture [34]. These studies collectively form the foundation upon which our work builds, as we aim to bridge the gap in shape bias between computer vision systems and human visual systems.
Improving Shape Bias of Vision Models Following the identification of texture bias in CNNs by (.. numerous studies sought to improve models' shape bias for better generalization. Training methods have been developed to make models more shape-biased, improving out-of-distribution generalization. Some approaches, like [T], involved training with stylized images to disconnect texture information from the class label. Such approach posed computational challenges and didn't 't
scale well. Others, like [15]. used human-like data augmentation to mitigate the problem, while [3 proposed gudded augmentation, generating different styles different sides of image's boundary. However, these techniques all rely on data augmentation. Our focus is on architectural improvements for shape bias, similar to [1] which created texture-biased model by reducing the CNN model's ' receptive field size. We propose using sparsity operations to enhance shape bias of CNNs. Furthermore, 7 proposes scale up the transformer model into 22 billion parameters and show near human shape bias evaluation results. We, on the other hand, are not comparing with their network since we focus on CNNs which requires less computation and doesn' 't require huge data to learn. We demonstrate in the supplementary that the same sparsity constraint could also be beneficial the ViT family as well, hinting the generalizability of our findings.
Robustness In Deep Learning Robustness in deep learning literature typically refers to robustness against the adversarial attack suggested by (35) which showed that minuscule perturbations to images, imperceptible to the human eye, can drastically alter a deep learning model's predictions. Subsequent research 14 2T corroborated with these findings, showing that deep neural networks (DNNs) are vulnerable to both sifiiialllyinducer ad adversarial attacks and naturally occurring, adversaraa corruptions. However, the robustness we are mentioning in this paper is about the robustness against confusing textures that are misaligned with the correct object class, as illustrated by the cue-conflict datasets provided by IT. Although sparsity has been shown be effective against the adversarial attack 25 explicit usage of Top-K in shape bias has not been explored.
3 Method
3.1 Spatial Top-K Operation in CNN
Sparse Top-K Layer We implement the sparse coding principle by applying Top-K operation which keeps the most significant K responses in each channel across all spatial locations in a particular layer. Specifically, for internal representation tensor X E Rexhxw, Top- K layer produces XTop_x : Top_K(X, K), where the XToP_K is defined as:
j[), if (X[.) 2 (anx 0,
XTop_x [1,
(1)
otherwise
Equation 1 sseccifies how each entry of feature tensor X E Rexhxw would be transformed inside the Top- K layer. The zero-out operation in Equation I suggests that the gradient w.n.t any non Top-K K
3

---

value, as well as the gradients that chain with it in the previous layers will become zero. However, our analysis later suggests that the network can still learn and get optimized, leading to improved dynamic spatial Top-K K selection.
Sparse Top-K With Mean Replacement To determine the relative importance of the Top-K values or the Top K positions in the Top-K operation, we create an alternative scheme in which all the Top-K responses in channel are replaced by the mean of the Top-K responses in the channel Top_ Bop___Mean__ai Rp1 as defined below:
Σ (To.K. jkk, if abs otherwise
>
XTop_K Nean Rpt [i.
0,
This Top_K_Meat Mean_enn_ppp [pl operation reduces the communication rate between layers by 1000 times. We study the impact of this operation on object performance in an object recognition network (See section 4.3 for the results) and to determine which type of information (values versus positions) is essential for inducing the shape bias.
3.2 Visualizing the Top-K code using Texture Synthesis
We used texture synthesis approach [9] with ablation of Top-K responses to explore the information contained in the Top-K K responses a particular layer using the following method.
Suppose program F(.)) denotes a pre-trained VGG16 network with number of parameters N 33 and TS xxxxxx3 x N - R2xxwx3 denotes the texture synthesis program from [9] where an input image is iteratively optimized by gradient descent best match the target image T internal activation when passing through F F(T). We detail the operations inside TS below. Denote the internal representation each layer of the VGG16 network when passing an input image I through F(-) as (I), and suppose there exist L layers in the VGG16 network. We update the image I as follows:
I I+ lr * (0l E/Grr(x(()) - Gr(X(((T))
.weee Gr(.) Rhxwxe + Rexe denotes the function that computes gram matrix of feature tensor, i.e. Gr(X,(T)) x,()) We adopt LBFGS (2 with initial learning rate 1.0 and 100 optimization steps in all our experiments with the texture synthesis program.
Utilizing the above texture synthesis program TS(, VGG16),, we can obtain the synthesis results in Swo Top-K by manipulating the internal representation of VGG16 such that we only use the non-Top-K responses compute the Gram matrix when forming the Gram matrix optimization objectives. This effectively computes synthesis such that it only matches with the internal non-Top-K neural response. For given target image T, this leads Swo Top-K:
Swlo Top-K TST,, Zero@uttInterraalTpp (VGGGGG
which would show the information encoded by non-Top-K responses. Next, we include the Top-K firing neurons when computing the Gram matrix to get Sw Top-K:
Swi Top K = (T' IdentifyFunction (VGG16))
Comparing these two results will allow us assess the information contained in the Top-K K responses.
3.3 Visualizing the Top-K neurons via Reconstruction
provide further visualization of the information Top-K neurons encode by iteratively optimizing an image to match the internal Top-K K activation directly. Mathematically, we redefine our optimization objective in section 3.2
Similar to section 3.2
I 1 - Ir
à Σ E (X, Mask, X2(T))) 57
where Mask; a controllable mask for each layer There are three types of mask we used in the experiments: TTop-K_Mask,
Hop-K_Maskk selects only the Top-K fired neurons while keeps the rest of the neurons zero, whereas Lon_Top-K_Man the opposite of the Top-K_ T_psk_Mask and Identity_Mann _Mak preserves all neurons. By comparing these three settings, one can easily tell the functional difference between Top-K and non Top-K fired neurons (See results in Figure 3)
Top-Too___ask
(dentity_Mask)
only selects

---

3.4 Shape Bias Benchmark
To demonstrate our proposal that the Top-K responses are encoding the structural and shape informa- tion, we silence the non-Top-K responses during inference when using pre-trained CNNs. To test the networks' shape bias, we directly integrate our code into the benchmark provided by [00. The benchmark contains a cue-conflict test which we use evaluate the Top-K operation. 'he benchmark also includes multiple widely adopted models with pre-trained checkpoints and human psychological evaluations the same cue-conflict testing images.
4 Results
4.1
Top-K Neurons Encode Structural Information
To test the hypothesis that the shape information is mostly encoded among the Top-K significant responses, whereas the non-Top-K responses are encoding primarily textures, used the method described in Section 3.2 and compared the texture images synthesized with and without the Top-K responses for the computation of the Gram matrix. Figure compares the TS output obtained by matching the early layers and the higher layers of the VGG16 network in the two conditions. One can observe that ablation of the Top-K responses eliminated much of the structural information, resulting in more texture images. We conclude from this experiment that (1) Top-K responses are encoding structural parts information; (2) Non Top-K responses primarily encoding texture information.
Original
ii. w. Top-K + Non Top-K
al. T5 w. Non Top-K
Original
TS Top-K Non Top-K
81. TS Non Top-K
Texture Synthesis (TS) using 2 shows the original image, shows the results of Sw/ Top K with both Top-K and Non Top-K activation intact, iii. shows the TS results Swis Top K with Top-K activation deleted before performing TS.
Figure 2:
To provide further insights about the different information Top-K and non Top-K neurons are encoding, we show another qualitative demonstration in Figure 3 where we optimize images that would excite the Top-K neurons alone and the non Top-K neurons alone respectively (See full description in Section 3.3
Original
Reconstruction
Reconstruction Top-K only
Reconstructio Non Top- only
Image
w/ Top-K Non Top-K
Figure 3: Visualizing Top-K and non Top-K neurons through optimizing input images to match their activation.
From Figure 3 it is clear that optimizing images to match the Top-K fired neurons yields high level scene structures with details abstracted away while optimization efforts to match the non Top-K fired neurons produce low level local textures of the target images. Together, provide evidence to
5

---

support our hypothesis that it is the strong firing neurons in the convolutional neural networks that provide structural information while the textures are encoded among the weakly activated neurons. Next, we demonstrate that this phenomenon will result in improved shape bias in both analysis and synthesis tasks.
4.2 Top-K Responses already have Shape Bias without Training
We test the Top-K activated CNNs with different degrees of sparsity the shape bias Benchmark proposed from [10). This benchmark evaluates the shape bias by using a texture-shape cue conflict dataset where the texture of an image is replaced with the texture from other classes of images. It defines the shape and texture bias in the following ways:
correct shape recognitions
# of correct recognitions shape bias
texture bias of correct texture recognitions
# of correct recognitions
has been shown in previous work 10 that CNNs perform poorly on shape-based decision tests whereas human subjects can make successful shape-based classification nearly all the evaluated cases. This results in CNN models having relatively low shape bias scores while humans have close shape bias score. Interestingly, it has been observed that Vision Transformers (ViT) model family has attained significant improvement in shape bias [10]
sparseAlexNet 5%
sparseVGG16 5%
Shape bias
sparseAlexNet,
10%
sparseVGG16 10%
sparseAlexNet, 20%
sparseVGG16
BDV//GG662200 20%
0.5
sparseAlexNet 30%
sparseVGG16 30%
sparse/lexNet, 40%
sparseVGG16 40%
sparseAlexNet_
50%
sparse
0.0
AlexNet
VGG16
VIT-L
humans
models
CLIP: ViT-B(400M)
Figure 4: Overall shape bias of sparse CNNs, CNNs, Transformer and humans
Adding the Top-K operation to simple pretrained such as AlexNet or VGG16 alone already can already induced a significant increase in shape bias, as shown in Fig 4 With the sparsity knob equal to 10% and 20%, the Top-K operation alone appears to achieve as much more shape bias as the state-of-the-arr Vision Transformer models the cue-conflict dataset, leading further support to the hypothesis that Top-K sparsity can lead to shape bias.
We plot the best of the Top-K sparsified AlexNet and VGG16 for each evaluation of 16 object classes in Fig(5 We can observe that sparsity constraint improve shape-biased biased decision -aaiin for most of the object classes, bringing the performance of the pre-trained model closer to human performance. With the proper settings of sparsity, certain classes (e.g. the bottle and the clock category) could attain human level performance in shape-biased scores.
However, we should note that the confidence interval is quite large, indicating that the network performs differently across the different classes. closer look shape bias for each class is shown in Figure
4.3
Top-K training induces Shape Bias in Recognition Networks
To evaluate the shape bias can be enhanced by training with Top-K operation, we trained ResNet- 18 [13] on different subsets of ImageNet dataset [8]. Each subset contains randomly selected 10 original categories from ImageNet,for all the training and evaluation. Every experiment is run three times obtaín an error bar. During the evaluation, we employ AdaIn style-transfer using programs adopted from [8] to transform the evaluation images into texture-ablated form as shown Figure The original texture of the image is replaced by styles of non-related images using style transform.

---

conflict Frection of 'shape' decsions
Fraction 'shape' decisions
sparseAlexNett 5%
sparscAlexNet 10%
sparseAlexNet 20%
sparseAlexNet 30%%
sparscAlexNcc 40%
sparseAlexNet_ 50%
AlexNet
Shage
sparseVGG16_55
sparseVGG16 10%
sparseVGG16_
sparseVGG16_30g
sparseVGG16 40%
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.9
0.2
0.2
0.6
0.7
spurseVGG16_
Traction teeture decisions
Fraztion
VGG16
Figure 5: The classification result on the Shape Bias Benchmark proposed from [0]. This plot shows the shape bias of sparse CNNs, CNNs and humans different class in texture-shape dataset. It also show the shape bias different sparsity degree. e.g. 5% means that only top 5% activation value would be passed to the next layer. Vertical lines means the average value.
This allows evaluate how much trained model is biased toward the original texture instead of the underlying shape.
(a)
(b)
(c)
(d)
(e)
(f)
Figure 6: Evaluating shape bias of the network with stylized ImageNet subsets. Three pairs images presented sampled from the our evaluation datasets. Specifically, transfer (a) (b), (c) (d) and (e) + (f) by AdaIN [18] and keep the original class labels. During the evaluation, the transferred images presented instead of the original test image measure the network's s texture bias sensitativity.
+
In this experiment, we train classification networks with two non-overlapping subsets of the ImageNet, namely IN-S, and IN-S2. We select the categories that are visually distinctive in their shapes. The details about the datasets can be found in the Supplementary Information. We trained ResNet-18 models over the selected IN-S, and IN-S2 dataset with the standard Stochastic Gradient Descent (SGD, batch size 32) and a cosine annealing learning rate decay scheduling protocal with Ir starting from 0.1. The same optimization is applied to ResNet-18 models with 20% spatial Top-K layer added after the second bottleneck block of the ResNet- 18. All models are then evaluated on the styleized version of IN-S, and IN-S, evaluation dataset after trained with 50 epochs.
Top-1 Acc.
IN-8,
Stylized-IN-S,
IN-Sa(t)
Stylized-IN-S; (f)
ResNet-18 18
[B]
87.8 +: 0.5
49.3 # 1.5
81.3 1.7
52.4 =2.2
ResNet-11 w. Top-K during training
89.4
55.4 0.8
83.4 0.9
59.7 0.6
ResNet-18 18 w. Top Mean Rpl during training
84.9 +: 0.3
56.8 1.7
75.5 + 2.5
53.1 + 1.0
Table 1: Evaluation for models trained on IN-S, and IN-S, datasets, each of which consists 10
classes of all train/val data from ImageNet- 1 dataset g
Table shows that (i) the classification accuracy on the original evaluation dataset doesn't drop: mean top-1 accuracy of 87.8 (baseline) v.s. 89.4 (w. Top-K) on IN-S, and 81.3 (baseline) v.s. 83.4 (w.

---

K)p-K) on IN-Sa respectively even when we push the sparsification to K= 20%; (ii) The shape bias improves significantly: mean top-1 accuracy of 55.4 (w. Top-K) v.s. 49.3 (baseline) on Stylized-IN-S, and 59.7 (w. Top-K) v.s 52.4 (baseline) on Stylized-IN-S, respectively. This supports our conjecture that the sparse code could introduce more shape bias, in comparison to the dense representation, during learning.
To further investigate why Top-K might induce shape bias, we evaluate whether the values of the Top-K responses matter by compressing the information in each channel to the mean responses of the Top-K responses of that channel at the Top-K K positions. This reduces the information of each channel to only a binary mask indicating the Top-K responding locations and single float number that relays the channel's weighted contribution to downstream layers, effective compressing the communication rate by 3 orders of magnitude (See Section 3.1 for detailed description of the Top K Mean Rpl).
Despite the enormous amount of data compression by replacing the Top-K values with the mean, the network can still maintain the shape bias comparable to the normal ResNet- 18 baseline (as indicated by the improved or on-par performance on the Stylized-IN-SIp w. Mean Top-K in Table D. This suggests that the spatial map of the Top-K activation is more important than the precise values of the Top-K responses. This suggests significant amount of the object shapes features are actually encoded in the occupancy map of significant neural activities, i.e. the binary mask of the Top-K.
between ResNet-18 and ResNet- 18
4.4 Towards Shape Biased Few Shot Image Synthesis
i. Jeep
ii. Fish
iii. Train
iv, Table
Ours
ii FastGAN
(a) Image samples from four training categories.
(b) Qualitative Comparison on Jeep-100 Synthesis.
Figure 7: Few shot image synthesis datasets and qualitative comparison results between our methods and FastGAN 27
Humans are great few-shot learners, i.e. learning from few examples. This ability might be related our cognitive ability learn generative model of the world that allows us reason and imagine. Recurrent feedback in the hierarchical visual system has been hypothesized to implement such a generative model. investigate whether the Top-K operation also induces shape bias in the synthesis network for the few-shot image synthesis task. We hypothesize that shape bias can benefit few-shot image synthesis by emphasizing on structural and shape information. Figure 77b shows that state-of-the-ar oh-the-art few-shot synthesis program (FastGAN (27) suffers severely from texture bias. We found that introducing Top-K operation in the fourth layer (the 32 x 32 layer) in FastGAN, sigmificant improvement in the synthesis results can be obtained on datasets selected from ImageNet [8] (100 samples each from four diverse classes, synthesizing each class independently, see Supplementary for details) shown in Table 7 Images from ImageNet possess rich structural complexity and population diversity. To be considered to be a good synthesis, generated samples would have to achieve strong global shape coherence in order to have good matching scores with the real images. A better quantitative evaluation result would suggest the emergence of a stronger shape or structural bias in the learned representation. Samples of the four classes shown in Figure 7a
To assess the image synthesis quality, we randomly sample 3000 latent noise vectors and pass them through the trained generators. The generated images are then compared with the original training images in the inception-v? encoder space by Fréchet Inception Distance (FID [66) and Kernel Inception Distance (KID [5] scores and documented in Table Each setting is run 3 times to produce an error bar.
First, the synthesis quality measurements for adding the Top-K operation to the FastGAN network show consistent improvement terms of FID and KID scores in Table Figure 7 shows that the Top-K operation leads the generation of objects (e.g. the jeep) that are more structurally coherent and distinct, compared to the original FastGAN. Specifically, we observe 21.1% 1% improvement on FID scores and 50.8% improvement on KID scores for the Jeep-100 100 class when K= 5% sparsity was

---

imposed during training (i.e. only keep 5% neurons active). Similarly, when 5% sparsity is imposed on Fish- 100 and Train-100 datasets the FID increased by 17.3% and 12% respectively and the KID performance is boosted by 48.5% and 33.4%. Lastly, we test the in-door table class which contains complex objects with many inter-connected parts and subparts. A K=15% sparsity leads to a gain of 9.3 % and 22.3% in FID and KID respectively for synthesizing similar in-door tables. Overall, our experiments show that introducing the Top-K operation a challenging few-shot synthesis task can significantly improve the network's generated results diverse set of complicated natural objects.
Jeep 100
Fish 100
Train 100
Table-100
Method
FID
KID*
FID 4
KID*
FID
KID* 1
FID 4
KID* 1
PasIGAN
27
49.0
1.4 12.0 + 1.9
46.2 18
13.4 0.6
46.1 188
11.2 + 0.8
67.2 + 0.1
19.7 - 0.3
FusGAN w. Top (ours)
38.7
0.5
5,9 4 0.7
38.2 14
4:
40.2 0.8
7.4 4 0.2
60.9 4:
15.3 =
Table 2: Few-shot Image Synthesis results measured in FID 17 and KID KID scaled by factor of 10 to demonstrate the difference.
[5] Note that KID) denotes
4.5 Parts Learning in Sparse Top-K
Finally, we study the training dynamies of the Top-K layer's internal representation. In Figure we can make the following observations. (1) By applying sparse constraint using Top-K, each channel is effectively performing binary spatial segmentation, i.e. the image spatial dimension is separated into either Top-K K region non Top-K territory. (2) Although there is no explicit constraint to force the Top-K K neurons to group together, the Top-K responses tend to become connected, forming object parts and subparts, training evolves.
Figure 8: Even though Top-K K Operation is not fully differentiable, the network is able to relocate the spatial activation mass smoothly towards a connected meaningful parts which eventually leads to component learning as shown in Figure
Figure 9: Synthesis network internal topk layers reveals semantic decomposition of parts and subparts.
We believe the development of this continuous map when training with Top-K operation might be due to two factors (1) CNN activations are locally smooth, i.e. two adjacent pixels in layer L,, are linked by the amount of overlap between their corresponding input patches in L,, 1, (2) Top-K increase the responsibility of each individual neuron to the loss function. When neurons and are being selected as Top-K, their responses are likely similar each other. However, their corresponding spatial location in the output has different semantic meanings, they will receive diverse gradients which will be then amplified by the increased individual responsibility. The diverging gradients would lead the value difference of two neuron i and j's gradients, resulting in one of them leaving the Top-K K group while only the semantic similar ones remaining the Top-K sets. This might suggest a principle we call neurons fire together, optimize together during CNN Top-K K training, which could lead to the observed emerging of semantic parts and subparts. This localist code could further connect to recognition by component theory 22 12 which constranints the representation by semantic parts segmentation, improving the network's 's focus on object components boundary, hence improving overall shape bias. In functional analysis of the brain, [29] also show that local smoothness constraint could lead to

---

the topological organization of the neurons, hinting that the hypothesized factors above could have neuroscientific grounding.
Top-K Sparsity Hyperparameter With the understanding from Section 4.5 we want to reiterate the importance of the sparsity hyperparameter we used. The amount the sparsity can be directly translated to "size of the parts". Thus, depending on the image type, the composition of the scene, the network architecture, and the layers which the Top-K sparsity operation is applied on, the results could be drastically different. We refers to the supplementary for more detailed ablation study.
5 Conclusion
In this study, we discovered that an operation inspired by a well-known neuroscience design motif of sparse coding induce shape bias in neural representation. We demonstrated this in object recognition networks and in few-shot image synthesis networks. We found that simply adding the Top-K sparsity operation
induce shape bias in pre-trained convolutional neural networks and that training of the CNNs and GAN with the simple Top-K operation can increase the shape bias further toward human performance, which makes object recognition variations and makes image synthesis generating structurally more coherent and distinct objects. Using texture synthesis, information, while the non Top-K responses carry more texture information. The observation that sparse coding operation can induce shape bias in deep learning networks suggests sparsity might also contribute to shape bias in human visual systems.
robust against texture
able demonstrate that Top-K responses carry more structural
6 Ethics Statement
This study investigates whether the sparse coding motif in neuroscience can induce shape bias deep learning networks. The positive results suggest that sparsity might also contribute to shape bias in the human visual systems, thus providing insights to our understanding of the brain. While deep learning ad advance science and technology, it comes with inherent risks to society. We acknowledge the importance ethical study in all works related to deep learning. better understanding of deep learning and the brain however is also crucial for combating the misuse of deep learning by bad actors in this technological arm race.
7 Acknowledgement
This work was supported by an NSF grant CISE RI 1816568 awarded Tai Sing Lee. This work is also partially supported by the graduate student fellowship from CMU Computer Science Department.
10

---

References
[1] Bahng, H., Chun, S., Yun, S., Choo, J., Oh, S.J., 2020. Learning de-biased representations with biased representations, in: International Conference on Machine Learning, PMLR. pp. 528-539.
[2] Baker, N., Lu, H., Erlikhman, G., Kellman, P.J., 2018. Deep convolutional networks do not classify based on global object shape. PLoS computational biology 14, c1006613.
[3] Barlow, H., 1961. Possible principles underlying the transformation of sensory messages. In W. A. Rosenblith (Ed.), Sensory Communication...
[4] Biederman, I., 1987. eecognition-by-coopooo
a theory of human image understanding. [5] Binkowski, M., Sutherland, D.J., Arbel, M., Gretton, A., 2018. Demystifying mmd gans. arXiv preprint arXiv:1801.014011
Psychological review 94, 115.
[6] Brendel, W. Bethge, M., 2019. Approximating cnns with bag-of-local
models works
supprssingy well imagenet, arXiv preprint urXiv:1904.00770
[7] Dehghani, M., Djolonga, J.. Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A., Caron, M., Geirhos, R., Alabdulmohsin, L., et al., 2023. Scaling vision transformers to 22 billion parameters. arXiv preprint uxivv2330.00544..
[8] Deng, J., Dong, W., Socher. R., Li, L.J. Li, K., Fei-Fei, L., 2009. Imagenet: A large- scale hierarchical image database, 2009 IEEE conference on computer vision and pattern recognition, leee. pp. 248-255.
[9] Gatys, L.A., Ecker, A.S., Bethge, M., 2016. Image style transfer using convolutional neural networks, in: Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2414-2423.
[10] Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F.A., Brendel, W., 2021. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems 34, 23885-23899
[11] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W., 2018. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 181 1.12311..
[12] Geng. Z., Guo, M.H., Chen, H., Li, X.. Wei, K., Lin, Z., 2021. Is attention better than matrix decomposition? arviv preprint arXiv:2109005553
[13] He, K., Zhang. X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778.
[14] Hendrycks, D., Dictterich, T., 2019. Benchmarking neural network robustness to common coruptiion and perturbations. arXiv preprint 122.11
[15] Hermann, K., Chen, T., Kornblith, S., 2020. The origins and prevalence of texture bias in convolutional neural networks. Advances Neural Information Processing Systems 33, 19000- 19015.
[16] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S., 2017. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems 30.
[17] Heusel, M., Ramsauer, H., Unterthiner, T.. Nessler, B., Hochreiter, S., 2018. Gans trained by two time-scale update rule converge to local nash equilibrium. arriiv 1706. 085000
[18] Huang, X., Belongie, S., 2017. Arbitrary style transfer real-time with adaptive instance normalization, in: Proceedings of the IEEE international conference computer vision, pp. 1501-1510.
[19] Krizhevsky, A., Sutskever, I., Hinton, G.E., 2017. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM 60, 84-90.
[20] Landau, B., Smith, L., Jones, S.S., 1988. The importance of shape in early lexical learning.
Cognitive development 3, 299-321.
[21] Laugros, A., Caplier, A., Ospici, M., 2019. Are adversarial robustness and common perturbation robustness independant attributes?, in: Proceedings of the IRFF/CVF International Conference on Computer Vision Workshops, pp. 0-0.

---

[22] Lee, D.D., Seung, H.S., 1999. Learning the parts of objects by non-negative matrix factorization. Nature 401, 788-791.
[23] Lee, S., Hwang, I., Kang, G.C., Zhang, B.T., 2022. Improving robustness texture bias via shape-focused augmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4323-4331
[24] Li, T., Li, Z... Luo, A., Rockwell, H., Farimani, A.B., Lee, T.S., 2022. Prototype memory and attention mechanisms for few shot image generation, in: International Conference on Learning Representations. URL: httpt:::pppppeeeeeeeeeeee
[25] Liao, N., Wang. S., Xiang, I.,, Ye, N., Shao, S., Chu, P., 2022. Achieving adversarial robustness via sparsity. Machine Learning. 1-27.
(26] Liao, R., Schwing, A., Zemel, R., Urtasun, R., 2016. Learning deep parsimonious representa- tions, in: Lec. D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (Eds.). Advances in Neural Information Processing Systems, Curran Associates, Inc. URL: https///roo/oceeeiinngg
[27] Liu, B., Zhu, Y., Song, K., Elgammal, A., 2021. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis, in: International Conference Learning Representa- tions.
[28] Liu, D.C., Nocedal, J., 1989. On the limited memory blgs method for large scale optimization. Mathematical programming 45, 503-528.
[29] Margalit, E., Lee, H., Finzi, D., DiCarlo, J.J., Grill-Spector, K., Yamins, D.L., 2023, A unifying principle for the functional organization of visual cortex. bioRxiv, 2023-05.
[30] Olshausen, B.A., 2013. Highly overcomplete sparse coding.
[31] Olshausen, B.A., Field, D.J., 1996. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607-609. URL: /tlL:/tttts:/doi.rrgg. (038/381607at) doi/10. 1038/ 0018/8110770
[32] Quiroga, R.Q., Reddy, L., Kreiman, G., Koch, C., Fried, I,, 2005. Invariant visual representation by single neurons in the human brain. Nature 435, 1102-1107.
[33] Simonyan, K., Zisserman, A., 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xiv: 1xxiv:1409. 1119.144.5550
[34] Smith, L.B., 2003, Learning to recognize objects. Psychological Science 14, 244-250.
[35] Szegedy, C., Zaremba, W., Sutskever, Bruna, Erhan, D., Goodfellow, I., Fergus, R., 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199
[36] Tang, S., Lee, T.S., Li, M., Zhang, Y., Xu, Y., Liu, F., Benjamin, T., Jiang, H., 2018a. Complex pattern selectivity in macaque primary visual cortex revealed by large-scale two-photon imaging. Current Biology 28, 38-48.
[37] Tang, S., Zhang, Y., Li, Z., Li, M., Liu, F., Jiang, H., Lee, T.S., 2018b. Large-scale two- two-photon imaging revealed spper-sparse population codes in the vl superficial layer of awake monkeys. clife.
[38] Wang, J., Zhang, Z., Xie, C., Zhou, Y., Premachandran, V., Zhu, J., Xie, L., Yuille, A., 2017. Visual concepts and compositional voting. arxiv: 1711. 04451
[39] Xing, X., Wu, T., Zhu, S.C., Wu, Y.N., 2020. Inducing hierarchical compositional model by sparsifying generator network. arXiv: 1 1909. 1909.04324.91
12


Reviews:
[{"review_id": "ugBDlSCyGN", "reviewer": "Reviewer_24RM", "summary": "This paper proposes enforcing sparsity within a neural network through a top-K mechanism and demonstrates that this leads in an emergent fashion an emphasis on shape rather texture in the learned representations of the network. This in turn leads to improvements in robustness for network behaviour.", "questions": "- In Figure 6(b) is there a difference between Ours 1 and Ours 2, or is that simply two different examples of output?\n\n- Does the use of a top-K mechanism lead to some of the network essentially becoming redundant (i.e. always ignored) and wasted computation?", "limitations": "The paper does not have any obvious limitations that were not discussed.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is clearly written and laid out.\n\n- The paper presents a novel and innovative contribution to a challenging goal in deep learning (namely, shifting neural networks more toward shape representations rather than texture representations). Although this topic has been studied before, as the paper points out existing methods to achieve this have significant drawbacks (such as the practicality of style-transfer augmentations), whereas the method proposed in this paper is extremely practical to implement.\n\n- The use of multiple different styles of experimental methods helps round out the conclusions (for example, Section 4.1 is a relatively straightforward qualitative experiment that is nevertheless compelling, naturally leading to the more comprehensive later experiments.\n\n- ", "weaknesses": "- I found it wasn't always clear whether Top-K was being applied to all layers or a subset of layers. Section 4.4 explicitly mentions that Top-K was used only in the fourth layer; why this layer, and how was this determined?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review and constructive feedback. The reviewer thinks our paper is technically solid with excellent contribution to a challenging goal in deep learning. We carefully prepare the following response aiming to address the reviewer’s questions:\n\n- Top-K layer specification:\n\n   We apply the Sparsity layer in a subset of the network. It is based on the intuition that the brain utilizes sparsity for long range communication but can allow local dense computation. We divide the networks into chunks where within each chunk the neuron’s activities are allowed to be dense (keep original) but the communication across different chunks is set to be sparse. To be specific, in the VGG Network, we divides the network into 5 chucks and apply the Top-K layer after each MaxPooling layer. Specifically, across a total of 31 feature layers in VGG, we apply the Top-K layer after the 5th, 10th, 17th, 23th, 31th original layers. In AlexNet, we also apply the Top-K layer after the MaxPooling except adding 2 additional Top-K layers at the 8th and 10th layer to make the sparse layer more evenly distributed. In the ResNet-18 training, we only apply the Top-K layer after the second layer to demonstrate the concept of sparse specialist code. We also apply the Top-K layer only in the 4th layer of the GAN generator as we observe empirically that it induces the best performance across other settings (such as applying multiple Top-K layers or the less severe sparsity). \n\n    Interestingly, our new experiments as shown in the Figure 1 in the general response indicates that ViT architectures seems to requires more dense activation, as we can observe that a moderate to high density (keep 50% - 80% of the original ranking neurons) yield the best performance for most of the classes (best mean shape score is achieved by keeping 60% of the original activation).\n\n- Difference in  Figure 6(b): We apologize for the confusion. Ours 1 and Ours 2 are from the same setting. We add more specification and synthesis examples in the camera ready version. \n\n- Redundancy in the network (Please see the general response).\n\n"}, {"review_id": "e9I6D4PoUK", "reviewer": "Reviewer_muuS", "summary": "Current deep learning models are known to have more texture bias for object recognition tasks while humans have more shape bias. They test the hypothesis that sparse coding may improve shape bias in deep learning models. Using Top-K operations, they show improved accuracy in pre-trained vison models as well as improved shape bias in models trained from scratch. Furthermore, in few-shot image systhesis tasks, models with sparsity constraints result in more structurally coherent images. The Top-K operations may be imposing binary masking on object parts, which can induce shape bias. \n\n", "questions": "- Section 4.2: How does the number of correct recognitions change when sparsity is imposed on the models? Are we sacrificing accrucy a lot for better shape bias?\n- Figure 3: For the boxplots, please explain the distribution, whether it is over multiple image categories or multiple seeds for applying sparsity, etc. \n", "limitations": "Currently, the study tested only convolutional architectures. Otherwise, limitations are adequately addressed. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Connecting sparsity with shape bias is novel.\n- The authors examine both classification and generative models based on the sparsity operations.\n- Their study has implicaitons in both building computational models and understanding human vision. Their empirical results suggest that sparsity can lead to improved shape bias, making the models potentially make judgments closer to humans. Conversely, their study suggests sparse encodings may play a role in inducing shape bias in human vision. \n- Presentation is clear with helpful figures. I enjoyed reading it.  \n\n", "weaknesses": "- Section 4.1: Additional comparisons with Ts w. Top-K only will be useful. It is unclear whether less structural information in TS w. Non Top-K is just due to a lack of representational power.  \n- Section 4.2: As noted in line 160, Vision Transformers show improved shape bias. How would you apply Top-K operations to Vision Transformers and do you expect to obtain more shape bias for those models? Since they are a popular model family, I think it's worth adding experiments or adding notes on them.\n- Section 4.3: Does the Top-K training lead to many dead neurons, essentially reducing the network size, which would work as an implicit regularization? \n- Section 4.5:  The idea suggested here seems related to models developed with topographic constraints (Margalit et al., 2023), where spatially smooth response patterns are learned. Comments on these models will be useful.\n- Section 4.6: The discussion here implies that an ideal sparsity rate may be different for each layer. Applying different K values depending on layer depth or not applying sparsity to some layers in the experiments (e.g. Figure 3) will be interesting to test the hypothesis further.\n- While I do think the study suggests an interesting perspective on shape bias using sparsity, the study may have less impact if models trained on larger datasets show more shape bias. For instance, in Figure 3, I wonder ConvNexts or ViTs trained on LAION-2B would give a stronger baseline.       \n\n\nMinor comments\n- Line 53 (\"indeed can indeed introduce\"): indeed is repeated\n- Line 161 (\"already can already induced\"): already is repeated\n- Figure 4: I assumed it indicates humans, but the label for purple markers is missing. \n   \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. The reviewer think our paper is novel and solid. Here we prepare the response to address the reviewer’s questions. \n- Comparison with  Ts w. Top-K only. As for the limitation of the response pages, we put the further experiments under our anonymous github repository: \nhttps://anonymous.4open.science/r/nips2023_shape_vs_texture-1057\n- Apply the Top-K to the transformer architecture. We conducted experiments and observe similar improvements on the shape bias score when applying sparsity to ViT. Please refer to our general response on the details. \n- Dead neurons and implicit regularization. Yes, indeed. We also address this issue in our general response. \n- Margalit et al., 2023: We will certainly include this work as a related work in our camera ready version. \n- Different layer of sparsity. Yes, we indeed apply different Top-K to different layers. We intend to mimic the long range communication yet local dense computation of the brain, i.e. we divide the network to several chunks and apply sparsity at the connections of these chunks. In VGG for example,  we apply the Top-K layer after the 5th, 10th, 17th, 23th, 31th original layers (after each MaxPooling Layer). In AlexNet, we also apply the Top-K layer after the MaxPooling except adding 2 additional Top-K layers at the 8th and 10th layer to make the sparse layer more evenly distributed. For the synthesis network, we only apply the sparsity on the 4th layer out of 8 layers as we observe that applying severe sparsity near the output layer would yield suboptimal performance. \n- Stronger baseline. We agreed on a strong baseline for evaluation. We included the ViT results in the general response. We believe our method is orthogonal to the scaling approach and our further study shows it can be combined with larger models to further improve the shape bias. However, due to the time limitation, we were unable to experiment with other large models that are trained on LION2B for example. We will try our best to apply the proposed method on the extremely large scale trained models in the following discussion period. \n- Accuracy v.s. Shape Bias. Our main manuscript Section 4.3 indicates that a network that is trained with Top-K operation can achieve competitive accuracy while increasing the shape bias robustness. In the general response Table 1, we can observe that Top-K training can lead to significant improvement compared to Top-K inference (see General Response Table 1 (2) v.s. (4)).\nIn terms of accuracy and shape bias trade-off, we agree with the reviewer that applying sparsity directly to inference could lead to degraded classification results. We interpolate between no sparsity and severe sparsity in the general response Figure 2(b). We vary the number of sparsity layers from 0 to 5 and observe that there is a sweet spot for balancing the shape bias and the classification accuracy. Therefore, in practice, we recommend training or fine tuning with Top-K operation to achieve the best performance and shape bias simultaneously. \n- The boxplot in Figure 3 and typos. We appreciate the reviewer’s question. To be specific, the distribution of the box plot is over different image categories. We will make further clarification and also correct the typo in our camera ready version. \n\n"}, {"review_id": "K8WUXQosUl", "reviewer": "Reviewer_ZUQs", "summary": "The paper proposes to use activation sparsity induced by the top-k operation to address the recently identified hard problem in DNNs, the texture bias. The operation is simple yet highly effective, as it shows good structural encoding capability (Sec 4.1), shape bias (even without training, Sec 4.2) and parts/subparts learning dynamics (Sec 4.5). Finally the operation (with training) is validated with tasks including style robustness (ResNet-18, Sec 4.3) and few-shot image synthesis (FastGAN, Sec 4.4) on ImageNet subsets, where it also shows promising results.", "questions": "1) Does the value of non-top-k units (now set to 0 in Eq 1) affect the experimental results (e.g. by overly shifting the mean or other statistics)? Shouldn’t some form of corrective offset (or learnable value other than 0) be applied?\n2) How does the top-k without training performance compare to others in Table 1?\n3) What do the colors mean in Fig 7? ", "limitations": "The proposed method itself doesn’t seem to have serious limitations as far as I can tell. However, the limitations of the standard protocols & datasets [10] currently used to analyze the results could be discussed in the paper.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ [Originality] The paper is sufficiently novel in my opinion. Although the idea of top-k sparsity has been commonly used in various applications (training/inference acceleration, brain-inspired algorithms, etc.), the fact that the simple idea works for this new, challenging topic should be interesting to the community.\n+ [Clarity] The paper is clear and easy to follow. The effort to clearly explain the motivations (e.g. Sec 4.4) and results (e.g. Sec 4.5) of the experiments is relatively noteworthy.\n+ [Quality] The paper is overall of good quality in my opinion. The proposed method is well motivated, simple and effective. The evaluation, though purely empirical, is relatively comprehensive and promising.\n+ [Significance] Given the simplicity and effectiveness of the proposed method, and the relevance of the topic, I think this work is significant and could be valuable to many in the community.", "weaknesses": "- [Evaluation, Minor] To further improve the impact of the paper, the authors could consider improving the following aspects.\n1) Fig 2 and A.1 are both valuable yet limited. The authors could consider including larger-scale experiments/visualization and quantitative analysis of the results (e.g. textureness [38]).\n2) The authors could consider including experiments combining with existing solutions (e.g. data augmentation based, given the orthogonality) on the benchmark [10, 39] to demonstrate this work’s potential to further boost performance, including on ViT-based architectures.\n\n[38] The Synthesizability of Texture Examples, CVPR, 2014.\\\n[39] https://github.com/bethgelab/model-vs-human/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the kind advice and constructive feedback. The reviewer consider our paper is technically solid and has novel contribution as well as good writing. We prepare the following feedback to address the reviewer’s remaining questions:\n\n- Quantitative analysis of the visualization results. We agree with the reviewer however due to the page limits, we will put the further quantitative results in our supplementary. \n\n- Combining the existing solution. We appreciate the comment of the reviewer. Indeed, we generalize our method to more architecture (ResNet and ViT) in the general response. We will apply our methods to SIN in our anonymous github repository during the discussion period to further demonstrate that our method can be combined with existing solutions. \n\n- Non top-k value offset. Our motivation is to mimic the efficient coding principle utilized inside the brain. A learnable offset would introduce further optimization instability. Also during training, setting the non-top-k to zero would make the gradient to become zeros automatically. However, it might improve the inference accuracy if we keep the distribution mean statistics during Top-K inference. However, due to the limited time, we haven’t observed any significant difference in our preliminary experiments and we will do more tests in the following days. \n\n- TopK without training. We appreciate the reviewers’ point. Please refer to our general response Table 1 for more details. In general, we observe that our optimal parameters for Top-K training is not optimal for inference but training with Top-K significantly improves both shape bias and accuracy (as shown in (2) and (4) in the general response Table 1). \n\n- Colors in Figure 7. The white indicates positive and black indicates negative while the gray indicates 0. \n- The limitation of the standard protocols & datasets includes the fixed and limited category as well as its practically in the real world. We will include this limitation and propose more comprehensive in the future work.\n"}, {"review_id": "pp37QTUZbi", "reviewer": "Reviewer_oHBy", "summary": "This work is motivated by the observation that deep neural networks may have a bias towards texture in classification decisions. The paper proposes a top-k sparsity selection process that is shown to result in emergence of shape sensitive neurons while improving performance across a variety of tasks.", "questions": "1. How does the rank based selection for sparsity compare with 'softer' methods for inducing sparsity (e.g. L1 regularization)?\n\n2. Is there any means of knowing the proper value for the k hyperparameter?", "limitations": "The authors are clear about limitations of the work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The demonstration that shape encoding occurs when the proposed sparsification is applied even in the absence of training is an important result and contribution.\n2. The paper demonstrates the value of this operation across a variety of tasks from recognition to synthesis and connections to neuroscience.\n3. The paper is generally well presented and gives a variety of examples that present a strong case for the claims made.", "weaknesses": "1. There are some editorial changes that would benefit the paper (e.g. which numbers are bold in tables, when is SD bold etc.) and some places in the text where the description could be refined.\n2. It would be nice to see something like figure 4 for the ResNet variants given their importance in the literature\n3. Some references might be added (e.g. Md Amirul Islam, Matthew Kowal, Patrick Esser, Sen Jia, Björn Ommer, Konstantinos G. Derpanis, Neil D. B. Bruce, Shape or Texture: Understanding Discriminative Features in CNNs, ICLR, 2021.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's kind comments and insightful feedbacks. We address the reviewers' concerns as follows:\n\n- ResNet variant of Figure 4. We thank the reviewer for bringing up this point. We conduct experiments on ResNet-50 (See general response Figure 1 (a)) and also on ViT (General response Figure 1 (b)). \n\n- L1 regularization: Indeed, we experimented with another soft version of sparsity, i.e. apply L1 norm to the loss function and document the results in the general response Table1 (5). We observe that the L1 induced sparsity introduces on-par shape bias as the Top-K do. We will perform more experiments on the effect of L1 activation loss on other tasks such as synthesis in the future. \n\n- Hyperparameter K. We currently perform grid search of the parameters but keep the Top-K value the same across different layers. We do observe that in ViT architecture, the fine shape bias would depend on the highest sparse level, e.g. if layer 1 keeps 20% of the activation and layer 2 keeps 40% activities, the final performance would be very similar to the one that both keep 20% of the activities. The underlying reason is still under investigation. \n\n- Editorial suggestion. We will further correct these minor errors and incorporate the suggested related work. \n\n"}]

Author rebuttal:
We appreciate the reviewers’ kind responses and constructive comments. In the following, we provide further experiments to support our paper. 

First, we generalize section 4.2 to ResNet-50 and ViT-B architectures. The results are documented in Figure 1 of the general response. The ResNet-50’s sparsity definition is the same as AlexNet and VGG. For ViT-B, we reshape the activation response from [n, h * w, d] to [n, d, h * w]  and apply the Top-K selection over dimension 2 before the activation is passed through the multiple head attention (Note that h and w is the height and weight of the latent tensor after reshape it to 2d, for ViT-B with patch size 16 on the 224x224 images, h=w=14). Please refer to our updated code for precise implementation. We observe that the similar shape bias gain from applying Top-K operation to ResNet-50 and ViT-B although ViT-B needs denser activation compared to the ConvNets.

We further add experiments for 4.3. Specifically, we add experimental results (2), (3) and (5) as highlighted in blue color. We first test how the model behaves when we apply Top-K operation on Non-Top-K trained models as Reviewer ZUQs points out. As the choice of hyper-parameter K in the proposed Top-K operation is critical to the model's performance, we apply the same Top-K parameter setting for (2) and (4). It can be observed that inference with Top-K with sub-optimal setting (2) would sometimes induce performance deficit but the models' performance will improve when applying the same Top-K sparsity during training (3). Second, we test the hypothesis that whether other soft sparsity constraints like L1 regularization would have the same effect (as suggested by Reviewer oHBy). We introduce a L1 term $ + \lambda || \texttt{NeuroActivation} ||_{1}$ during training (we tried $\lambda$ choice of $\{10, 1, 0.1\}$ and found $\lambda=1$ works the best.) and provide its results in (5). We can observe that (5) has similar performance as trained with Top-K layers (4), suggesting that L1 regularization induced sparsity could also help for improve the out-of-domain generalization of the model.

Next, we investigate the internal activation when applying Top-K during training. In General Response Figure 2(a), we find that the Top-K training would lead to “dead neurons”, i.e. the neurons that will not be used across 100k inputs. We produce this plot by logging the frequency of the times each neuron is used. Specifically, we take the GAN generator trained in Section 4.4 and pass 100k random noise from the standard normal distribution. As the network is trained to map the noise to a real image (jeep class), we analyze the activation tensor inside the Top-K layer. We consider each neuron to be used once if it’s inside the Top-K selection. We normalize this utilization number by 100k and get the utilization frequency for each neuron. We can see that 15% of neurons are alive and the rest doesn’t respond at all. Therefore we can understand our proposed method as an implicit regularization to reduce the model size. 

Finally, we also provide analysis between the trade-off between shape bias gain and the classification accuracy sacrifice (General Response Figure 2(b)). We vary the number layers we install Top-K during inference on the AlexNet and evaluate the mean shape score as Figure3 in the main text and the normal classification accuracy. We can observe that there is a sweet spot that can balance the shape bias score and the classification accuracy. Combining with the experiments from general response Table1 (2) and (4), where Top-K training could increase both classification accuracy and the shape bias score, we recommend training or fine tuning with Top-K to obtain improvement on both classification accuracy and the shape bias robustness. 


Meta-review: The paper shows that enforcing activation sparsity in deep neural networks biases their decision-making towards using shape cues, similar to human perception. All reviewers agree that it is a strong paper that makes a valuable contribution. They noted some minor concerns which were addressed well during the response and discussion period.
Decision: Accept (oral)