Comparative Analysis between Vision Transformers and CNNs from the view of Neuroscience
Authors: Xiuyuan Hu
Hao Zhang
Yang Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Vision Transformer
CNN
neuroscience
sparsity

Abstract: Neuroscience has provide many inspirations for the development of artificial intelligence, especially for neural networks for computer vision tasks. Recent research on animals' visual systems builds the connection between neural sparsity and animals' levels of evolution, based on which comparisons between two most influential vision architecture, Transformer and CNN, are carried out. In particular, the sparsity of attentions in Transformers is comprehensively studied, and previous knowledge on sparsity of neurons in CNNs is reviewed. In addition, a novel metric for neural sparsity is defined and ablation experiments are launched on various types of Transformer and CNN models. Finally, we draw the conclusion that more layers in models will result in higher sparsity, however, too many heads in Transformers may cause reduction of sparsity, which attributes to the significant overlap among effects of attention units.

Under review as a conference paper at ICLR 2023
COMPARATIVE ANALYSIS BETWEEN VISION TRANSFORMERS AND CNNs FROM THE VIEW OF NEUROSCIENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Neuroscience has provide many inspirations for the development of artificial intel- ligence, especially for neural networks for computer vision tasks. Recent research animals' visual systems builds the connection between neural sparsity and an- imals' levels of evolution, based on which comparisons between two most influ- ential vision architecture, Transformer and CNN, are carried out. In particular, the sparsity of attentions in Transformers is comprehensively studied, and previ- ous knowledge on sparsity of neurons in CNNs is reviewed. In addition, a novel metric for neural sparsity is defined and ablation experiments are launched on var- ious types of Transformer and CNN models. Finally, we draw the conclusion that more layers in models will result in higher sparsity, however, too many heads in Transformers may cause reduction of sparsity, which attributes to the significant overlap among effects of attention units.
INTRODUCTION
Visual perception is not only the most significant kind of humans' perception, but also the most typ- ical characteristic of higher animals' intelligence As a consequence, computer vision becomes one of the most high-profile research fields in the history of artificial intelligence, in which various ma- chine vision tasks were uniformly defined for practical applications in the past several decades, and numerous algorithms and models emerged improve performance of computers all vision architectures, CNN (convolutional neural network) is the most influential one, which lead machine learning enter the deep era, and dominated almost all the fundamental vision tasks in the 2010s, including image classification (Krizhevsky et al. 2012 He et al. 2016 Tan & Le 2019), object detection (Redmon al. 2016 Ren et al. 2015 al. 2017) and semantic segmentation Ronneberger et al. 2015 Chen et al. 2018).
them. Among
CNN architecture was initially inspired by studying animals' visual system. Through biological ex- periments on mammals (one of the most evolved species in the animal kingdom) (Hubel & Wiesel] 1959,, some essential properties of visual systems were observed, such as hierarchical structure, receptive field and translation invariance. These discoveries laid the foundation for the design of CNN architecture, which, in turn, demonstrated its striking performance firstly in vision tasks (Le- Cun et al. 1989). And in recent years, some works concentrating on comparison between CNNs and higher animals like humans have been launched, providing helpful points for research on inter- pretation of deep learning and brain-inspired intelligence (Geirhos et al. 2020).
Starting from 2020, Transformer architecture began to replace CNN as the new focus of research in computer vision field. Though Transformer had swept the natural language processing field be- fore that (Vaswani et al., 2017] Devlin et al. 2019 Brown et al. 2020), models applying attention mechanism could not surpass the performance of Resnet-based CNNs (He et al.] 2016 Xie al.) 2017 Tan & Le 2019; Radosavovic et al.) (020) in vision problems. ViT (Vision Transformer) put forward in Dosovitskiy al. (2021), the milestone labeling the new era of computer vision, which depends completely on attention mechanism and has nothing to do with convolution, became state
For animals, lower and higher are descriptions for relative levels of evolution of biological complexity. For instance, primates are higher than non-primate mammals, mammals are higher than other vertebrates and vertebrates higher than invertebrates.

---

Under review as a conference paper at ICLR 2023
of the art in the task of image recognition scale (represented by ImageNet (Russakovsky al., 2013). After that, hundreds of works on computer vision based on Transformer architecture have been published. contributing
architecture (Liu et al. 2021; Wang al., [02T] Chu et al., 2021), novel training techniques (Touvron et al. 2021 2022), ) expansion for other tasks
innovations
(Carion al. 2020 Chen et al.. 2021b; Jiang et 2021 Chen al. (021a), etc.
Attention mechanism is also recognized as an essential property of animals' perception, therefore, some researchers have attempted to observe and study Transformer with prior knowledge of bio- science. Meanwhile, since Transformers perform better than CNNs currently in many tasks, people tend to find evidence supporting that Transformer instance. Tuli et al. (2021) proposes that Transformer is more similar to humans' visual system in terms of behavioral analyses. However, this statement is not well supported, since there are many other properties of humans' and animals' visual systems remaining not having been measured and analysed in vision models.
more advanced architecture than CNN. For
Inspired by the recent research about sparsity in animals' visual system, we discuss the sparsity of attentions in Vision Transformers in depth, and compare with sparsity of neurons in CNNs set of vision models, including classic CNNs and Transformers of different configurations. From the experimental results, the conclusion is drawn that adding layers to models will enhance the effect of sparsity, but adding heads to Transformers may play the opposite role, when the number of heads is too large. Specifically, our contributions mainly include:
through systematic experiments
In section some related works reviewed.
section 3 sparsity attentions in Vision Transformers is discovered and strictly defined, and its distribution is analysed from different perspectives.
In section previous works on sparsity of neurons in CNNs and that in animals' visual systems are reviewed.
In section ablation experiments and metric for neural sparsity are designed, and exper- imental results are reported and analysed.
Please refer Appendix A for experimental details, and codes for experiments are publicly
available at gigggghhhhugg.g.. com http:////maaaataatooooooo/oeeeee for- ICLR-22 2023
RELATED WORKS
2.1 COMPARISONS BETWEEN TRANSFORMERS AND CNNS
Intuitively, Transformers have less bias for vision than CNNs, which is generally acknowledged. Besides, Raghu et al. (2021) points out that Transformers have more uniform internal representa- tions than CNNs, and depend more on dataset scale. In addition, it is observed by Park & Kim (202)) that MSAs (multi-head self attentions) are low-pass filters, while convolutions filters, so they are complementary to some degree. Similarly, Zhao et al. (2021) argues that hybrid design containing both convolution and Transformer modules is better than either one. Moreover, theoretical proof is given in Li et al. (202T) that MSA layer with enough heads can perform any convolution operation. And in terms of behaviors, Bai al. (202T) claims that Transformers are not more robust than CNNs, and those opposite results obtained by previous works may be caused by unfair experimental settings.
high-pass
2.2 ANALYSIS OF NEURAL NETWORKS FROM THE VIEW OF NEUROSCIENCE
Understanding of brains and that of artificial networks always promote each other, in which observa- tions in neuroscience have provide a lot of inspirations for design both algorithms and hardware RRoy et al. 2019). Besides the history that the study on mammals' visual systems contributed to the development of visual computing, Marblestone et al. (2016) puts forward several hypotheses of mechanism of humans' brains, which may guide researchers to novel directions of network mod- eling. Additionally, Yang et al. (2019) finds that network models can be trained to be functionally specialized for different cognitive processes of brains spontaneously.
2

---

Under review as a conference paper at ICLR 2023
3 SPARSITY OF ATTENTIONS IN VISION TRANSFORMER
3.1 ATTENTIONS
Attention mechanism is adopted thoroughly in Transformers, which is their main difference pared with CNNs. In this subsection we use the Vision Transformer standard version (ViTT-base in Dosovitskiy al. (2021) example illustrate how attention mechanism works in image recognition tasks.
In Transformer containing L (in ViT-base L 12) Transformer Encoders (layers), each one carries out the following process during inference:
z MSA(LN(Z)-1))) + Zt-1.
1, L 1 1, L
(1)
Z MLP(LN(Z))) + +2Z,
in which Zu is the original patch embeddings, MSA refers to multi-head self-attention function, LN represents layer normalization and MLP is a multi-layer perceptron.
Specifically, each head in MSA (in ViT-base one MSA contains 12 heads) calculates in the way that:
head, Attention(QW KW.K, vw.) softmax (QW JKKWKK (VW))
(2)
Vd
which Q, K, V are respectively query, key and value matrices, W.O. W.K K W.V are corresponding weights, and is a scaling factor determined by the model. Intuitively, VW can recognized as containing the information in features, and the softmax term is coefficient matrix for transferring information between pairs of features, which plays a pivotal role attention mechanism. Here we name it by attention map, represented by AttnMap:
(QW )KWW
AttnMap = softmax
(3)
Vd
Here AttnMap E [0, N| in which is the number embeddings (in ViT-base N 197). The sum of each row of attention map is ensured by softmax.
In the following parts we are going to visualize and analyse AttnMaps in ViT-base, and demonstrate the patterns we discovered in attentions.
3.2 SPARSITY OF ATTENTIONS
Sparse activation a common phenomenon in deep neuron networks, which has already been ob- served in CNNs and in Transformers. In the attention maps of deeper layers in Vision Transformers, we also discover evident sparsity of columns (vertical lines), as shown in Figure] T
0.30
0225
0300
0.15
100
(c)
Figure 1: (a) The input image (transformed to 224 x 224) selected from ImageNet (uussakovsky
et al. 2015); (b) the attention map of one head of the last layer in the ViT-base, generated by inputting
image (a); (c) the corresponding patches (by index) of the top-3 "bright" lines (b).
3

---

Under review as a conference paper at ICLR 2023
Figure 1 (b) presents typical attention map, in which several vertical lines are significantly "brighter" than others. In other words, those columns contain coefficients which are particularly large. As the mean value of all values in an AttuMap is 1/N 197 < 0.01, it is not surprising that most of the area in an attention map are "black".
Large values in AttnMaps of deeper layers distributed in certain columns, instead of be- ing scattered in different columns. This general phenomenon indicates that some certain features (vectors) are paid by more attention in deeper layers, as result of which they are likely to be more significant than other features. As those features are considerably prominent, they are recognized as sparsity of attentions and we are interested in their distributions (see the following parts of section 3) and effectiveness (see section (5).
The distributions of sparsity of attentions among heads in the same layer are similar. It is ob- served in Figure [ST that all the twelve heads in a deeper layer share similar locations (indexes) of "bright" columns, which further verifies that their corresponding features are paid by more atten- tion in all heads. More generally, shown in Figure 2 (a), most of the means of the correlation coefficients between AttuMaps of pairs of heads are large, demonstrating that all the heads one Transformer model share similar patterns of sparsely activation.
3,000
0.840
0.555
0.814
3.840
0.800
0.899
872
3.33
0.292
3.000
0315
0.357
0.048
0.001
070-0.99
10
0.826
2,540
1.000
0.375
0:817
0.791
9.813
0.837
0.417
0.883
6
856
3.825
0.203
5.135
L000
0:323
0.119
0,098
2:071
0.024
2.028
0.087
-66
0.805
2,833
0.817
2:442
1.000
0.794
3.522
0.810
0.472
0.841
0 3.529
0.073
3.048
0329
0.382
1,000
0.293
5:257
0.229
0.022
5,422
0:033
0.215
0.81
5.646
0.822
6.796
1:00
0.814
8:9
0.007
001
0.257
0.00
1000
8.656
0.003
0.000
5:482
0.818
0.610
5.859
1.000
0.502
0JB
876
8.860
0.004
0.200
0.299
8.509
5.056
1.000
0.693
0.717
0.727
0.039
022
0.833
3,589 0:833
2.450
0.841
0,848
3,500
0.854
0.472
1,000
0.833
0.572
0.042
2.071-0.014
0,042
8.404
R.404
0.737
0.837
1,000
0.751
0.724
-1.
0.035
8.78L
A.812 5.831 0.82
5.501
0.824
6.791
0.866
0.870
1.500
0.025
0.003.15
5.515
0,815
0.314
0.614
0.705
3.724
0.721
1.500
12
(a)
(b)
Figure 2: (a) The mean of the correlation matrix of Attn of heads in the last layer of ViT-base, in which each value represents the mean of the correlation coefficient between a pair of heads; (b) the mean of the correlation matrix of AttnMaps of layers in ViT-base, in which each value represents the mean of the correlation coefficient between pair of layers. Both results are calculated by attentions generated while inference of images of all categories of ImageNet.
Sparsity of attentions in columns gets more prominent as layers get deeper, and the distribu- tions of sparsity of attentions among deeper layers are similar. Figures S2 and S3 show that the patterns of AttnMaps gradually change from being prominent on diagonals to columns as layers get deeper, and the deeper layers share similar locations (indexes) of "bright" columns, indicating that their corresponding features are paid by more attention in all deeper layers. This property of attentions is further supported by Figure (b), in which the means of the correlation coefficients between AttnMaps of pairs of layers get larger as layers get deeper.
The distributions of sparsity of attentions among input images are dissimilar. This statement is verified only to guarantee that attention is not always concentrated on some certain features, but is distributed differently among all input images. Otherwise, the distribution of sparsity of attentions would be only determined by the Transformer model and weights, and all of our analysis would be meaningless. This argument is further proved by results in subsection 3.4
4

---

Under review as a conference paper at ICLR 2023
3.3 NUMERICAL DISTRIBUTION OF SPARSITY OF ATTENTIONS
For more intuitive demonstration of the numerical distribution of sparsity of attentions, f is defined as the mean of values columns of certain index of attention maps of one layer, and is defined as the negative denary logarithm of J.:
N ΣΣ
PI
AutnMaprans E [0, 1], Vt.j - Ml, E [0, +oc)
(4)
HN 1=li=1
in which h, j are the indexes of layers, heads, rows on AttnMaps and columns on AttnMaps,
respectively.
As shown in FigureBi (a). most of y/ of the last layer is larger than logio P. while small portion of lies around peak smaller than 1. other words, most of ja around the order of magnitude of 10-3, while small portion of is gathered around the order of magnitude of 10-1 This result provide direct evidence for the existence and significance sparsity of attentions of deeper layers in Transformers (the density curves of of all layers are shown in Figure S4
6836.2020 0038.2000 0020.2004 0S 6020 podi J01h Ao aaso se to an se covs aavin coceion
17
14
p00-6 anni soes ar sona oeto anio
6626.2040 0036.20-4 00 eeet 2010 at 60 1200..0000 0538
XX066.244 68 KBS uo 20.0 18,2000 0ob 0030
K35
2o
0620.000 60.6.7110 6655 20.0 2030 0150.00 2010
60--6 20:00 6046.20 20 6644 so Jonan oo Jomp
0028 3010 66 6628.01
6020.00
00400,4..00 6230 682t
2040,0000
t020 0930
(b)
Figure 3: (a) The numerical distribution of (i.. v of the last layer) ViT-base, which the green line refers to the negative denary logarithm of the mean of μr; (b) the spacial distribution of top-5% large (i.. a of the last layer) ViT-base, shown by 14 x 14 patches corresponding to the input images. Both results calculated by attentions generated while inference of images of all categories of ImageNet.
3.4 SPATIAL DISTRIBUTION SPARSITY OF ATTENTIONS
It has been illustrated that attentions are distributed sparsely among columns of attention maps, corresponding to some features. Then another question emerges: are all the features equally likely to become the focus of attentions? Figure (b) shows the probability distribution of top-5% large of the last layer (not considering
feature), and draw it on patches corresponding to those of the input images. Apparently, the distribution is not completely uniform, but all the probabilities lie in |0.002, 0. 0.028), which does not manifest great dispersion. So it is reasonable carry out the ablation experiments in section 5 Transformers.
Moreover, seems interesting that the locations of the top-4 large probabilities are symmetrical in sense, which is beyond explanation currently.
In summary, in this section it is shown in detail that there exists sparsity of attentions in deeper layers Vision Transformer, and its distribution is similar to sparsity in animals' visual systems (illustrated in subsection 4.2)
5

---

Under review as a conference paper at ICLR 2023
4 SPARSITY IN CONVOLUTION NEURAL NETWORK AND ANIMALS' VISUAL SYSTEMS
4.1 SPARSITY IN CONVOLUTION NEURAL NETWORK
In contrast with Transformers, the sparsity in CNN usually refers to the phenomenon that connec- tions between neurons are sparsely activated (some values are zeros or they will not affect calculation significantly). which is mainly caused by nonlinear activation function such as ReLU (rectified lin- ear unit) (Hara al. 2013). Actually, CNN itself is an architecture sparsified from fully connected network, mainly according to locality principle. So far, sparsity in CNN has been well studied and widely applied in model compression and efficient inference and training, through approaches like pruning and sparsely training (Cheng et al.) 2017: Hoefler et al., 2021 Perrinet, 2017). Meanwhile, sparsity has been also used to analyze and illustrate CNN model performance from the view of neuroscience (Zhao & Zhang 2022).
In our experiments on CNNs, the l norms of outputs neurons after the activation layers are calculated and sorted decreasingly, and the corresponding neurons of a percentage of largest norms are recognized as the sparse ones.
4.2 SPARSITY IN ANIMALS' VISUAL SYSTEMS
The basic units in animals' neural systems are neurons, and they process information by generating sequences of electrical impulses. Sparse encoding has been theoretically proved and physically ob- served to be commonly adpoted in brains, which refers to the phenomenon that states and events are encoded only using small subset of neurons (Dayan & Abboti, (001). Particularly, experimental evidence for sparse firing in the animals' visual cortex discovered (Willmore et al. 2011: Barth & Poulet 2012). especially in VI (the primary visual cortex of primates).
Moreover, the latest biological research points out that the neural sparsity is more prominent in higher animals, compared with lower animals (Wildenberg et al. 2021 This result inspires us to launch comparative analysis of sparsity among vision models.
EXPERIMENTS
5.1 DESIGN OF ABLATION EXPERIMENTS
In order to fairly compare the effectiveness of sparsity of different vision models, series of ablation experiments are designed. Just as their names imply, the effect of sparsity is measured by the change of prediction accuracy when certain percentage of basic units (attentions in Transformers and neurons in CNNs) of certain layers are dropped (set as O). The prediction accuracies of dropping the top-p sparse units of the last n layers] and randomly dropping p units of the last layers respectively denoted as Ar(p, n) and A, n), and the effect of sparsity is reported by v' n):
Ag (p,
(p, n) Ag (p,
(5)
in which Ao is the prediction accuracy of the full model. (b, n) is reasonable metric for functional sparsity, which is supposed to be larger than if the sparsity is effective. And the larger (pp, n) is, the more effective the sparsity is.
To comprehensively study the sparsity of Vision Transformers and CNNs, ablation experiments carried out the following models of different configurations: ViT (Dosovitskiy et al.) 2021 Steiner et al. 2021). DeiT (Touvron et al. 2021). Swin (Liu et al. 2021). VGG (Simonyan &
Zisserman 2015) and ResNet (He et al. 2016). Parameters are selected as p 5%, 10%. 20%, 30% and n E [1, (1,2, 2, 3). Additionally, all A, (p. n) are reported using the means of results of 3 replications with different seeds.
2 Here we choose dropping units in the last n layers, because: (1) sparsity exists only in deeper layers of Transformers, and the sparsity of shallower layers of CNNs are mainly due to locality, which is not our concern; (2) once the the sparse units of one layer are dropped, sparsity of its following layers will change instead of disappearing, which is not in line with needs.

---

Under review as a conference paper at ICLR 2023
5.2 COMPARING SPARSITY IN TRANSFORMERS AND CNNS
Results of ablation experiments are detailedly demonstrated in Table 12 and Figure
Table 1: Results of ablation experiments Transformer models. For experimental details,
Appendix [
5% dropped
10% dropped
20% dropped 30% dropped
Model Ao
Layers
Ar
A,
A,
A,
v
A,
A
Ar
A
75. 75.40%
75.32%
75. 15%
74.
17.45
16.65
13.27
7.32
74.26%
73.02%
71.24%
69 99.94%
ViT-tiny 75.47%
75, 50.40
75.23%
74.90%
74.29%
2
29.80
17.70
13.79
9.29
73.40%
71.25%
67.68%
64.55%
75.34% 3
75.15%
74.64%
73.75%
22.17
20.32
15.16
10.86
72.67%
69.10%
62.99%
56.81 %
81.38% %
81.32%
81.19%
80.95%
109.07
38.63
28.12
18.45
80.37%
78.71%
75.86%
73.35%
ViT-small 81.39%
81.36%
81.27%
80.99%
80.53%
2
57.98
39.87
26.79
19.18
79.61 %
76.82%
70.85%
64.97%
81.31 %
81.23%
80.79%
80.08%
3
27.33
37.60
23.35
17.06
79.22%
75.50%
67.53%
59. 16%
84,49%
84.47%
84.34%
84.12%
6.16
14.75
12.48
9.93
84.29%
83.70%
82.24%
80.47%
ViT-base 84.53%
84.45% 2
84.35%
84.10%
83.67%
11.55
10.65
11.41
10.16
83.65%
82.60%
79.67%
75.84%
84.42% 3
84.29%
83.97%
83.30%
8.15
9.34
10.92
9.06
83.63% %
82.29%
78.40%
73.43%
74.45%
74.43%
74.36%
74.30%
9.94
11.67
8.21
6.14
73.95%
73.59%
73.32%
73.24%
DeiT-tiny 74.50%
74.43%
74.35%
74. 74.18%
73.94%
26.66
21.86
17.45
12.43
72.62%
71.04%
68.88%
67.51%
74.44% %
74.28%
73.95%
73.52%
3
40.65
26.52
20.76
16.91
71.79%
68.62%
62.95%
57.79%
81.21% %
81.19%
81.19%
81.11%
44.67
13.87
19.36
5.09
80.95%
80.85%
80.67%
80.66%
Deil-small 81.22%
81.17%
81.10%
81.04%
80.88%
80.22% 2
21.61
14.88
13.70
8.57
79.55%
78.75%
78.37%
81.15% 3
81.05%
80.88%
80.67%
28.42
24.81
21.70
17.72
79.21%
77.11%
74.00%
71.57%
83.37%
83.38%
83.31
83.19%
6.63
23.00
6.95
3.39
83.30%
83.16%
82.83%
82.73%
DeiT-base
83.35% 2
83.27%
83.09%
82.83%
10.31
9.67
8.18
6.51
83.39%
83.06%
82.29%
80.93%
79.76%
83.31% %
83.24%
82.96%
82.55%
8.19
12.18
11.67
9.74
82.80%
81.58%
78.44%
75.27%
81.31%
81.25%
81.19%
3.09
4.71
4.14
Swin-tiny 81.37%
81.16%
80.78%
80.61 %
81.19%
81.07%
80.73%
2.74
7.05
4.72
80.87%
79.24%
78.34%
83.21 %
83.18%
83.15%
2.86
3.16
2.54
Swin-small 83.23%
83.18%
83.07%
83.04%
83.15%
83.10%
82.99%
2
1.83
4.33
3.42
83.09%
82.67%
82.43%
85.23%
85,17%
85, 85.14%
4.42
3.87
3.54
twin-base 85.27%
85.09%
84.88%
84.83%
85.14%
85.01 %
84.90%
2
4.73
5.22
4.68
84.67%
83.91%
83.53%

---

Under review as a conference paper at ICLR 2023
Table 2: Results of ablation experiments CNN models. For experimental details, Appendix
5% dropped
10% dropped
20% dropped
30% dropped
Model Ao
Layers 71
A,
Ar
A,
Ar
A
A,
A₁
A,
67.80%
66.64%
63.69%
59.86%
12.67
11.56
8.48
6.15
53.59%
41.49%
23.78%
12.66%
VGG11
66.31%
63.19%
55.25%
44.89%
2
10.44
7.49
4.22
2.66
69.02%
40.69%
25.33%
10.84%
4.89%
64.79% 3
59.94%
47.24%
31.87%
7.77
5.20
2.81
1.78
13%
21.78%
7.91%
2.80%
68.75%
67.63%
64.76%
61.11%
1
12.63
11.68
8.74
6.39
55.05%
43.12%
24.74%
13.61 %
VGG13
67.43%
64.54%
57.29%
47.13%
2
11.25
8.10
4.67
2.85
69.93%
41.81% %
26.32%
10.92%
4.92%
66.13%
61.72%
49.60%
34.77%
37.37% 3
8.58
5.79
3.02
1.90
22.42%
8.61%
3,28%
70.67%
69.89%
67.45%
64.58%
54.49% 1
18.60
18.28
12.45
8.88
40.50%
20.09%
9.31%
VGG16
69.84% 2
67.78%
62.02%
54.07%
16.60
12.07
6.47
3.86
71.59%
42.48
25.53%
9.68%
4.00%
68.67% 3
65.13%
55.20%
41.86%
12.08
7.62
3.82
2.30
36.28%
22.40%
9.03%
3.22%
71.65%
70.80%
68.96%
66.33%
27.39
22.96
16.72
10.93
52.49% 2
36.10%
15.32%
6.33%
VGG19
70.84%
69.17%
64.54%
57.88%
19.98
15.01
8.13
4.72
72.38%
41.75%
24.29%
8.64%
3.92%
69.95% 3
67.16%
59.40%
47.85%
15.01
9.85
4.94
2.82
36.03%
21.01%
8.24%
3.18%
74.76%
74.47%
73.86%
73.04%
10.59
6.48
3.43
2.23
71.37%
70.97%
70.81%
70.48%
ResNet34
74.49% 2
73.88%
72.77%
71.25%
20.59
14.56
8.79
5.03
75.11%
62.38%
57.16%
54.55%
55.69%
73.55% 3
71.98%
68. 18.188
63.08%
32.31
21.82
10.57
5.94
24.58%
6.68%
1.78%
3.70%
78.91%
78.75%
78.57%
78.18%
19.47
14.48
17.15
12.14
76.67%
74.94%
71.12%
68.67%
ResNet50
78.70%
78.29%
77.36%
76.24%
2
27.71
19.73
14.11
10.76
79.03%
69.81% %
64.31%
55.48%
49.02%
78.34%
77.44%
75.63%
72.76%
62.98% 3
23.20
20.67
18.18
11.40
16.1
17.13%
7.50%
77.04%
76.83%
76.31%
75.72%
24.70
20.14
11.69
6.94
72.05%
68.79%
66.23%
66.60%
ResNet101
76.81% 2
76.31%
74.81%
72.94%
42.24
46.34
29.11
17.26
77.25%
58.72%
33.54%
6.17%
2.81%
76.37% 3
75.37%
72.85%
68.78%
38.21
35.08
17.39
9.10
43.40%
11.14%
0.64%
0.17%
78.05%
77.86%
77.36%
76.74%
23.97
19.06
11.10
6.64
73.61 %
70.98%
68.52%
68.30%
ResNet152
77.81 %
77.32%
76.19%
74.60%
2
35.59
40.68
33,01
19.92
78.24%
63.08%
40.89%
10.63%
5,65%
77.53%
76.64%
74.58%
71.38%
3
39,90
38.87
21.09
11.37
49.83%
15.86%
0.98%
0.20%
From the results of ablation experiments, the following discoveries are summarized:
All the ie Table and 2 are significantly larger than 1, indicating that the sparsity dis- covered in attentions of Transformers and neurons of CNNs is effective indeed.
8

---

Under review as a conference paper at ICLR 2023
2. In the same model, A, and A, always decrease as the percentage of dropping units increases or the number of layers with dropping units increases, which is consistent with our intuitive expectations that all the units have positive impacts model performance.
3. Attention mechanism is more robust than convolution in terms of dropping basic com- puting unit As shown Figure 4 (a), the loss accuracies of CNNs (VGG, ResNet) when certain percentage of units are dropped are much larger than those of Transformers (ViT, DeiT, Swin). The great loss of accuracy is not surprising in models without residual connections such VGG, but in the comparisons between oftte-of-theart Transformers and CNNs, the discovery is meaningful. Furthermore, Figure 4 (b) shows that in ViT-base, the loss of accuracies when randomly dropping p attention units is less than p of the loss of accuracies when dropping the whole attention layer (the dotted lines are "upper convex"), and the loss of accuracies when dropping the top-p attention units is also not large their proportions of values. This means that the effects on prediction of those attention units are overlapping greatly.
5
VISAA.
3ipes
V9918A A
v
Reahebb
20
15
30
0
60
300
percertage:
setcentaper
(a)
(b)
Figure 4: (a) The accuracy curves of several models for ablation the last layers of them; (b) The accuracy curves of ViT-base for ablation on the last n layers of it.
4. According D in Table and 2 for CNN models more layers lead to higher sparsity. while for Transformer models, more heads do not always result in higher sparsity The sparsity of Swin significantly lower than ViT and DeiT, which likely to be caused by the larger number of heads (Appendix A.. This result reveals a side effect of using too many heads in a Transformer model, i.e., loss sparsity and more dissimilar to higher animals' neural systems.
6 CONCLUSION
In our works, the sparsity of attentions in Transformers is proved to be existent, and its distribution is quantitatively analysed. What more, inspired by recent achievements neuroscience, metric for the effect of sparsity in vision models is defined based on ablation experiments, which are conducted on Vision Transformer models and CNN models of different structures and configurations.
We finally draw the conclusion that generally, increasing the number of layers in CNNs (also likely Transformers) conduces to improve neural sparsity deep layers, while overly increasing the number of heads in Transformers does not, which is likely cause overlap of effects among at- tention units. This discovery will be helpful for understanding attention mechanism and designing more efficient and neurally advanced models for vision tasks.
'eer only concentrate the deeper layers, whose units have global receptive tields, since discussion units with local receptive fields is meaningless.
'For VGG and ResNet models, configurations mainly differ in numbers of layers; while for VIT and DeiT, (base, small, tiny) models mainly differ in numbers of heads. For Swin, small model contain less heads compared with base one, and tiny model contain less layers compared with small

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yutong Bai, Jieru Mei, Alan L. Yuille, and Cihang Xie. Are transformers more robust than cnns?
Advances in Neural Information Processing Systems, NIPS, 2021.
Alison L. Barth and James F.A. Poulet. Experimental evidence for sparse firing in the neocortex.
Trends in Neurosciences, 35:345-355,
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Vierbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neu- ral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, ECCV, 2020.
Hanting Chen, Yunhe Wang. Tianyu Guo. Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chun jing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2021a.
Liang- Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In European Con- ference Computer Vision ECCV 2018, 2018.
Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang. Xiaoyun Yang, and Huchuan Lu. Transformer track- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 8126-8135. 2021b.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. ArXiv, abs/1710.09225 2017.
Xiangxiang Chu, Zhi Tian, Yuqing Wang. Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, NIPS, 2021.
Peter Dayan and I.. F. Abbott. Theoretical Neuroscience: Computational and Mathematical Model- ing of Neural Systems. 2001.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale. In 9th International Conference on Learning Representations, ICLR, 2021.
Robert Geirhos, Kristof Meding, and Felix A. Wichmann. Beyond accuracy: quantifying trial-by- trial behaviour of cuns and humans by measuring error consistency. Advances Neural In- formation Processing Systems 33: Annual Conference on Neural Information Processing Systems NeurlPS, 2020.
Kazuyuki Hara, Daisuke Saitoh, and Hayaru Shouno. Analysis of function of rectified linear unit used in deep learning. In 2015 International Joint Conference on Neural Networks, IJCNN, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE Inter- national Conference Computer Vision, ICCV, 2017.
10

---

Under review as a conference paper at ICLR 2023
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22:241:1-241:224 2021.
David H. Hubel and Torsten N. Wiesel. Receptive fields of single neurones in the cat's striate cortex. The Journal of Physiology, 148, 1959.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances Neural Information Processing Systems, NIPS, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. 26th Annual Conference on Neural Information Processing Systems NIPS, 2012.
Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1:541-551, 1989.
Shanda Li, Xiangning Chen, Di He, and Cho-Jui Hsich. Can vision transformers perform convolu- tion? arXiv preprint 1:Xiv:21/1.0..: 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEF/CVF International Conference Computer Vision, ICCV, 2021.
Adam H. Marblestone, Greg Wayne, and Konrad Paul Kording. Toward integration of deep learning and neuroscience. Frontiers in Computational Neuroscience, 10, 2016.
Namuk Park and Songkuk Kim. How do vision transformers work? arXiv preprint
060,79 2022.
Laurent Udo Perrinet. Sparse models for computer vision. ArXiv, abs/1701.06859) 2017.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers like convolutional neural networks? Advances in Neural Information Processing Systems, NIPS, 2021.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems NIPS, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention MICCAI, 2015.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575:607-617, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference Learning Representations, ICLR, 2015.
11

---

Under review as a conference paper at ICLR 2023
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. ArXiv, abs/2106.102700 2021.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proceedings the 38th International Conference on Machine Learning, ICML, 2021.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. arXiv preprint arXiv:2204.07118, 2022.
Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas Griffiths. Are convolutional neural net- works or transformers more like human vision? ArXiv, abs/2105.07197, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances Neural Infor- mation Processing Systems 30: Annual Conference Neural Information Processing Systems NIPS, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
ICCV, 2021.
Ross Wightman. Pytorch image models.
immbbrrch-imagmmmode 2019.
G.A. Wildenberg, Matt R. Rosen, Jack Lundell, Dawn Paukner, David J. Freedman, and Narayanan Kasthuri. Primate neuronal connections are sparse in cortex as compared to mouse. Cell reports, 36 11:1 11:109709, 2021.
Ben D. B. Willmore, James A. Mazer, and Jack L. Gallant. Sparse coding in striate and extrastriate visual cortex. Journal of neurophysiology. 105 6:2907-19, 2011.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, CVPR, 2017.
Guangyu Robert Yang, Madhura R. Joglekar, H. Francis Song, William T. Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature Neuroscience, 22:297-306. 2019.
Yang Zhao and Hao Zhang. Quantitative performance assessment of CNN units via topological entropy calculation. In The Tenth International Conference on Learning Representations, ICLR, 2022.
Yucheng Zhao, Guangting Wang, Chuanxin Tang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha. A battle of network structures: An empirical study of enn, transformer, and mlp. arXiv preprint arXiv:2108. 13002, 2021.
12

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
A. BASIC INFORMATION
We adopt ImageNet (Russakovsky et al. 2013). the most acknowledged dataset for image recogni- tion, and timm code library (Wightman, 2019). library of various image models (with pretrained weights) implemented by PyTorch, for experiments. They accept applications of non-commercial research purposes.
The specific process of ablation experiments:
For each image:
1. inputting it into the Transformer or CNN model and sorting Midi
2. for a percentage p, dropping out the top-p sparse units of the
last layers and doing inference; Then calculating the classification accuracy (l,p) among all input images.
must be pointed out that in ablation experiments, the values of de may be not precise on account of the randomness when measuring A,, especially when A, is close Ag. Replications of random experiments are adopted to alleviate this problem, and multiple experiments with different configu- rations also contribute to draw stable conclusions.
For details of implementation, please refer to our codes at https://github. com/ SmarttAnonnyoous/Coeeee--- for--
A.2 MODEL CONFIGURATIONS AND DETAILS
In ablation experiments. ViT Dosovitskiy et al. 2022 Steiner et al. 2021), DeiT TTourron et al. 2021) and Swin (Liu et al. 2021) are selected among Transformer models, and VGG (Simonyan & Zisserman 2015) and ResNet (He et al. 2016) are selected among CNN models.
For VIT and DeiT, the base, small and tiny versions of models are selected, which all contain 12 layers and respectively contain 12, 6 and heads. All the models take 224 x 224 as the size of input images and 16 16 as the size of patches.
ViT has class token for prediction, and DeiT' has class token and distillation token, which are all not considered into discussion of sparsity and ablation experiments. This is because they are not equivalent in status with other features.
For Swin, the base, small and tiny versions of models selected, and the numbers of layers and heads are shown in the table below. All the models take 224 x 224 as the size of input images, 4 the size of patches and 7 the size of windows.
Models
Layers
Heads
Swin-bace base
(2, 2, 18, 2)
(4, 8, 16, 32)
Swin-small
(2, 2, 18,
(3, 6, 12, 24)
Swin-tiny
(2, 2, 2)
(3, 6, 12, 24)
For VGG, the 11, 13, 16 and 19 layer versions of models are selected, and all the models take 224 x 224 as the size of input images.
For ResNet, the 34, 50, 101 and 152 layer versions of models selected, and all the models take 224 224 as the size of input images. In ResNet, we only consider the sparsity in layers with 3 x convolution kernels.
13

---

Under review as a conference paper at ICLR 2023
B SUPPLEMENTARY RESULTS
Head
Head 1
Head
Head
Head
Head 9
Head 10
Head 11
Figure S1: The attention maps all the 12 heads the last layer of ViT-base.
14

---

Under review as a conference paper at ICLR 2023
Layer
Layer
Layer
Layer
0.5
0.4
Layer
Layer
Layer
Layer
Layer
Layer 9
Layer 10
Layor 11
Figure S2: The attention maps all the 12 layers of ViT-base.
15

---

Under review as a conference paper at ICLR 2023
Layer
Layer
Layer
Layer
14
12
Layer
Layer
Layer
Layer
Layer
Layer 9
Layer 10
Layer 11
Figure S3: The negative natural logarithmic attention maps of all the 12 layers of ViT-base.
16

---

Under review as a conference paper at ICLR 2023
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Density
Layer 10
Layer 11
2
0
2
4
6
Figure S4: The numerical distribution of (i.e of the / layer) of ViT-base for / E (0,1,...,113 11), which the green line refers to the negative denary logarithm of the mean of μ..
17


Reviews:
[{"review_id": "gLuUSm1vhLQ", "reviewer": "Reviewer_LMCh", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors explore the role of sparsity of attention in Vision Transformers and sparsity of neurons in CNNs.   It minimally reviews some literature surrounding sparsity in neuroscience and neural networks (but does not link the two).  Ablation studies are performed where sparse units of later layers are dropped the performance is measured.", "summary_of_the_review": "This paper is not reproducible as currently written.  Results are stated for \"Transformers\", but no where is any information provided as to the dataset, training protocol, specific architecture, etc.  \n\nFurthermore, the paper is strongly lacking in direction and focus.  Claims are not demonstrated to hold over a wide range of datasets, training protocols, initializations, tasks, etc.  Error bars are not given on the ablation studies.  What this analysis has to do with neuroscience (beyond the fact that sparsity is observed in the brain, but also not in the way seen here, which the authors don't discuss) is unclear.  \n\nThe claim that the ablation studies  \"compare the effectiveness of sparsity of different vision models\" is strongly false.  Dropping sparsely activated units is not the same as evaluating the impact of sparsity.  In fact, in a sparse system, one has many very inactive neurons.  Many other claims made around CNN and Transformer models in general are 1. strongly false 2. not justified by the results.", "strength_and_weaknesses": "\nWeaknesses:\nThe main claim is that the authors demonstrate that more layers results in higher sparsity in transformers but more heads reduces sparsity.  This is not proven through theoretical justification, but only a handful of results.  From these results, one cannot assert that claim is true- there is not enough evaluation on different datasets, tasks, or transformer architectures.  Furthermore, it's not clear why this is relevant.  Just because sparsity exists in the attention activations does not tell us something important about the brain or transformers- the link between these two domains is not utilized in this analysis (nor should it be necessarily). \n\nThe abstract claims that the sparsity of transformers is extensively studied, but there are minimal experiments in this work. \n\nThe related works section is extremely sort.  While there are a number of works cited in the intro, it's not clear what value Section 2 (less than half a page) has.  \n\n\"Results\" are given beginning in Figure 1, but nowhere is the dataset, specific architecture, training paradigm, etc. ever stated prior to that.\n\nThe paper is written as if there is \"A Vision Transformer\"- one model, one dataset, one training protocol- one grand model in the universe.   But we know initialization, training, all have huge effects on the end weights.  Do these results hold up under different initializations, training protocols, etc. \n\nSection 4.2 has nothing to do with the rest of the paper.  Observing sparsity in the human brain is well known.  What implication this has for neural networks, transformers in particular, is not given.\n\nThe authors conclude that the ablation studies \"compare the effectiveness of sparsity of different vision models\" , but this is not accurate.  Dropping sparsely activated units is not the same as evaluating the impact of sparsity.  In fact, in a sparse system, one has many very inactive neurons.  \n\nOther claims made by the authors like \"for CNN models more layers leads to higher sparsity\" are strongly false.  Sparsity can be introduced into a model- in fact it is a design choice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Lx_7wrYfzv", "reviewer": "Reviewer_3yeY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to compare sparsity in different ANN models, such as transformers and CNNs to sparsity in the brain. The authors further conduct ablation experiment to assess the functional significance/effect of sparsity (as the effect of dropping sparse units on downstream accuracy). ", "summary_of_the_review": "This work, in its present form, falls short in terms of the presentation style (writing) and the relevance of the experimental results. ", "strength_and_weaknesses": "\nStrengths: \n- The topic of (emergent) sparsity in ANN models is highly interesting \n- The authors compare a wide range of models (transformers, CNN) in terms of the effectiveness of sparsity\n\nWeaknesses: \n- The main contributions of the paper are unclear. In particular, the authors claim to compare CNNs and transformers from the perspective of neuroscience (as in the title too) but the only aspect in which they are compared is the effectiveness of sparsity. The relevance of neuroscience to the experiments/results in the paper is not entirely clear. \n- The comparison between transformers and CNNs in terms of the proposed metric (the downstream effect of ablating sparse units on prediction accuracy) seems like an apples to oranges comparison due to the very different architectures and it is difficult to meaningfully interpret the authors observation that ViT is more robust than CNNs in terms of dropping units. \n- Several claims/statements are not well justified: \ne.g. The authors claim that the distribution of sparsity in ViT is similar to the brain but no data is presented to support this claim. \n- The organization of the paper is confusing. In particular, the separation of lit review (related works) in two sections intermingled with experiments and results is confusing. It would be better to discuss all related work in one section and experiments in another\n- Tables 1 & 2 which present the main results are very busy and it is hard to decipher so many numbers at once. It would be helpful is the authors could bold the relevant numbers (ones with lowest psi for example)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DEBUCL08cN", "reviewer": "Reviewer_Kv5z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The enclosed manuscript presents an analysis of unit activations in vision transformer models and compares them to CNNs using ablation experiments. They examine activations in VIT base, and show that there is sparsity in the columns of ‘attention maps’, which correspond to sparse activation in patches present. They also show high correlations of activated columns across deeper layers and heads. They then perform ablation experiments across vision transformers with different numbers of heads, and CNNS with different numbers of layers, ablating either a random fraction of units or the most active units in the bottom 1-3 layers. They show that transformers are more robust to loss of units than CNNs, and claim results about the effect of greater numbers of heads and layers for transformers and CNNs, respectively. They note the presence of sparsity in biological vision and suggest links between artificial network and biological sparsity.", "summary_of_the_review": "\n\nOverall, while I believe the investigations of vision transformer representations is very useful for the field, and there is novelty in the analyses, I think the manuscript is hampered by issues with the metrics used for analysis, the interpretation of results and a somewhat qualitative treatment of the term sparsity. I would be happy to look at a revision but I don’t think its ready at this time.", "strength_and_weaknesses": "Strengths\n(1)\tThere are some intriguing results about the sparsity of vision transformers across layers. I thought the demonstration of correlation of attention across heads was interesting, as well as the ‘columnar’ sparsity of the attention maps presented. I think doing model analysis fits squarely in the purview of ICLR and so the work is plausibly suited.\n(2)\tThere is a large and fairly comprehensive ablations comparison of transformer models and CNNS with respect to the importance of highly active units in lower layers. \n(3)\tI find the difference between CNN and transformers in robustness somewhat interesting, and it suggests the transformer representations are more distributed. \n\nWeaknesses\n (1) links with neuroscience and evolution are mostly asserted, and qualitative at best. I am not sure they add anything to the body of the paper without a more direct comparison of the sparsity distribution. Yes, higher order visual areas in primates, for instance object or face selective patches are thought to have fairly sparse representations, but there are many ways to achieve sparsity and simply asserting that both systems are sparse is not enough. The transformer models discussed could have sparse responses to different categories, but they could also have sparse responses to other features or qualities as well. See also nits below on discussion of neuroscience. \n\n(2) The word sparsity is used somewhat loosely. The distribution in figure 3a does appear sparse, but it would be nice to use a quantification of this (e.g. L1 or gini index). In the ablation experiments it is highly active units that are removed, and you could imagine \n(3) I find the metric \\psi proposed for measuring sparsity to be problematic. Because it measures the ratio of dropping highly active (‘sparse’) units compared to the difference of dropping random units vs. baseline, it is more sensitive to the effect size of dropping random units than dropping highly active (‘sparse’) units. So for instance ViT-small has a very large psi (100!), but only a 1% difference in effect size between dropping random and ‘sparse’ units, whereas VGG11 has a more moderate psi (10), but a 15-30% effect size for dropping ‘sparse’ units. \nReporting the uncertainty of these values (e.g. from multiple model seeds) would help to evaluate the results, especially because in many cases the effect of randomly dropping units is so small. \n(4) The issues with the psi metric above I believe invalidate the claim that “more layers in models will result in higher sparsity, however, too many heads in Transformers may cause reduction of sparsity, ”. Looking at the table, the effect of removing top units (A_t) is almost identical across CNNs of different layer numbers. For the transformers, the changing of \\psi is not driven by A_t, but rather by both A_r and A_t together.\n\nNits on neuro writing:\nThe writing overall is not great, and some sections, especially those discussing neuroscience, should be re-written for correctness. For instance:\n“Sparse encoding has been theoretically proved and physically observed to be commonly adpoted in brains,”\nIs grammatically off.\nThe discussion of evolution, while appropriate somewhat in the differentiation of higher visual areas, is problematic, for instance the ‘scala natura’ idea that there are ‘higher’ and ‘lower’ organisms:\n“For animals, lower and higher are descriptions for relative levels of evolution of biological complexity. For instance, primates are higher than non-primate mammals, mammals are higher than other vertebrates and vertebrates are higher than invertebrates “\n“the neural sparsity is more prominent in higher animals, compared with lower animals” – the cited paper here has to do with anatomy, and not neural representations as discussed here. \n\nOther nits:\nThe use of attention map is a bit close to the attention heatmaps that are often used to convey receptive fields, you might consider a different name or doing the analysis in a more interpretable basis. \nFigure S2, S3 – are these results for one example image? Results mostly anecdotal without summary statistics. \nFigure 1 b – missing y axis label\nFigure 2 – axes labels and colorbar label will help\nFigure 3 caption spacial->spatial\n\nWhat is upper convex in this section:\nFurthermore, Figure 4 (b) shows that in ViT-base, the loss of accuracies when randomly dropping p attention units is less than p of the loss of accuracies when dropping the whole attention layer (the dotted lines are ”upper convex”)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5fH9qIfbBDt", "reviewer": "Reviewer_bMQh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission studies attention patterns in vision transformer models, finding significant sparsity across layers. Manipulations that enforce greater sparsity do not severely reduce the performance of these models. A comparison between attentional sparsity in transformers and activation sparsity in convolutional neural networks (convnets) reveals that convnets are less robust to sparsification.", "summary_of_the_review": "The submission presents many results but does not make a clear case that its empirical investigations into sparsity in vision transformers bring new and important insights.", "strength_and_weaknesses": "#### Strengths:\n\n1. The empirical investigations are thorough, and the causal manipulation makes sense to probe the limits of sparsity in vision transformers.\n\n\n#### Weaknesses:\n\n1. **Motivation.**  There are no strong arguments made as to the *specific* biological relevance of attention in transformer models to attentional mechanisms in animals and humans; the connection is very high-level.\n\n\n2. **Prior art.** No references made to existing work in sparse attention in (vision) transformers, despite its relevance  to the empirical investigations of attention sparsity in the paper; \ne.g.:\n\n    - Guo, Q., Qiu, X., Liu, P., Shao, Y., Xue, X., and Zhang, Z. Star-Transformer. In North American Chapter of the Association for Computational Linguistics, 2019.\n\n    - Child, R., Gray, S., Radford, A., and Sutskever, I. Generating Long Sequences with Sparse Transformers. Preprint\n    arXiv:1904.10509, 2019.\n\n    - Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y., and Yan, X. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. In Neural Information Processing Systems, 2019.\n\n    - Beltagy, I., Peters, M., and Cohan, A. Longformer: The Long-Document Transformer. Preprint arXiv:2004.05150, 2020.\n\n    - Martins, P. H., Niculae, V., Marinho, Z., & Martins, A. F. (2021, September). Sparse and structured visual attention. In 2021 IEEE International Conference on Image Processing (ICIP) (pp. 379-383). IEEE.\n\n     - Shi, B., Song, Y., Joshi, N., Darrell, T., & Wang, X. (2022). Visual Attention Emerges from Recurrent Sparse Reconstruction. arXiv preprint arXiv:2204.10962.\n\n\n    Also related but not referenced are works in activation sparsity in convolutional neural networks.\n\n\n   - Kurtz, M., Kopinsky, J., Gelashvili, R., Matveev, A., Carr, J., Goin, M., ... & Alistarh, D. (2020). Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference on Machine Learning (pp. 5533-5543). PMLR.\n\n    Given the above works demonstrating that sparsity is already well-valued in vision transformers, it's not clear what this submission adds.\n\n\n#### Minor points:\n\n- Many typos; e.g., \"deifned\"; \"adpoted\"; \"detailedly\"\n\n> \"These discoveries laid the foundation for the design of CNN architecture, which, in turn, demonstrated its striking performance firstly in vision tasks\"\n\n  Please also cite Fukushima's \"Neocognitron.\"\n\n> \"Attention mechanism is also recognized as an essential property of animals' perception, therefore,\nsome researchers have attempted to observe and study Transformer with prior knowledge of bioscience.\"  \n> \"Inspired by the recent research about sparsity in animals' visual system...\"\n\n  These statements require references to prior work, but none are given.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject