The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
Authors: Zonglin Li
Chong You
Srinadh Bhojanapalli
Daliang Li
Ankit Singh Rawat
Sashank J. Reddi
Ke Ye
Felix Chern
Felix Yu
Ruiqi Guo
Sanjiv Kumar
Venue: ICLR 2023 poster (2023)
Keywords: Transformers
Sparse
Calibration
Robustness
Label Noise
Efficiency

Abstract: This paper studies a curious phenomenon that machine learning model with Transformer architectures have sparse activation maps. By activation map we refer to the intermediate output of the multi-layer perceptrons (MLPs) after a ReLU activation function, and by "sparse" we mean that on average very few entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for each input to MLP. Moreover, larger Transformers with more layers and wider MLP hidden dimensions are sparser as measured by the percentage of nonzero entries. Through extensive experiments we demonstrate that the emergence of sparsity is a prevalent phenomenon that occurs for both natural language processing and vision tasks, on both training and evaluation data, for Transformers of various configurations, at layers of all depth levels. We discuss how sparsity immediately implies a way to significantly reduce the FLOP count and improve efficiency for Transformers. Moreover, we demonstrate perhaps surprisingly that enforcing an even sparser activation via Top-k thresholding with a small k brings a collection of desired properties, namely less sensitivity to noisy training data, more robustness to input corruptions, and better calibration for their prediction confidence.

Published as a conference paper at ICLR 2023
THE LAZY NEURON PHENOMENON: ON EMERGENCE OF ACTIVATION SPARSITY IN TRANSFORMERS
Zonglin Li; Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, and Sanjiv Kumar Google Research, New York City, USA
(lizonglin in,, cyou, bsrinadh, dal ianglin i, ankit @google. comommemmmmmmmmmmmm.ooom.mmmm com (sashank, kkye, Echern, felixyu, guorq, sanjivk] @google com
ABSTRACT
This paper studies a curious phenomenon that machine learning model with Trans- former architectures have sparse activation maps. By activation map we refer to the intermediate output of the multi-layer perceptrons (MLPs) after a ReLU activation function, and by "sparse" we mean that on average very few entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for each input to ML.. Moreover, larger Transformers with more layers and wider MLP hidden dimensions are spanser as measured by the percentage of nonzero entries. Through extensive experiments we demonstrate that the emergence sparsity is a prevalent phenomenon that occurs for both natural language processing and vision tasks, on both training and evaluation data, for Transformers of various configurations, at layers of all depth levels. We discuss how sparsity immediately implies a way to significantly reduce the FLOP count and improve efficiency for Transformers. Moreover, demonstrate perhaps surprisingly that enforcing even sparser activation via Top-k thresholding with a small & brings a collection of desired properties, namely less sensitivity noisy training data, more robustness to input corruptions, and better calibration for their prediction confidence.
INTRODUCTION
The great success of modern machine learning for tasks in computer vision, natural language processing, game playing and beyond is driven primarily by the computational model known as deep neural networks (DNNs) (LeCun al., 2015). With inspirations drawn from biological intelligent systems, DNNs are massive systems of distributed computational nodes (a.k.. neurons) with learned inter-connections, which possess the capacity of accomplishing complex real-world tasks.
Although originally motivated from biological brains, there are differences at very fundamental levels on how DNNs work compared to biological neural networks. One of such differences is in the sparsity of neural activities. Evidence from neuroscience suggests that neural activity in biological brains is sparse, namely, only a small percentage of all neurons fire at each time (Ahmed et al., 2020; Barth & Poulet, 2012; Kerr et al., 2005; Poo & Isaacson, 2009). Sparse firing suggests that despite having billions of neurons, only a small fraction of the brain participates in computation at each time, which may explain why brains can sustain at very low energy cost, In contrast, learning and inference with DNNs rely primarily on dense computations where all neurons are involved for any input. In fact, modern computational hardware for deep neural networks, such as GPUs and TPUs, are designed to facilitate massive scale dense computations. Even with such dedicated hardware, DNNs are still notoriously resourie-emmanding to train and deploy. Aside from computation efficiency, DNNs also lag far behind biological brains in terms of robustness to input perturbation, error correction for erroncous training labels, confidence calibration for the predictions, etc.
1.1
AN INTRIGUING OBSERVATION: ACTIVATIONS ARE SPARSE IN TRAINED TRANSFORMERS
This paper provides an extensive study on surprising observation that despite performing dense com- putations, DNNs produce very sparse activation in its intermediate layers once
"qqual contribution

---

Published as a conference paper at ICLR 2023
Percentage nonzeros
Layer
Percentage of nonzeros
10-
Layer3
Layer
Layer?
Layer
Layer 9
Layer
50000
60000
10000
Steps
Steps
(a) T5 Encoder
(b) T5 Decoder
Figure 1: Percentage of nonzero entries (y-axis, log scale) in the activation map as a function of number of training steps (x-axis) for T5-Base model trained with the span comupption objective the C4 dataset. Left: layers (from shallow to deep) of the encoder. Right: layers of the decoder.
we study Transformer (Vaswani al., 2017), DNN model architecture that has become workhorse for modern applications. Transformers are constructed by interleaving a self-attention module and a multi-layer perceptrons (MLPs) of depth 2, and the focus of this paper is on the activation map of the first MLP layer. Figure shows the sparsity of the activation maps, measured by the percentage of nonzeros, in all MLP layers of a T5-Base model (Raffel et al., 2020) computed on the training set of C4. We see that the percentage of nonzero entries is around 50% at initialization, which is expected: randomly initialized weights produce roughly equal numbers of positive and negative entries in the pre-activation map, resulting in about 50 % non-zeros after the ReLU. However, at the end of training the percentage of nonzero entries reduces drastically: the average value across all encoder-decoder layers is 2.7% with the largest one being 12.0% and the smallest one being only 1.1%. The emergence of sparse activation Transformers bears a similarity to the sparsity of neural activities in biological brains, revealing interesting connection between artificial and biological networks. Moreover, unlike classical sparse methods where such connection is established via explicit sparse regularization (Olshausen & Field, 1996), the sparsity observed in Transformers is emergent without any explicit design.
1.2 PREVALENCE, BENEFITS, AND CAUSES OF SPARSITY
This paper studies the aforementioned phenomenon of sparse activation in trained Transformers, with focus on the following questions. First, is the phenomenon shown in Figure a corner case does it occur broadly? Second, why should we care about the sparsity in DNNs, other than the appeal of its similarity biological brains? Our main results along these two lines summarized below.
1. Sparsity is prevalent phenomenon. We show in Section 2 that the emergence of sparse activation reported in Figure not an isolated and cherry-picked case. Rather, sparsity is prevalent, and occurs broadly in Transformer models: it emerges in all layers of Transformer, for Transformers trained on both vision and natural language data, for Transformers of various configurations, and for activation maps computed both train and test data, etc. Moreover, through controlled experiments on the width and depth of Transformers, we reveal that larger models sparser, measured by percentage of nonzero entries. We also show in the Appendix that sparsity emerges with many other architectures and with different optimizers.
2. Sparsity improves efficiency. Sparsity of activation map in trained Transformers implies that large proportion of the computation during inference spent on multiplying values by zero. Hence, FLOPs can be drastically reduced by avoiding all such computations, which we discuss in Section 3. Motivated by this observation, and to obtain reduced FLOPs not only after training but throughout training, we introduce Top-k Transformer in Section 3.2, simple modification of Transformers where a Top-k thresholding is applied to the activation maps'. We show that Top-k Transformers with reasonable sized has par performance with vanilla Transformers. To demonstrate the computation benefits of Top-k Transformers, we provide proof-of-concep results on wall time reduction for the task of unbatched decoding TPUv4 with large Top-k T55 Meanwhile, we emphasise that this result is far from fully realizing the benefit of sparse activation, due a lack of hardware support for sparse computation.
'The approach is previously adopted in ConvNets for improving model robustness (Ahmad & Scheinkman, 2019), and recently in Gupta al. (2021) for improving memory efficiency of Transformers.
2

---

Published as a conference paper at ICLR 2023
3. Sparsity improves robustness and calibration. We further show in Section 3.3 that enforcing explicit sparsity via Top-k Transformers improves model performance in terms of less sensitivity to noisy training data, less sensitivity to input corruptions, and better confidence calibration.
In addition, we provide study the causes of sparsity in the Appendix D, showing that sparsity likely not an artifact of the training data, and may be attributed to the training dynamics in the optimization process.
1.3 EXPERIMENTAL SETUP
We study the sparsity in activation maps of Transformers with two commonly used Transformer models, namely Text-to-Tens Transfer Transformer (i.c., T5) and Vision Transformer (i.c., ViT).
T5 is an encoder-decoder model for natural language processing tasks (Raffel et al., 2020). We train T5 on the Colossal Clean Crawled Corpus (C4) using the span corruption task.
VIT is an encoder model for vision tasks (Dosovitskiy et al., 2021). Unless specified otherwise, we train ViT on ImageNet-2Ik (Deng et al., 2009), an image classification dataset with 14M images and 21k classes. For certain cases we also use ImageNet- Ik which is subset of ImageNet-2Ik with 1.3M images and k classes.
We measure the sparsity level (computed on training set unless specified otherwise) at the intermediate output of the two-layer MLPs in Transformer. Recall that an MLP performs the following mapping
dn
f(x; KVKV
Σ (((()))))) .vs), equivalently, f(;; K, V) = Vo(K x),
(1)
where x. Ru is the input, K [ki,.... kddl E RR X and V [e1...... Vdl E Ia du are learnable layer parameters, and o is nonlinear activation function. We use ReLU as the activation function o() for both T5 and ViT². A two-layer MLP may be regarded as having du neurons in the hidden layer, where the i-th neuron performs the computation or (((k,,))) Us and the final layer output the sum of the output of all neurons. Each neuron called activated if @ (k; a)) is strictly positive. Hence, the sparsity of neuron activation be measured by the number of nonzero entries in the feature map
(2) Both T5 and VIT come with several configurations for dmoodel de, number of layers, etc. Unless specified otherwise, we will use the Base models (i.e., TS-Base and ViT-B/16) which have dmotel = 768, dr 3072, and 12 layers (for ViT) and 12 encoder layers +12 decoder layers (for T5). Our experiment with T5 and ViT uses the T5X (Roberts et al., 2022) and the Scenic codebase (Dehghani al., 2022), respectively. More training details of T5 and ViT are provided in Appendix A.
a a=o(K x:) E TR.""
2 PREVALENCE OF SPARSITY IN LEARNED TRANSFORMERS
This section shows thorough experiments on commonly used Transformers that sparsity in activation maps is prevalent phenomenon. We also show through some controlled experiments that deeper and wider Transformers tend to be sparser measured by percentage of nonzero entries in activation maps.
2.1 SPARSITY IS A UBIQUITOUS PHENOMENON
We start by providing experimental evidence that the emergence of sparse activation in trained Transformers is ubiquitous phenomenon. To this end, we plot the percentage of nonzero entries of activation maps in different Transformers, and present the results in Figure 2. These results demonstrate the following.
Sparsity emerges for both Vision and NLP tasks. Figure 2a shows the percentage of nonzero entries of trained T5 and VIT models evaluated on their respective training datasets. We see that both encoder and decoder T5, as well as the ViT, all exhibit sparsity.
'VIT uses GeLU as its activation function (Dosovitskiy et al., 2021). Here we switch to ReLU as it allows us to more easily measure the sparsity level using the number of nonzero entries with a very small performance drop (c.g., 47. 47.78% with GeL.U vs 47.58% with ReLU for Top-1 evaluation accuracy ImageNet-21K)]
3

---

Published as a conference paper at ICLR 2023
50
50-
Nercetaage nonzeros
VIT
Porsortaee nonzeras.
VIT itrain)
Porconage nonzeras
mageNatt221 21K mageNet
15 encoder decoder
1
0.5
0.5
0.5
Layar Indox
Layer indcx
Layar Irdcx
(a) TS vs VIT
(b) Train vs evaluation data
(c) Different training data size
50
50
Percentage noccrros
VT-916
Percentage noverros
T5-Small
Parcentage nonzeros
T5-Small
T5-Base
10
WI-H114
10
La-Large
10
L5-Lerge
75-38
T5-38
0.5
0.5
10
20
30
10
15
20
10 -5
20
Index
Index
Layer Index
(d) Varying configuration (VIT)
(c) Varying config. (T5 Encoder)
Varying config. Decoder)
Figure 2: Percentage of nonzero entries across different layers of trained Transformers (a) for both language data with T5 and vision data with VIT, (b) on both train and evaluation data, (c) for VIT trained on ImageNet of 21k Ik classes, (d) on ViT of varying configurations, and (e, f) on T5 of varying configurations. Note that the y-axis is in log scale. Sparsity emerges in all cases.
Sparsity emerges both truining and evaluation data. Figure 2b shows the percentage of nonzero entries in a trained T5 model measured on both the training data and the evaluation data. We see that the property of sparsity generalizes very well to evaluation data the curves for training and evaluation data align very closely with each other.
Sparsity emerges on datasets of varying scale. Figure 2c shows the percentage of nonzero entries in ViT trained on both ImageNet-21k and ImageNet-Ik, where the former is a superset of the later with approximately 10x more images and 21 more classes. We see that the scale of data does not affect much of the sparsity level.
Sparsity emerges on Transformers of varying configurations. Figure 2d shows the percentage of nonzero entries for ViT of varying configurations in model size. Figure 2e and 21 show the percentage of nonzero entries for encoder and decoder, respectively, of T5 with varying configurations in model size. We see that sparsity persists for all cases.
Sparsity emerges across all layers of Transformer. Finally, all plots in Figure 2 show that sparsity emerges in all layers of a Transformer. Moreover, in all cases the first few and last few layers tend to be denser than intermediate layers.
The presence of sparsity in activation maps does not rule out the possibility that small percentage of the neurons are always activated for all inputs, whereas the rest of the neurons never activated. To illustrate that this is not the case, we experiment with a pretrained T5 base model to plot the percentage of layer inputs for which each of the dr neurons is activated when evaluated on 800 examples taken from C4 dataset with span corruption task. Note that there are 800 x 512 409600 samples as MLP activation computed per token. The results are presented in Figure with x-axis being indices of neurons in the first encoder layer of T5 sorted in descending order according to percentage of layer inputs on which they are activated. It can be seen that while few neurons are activated for around 50% of the time, the vast majority of neurons (around 93.5%) are activated less than 10% of the time. Moreover,
50
Percenn of Activations
10
1000
2000
3000
Neuron
Figure 3: Percentage of times that each neuron in the first MLP layer of trained T5 is activated on C4 dataset.
Pttppa // htw.://t............... goog Me reseerr /t oob/mml //doa /mode 5 checkpoints
4

---

Published as a conference paper at ICLR 2023
8 layers
Necertage of nonzeros
Percentage nonzeros
Number nanzerax
layers
10*
24 layers
1:
0.5
0.3
Layer index
.ayor Index
Layer Index
(a) Sparsity vs. depth
(b) Sparsity (percentage) vs. width
(c) Sparsity (count) vs. width
Figure 4: Activation sparsity across different encoder layers of trained T5 Transformers of (a) varying depth and (b, c) varying width (i.e., da). Since with varying width the dimension of activation maps also changes, evaluate sparsity both in term of the percentage (as in (b)) and the count (as in (c)) of nonzeros. Deeper and wider models are sparser in terms of percentage of activated neurons.
there are no dead neurons that are never activated, and the least activated neuron is activated for around 0.001 % of the time, and 99% of neurons activated over 1% of the time. Finally, while the results here are for neurons in the first MLP layer of a pretrained T5 base encoder, all other MLP layers show qualitatively similar behavior.
2.2 THE LARGER, THE SPARSER
We next examine the effect of model size on the sparsity level of activation maps. Note that Figure 2e and Figure 2f provide evidence with T5 of varying configuration that larger models tend be sparser. Here we perform controlled experiments to examine the effect of model depth, measured by the number of Transformer layers, well the effect of model width, measured by the dimension of activation map of MLPs (i.e., da), separately. Towards that, we take standard T5 model and vary the depth and width, respectively while keeping the rest of the configuration fixed, and examine their sparsity level after training. The results are presented in Figure 4 for the encoder, whereas we omit the results for the decoder as they are qualitatively the same as those for encoder.
It can be seen from Figure 4a that deeper Transformers are arguably sparser. For example, many of the middle layers of the 32-layer model have less than 1% nonzero entries while all shallower models have more than 1% nonzero entries across all layers. For comparing networks of different widths, we measure the sparsity with the percentage and the count of nonzero entries Figure 4b and Figure 4c, respectively. It can be seen that wider models have lower percentage of nonzero entries, though higher count of nonzero entries.
3 EFFICIENT, ROBUST, AND CALIBRATED: SPARSITY IS ALL YOU NEED?
In this section we show that activation sparsity provides several practical benefits. In Section 3.1 we discuss how the free sparsity in trained Transformers brings us free computation efficiency in terms of FLOPs count during inference. order to obtain sparsity hence FLOPs reduction throughout training, in Section 3.2 we introduce Top-k Transformers, a simple modification of Transformers where top-k thresholding operation is applied to the activation maps in all MLPs. While existing hardware cannot well support sparse computation and fully realize the benefit of FLOPs reduction, we provide a proof-of-concep experiment on preliminary benefits of Top-k Transformer. Finally, in Section 3.3 we show that sparsity in activation is a good regularization for Transformers. Namely, enforcing sparser activation with smaller values in Top-k Transformer (without any other hacks, tweaks and hyperparameter tuning) bestows Transformers several desired properties, namely, robustness of training with erroneous annotations, less sensitivity to input noise/perturbation, and better confidence calibration of the predictions.
3.1 EFFICIENCY FOR FREE
Given an embedding dimension dmedel and an MLP intermediate dimension dg, the computational complexity of a Transformer for input sequence of length N is (f d d where the first term comes from computing the key, query, and value matrices. the second term comes from computing the self-attention matrix, and the third term comes from the MLP. For fixed
5

---

Published as a conference paper at ICLR 2023
sequence length N, and considering the fact that de is often much larger than dmodet. it is arguable that MLP poses the computational bottleneck in large Transformers. In the following, we explain how sparsity in activation map of MLP can be leveraged to significantly reduce its computational cost, without affecting the model performance.
Efficiency for the Second MLP Layer. The sparse activation immediately suggests that lot of the computation for inference with Transformers is not needed at all. That is, while doing dense matrix-matrix multiplications, much of it is about multiplying a vector by value of zero, which can be avoided save computation.
Specifically, we consider the second layer of the MLP in (1) which performs the computation
Va,
(3)
where E Rd is the intermediate activation map of MLP (see (2)) and V E R d is the layer parameter. Eq. (3) involves a simple matrix-vector multiplication which has FLOP count of modmodel X dæ. However, if a is sparse with, say nonzero entries, then the FLOP count for (3) reduces to mddee 8. Hence,
FLOP in the second MLP layer is reduced by factor of - dy
Note that is exactly the percentage of nonzeros plotted in the y-axis of e.g. Figure 1, which is 2.7% averaged across all layers, Hence, the computational cost of the second MLP layer can be reduced by a significant amount. More excitingly, the reduction factor di likely to even bigger for larger Transformer models (see Figures and 4b), pointing to greater reduction in computation.
5.
The sparsity in the intermediate activation map of MI.P does not immediately suggest reduction in computation for the first MLP layer. Nonetheless, it is possible to significantly reduce the computation in the first MLP layer by leveraging approximate nearest neighbor search, which we explain next.
Efficiency for the First MLP Layer.
Recall from (1) that the computation the first MLP layer is given by
(4) with K = [k.,..., kig] E IR being the layer parameter and being the layer input. If the output is sparse with nonzero entries, then the calculation in (4) may be formulated as finding points from the set } ) that are "closest" to the imput x measured by values of inner product. Such problem is well-known the nearest neighbor search (NNS) problem the maximum inner product search problem. While naive solution of the NNS problem has linear complexity in dn. there exists approximate algorithms (Guo et al., 2020; Johnson et al., 2019; Shrivastava & Li, 2014) that are of sublinear complexity, and using them in Transformers means that
o(K') x)),
FLOP in the first MLP layer may be reduced to have sublinear complexity in dn-
There are of course the questions of whether such approximate NNS algorithms could hurt Trans- former performance, which we leave for future study.
3.2 SPARSITY IN TRAINING VIA TOP- TRANSFORMERS
The benefit of efficiency from sparsity in Section 3.1 comes with caveats. First, while the activation maps are sparse on average, there the possibility that some of the activation maps for certain inputs are denser hence cannot benefit from sparse computation. Second, sparsity occurs only in trained Transformers while the computation is dense during and particularly the beginning of training.
Here we present Top-k Transformer, simple modification to Transformer architecture that allows us to control sparsity level for all model inputs, and throughout training. Top-k Transformer is built upon regular Transformer with the only modification being the MLP layers, where at the output of the activation function o() (see (1)) we add a Top-k thresholding operator. That is, the MLPs of Top-k Transformers perform the following computation
f(x; K, V) = V Tops (o(KT))) x)).
(5)
where (-) performs a thresholding that all entries other than those the largest & values are set to zero with & being byper-parameter subject to design choices. Note that Top-k Transformer reduces to a regular Transformer if we set & = dg. By using a small value lt the benefit of efficiency in terms of reduction in FL.P as discussed in Section 3.1 applies to Transformer training well.

---

Published as a conference paper at ICLR 2023
T5 base (Train)
Top- TS base (Trin)
T5
Top-K TS base (Eval)
15 (Triin)
64
large [Taan]
15 large (Eva))
Top-K VIT (Train) (Eval) Top-K VIT (Eval)
(Eval)
T5 38 (Train)
Tu--K T5 36 (Train)
100
125
T5 (Eval)
250
Top K TS 36 (Eval)
(a) T5
(b) VIT
Figure 5: Training and evaluation accuracy of Top-k T5 for three different sizes: base, large and 3B (left) and Top-k ViT (right) with varying Top-k Transformer is on par with regular Transformer for a large enough k. e.g. for T5 3B with k 128, and ViT with k 256, the drop is around 0.3%.
The immediate question for Top-k Transformer is whether it offers training sparsity at the cost of reduced performance. Here we conduct experiments with Top-k T5 and Top-k ViT, and evaluate their performance measured by prediction accuracy for C4 span corruption and ImageNet-2I1 k classification tasks, respectively. The results are provided in Figure 5. We see that with the Top-k T5- Base, Large, 3B) (resp.. Top-k ViT) Transformer, taking k to be 128 (resp.. 256) is sufficient for closely matching the test performance of the vanilla T5- (Base, Large, 3B) (resp., ViT). Note that this is achieved without any other syper-parameter tuning for the Top-k Transformers upon those used for a regular Transformer, and other hyper-parann parameter choices may further improve the performance of Top-k Transformers.
We now provide experimental results with Top-k Transform- wall-time benefits from FLOPs reduction discussed in Section 3.1. In particular, we evaluate the inference time our experiment,
% decodinn wall sime improvement
latency reduction of Top-k Transformer. we add a Top-k thresholding to T5X (Roberts et al., 2022)*.
T5 lerge
T5 36
11B
We gain efficiency the second MLP layer by an implemen-
tation that avoids all multiplication by zero described in Section 3.1. The decoder per-token wall time for unbatched greedy decoding during inference on single TPUv4 chip is presented in Figure 6. We observe that larger models have more wall time reduction, due to the fact that they have larger dg hence more FL.OPs reduction. In particular, for T5-1 1B observe around 10% wall time reduction with < 128, though this amount becomes smaller with a larger 256.
-10
-15
-20
32
64
128
256
Figure 6: Latency reduction for un- batched greedy decoding in decoder of Top-k Transformers on TPUv4.
Finally, we emphasize that the sparsity in Top-k Transformers is unstructured and data-dependent, which is not well supported existing computation hardwares such TPUs and GPUs. Hence, the results in Figure 6 are for proof-of-concep purposes, and are far from fully realizing the benefit of FLOPs reduction via sparsity. We leave study of better implementation of sparse computation for obtaining higher wall time reduction to future work.
3.3 BONUS! IMPROVED ROBUSTNESS AND CALIBRATION
Despite not being explicitly designed for such purposes, inducing sparse activation via Top-k Trans- former has the benefits of improving model robustness and confidence calibration. We demonstrate this using the image classification task with the ImageNet-Il lk dataset, and present the results in Table 1. All results for Top-k ViT are obtained without any model and training typer-parameter tuning upon those for VIT. Contexts and details are presented below. More results are presented in Appendix C.
"We use the implementation of jax. 1ax, approx_max__ (Cher et al., 2022) with recall target of 0.95. Thii previously demonstrated Ahmad & Scheinkman (2019) for Conv-Nets.
7

---

Published as a conference paper at ICLR 2023
Table 1: Evaluation of Top-128 ViT for ImageNet-Ik classification in terms of 1) natural accuracy with ImageNet- Ik evaluation set, 2) robust accuracy with (40%, 80%) | corrupted training labels, 3) robust accuracy under input perturbation with additive (Gaussian, Impulse, Shot) noise on evaluation images, and 4) calibration error on evaluation data measured by ECE. Top-128 VIT is on par with ViT for natural accuracy while is significantly better for model robustness and calibration.
Natural Accuracy
Accuracy w/
Accuracy under
Expected Calibration Error (ECE)
Methods
Train Label Noise
Input Perturbation
40%
80%
Gaussian
Impulse
Shot
VIT
74.85%
59.44%
25.35%
39.54%
37.37%
38.56%
8.42%
Top- 128 ViT
74.83%
62.13%
30.80%
42.29%
40.07%
40.68%
7.48%
Robustness to Label Noise. An important challenge for DNNs is that they are highly susceptible to label noise, the problem where certain percentage of training labels are corrupted or erroneously generated. This may be attributed to the fact that DNNs are often wver-parameterized. hence too "capable" that they tend to overfit, or "memorize" the noisy labels without generalizing to test data. While many dedicated techniques exist (see e.g., Algan & Ulusoy (2021); Song et al. (2022) for a review), here we show that simple Top-k Transformer can effectively address the label noise issue.
We conduct experiments using the ImageNet- Ik dataset for which we replace p% of the labels in the training set with a random label drawn uniformly from the set of all possible labels. The evaluation performance under p E (40%, 80%} label noise is presented in Table 1. It shows that Top-k offers a consistent performance gain with label noise.
Confidence Calibration. Aside from label noise, another symptom of over wee-parameterizzttion of DNNs is that they tend to be overly confident in their predictions. In the context of classification problems, they tend to assign a high (i.e., close to 1) probability to the class of its prediction, while it is more desirable that they produce a probability that is commensurate with its confidence level (Guo et al., 2017). commonly used metric for confidence calibration is the expected calibration error (ECE) (Nacini et al., 2015), which is the discrepancy between the probability to the class of model's s prediction and the probability that its prediction is actually correct.
Here we measure the calibration of Top-k VIT via ECE and report the results in Table 1. It shows that Top-k with k = 128 enables the Transformer to be more calibrated when compared to a vanilla Transformer. Furthermore, results reported in Appendix C show that ECE monotonically decreases as k: is decreased from 128 to 32.
Robustness to Input Perturbation. Another important challenge with DNNs is that their outputs tend to be sensitive naturally occurring image corruptions, which limits their application to mission critical tasks (Bhojanapalli et al., 2021). Here we evaluate the robustness of Top-k VIT to three types of additive noises, namely Gaussian noise, impulse noise, and shot noise. For that purpose, we train Top-k ViT on standard ImageNet-IK Ik training data and report their classification accuracy on ImageNet-C (Hendrycks
Dietterich, 2019), a benchmark that contains algorithmically generated Gaussian, impulse, and shot noise (among many others types) applied to the ImageNet- Ik test dataset. For each noise type, there are five severity levels. We report the averaged performance over all severity levels of each corruption type in Table I for & 128, and in Appendix C for few other values of k. We see that robust accuracy is the highest with = 64, while taking k = 128 or k = 32 also provides benefits compared to the vanilla Transformer.
RELATED WORK
Prior efforts on introducing sparsity in deep neural networks abound, though often with diverse motivations and objectives. Here we provide brief overview of several popular lines of work.
Sparsity for Efficiency. Sparsity in either model weights or activation maps is often used for improving training and inference efficiency (see e.g. Hoefler et al. (2021) for review). For activation sparsity in particular, sparsity for efficiency is explored perhaps first in ConvNets (Georgiadis, 2019; Kurtz et al., 2020; Rhu et al., 2018) before subsequently becoming key design component in many of the largest Transformer based language and vision models (Du et al., 2022; Fedus et al., b022a;b; Rajbhandari et al., 2022). The Top-k thresholding that we use in Top-k Transformer has also been
8

---

Published as a conference paper at ICLR 2023
previously used in Gupta et al. (2021) to improve memory efficiency of Transformers. However, it has been unclear a priori whether sparsity hurts model performance, hence the practice often relies on wishful design, trial-and-error, and post-hot justification (Baykal et al., 2022). Our discovery that Transformers naturally produce sparse activation maps, and that larger models are even sparser, may provide principled perspectives towards efficiently training future large models.
Sparsity for Robustness. Many works find that smaller and sparser networks obtained by model compression are more robust to adversarial perturbation (Chen et al., 2022; Guo et al., 2018; Jordao & Pedrini, 2021) and label noise (Xue et al., 2022). Another line of work that uses sparsity for robustness leverages the property that practical data corruption is often sparse (Ghosh et al., 2017; Liu et al., 2022; You et al., 2020). None of the work mentioned above based on sparsity in activation maps. More closely related to ours is the work of Ahmad & Scheinkman (2019) where sparsity in activation map of convolutional DNNs is shown to improve robustness to input perturbation, and Muthukumar & Sulam (2022) that leverages sparse activation to derive robust generalization error bounds
Sparsity for Explainability. Work on leveraging sparsity for interpreting deep learning models long exist but often in post-hoc fashion for examining the semantic meanings encoded by a neuron of a trained model (Dalvi et al., 2019). For Transformers evidence suggests that the learned knowledge is encoded mainly in its MLPs with individual neurons expressing specific factual knowledge (Dai et al., 2022). Moreover, enforcing neuron activation sparsity in MLPs helps to improve the percentage of neurons that are interpretable (Elhage et al., 2022). Hence, our discovery may point to new directions towards developing more interpretable DNNs (Cuadros et al., 2022; Sajjad et al., 2021).
Sparsity for Data Modeling. Following the seminal work of Olshausen & Field (1996), there are lot of interests in sparsity as an effective modeling of natural signals (Mairal et al., 2014). With the close resemblance of the computational structure of ReLU networks and sparse encoding algorithms (Gregor& & LeCun, 2010), it became natural to study DNN as a multi-layer sparse modeling of the data (Papyan et al., 2018). Along with substantial theoretical understanding of such a modeling are obtained (Papyan et al., 2017; Sulam et al., 2018), there are also experimental results on their practical benefits (Sun et al., 2018) though less often on modern large-scale data.
Sparsity for Theory of Over-parameetrrzeed Models. Because of its simplicity and well-develped theory in classical machine learning (Candès & Wakin, 2008; Vidal et al., 2015; Wright & Ma, 2022), sparse modeling is often used to provide theoretical understanding of modern large and over- parameterized models. This include works on implicit regularization (Chou al., 2021; Nacson et al., 2022; Vaskevicius et al.. 2019; Woodworth et al., 2020; Zhao et al., 2019), nonconvex optimization (Buhai al., 2020; Sulam et al., 2022), noise interpolators (Chinot et al., 2022; Donhauser et al., 2022; Koehler et al., 2021), etc. However, the aforementioned work uses sparsity as testbed or toy model gain insights, without implication of existence of sparsity in DNNs.
DISCUSSION
This work demonstrates the natural emergence of sparse activation in commonly used Transformer models (Section 2). The notion of sparsity pertains to the law of parsimony, a.k.a. Occam's 's razor, where among all possible explanations of observed data, the simplest ones are preferred. It is a fundamental scientific principle broadly used in various scientific and engineering subjects (Domingos, 1999; Epstein, 1984). including classical machine learning (Tibshirani, 1996). Hence, our discovery may be suggesting that the law of parsimony is playing a role in Transformers even though they are not explicitly designed so, resonating with recent view on the role of sparsity for intelligence systems (LeC'n, 2022; Ma et al., 2022; Roberts, 2021; Vasudevan al., 2021). More importantly, we back such a perspective by providing evidence of improved robustness and calibration via enforcing sparsity using Top-k thresholding (Section 3), which indicates that sparsity is indeed a pertinent prior for good generalization. We hope that our work may motivate future effort introducing sparsity in deep learning models in a more principled way for obtaining more efficient, robust, and calibrated models. Finally, while motivation of studying sparse activation in Transformers comes (partly) from study of biological brains, establishing such a connection may reciprocally benefits efforts on applying artificial intelligence to the study of biology and neuroscience (Richards et al., 2022).
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like to acknowledge helpful discussions with René Vidal and Jeremias Sulam from Johns Hopkins University, with Weijie Su from UPenn, with Yuxiang Wang from UC Santa Barbara, with Atlas Wang from UT Austin, with Nishanth Dikkala, Nikhil Vyas, Preston McAfee and Mukund Sundararajan from Google, with Subutai Ahmad from Numenta, with Wei Hu, Salar Fattahi, and Jianhao Ma from University of Michigan, with Tuo Zhao from Georgia Tech. We particularly thank Donhauser Konstantin from ETH Zurich for interesting discussion on hypothesis for emergence of sparsity.
REFERENCES
Subutai Ahmad and Luiz Scheinkman. How can we be so dense? the benefits of using highly sparse representations. arXiv preprint /903. /257, 2019.
Mohsin S Ahmed, James Priestley, Angel Castro, Fabio Stefanini, Ana Sofia Solis Canales, Eliza- beth M Balough, Erin Lavoie, Luca Mazzucato, Stefano Fusi, and Attila Losonczy. Hippocampal network reorganization underlies the formation of a temporal association memory. Neuron, 107(2): 283-291, 2020.
Görkem Algan and Ilkay Ulusoy. Image classification with deep learning the presence of noisy labels: A survey. Knowledge-Based Systems, 215: 2021.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley, Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp. 1352-1361. PMLR, 2021.
Alison L Barth and James FA Poulet. Experimental evidence for sparse firing in the neocortex. Trends in neurosciences, 3566)334-355, 2012.
Cenk Baykal, Nishanth Dikkala, Rina Panigrahy, Cyrus Rashtchian, and Xin Wang. A theoretical view on sparsely activated networks. arXiv preprint arXiv:2208. 044611, 2022.
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li. Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10231-10244 2021.
Cares-Darius Buhai, Yoni Halpern, Yoon Kim, Andrej Risteski, and David Sontag, Empirical study of the benefits of overparameterrizaton in learning latent variable models. In International Conference on Machine Learning. pp. 1211 1211-1219. PMLR, 2020.
Emmanuel J Candès and Michael B Wakin. An introduction to compressive sampling. IEEE signal processing magazine, 25(2))21-30, 2008.
Tianlong Chen, Zhenyu Zhang, Santosh Balachandra, Haoyu Ma, Zehao Wang, Zhangyang Wang, et al. Sparsity winning twice: Better robust generalization from more efficient training. In International Conference Learning Representations, 2022.
Felix Chern, Blake Hechtman, Andy Davis, Ruiqi Guo, David Majnemer, and Sanjiv Kumar. Tpu-knn: K nearest neighbor search at peak flop/s. arXiv preprint arXiv:2206.14288. 2022.
Geoffrey Chinot, Matthias Löffler, and Sara van de Geer. On the robustness of minimum norm interpolators and regularized empirical risk minimizers. The Annals of Statistics, 50(4):2306-233; 2022.
Hung-Hsu Chou, Johannes Maly, and Holger Rauhut. More is less: Inducing sparsity via overparam- eterization. arXiv preprint arXiv:21 1027, 2021.
Xavier Suau Cuadros, Luca Zappella, and Nicholas Apostoloff. Self-conditioning pre-trained language models. In International Conference on Machine Learning, pp. 4455 4455-447,, PMLR, 2022.
10

---

Published as a conference paper at ICLR 2023
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1493-8502, 2022.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6309-6317, 2019.
Mostafa Dehghani, Alexey Gritsenko, Anurag Amab, Matthias Minderer, and Yi Tay. Scenic: A jax library for computer vision research and beyond. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21393-21311 21398, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HILT, pp. 4171- 4186, 2019.
Pedro Domingos. The role of occam's razor in knowledge discovery. Data mining and knowledge discovery, 3(4):409-425, 1999.
Konstantin Donhauser, Nicolo Ruggeri, Stefan Stojanovic, and Fanny Yang. Fast rates for noisy interpolation require rethinking the effects of inductive bias. arXiv preprint 2rXiv:2203.0333 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pp. 5547-5569, PMLR, 2022.
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hathiel-Doodde Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022. http://meeeeeeiiiuisup//////////
Robert Epstein. The principle of parsimony and some applications in psychology. The Journal of Mind and Behavior, pp. 130, 1984.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01663 016777 2022a.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022b.
Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp.
7085-7095, 2019.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
11

---

Published as a conference paper at ICLR 2023
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 026777 2017.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference international conference machine learning, pp. 399 -99-406, 2010.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chem, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference Machine Learning, pp. 3887-3896. PMLR, 2020.
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse dnns with improved adversarial robustness. Advances neural information processing systems, 31, 2018.
Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. Memory-efficient transformers via top-k attention. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pp. 39-52, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026- 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22(241):1-124 2021.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering, 2020. URL /tttps://arxttai wttp:/wwwwww...///////a///////// 2000. 0128.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
Artur Jordao and Hélio Pedrini. On the effect of pruning on adversarial robustness. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1-11, 2021.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 1..11865//1/2220.0. 10210003/1///20.mmmm.lll.. main.550. URL https: /www aclwes org/ anthology/ 550.
2020.
Jason ND Kerr, David Greenberg, and Fritjof Helmchen. Imaging input and output of neocortical networks in vivo. Proceedings of the National Academy of Sciences,
2005.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Frederic Koehler, Lijia Zhou, Danica J Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting. In Advances in Neural Information Processing Systems, 2021.
12

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, (016):84900, 2017.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference Machine Learning, pp. 5533-5543. PMLR, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Damielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Yann LeCun, A path towards autonomous machine intelligence version 0.9. 2, 2022-06-77. 2022.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521 521(7553):436-444 2015.
Zonglin Li, Ruiqi Guo, and Sanjiv Kumar. Decoupled context processing for context augmented language modeling. arXiv preprint 10.1522200.05758 2022.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization. 2022.
Yi Ma, Doris Tsao, and Heung- Yeung Shum. On the principles of parsimony and self-consistency for the emergence of intelligence. Frontiers of Information Technology & Electronic Engineering, pp. 1-26, 2022.
Julien Mairal, Francis Bach, Jean Ponce, et al. Sparse modeling for image and vision processing. Foundations and Trends in Computer Graphies and Vision, 8(2-3):85 283, 2014.
Ramchandran Muthukumar and Jeremias Sulam. Adversarial robustness of sparse local lipschitz predictors. arXiv preprint 1xXi::2202.13216,, 2022.
Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference Machine Learning, pp. 16270-16295. PMLR, 2022.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, (816633)) -69, 1996.
Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research, 18(1):2887-2938, 2017.
Vardan Papyan, Yaniv Romano, Jeremias Sulam, and Michael Elad. Theoretical foundations of deep learning via sparse representations: multilayer sparse model and its connection to convolutional neural networks. IEEE Signal Processing Magazine, 35(4):72--9, 2018
Cindy Poo and Jeffry S Isaacson. Odor representations in olfactory cortex: "sparse" coding, global inhibition, and oscillations. Newron, 62(6):850-811, 2009.
Haozhi Qi, Chong You, Xiaolong Wang, Ma, and Jitendra Malik. Deep isometric learning for visual recognition. In International Conference on Machine Learning, pp. 7824-7835. PMLR, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140): 2020.
13

---

Published as a conference paper at ICLR 2023
Samyam Rajbhandari, Conglong Li, Zhewei Yao. Minjia Zhang. Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-most moc: Advancing mixture-of-expers inference and training to power next-generation ai scale. arXiv preprint Mxki::201.0555000 2022.
Minsoo Rhu, Mike O' O'Conoo,, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W Keckler. Compressing dma engine: Leveraging activation sparsity for training deep neural networks. In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 78 91. IEFF, 2018.
Blake Richards, Doris Tsao, and Anthony Zador. The application of artificial intelligence to biology and neuroscience. Cell, 185(15):240-2643, 2022.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyzz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov. Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with 5x and seqio. arXiv preprint 20Xi::203.177189 2022. URL https: /arxiv. org/ hog//bb//20 2203222332. 17189.
Daniel A. Roberts. Why is ai hard and physics simple?, 2021.
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep nlp models: A survey. arXiv preprint arXiv: 108. 13138, 2021.
Anshumali Shrivastava and Ping Li. Asymmetric Ish (alsh) for sublinear time maximum inner product search (mips). Advances neural information processing systems, 27, 2014.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil ne--il Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions Neural Networks and Learning Systems, 2022.
Jeremias Sulam, Vardan Papyan, Yaniv Romano, and Michael Elad. Multilayer convolutional sparse modeling: Pursuit and dictionary learning. IEEE Transactions on Signal Processing. 66(15): 4090-4104, 2018.
Jeremias Sulam, Chong You, and Zhihui Zhu. Recovery and generalization in over-realized dictionary learning. Journal of Machine Learning Research, 23(135): 2022.
Xiaoxia Sun, Nasser M Nasrabadi, and Trac D Tran. Supervised deep sparse coding networks. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 346-350. IEEE, 2018
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1 2627-288 1996.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: all-mip architecture for vision. Advances in Neural Information Processing Systems, 34: 24261-24272 2021.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 32 42, 2021.
Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems, 32, 2019.
Rama K Vasudevan, Maxim Ziatdinov, Lukas Vlcek, and Sergei V Kalinin. Off-the-shelf deep learning is not enough, and requires parsimony, bayesianity, and causality. npj Computational Materials, 7(1):1-6, 2021.
14

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems. 30, 2017.
Rene Vidal, Yi Ma, and Shankar Sastry. Generalized principal component analysis. Interdisciplinary Applied Mathematics, 43:22-23, 2015.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference Learning Theory, pp. 3635-3673, PMLR, 2020.
John Wright and Yi Ma. High-Dimensional Data Analysis with Now-Dimensional Models: Principles, Computation, and Applications. Cambridge University Press, 2022.
Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Superior generalization of smaller models in the presence of significant label noise. arXiv preprint arXiv:2208. 08003, 2022.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pp. 10767-10777. PMIR, 2020.
Chong You, Zhihui Zhu, Qing Qu, and Yi Ma. Robust recovery via implicit bias discrepant learning rates for double over parameterization. Advances in Neural Information Processing Systems, 33: 17733-17744, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representations, 2019.
Peng Zhao, Yun Yang, and Qiao-Chu He. Implicit regularization via hadamard product over- parametrization in high-dimensional linear regression. arXiv preprint
2019.
15

---

Published as a conference paper at ICLR 2023
Appendices
The appendices are organized follows. In Section A we provide the implementation details for experiments conducted in this paper. In Section B we demonstrate the emergence of sparse activation in other architectures and with other optimizers than those used in Section In Section we provide additional experiments upon those in Section 3 to demonstrate the benefits of sparsity. In Section D we explore the potential causes of sparsity, with a focus on the effect of training data. Section E we present derivation to show that during early training, the final MLP layer's intermediate activation tends to get sparse. Finally in Section F we present insights on the emergence of activation sparsity from experiments 2-layer MLP models.
A IMPLEMENTATION DETAILS
A.1 T5
For most of the experiments, except the Top-k transformer, we used vanilla T5 architecture (Raffel et al., 2020). We trained model with Adafactor optimizer, an inverse square root learning rate schedule, and no dropout. For the first 10,000 steps we also use a fixed learning rate of 0.01 as warm-up. The training task is span corruption without any mixture, and unless specified otherwise, we train the model for 100,000 steps with batch size of 256 to save compute and time, as the sparsity or accuracy trend is already clear by then. We used 512 tokens the encoder side and 114 tokens the decoder side.
A.2 VIT
Following Dosovitskiy et al. (2021), we train ViT using ADAM (Kingma & Ba, 2015) as the optimizer with 31 0.9, 32 0.999. Other training details such weight decay, dropout rate, and learning rate all follow the description in (Dosovitskiy et al., 2021, Section B.1) except that we train for 180 epochs (as opposed 300) on ImageNet-Ile Ik.
A.3 T5 / VIT CONFIGURATIONS
For the reader's s convenience, we summarize the configuration of varying T5 ViT models used in paper in Table A.I.
Table A.I: Configuration of T5 and ViT that are used the experiments. dmodel and di are defined in Section 1.3. # Layers is the number of encoder decoder layers for T5 and encoder layers for ViT.
T5
ViT Large
Small
Base
Large
3B
11B
Base
Huge
dmodel
512
768
1024
1024
1024
768
1024
1280
de
2048
3072
4096
16384
65536
3072
4096
5120
Layers
6
12 12
24 + 24
24 +
24 + 24
12
24
32
Parameters
60M
220M
770M
2,800M
11,000M
86M
307M
632M
B ADDITIONAL RESULTS ON PREVALENCE OF SPARSITY
B. SPARSITY AND NETWORK ARCHITECTURE
We evaluate the sparsity level of activation map in several commonly used network architectures beyond T5 and ViT. This includes BERT which is also Transformer based architecture, as well as non-Transformer based architectures such MLP-Mixer and ConvNets. We also examine whether residual connection accounts for the emergence of sparsity.
16

---

Published as a conference paper at ICLR 2023
Liga
(a) BERT Base
(b) BERT Large
(c) Pre-activations: Layer (d) Pre-activations: Layer 12
Figure B.2: Plots a, b: Percentage of nonzero entries in activation maps of BERT Base and Large models (Devlin et al., 2019) trained on Wikipedia dataset. We observe high levels of sparsity (<10%) similar to other Transformer models. Plots c, d: Histograms of pre-activation values for layers and 12 of a Bert Base model. We notice that while at initialization the activations are distributed with mean O, the mean quickly shifts negative as the training progresses, resulting high levels of sparse activation values.
(a) GeLU Activation
(b) Sigmoid Activation
(c) Tanh Activation
Figure B.3: Layer Preactivation histograms for BERT Base models with different activation functions. We observe similar behavior as ReLU with GeLU and Sigmoid activations. However Tanh activation has different distribution of preactivation values. The network doesn't 't show sparsity and the accuracy also worse in comparison to ReLU/GeLU.
BERT. We evaluate the sparsity level of BERT models (Devlin et al., 2019). We specifically consider BERT Base (12 layers) and BERT Large (24 layers) Transformer models, with ReLU activation in the MLP layers. We follow the same training receipe as Devlin et al. (2019) and pre-train these models on Wikipedia and Books dataset using Masked Language Mod- elling (MLM) objective. We train for 450000 steps with a batch size of 1024 using AdamW optimizer with le learning rate.
Percentage of nonzaros
10'
10"
(VTT)
Token-mixing MLP
Channel mixing MLF
Layer Index
In Figure B.2 we plot the sparsity levels of both BERT models for all the intermediate MLP layers (plots and b). We observe that both these models exhibit high levels of sparsity (< 10%) as other Transformer models. We further visualize the pre- activation values of the MI. layers as histograms in plots and We observe that while they have mean O at initialization, the mean quickly becomes negative as training progresses, resulting in high sparsity levels. Finally, in Figure B.3 we provide visualization of pre-activation of values with several popular activation functions, including GeLU, Sigmoid, and Tanh. We observe similar distribution of preactivation values as ReLU with GeLU and Sigmoid activations. However Tanh activation has different distribution of preactivation values. With Tanh activation, the network does not show sparsity and the accuracy is significantly worse in comparison to ReLU/GeLUU
Figure B.I: Percentage of nonzero entries in activation maps of MLP-Mixer trained on ImageNet- k.. Results for token- miiennmixing and channel-mixing MLPs are plotted in separate curves.
MLP-Mixer. We evaluate the sparsity level of the MI.--Mxer (Tolstikhin et al., 2021), an all-MLP architecture constructed from cascading moken-mixing and channel-mixing MIPs. Specifically, we use Mixer-B16 the architecture, ADAM with 81 0.9, B8 0.999 the optimizer, and train on ImageNet-22Ik for 300 epochs. While Tolstikhin et al. (2021) sweeps over a product set of hyper-parameters, here for simplicity 0.03, gradient norm clipping at 1.0, base learning rate of 0.003, RandAugment magnitude of 10, no mixup, no stochastic depth, and no dropout.
fixed set of syper-parameete with weight decay of
Figure B.I shows the sparsity level at the intermediate layer of both token mixing and channel mixing MLPs of Mixer-B16. We also plot the sparsity level of VIT (i.e., the plot in Figure 2a) to Figure B.
17

---

Published as a conference paper at ICLR 2023
100
100
Percentape nonzercs
Percentage
10
1st Layer layer
layer
2nd Layer
3rd Loyer
10
Block Index
Block Index
(a) R8
(b) ResNet-50
Figure B.4: Percentage of nonzero entries in activation maps of ResNet-18 and ResNet-50 trained on ImageNet- Ik. Results for the two (resp., three) layers in each residual block (resp., bottleneck residual block) of ResNet-18 (resp., ResNet-50) are plotted in separate curves.
100
100
Percertzge nonzeros
Percentage monoetos
10
3-
Standadd Traning
*
Standar: Traning
n. IR
0.14 LR
0.1x UR, No
0.1 LR, No
Block Index
Block Incex
(a) 1st Layers
(b) 2nd Layers
Figure B.5: Effect of batch normalization (BN) on sparsity level across layers ResNet-18 18. Because ResNets cannot be effectively train without BN, reduce the learning rate (LR) by factor of 10, and multiply the residual branch by a trainable scalar initialized at
for comparison, It can be seen that the first four layers of channel mixing MI.P and VIT have almost identical sparsity levels, while the rest of the layers (other than the last one) of channel mixing MLP are denser than the corresponding layer of VIT. On the other hand, the token mixing MLPs produce dense activation maps with more than 50% nonzero entries, probably because the dimension of the activation maps (384) is too small.
Convolutional Neural Network (Ctn Sparsity in activation maps has been studied for Con- vNets such as the AlexNet (Krizhevsky et al., 2017) at least as early as in the work of Rhu et al. (2018). There also follow-up work (Georgiadis, 2019; Kurtz et al., 2020) on how enforcing sparse activation maps can help to gain computation efficiency. For completeness, we evaluate and present results for the sparsity level of residual networks (ResNets) (He et al., 2016), which is of the most commonly used CenvNets, trained on ImageNet-1k: In particular, we focus on ResNet- 18 and ResNet-50 which are constructed from stacking standard residual blocks and 16 "bottleneck" residual blocks, respectively, where each block has two and three convolutional and ReLU layers, respectively. We examine the sparsity of activation maps after each of the ReLU layers in each residual block.
The results for ResNet-18 and ResNet-50 are reported in Figure B.4a and Figure B.4b, respectively. Here, the x-axis is the index of the residual block, and the sparsity of different layers in the residual blocks are plotted with separated curves in each figure. It can be observed that
Layers near the network output tend produce sparser activation maps than layers near the network input. This is aligned with the observation with ViT trained on ImageNet- 1k (see Figure 2b). For each residual block, the intermediate layers (i.e., the 1st layer for ResNet-18 18 and the 1st & 2nd layers for ResNet-50) produce sparser activation maps than the output layer (i.e., 2nd layer for ResNet- 18 and 3rd layer for ResNet-50).
In addition, all residual blocks are divided into four stages that have different output feature map sizes. For ResNet-50, the four stages composed of blocks 0 2, 6, 7 12, and 13 15. Figure B.4b
18

---

Published as a conference paper at ICLR 2023
100
100
Percentape nonzeres
Percentage
10
ResNet1B-We4
ResNat1B-WGG ResNer18-W13
ResNer18-w118
ResNet10-W256
ResNet18 W256
Block Index
Block Index
(a) Layers
(b) 2nd Layers
Figure B.6: Effect of network width E {64, 128, 256} sparsity level across layers of ResNet-18.
shows that there are patterns how sparsity level varies within each stage and across the boundary of the stages.
For the 1st layers, percentage of nonzeros decreases within each stage, and jumps up from the last layer of each stage to the first layer of next stage.
For the 2nd layers, percentage of nonzeros decreases quickly the beginning each stage then becomes stable.
For the 3rd layers, percentage of nonzeros tend to increase slightly within each stage, and jumps down from the last layer of each stage to the first layer of next stage.
Such observations may help to understand the role each stage in ResNets.
Comparing the percentage of nonzero entries in ResNets (shown in Figure B.4a and Figure B.4b) and for Transformers (shown in Figure 2b), both which are trained on ImageNet- -keettt, we see that ResNets produce much denser activation maps with more than 10% nonzero entries in all layers. One possible explanation is that ResNet uses batch normalization (BN) before each activation function, while Transformer's MLP does not have BN before the activation function. To understand the effect of BN on sparsity, we conduct an experiment with BN ResNet removed. Because ResNet cannot be effectively trained without BN, we decrease the learning rate from standard ResNet training by factor of 10. Moreover, we add a learnable scalar multiplier that is initialized as to all the residual branches, following the study Bachlechner et (2021); Qi et al. (2020). The results for comparing with standard ResNet are reported in Figure B.5, where to separate the effect of using a smaller learning rate, we also compare with the method training a regular ResNet but with a small learning rate compared to standard training. The two subfigures of Figure B.5 show the effect of width on sparsity of the first and second layers each residual block, respectively. It can be observed that, removing BN does not significantly change the sparsity level, except for small set of layers.
Meanwhile, the trend that larger models are sparser for Transformers (see Section 2.2) holds for ResNets well, seen in Figure B.6. Here, we vary the width of ResNet-18 by multiplying the number of output channels of each convolutional layer by factor 128), and 4 (for width 256). The two subfigures show the effect of width sparsity of the first and second layers in each residual block, respectively. In both cases, wider models have smaller percentage of nonzero entries across all layers, except for the very last layer (i.e., the 2nd layer in block 47 shown in Figure B.6b).
(for width 64), 2 (for width
Sparsity and Residual Learning. We provide a study on the effect of residual connections on activation sparsity. Each Transformer block contains two types of residual connections: the one that is in parallel with the attention blocks, and the one that is in parallel with the MLP blocks. We focus on the residual connection parallel to the MLP blocks. We perform two different studies.
Effect of shortcut connection. Towards that, we train two T5-Large models, one using the vanilla Transformer block and the other with residual connection removed for the Transformer block on encoder layer 6 (i.e., the 7th encoder layer, as we count from 0). There is 1.6% evaluation accuracy drop with the latter model compared to the former model.
The percentage of nonzero entries of these two Transformers encoder layers and in Figure B.8 for the decoder layers. It can be seen that in encoder layer 6 for which the residual connection removed, the sparsity has very different trend during
presented in Figure B.7 for the
19

---

Published as a conference paper at ICLR 2023
training compared to the corresponding layer of the vanilla Transformer. Moreover, the sparsity level at all other layers also changes, though to a much smaller extend.
Effect of initialization scale of the residual branch. Many works have found that having the residual branch initialized at smaller scale helps with stabilizing and accelerating the training of residual (convolutional) networks (Goyal et al., 2017; Zhang et al., 2019) and Transformers (Touvron et al., 2021). Here for simplicity consider the idea from Bachlechner et al. (2021); Qi et al. (2020) where a trainable scalar multiplier that is initialized at zero is applied to the residual branch (a.k.a., ReZero).
We consider ViT trained on ImageNet-21k with ReZero added to the MLP modules. We find that this increases the training accuracy from 46.15 46.15% to 46.85% but reduces the validation accuracy from 47.58% to 46.75%. We plot the sparsity level of ViT with ReZero and compare it with the vanilla ViT in Figure B.9, It can be seen that ReZero reduces the percentage of nonzeros in layers near the network output.
encoder aayer
crooder uyer
erooeer layer
Gorlo
05
Van
14-
C3
9:2
0.2
2001
2081
Vedia
Venila 15
04-
Van 75
0:2
0.C
crooder
0.40
VVa 15
M.Presiiall
0:50
0.00
6.27
6.19
6.0
6.0
20000
46900
00000
B00CE
100000
20002
40893
50493
B3000
100830
20093
40043
50009
00300
3000
encodar ayer
encoder layer 10
03c2cer layer 11
0.0
Mi more tayer
0.25
4.20
0.23
6.00
530
0.00
20000
48000
30000
30008
100900
20090
40003
50093
B3300
100899
20099
40003
50009
50300
330009
Figure B.7: Percentage of nonzero entries in activation maps in vanilla T5-Large and in a L5-Large with the residual connection parallel to MLP removed in the 7th encoder layer (i.e., encoder layer 6). Different subplots correspond to different encoder layers (see Figure B.8 for results on decoder layers). The encoder layer 6, which has its residual connection removed, shows significant difference in both sparsity and the trend of sparsity during training. Sparsity level in other layers changes from vanilla T5- as well, though to a smaller extent.
20

---

Published as a conference paper at ICLR 2023
decoder layer 12
deccda
decedar
0204
0703
0333
6375
6154
6325
50m
0.00
0002
0.00
8:22
0/01
0.02
23582
50083
60C2
10000
2200
1350
LI000
2088
40300
10000
30030
dccoctt laye 15
descdel laya
cocodal uyy 17
1.75
MLPresiuu removedion encoder Syer
-al-ordatreeneee encater lyee
NBT
MLF resiratt sumoredes encter kyet
615e
0.100
1335
1.39
0.00
8.055
6054
0.00
23088
10003)
3.35
Vakiaa 15
Variaa
0.12
0.10
0.58
0.00
0.04
0.0
layer
M1Prendua arcoder
HL.
0.14
5.37
0.12
1.31
E.44
0.04
20439
40003
5001
30209
100000
20008
4300
E300e
B0006
LOC830
0
20006
43000
68000
60300
13000
Figure B.8: Same setup as Figure B.7, but showing the results for the last 12 layers of the decoder.
50
Percentage of nanzeros
ViT
VIT with ReZero
10
0.5
0.0
2.5
5.0
7.5
10.0
Layer Index
Figure B.9: Effect of initialization scale the residual branch. We add a scalar multiplier that is initialized at 0 on the residual branch (a.k.a., ReZero Bachlechner et al. (2021)) of the ML. modules of a ViT, and train the model on ImageNet-21k. The percentage of nonzero entries is compared with those obtained with a regular VIT.
21

---

Published as a conference paper at ICLR 2023
.00
100
Percentape nonzeres
SGD ADAM
Percentage nonzerros
3-
3-
ADAM
Block Index
Block Index
(a) Layers
(b) 2nd Layers
Figure 1010: Effect of optimizer sparsity level across layers of ResNet-18.
100
SCD ADAM
Percentage of nonzeros
Berrantgg of nanowroo
20
10
10
Percentage
SGD ADAM
3
ADAM
15
10
Block Index
Black Index
(a) 1st Layers
(b) 2nd Layers
(c) 3rd Layers
Figure B.11: Effect of optimizer sparsity level across layers of ResNet-50.
B.2 SPARSITY AND OPTIMIZER
Transformers are usually trained using ADAM or its variants as the optimizer (Kingma Ba, 2015). It may be curious to ask whether the emergence of sparsity is specific to such optimizers and whether other optimizers. such as stochastic gradient descent (SGD), also leads to sparse activation maps. However, we find that SGD cannot effectively train Transformer architectures such as T5 and ViT. Hence, we study the effect of optimizer on activation sparsity by looking at ResNet trained on ImageNet-16 1k following the setup in Section B.1, since both SGD and ADAM can effectively train the network. To train ResNet with ADAM, we use the same hyper-parameter with the only difference being that the optimizer is ADAM with 81 0.9, _3 0.999. To make the comparison with SGD fair, we tune the base learning rate for ADAM and select 3e - 3, which the one that gives the highest training accuracy among the set of {le 4, 3e 4, 1e 3, 3e - 3, le 2). The training accuracy obtained by ADAM with base learning rate 3e 3 similar to that obtained by SGD, namely. 67.5 57.8% by ADAM vs 69.3% by SGD with ResNet-18, and 75.0% by ADAM vs 78.5% by SGD with ResNet-50.
as those used SGD,
The results for ResNet- 18 and ResNet-50 are presented in Figure B.10 and Figure B. respectively. For ResNet-18, 18, we see that ADAM leads to a smaller percentage of nonzero entries particularly towards the output of the network for the first layers of each residual block. In contrast, ADAM and SGD have very similar sparsity level at the second layers of each residual block. Similar observation holds for ResNet-50, where the percentage of nonzero entries smaller with ADAM for the first and second layers of each residual block, while for the third layer the sparsity level does not change much.
B.3 SPARSITY FINETUNING
In this section we show that activation sparsity not only occurs after model pretraining but persists after further finetuning downstream tasks. Here we take a T5 that has been pretrained on C4 as described in Section 1.3, and finetune the model on a open domain Natural Question (Kwiatkowski et al., 2019) QA task. We follow the set up in Li al. (2022), where the retrieved passages independently encoded by the encoder, and then passed to the decoder via cross attention. The decoder takes the question the prefix and produces the answer. The decoder is standard auto-
22

---

Published as a conference paper at ICLR 2023
Percentage of nonzeros
T5-Pretrain (Encoder) 15-Finetune (Encoder)
Percentage of nonzeros
T5-Pretrain (Decoder)
13-Finetune (Decoder)
0.5
0.5
10
10
Layer Index
Layer Index
(a) Encoder
(b) Decoder
Figure 1.12: Percentage of nonzero entries across different layers trained T5 after pretraining vs after finetuning on question answering task. Left: Results encoder layers. Right: Results on decoder layers.
regressive setup, despite passing the question as prefix. We used the same Wikipedia passages retrieved by FiD (Izacard & Grave, 2020) which uses DPR (Karpukhin et al., 2020) as retriever. For this experiment, we used 20 passages for each question. When calculating the sparsity level, we ignored the activation produced from paddings, for both encoder and decoder. The sparsity level of different encoder and decoder layers after finetuning reported in Figure B.12 and is compared to those before the finetuning. can be seen that finetuning does not drastically change the sparsity level.
C ADDITIONAL RESULTS ON BENEFITS OF SPARSITY
C.I TOP- DOES NOT SIGNIFICANTLY AFFECT TRAINING CONVERGENCE
One may argue that the convergence of Top-k Trans- former can be slower than that of vanilla Trans- former because, with fewer neurons being activated, the amount of parameters that have nonzero gradi- ent associated with each training sample is smaller.
0.4
Accuracy
(Eval)
Tap-64 VIT (Tain)
Here we plot the training curves for VIT as well as Top-k VIT with h E [64, 128, 256} on ImageNet- 21K (this is the same setup as the experiments in
0.2
Tap-128 VIT
VIT (Eval)
Top 256 VIT (Evall
VIT (Train)
(Pval) VIT (Tratn)
Figure 5b), and report the results in Figure C.I. It can observed that taking Top-k does not signifi- cantly reduce convergence speed, particularly when is relatively large (c.g., k = 256).
0.0
100000
200000
Step
Figure C.1: Learning curves for results ported in Figure 5b. ViT and Top-k VIT have similar convergence rate.
C.2 BENEFITS OF SPARSITY PERSISTS WITH & NNRM INDUCED SPARSITY
While Top-k thresholding used in Section 3.3 to demonstrate the benefit of sparsity, we show that other means of obtaining sparsity, such as an explicit norm regularization, also provides such benefits.
We experiment with VIT for ImageNet-11 k classification under the same setup as in Section 3.3. Here instead of the Top-k VIT, we train a regular ViT but with additional loss term, which is the sum of the & norm of all activation maps of ViT across all layers. We refer to the method LI-ViT. We vary the weight& on the E loss in the set À E 0.5, 1.03 to control the strength of the regularization, and denote the corresponding methods L1-ViT- 0.5, 1.0).
The sparsity level, natural accuracy, robust accuracy under input perturbation, and ECE of LI-VIT are reported in Table C.I. We see that with À 0.1 or 0.5, the averaged percentage of nonzero entries do not change much, but already demonstrates performance gain in terms of accuracy under input perturbation and calibration without hurting the natural accuracy. Using À 1.0 drastically reduces
23

---

Published as a conference paper at ICLR 2023
VT
50
Tup-32
Top-32
Top-32
Top-64
50
Top-64
Tap-64
Accuracy
40
Top
128
Accuracy
40
Top-256
Tap-256
Top-256
30
20
20
10
Sevenity Comruptton
Severity Corruption
Severity Corruption
(a) Gaussian Noise
(b) Impulse Noise
(c) Shot Noise
Figure C.3: Performance of Top-k ViT on corrupted ImageNet- 1k test data with Gaussian noise (left), impulse noise (middle), and shot noise (right), each under five severity levels. Top-k improves robustness for all noise types and on corruption levels with a suitable choice of k.
Table C.I: Evaluation of VIT with varying weight E {0.1, 0.5, 1.0) & regularization upon activation maps for ImageNet- Ik classification in terms of 1) averaged percentage of nonzero entries in activation maps across all layers, 2) natural accuracy (i.e., on ImageNet- evaluation set), 3) robust accuracy under input perturbation with additive (Gaussian, Impulse, Shot) noise, and 4)
calibration error measured by ECE.
Avg. Perc. of Nonzeros
Natural Accuracy
Accuracy under Input Perturbation Impulse
Expected Calibration Error (ECE)
Methods
Gaussian
Shot
VIT
5.67%
74.85%
39.54%
37.37%
38.56%
8.42%
LI-VIT-0.1
5.66%
75.02%
40.70%
38.52%
39.56%
8.37%
L1-ViT-0.5
5.76%
74.83%
42.46%
40.64%
40.96%
8.22%
L1-ViT-1-0
1.60%
73.21% %
40.26%
38.01% %
38.95%
6.34%
the percentage of nonzero entries but it offers better performance in terms of accuracy under input perturbation and calibration compared to VIT.
C.3 ADDITIONAL RESULTS FOR CALIBRATION
Under the same setup as in Section 3.3, we report the calibration of Top-k VIT during training and with vary- ing values of k. and report the results in Figure C.2. At the beginning of training the model has a low ECE because the output probabilities are mostly uniformly distributed across all classes hence the model not con- fident, and that its prediction is purely random hence wrong with high probability. The model tends to be- come overly confident the training progresses, hence the ECE increases particularly towards the end of train- ing. What we can observe is that Top-k enables the Transformer to be more calibrated when compared to vanilla Transformer, particularly for small values of The results with & = 128 and its comparison with the vanilla ViT is also presented in Table
Tap-32
Top 64
ECE 196)
Tup-128
-3
2
10000
30000
50000
Steps
Figure C.2: Confidence calibration of Top-
ViT for ImageNet-Il classification.
C.4 ADDITIONAL RESULTS FOR ROBUSTNESS TO INPUT PERTURBATION
Under the same setup as in Section 3.3, we report the performance of Top-k ViT with varying value of A in Figure C.3. We with & = 64 or = 128 and the severity level is high. The averaged performance over all severity levels of each corruption type reported in Table 1.
that Top-k ViT offers performance gain the vanilla ViT, particular
24

---

Published as a conference paper at ICLR 2023
50
50
Nercetaage nonzeros
Porsentage nonzerss
Random
Baseline (IK-21K)
Porconage or nonzoraa
1
0.5
70t random into
0.5
0.5
Baseline (IM 2Ik)
100% roncor obat
Layar Indox
Layer indcx
Layer trocx
(a) Random label
(b) Random image
(c) Infinite data
Figure D.I: Percentage of nonzero entries in VIT trained on ImageNet- (IM-21K) with (a) random labels where p% labels replaced by labels drawn from uniform distribution with E (50%, 70%, 100%]. (b) random images where each image is replaced by one where the pixels drawn from i.i.d. uniform distribution in [-1, and (c) infinite data where sufficient training data is generated by drawing random image and random label pairs so that the model is never trained on the same pair twice.
D SPARSITY FROM TRAINING DYNAMIC?
this section we study the causes of sparsity activation maps of trained Transformers. Towards that, in Section D. D.2, and D.3 we present set of hypotheses and design corresponding experiments to validate or disprove them. We discuss the observation from the experiments and draw a tentative conclusion in Section D.4 on attributing sparsity to the training dynamic, with theoretical evidence.
D.1 SPARSITY FROM LABELS?
Transformers are usually trained via supervised learning (e.g.. using the ImageNet dataset for ViT) or self-supervised learning (e-g., using the span corruption task for T5). In both cases, the training labels provide a pertinent and meaningful description of the corresponding training data (e.g.. image for ViT and text for T5). Sparsity may arise because the label set provides structured organization of the massive training data, hence the training dataset admits a compact representation. This motivates us formulate the following hypothesis.
Hypothesis D.1 (Sparsity from labels). Sparsity trained Transformers arises from the labels for Transformer training, e.g., human annotations in supervised training generated labels from data itself in self-supervised learning.
We use a random label experiment with ViT for image classification to test Hypothesis D.1. Specifi- cally, we generate with random labels drawn uniformly at random from the set of all possible labels, where p is varied to examine the effects. With such dataset, the labels for certain percentage of images do not provide a meaningful description for the content the image. Hence, Hypothesis D.1 is valid, then the activation map will become dense.
training dataset by replacing p% of the labels in the ImageNet-21k dataset
The sparsity level of ViT trained on the random label datasets is shown in Figure D. la. It can be seen that the percentage of activated neurons decreases with an increasing percentage of label noise up 70%. An even higher label noise level at 100% changes the sparsity level across layers as the shallow layers (i.e., layers 0 4) becomes sparser, while the deep layers (i.e., layers 5 11) becomes denser. Nonetheless, even with 100% label noise, all layers have 10% activated neurons.
D.2 SPARSITY FROM DATA?
While modern image and text data are often of high-dimensional,, their intrinsic degree freedom much smaller, i.e., they are low-dimensional and admit compact representations (Vidal al., 2015; Wright & Ma, 2022). Hence, even if the labels do not provide meaningful descriptions of the data, it may still be possible that Transformers extract now-dimensional structures from data and produce compact representations in the form sparse activation maps. This motivates the following hypothesis.
25

---

Published as a conference paper at ICLR 2023
Hypothesis D.2 (Sparsity from natural data). Sparsity in trained Transformers arises from natural training data (e.g.. images for ViT and texts for T5).
We use random image experiment test Hypothesis D.2. With the ImageNet-211 dataset, we replace each image with random image generated by drawing pixel values from an i.i.d. Uniform distribution in the range of [0, 255], and use these images (instead of the original images in ImageNet- 21k) for model training. Such random images do not contain any low-dimensional structures nor compact representations.
The percentage of nonzero entries of VIT trained on random image dataset is shown in Figure D.1b. can be seen that the first four layers become sparser while the last few layers become relatively denser compared to training with natural images in ImageNet-2Ik. Nonetheless, all layers have < 10% activated neurons.
D.3 SPARSITY FROM DATA DATA-FITTING?
Modern deep neural networks are often vver-parameterized, with sufficient capacity fit practical training datasets and obtain close-to-zero training error. There is evidence suggesting that this result holds true even if the data and label are generated in random (Zhang et al., 2021). Hence, there is the possibility that sparsity arises because the training data, even if generated in random, is scarce relative to the scale modern pver-pareeeterized models.
Hypothesis D.3 (Sparsity from data-fitting). Sparsity trained Transformers arises from the fact that models have more than sufficient capacity to fit training data of practical scale.
To test Hypothesis D.3, we design an infinite data experiment where the amount of training data is infinitely large so that any practical Transformer becomes under- parameterized relative to the data and cannot fit the data. The way we generate infinite training data is to sample images with random pixels in the random image experiment, and for each image sample a random label as in the random label experiment. Moreover, we generate sufficient amount of such training data to make sure that the model never sees the same data point twice during the training. The number of training iterations in the infinite data experiment is kept the same as that of the random image and random label experiments.
The result of this experiment is presented in Figure D.Ic. It can be seen that the first four layers produce sparser activation maps, while middle layers with index are considerably denser compared to the baseline with near 10% to 20% nonzero entries.
D.4 DISCUSSION: SPARSITY FROM TRAINING DYNAMIC?
The results of random label, random image, and infinite data experiments in Figure D. show that labels, data, and data-fitting as conjectured in Hypothesis D. D.2, and D.3, respectively, all affect the sparsity level the activation map. Nonetheless, none of them fully explains the emergence of sparsity since for all results Figure D.I, the percentage of nonzero entries is considerably smaller than at the initialization (i.e., 50%).
Our results point the possibility that sparsity comes from the training dynamic. Namely, at early training stage with any training data and a random initialization of network parameters, the descending direction of the gradient on the Transformer parameters tends to point regime where their MLPs produce sparse activation maps. In the following, we provide theoretical evidence for this argument by looking at the gradient on the positive activation maps for DNN with last two layers being a ReLU followed by a fully connected layer. In particular, have the follow result.
Theorem D.I. Let fxx V.O) 0) IR" + RK be neural network given by
f(x) = (p(p(x;)))
(D.1)
where V = [[ val1 E IR Xdy is network parameter for the last layer drawn from random distribution, o() is the ReLU activation function, and p(x; 0) denotes all other layers with parameter 0. We write p pm:; 0) for simplicity.
26

---

Published as a conference paper at ICLR 2023
Consider the mean squared ervor (MSE) loss (f (x). y) = 2 1l((() arbitrary vector independent of V. Assume that V sattifies
211.
where y is
0,
if
V = 0, and E [[ w,
(D.2)
O,
otherwise
If there exist an such that p.. 0, then we have
Osssn( (x))
I
(D.3)
Öp:-
where the expectation is taken with respect to randomness in V.
Consider the cross-entropy (CE) loss (ce(f(x))))) = (y. log (exp(] where is an arbitrary vector that sums up to one and independent of V. Assume that the entries of V are drawn from independent distributions, the probability of any entry of being 0 less than and E. [V] 0. If there exist an it such that p;- >> 0, then we have
exp(ftæ)
OlcE([(x) y)
E
O,
(D.4)
Öp:-
where the expectation is taken with respect to randomness in V.
The proof of Theorem 1.1 is provided in Appendix E. Theorem 1.1 states that the gradient of either the MSE or CE loss with respect to any positive activation pi- is positive in expectation. Hence, any training algorithm based on negative gradient directions tends to reduce the magnitude of such positive activations, which will lead to smaller training loss. Here, the expectation is taken with respect to the randomness in the last layer parameter Hence, our result can considered as an analysis for DNNs at initialization where weights are often chosen randomly from fixed distribution. In particular, the required properties for the distribution of V in Theorem D. for both MSE and CE losses are satisfied by commonly used initialization methods, such the one in He et al. (2015). the other hand, Theorem D. does not apply to subsequent training iterations since the label y no longer independent of V. However, it be empirically from Figure 1 that the trend of a decreasing percentage of nonzero entries persists for certain number of iterations during the beginning of training until such percentage reaches low level and stays relatively stable until the end of training.
E PROOF OF THEOREM D.1
Proof of Theorem D.I. For arbitrary loss ((f(x)) y), we have Ol
OL
de 0
0f Vi:
(E.1)
Op.-
0f 0p:-
First, Consider e LMSSE We have
dl UCMSE
= f(x) - - = Σ a(p)) -U; y.
(E.2)
0f
Plugging this into (E.1), obtain
MEMEE
Σ Copp))on
- (0r
Op:-
(E.3)
Σ (.,,)
+ op.. (ew,, (wr- 8)
Taking the expectation, and noting the conditions in (D.2), we have
OLMSE
E
= 0 + a B (r wz)) }] + 0 0.
(E.4)
Op:-
This finishes the proof for MSE loss.
Thii requirement is generally satisfied unless the probability of e,
27

---

Published as a conference paper at ICLR 2023
In the rest of the proof we consider e = LCE. We have
dlcE exp(f(x))
exp(2 o (pp) -0,) (exp(E. (p:)) )- v;). 1)
0f (exp(f(x)). 1)
y.
(E.5)
Plugging this into (E.1), we obtain
pccE
(exp(E E; (pi) vz), 0/-)
(v++, y)
(E.6)
Pp.-
(exp E; (p;) U,), 1)
For the enumerator in the first term on the RHS of the equation above, we have
exp Eo(rr)
mi) mx) vr- Σ
[p(ps) Vim ))
Ujt,m exp
Σ
Uje,m exp (pi- (/=,m) exp
Σ (ps))
Vimm
171
(F.7)
Plugging this into (E.6) and denoting
Ca exp Σ c((p)) vs.
we obtain
OccE
Σ
Us+ m exp (p:- U/a m))
C (v..,)) y)
(E.8)
Op.
(e ((p)) vz),
For the denominator in the first term the RHS of the equation above, we have exp
Σ (pi)) vz
.1 .1)
Σ
exp Σ o(p)) Vim'
m
tim )
Σ
exp (pi- U/- We-mme) exp
Σ ix
api))
m
(E.9)
exp (pi- 1/- vm-mm) exp
Σ (p.))
22.
Vim ))
exp (pi.
Ui- (mmmm) exp
Σ ops))
m mm
Plugging this into (E.8) and denoting
((2) = exp
Σ (ps) vimm
(E.10)
$$1
V'm ))
(((3
Σ
cxp (ps. Up mm)) exp Σ (pp))
(E.11)
we obtain
δcCE
Vm cxp (ps- Us. Uy. m) ((1)
Σ
(wr-, v).
(E.12)
Op.
exp (pi- -0-. m)) ((2) [3)
m
28

---

Published as a conference paper at ICLR 2023
Taking expectation with respect to V on both sides, and using the assumption that all entries of V are independent, we have
Oloe Öp:-
Uarm exp (ps- ve ml
(.)))
E
Σ
E
[(vi- y)]
exp (pe (1mm))) m)
O
C
Vit.m exp (pi- 0i m)) (cl) exp (p:- U/ C Cm
Σ E (o
Bug..m
TB ( y)
(E.13)
In above, Er.. 0 means expectation with respect to .n and E (e m)) means expecta- tion with respect to all other entries in V. Note that Cll Cle and C are independent of te-.m. .m- By Lemma F.1 and using the assumption that the expectation of V is zero, have
Ue exp (p+-
U/. Us+.m) .m)
Eur,,
> 0,
(E.14)
exp (pr .m) ((22) + Cm
and
E (u y)l = 0.
(E.15)
Plugging the above two relations into (E.13). we obtain
Occ
E
> 0.
(E.16)
Opi-
The following lemma is used in the proof above.
Lemma E.1. Ler V be a random variable with a probabilistic density fumction p(v) that satisfies P(V= = 0) # 1. Let C1, C2, C3 and p be positive numbers. Then,
C.V exVlpx)
C₁
E
C3 EM.
(E.17)
C2 C2exp(pV)
C3.
Proof.
We may calculate the expectation by using the probabilistic density function p(v) as 1 E C2 C's exp((pV))
C.V . exp(pV)
C1V
Ciu
E
C2 C3 expe pvv) p(v)dv = g(v) - up(v)de.
C2exp(pV) + Co
(E.18)
Since g(v) is monotonically increasing for e E R. we have g(v) 2 g(0) for v 2 and g(v) S g(0) for S 0. Hence,
f g(v) up(u)dr 2 9(0) [" up(u)du,
(E.19)
[. g(v) up(u)du 2 ,(0)
[ up(v)dv.
(E.20)
Moreover, since PVV= = 0) + 1, there exists an interval (a, b) such that Se p(u)du > 0. Without loss of generality assume that b > 0. Then,
L s(v) up(u)do > 9(0) f. vp(x)dd.
(R.21)
That is, the inequality in (E.20) holds with strict inequality. Hence we have I g(v) vp(v)de > g(0) M
CiV exp(pV)
C E
(E.22)
exe((pV + Cs
C2 + Cs
F INSIGHTS FROM SPARSITY IN MLPs
We study the sparsity of activation maps in two-layer MLPs. By showing that sparsity emerges, the result here extends the scope of prevalence of activation sparsity from modern DNNs to two-layer
29

---

Published as a conference paper at ICLR 2023
Sparsity and Over/Under-Fitting in 2-Layer MLPs
non zord rate
5gd
1.0
0.8
0.6
0.2
0.0
0.8
0.6
0.4
0.2
10a
10a
10a
10a
10a 10a
hidden layer width
Figure F.1: Training accuracy and percentage of nonzero entries (both on the y-axis) in activation maps of two-layer MIPs of varying width (on the x-axis, in log scale) after 200 epochs of training. Rows correspond to different training datasets, and columns correspond to different training algorithms.
MLPs which are one of the simplest neural network architectures. Moreover, by training such two- layer MLPs with different types data, we provide additional insights on the causes for emergence of sparsity.
Datasets. We conduct experiment with the MNIST dataset, which contains 60,000 grey scale images handwritten digits. Similar to the experiment in Section D, we also consider a dataset with random data, well dataset with infinite data. For the random data, replace each image of MNIST with a random one drawn from sampling i.i.d. pixels from uniform distribution, and each label with a random class amongst 10. Note that the image-label pairs are fixed throughout training. For the infinite data, the random images and random labels are generated on-the-fly, representing random dataset of infinite size.
Models and Training, We train two-layer MLPs with ReLU activation maps with varying width (i.c., hidden dimension): 32, 128, 512, 2048, 8192, 32768 and 131072. We three different optimizers: SGD, SGD with momentum, and Adam, all for 200 epochs (for the infinite data case, we use the same number of iterations that for training MNIST and random data). We find that 200 epochs is sufficient for the reported metrics to converge in most of the cases.
Results. We report training accuracy and the percentage of nonzero entries in the intermediate activation map (i.e., non-zero rate) at the end of the training in Figure F.I. We have the following observations.
30

---

Published as a conference paper at ICLR 2023
Sparsity with Increased Learning Rate in 2-Layer MLPs
non zord rate
sgd
1.0
0.8
0.6
0.2
0.0
0.8
0.6
0.4
0.2
10a
10a
10a
10a
10a
10a
hidden layer width
Figure F.2: The same as in F.1 except that for each optimizer we use a larger learning rate.
For random data, we observe uni-modal shaped curve for sparsity level. Namely, when the model width small hence the model cannot well-fit the training data, the percentage of nonzero entries is small. As the model width increases, where the model is able to fit the training data evidenced by the fact that the training accuracy increases, we observe that the percentage nonzero entries starts to increase. However, as we further increase the model size in the regime where model is able to perfectly fit the training data, we see that percentage of nonzeros starts to decrease.
For infinite data, where the model cannot fit the training data (hence training accuracy is 0.1 which is the same as result from random guessing), the percentage of nonzero entries is close to 0. This is aligned with the result of random data experiment with a small model width.
For MNIST, where the model of varying width in our experiment is able to fit the training data, observe that the percentage of nonzero entries decreases. This trend aligns with the random data experiment with large model width
The evidence above suggest that the sparsity level may be associated with the under- and over- parameterization of the models. Namely, the percentage of nonzero entries is the highest when the model size is close to the point that the model can start to fit the training data (i.e., the interpolation threshold), and is lower in both under and over-parameterized regimes. It may be intriguing to note that a similar pattern exists for the variance (as in the bias-variance tradeoff) curve of deep learning models, which shown in Yang et al. (2020) exhibit uni-moda shape well. Such connection may help us understand the interplay between generalization and sparsity of activation in deep learning models.
31

---

Published as a conference paper at ICLR 2023
Finally, in Figure F.2 report the results obtained with optimizers with larger learning rates compared to those used in Figure F.1. can be observed that larger learning rate produces sparser activation maps.
32


Reviews:
[{"review_id": "wxIupT5NMe", "reviewer": "Reviewer_cu8i", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study sparsity in the MLP portion of transformers as a means to speed up inference time computation. They also show that sparsity enables robustness to input noise. \n", "summary_of_the_review": "I am on the fence about accepting this paper. It contains a lot of interesting experiments but is lacking a strong message. I hope the authors can address some of the weaknesses: I am open to increasing my score. \n", "strength_and_weaknesses": "+ Reducing the number of operations in a dense matrix multiplication makes sense as a route to speedups.\n+ The writing is easy to understand.\n+ Exhaustive empirical evaluation\n\n- I'm not convinced by reading the paper that sparsity it the solution to more computationally efficient models. Right above section 3.3, the authors claim a 10% wall-time reduction. Is this significant? I wonder if it's worth retraining a model to be sparse for only a 10% wall-time reduction.\n- I'm curious why the authors mainly studied the activations of the MLP portion of transformers and not so much the activation maps prior to the MLP.\n- Section 3.3 contains interesting information but feels rushed and too short. I was hoping the authors could expand on how much sparsity improves robustness to noise and confidence calibration.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ouT42gU34G", "reviewer": "Reviewer_x5F1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study the role of sparsity in deep neural networks that are based on transformers. In particular, the authors show that for a trained transformer only a small percentage of the hidden neurons in Multi-Layer Perceptrons (MLP) are non-zero. Motivated by that they propose a variant of the transformer networks, for a MLP layer only the top k hidden neurons are kept. They conduct numerical experiments for two instances of transformer networks for images and one for natural language processing. When comparing their version with the base models, they show that this model achieves comparable metrics, when the top 128 neurons (k=128) are kept. They further investigate the robustness of this model by introducing noisy labels and adding noise to the inputs. Overall, this new model appears to be outperform the baseline in this noisy conditions.   ", "summary_of_the_review": "Overall, the authors show intersting insights into the transformers sparsity patterns. However, I believe that the the current numerical results are not convincing to immediately adopt this variant. ", "strength_and_weaknesses": "Strengths:\n\n(a) The work presents an interesting application that can be easily applied to all transformer based models.  \n(b) The paper is in general well-written and easy to follow. \n(c) The authors conduct additional experiments on the robustness of the models. \n\nWeaknesses:\n\n(a) From Fg. 6 I see that in quite a few cases using this \"sparse\" model does not lead to improvement in inference time. For instance for T5 Large all the numbers are negative. This undermines the main goal of this model, which was to improve running time. \n(b) On page 6, the authors discuss the possibility of using approximate algorithms for finding the top K values with sublinear running times in hidden dimension. However, it is not clear to me of the authors actually used this approximate method or not. \n(c) Minor comments/questions:\n    (c1) Are the results in Fig. 1 for train set or test set? \n    (c2) Is dim in the legend of Fig. 4 the same as dff?\n    (c3) What is dff in Fig. 5? \n    (c4) On page 9, change \"many work...\" to \"many works...\"\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nmEJY_5N7_j", "reviewer": "Reviewer_KPJJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work shows that sparsity is prevalent in transformer models, including T5 model for NLP and ViT model for vision with extensive experiments. Such a discovery might be suggesting that the law of parsimony is playing a role in Transformers even though they are not explicitly designed so. Furthermore, a top-k transformer is proposed by using top-k thresholding, which achieves good generalization.", "summary_of_the_review": "To sum up, the work shows the emergence of sparse activation in transformer models with amounts of experiments and proposes a top-k transformer based on this discovery, which is promising. Therefore, I tend to vote accept. Some experiments for reasonable comparisons should also be conducted.\n\n\n-------- After rebuttal --------\n\nI have read the authors' responses and other reviewers' comments. I keep my 'accept' rating.", "strength_and_weaknesses": "Strength:\n\n(1)  The authors show the natural emergence of sparse activation in commonly used Transformer models.\n\n(2)  Inspired by the experiments on sparsity, a top-k transformer is proposed to obtain good performance and generalization.\n\n(3)  Extensive experiments are conducted to support their arguments.\n\n\nWeaknesses:\n\nThis work shows the emergence of sparse activation in Transformer models. I have some questions as follows. \n\n(1)  In Sec. 1.3, the authors state that they use ReLU as the activation function instead of GeLU.  However, what is the performance gap between ReLU and GeLU? If GeLU is employed, Does sparsity still exist? Furthermore, the authors also propose a top-k transformer. The authors should list the specific performance with the top-k transformer and the original GeLU for comparisons. \n\n(2)  The authors choose the top k value for the transformer. However, it might affect the data parallelism thereby reducing the inference speed. The authors should provide a reasonable comparison, e.g., throughput. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "XFUZ7J7kqA", "reviewer": "Reviewer_HpEq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work demonstrates the natural emergence of sparsity in commonly used Transformer models. \n\nThis paper proposed  Top-k thresholding to enforce sparsity, which brings robustness of training with erroneous annotations, less sensitivity to input noise/perturbation, and better confidence calibration of the predictions.", "summary_of_the_review": "This paper observed the sparsity in Transformers and proposed Top-k thresholding to enforce sparsity. Based on the proposed method, the efficiency, robustness and calibration are improved. However, this paper only considered the classification task, and I'd like to see the sparsity existing in the downstream tasks. ", "strength_and_weaknesses": "Strength:\n(1)This paper observed the sparsity in Transformers and this sparsity is emergent without any explicit design.\n\n(2) This paper found that sparsity in Transformers is a prevalent phenomenon, and can improve efficiency, robustness and calibration.\n\n(3) This paper further presented Top-k Transformer, a simple modification to Transformer architecture that allows control on the sparsity level for all model inputs throughout training.\n\nWeakness:\n(1) This paper only shows sparsity on the classification task. Is the sparsity existing for downstream tasks? such as detection and segmentation.\n\n(2) Tab 1 shows that Top-128 ViT is on par with ViT for natural accuracy while is significantly better for model robustness and calibration. How about the other Top-k results? (please move some experiments from the appendix). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FKLzZAIZZN", "reviewer": "Reviewer_urTS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper experimentally discusses the emergence of sparse activations in the feed-forward layer of Transformer neural networks. The authors highlight that sparse activations emerge for a variety of tasks and architectures. They then discuss how sparsity can be taken advantage of to reduce FLOP count and the implications of sparsity for robustness under different sources of noise.", "summary_of_the_review": "All in all the paper offers some valuable insights on the activation sparsity in (Transformer) networks. Plethora of experiments demonstrate that this phenomenon is present across architectures and tasks. Claims regarded FLOP efficiency are briefly mentioned, but not adequately backed. A new Top-k layer is proposed, with limited insights given regarding drops in expected accuracy and expected gains in wall time inference. Robustness claims are also linked to previous work. The authors do make attempts to explain possible causes (mainly in the appendix). Contributions are incremental and not totally clear.", "strength_and_weaknesses": "Strengths:\n- Many experiments emphasize that this phenomenon happens for a variety of architectures and tasks. \n- The appendix offers some very good insights. The relevance of the optimizer and the residual connections are the first candidates that fall in mind when first reading the paper and could be emphasized/briefly mentioned in the main text as well.\n- It is nice to see that depending on the task and the rank of the representation required, different sparsity patterns with layer index can be observed. \n\nWeaknesses:\n- As mentioned, time-wall performance is not guaranteed apart from unbatched greedy decoding for larger models.\n- Would be beneficial to discuss more (dis)similarities with sparsity in the attention matrix. I suspect similar results to Table 1 would come out if the Top-k operator was applied to the attention matrix.\n\nQuestions/Remarks:\n- Would be interesting to see what happens when residual connections weights are learned, as e.g. in ReZero [1].\n- The Role of the optimizer is discussed extensively in the Appendix. [2] might offer an alternative to train Transformers via SGD, which would be a valuable comparison.\n- It is not clearly described what the legend in Figure 5 means. I guess (Train) indicates that the model was trained from scratch with a fixed k? If Top-k Transformer models are trained from scratch, it would be interesting to know if more steps are required to reach convergence.\n- Studying the relationship with ResNets is not fair, as ResNets have batch normalization layers before the activations. Comparison with MLP should also include residual connections to be fairer.\n\n[1]: Bachlechner, Thomas, et al. \"Rezero is all you need: Fast convergence at large depth.\" Uncertainty in Artificial Intelligence. PMLR, 2021.\n[2]: Noci, Lorenzo, et al. \"Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse.\" arXiv preprint arXiv:2206.03126 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster