Sparse Weight Activation Training
Authors: Md Aamir Raihan
Tor M. Aamodt
Venue:  (2020)
Keywords: Sparsity
Training
Acceleration
Pruning
Compression

Abstract: Training convolutional neural networks (CNNs) is time consuming. Prior work has explored how to reduce the computational demands of training by eliminating gradients with relatively small magnitude. We show that eliminating small magnitude components has limited impact on the direction of high-dimensional vectors. However, in the context of training a CNN, we find that eliminating small magnitude components of weight and activation vectors allows us to train deeper networks on more complex datasets versus eliminating small magnitude components of gradients. We propose Sparse Weight Activation Training (SWAT), an algorithm that embodies these observations. SWAT reduces computations by 50% to 80% with better accuracy at a given level of sparsity versus the Dynamic Sparse Graph algorithm. SWAT also reduces memory footprint by 23% to 37% for activations and 50% to 80% for weights.

Under review as a conference paper at ICLR 2020
SPARSE WEIGHT ACTIVATION TRAINING
Anonymous authors
Paper under double-blind review
ABSTRACT
Training convolutional neural networks (CNNs) is time consuming. Prior work has explored how to reduce the computational demands of training by eliminat- ing gradients with relatively small magnitude. We show that eliminating small magnitude components has limited impact on the direction of high-dimensional vectors. However, in the context training a CNN, we find that eliminating small magnitude components of weight and activation vectors allows to train deeper networks on more complex datasets versus climinating small magnitude compo- nents of gradients. We propose Sparse Weight Activation Training (SWTT), an algorithm that embodies these observations. SWAT reduces computations by 50% 80% with better accuracy a given level of sparsity versus the Dynamic Sparse Graph algorithm. SWAT also reduces memory footprint by 23% to 37% % for acti- vations and 50% 80% for weights.
INTRODUCTION
The usage of convolutional neural networks (CNNs) has dominated wide variety of complex com-
puter vision tasks, such as object recognition (Krizhevsky et al. 2012, Szegedy et al. 2015). object
detection (Szegedy et al., 2013 Ren al. 2013), and image restoration (Dong et al. 2014 Zhang
eet al. 2017). However, CNNs are compute and memory intensive; even moderately sized CNN model, like ResNet-50 with tens of millions of parameters, requires billions of floating-poit opera- tions and consumes tens of gigabytes store weights and activations during training.
Previous works propose techniques for reducing computations and memory consumption during CNN training. Such techniques include quantization where every operation is quantized in low- precision during training such a (Zhou et al.] 2016 Choi et al.) 2018 Wu et al. 2016: Wang et al., 2018). or, fixed-point integers instead of floating-point numbers (Wu et al. 2018 Das et al. 2018
An orthogonal approach reduce computations is sparsification. a process in which we eliminate computations involving small values. meProp (Sun et al., 2017]] Wei et al.] (2017) sparsifies back- propagating by selecting subset of output gradients in each layer. Using only the top 5% of the gradients (ranked by magnitude), meProp can train a CNN and MLP on MNIST dataset without accuracy loss. The computational flow of meProp is shown in Figure Ta and lb meProp does not modify the forward pass. In the backward pass meProp performs a "Top-K" operation on the output activation gradients which sets components not ranked in the Top-K by magnitude to zero. It then uses the sparsified output activation gradients to (potentially more efficiently) compute the input activation and weight gradients. Our experiments suggest meProp fails to converge on larger networks and datasets.
Recently, Liu et al. (219)) proposed a method of reducing computation during training and infer- ence by constructing dynamic sparse graph (DSG) using random projection for dimensionality reduction. DSG loses accuracy on ImageNet dataset.
In this work, we propose an alternative technique, Sparse Weight Activation Training (SWAT), that can train deep CNNs on complex data sets like ImageNet. Compared to DSG, SWAT is a straight- forward technique which uses less expensive Top-K operation, inspired by meProp, while achieving better accuracy than DSG on ImageNet.
This paper provides the following contributions:

---

Under review as a conference paper at ICLR 2020
It shows that dropping gradients during back-propase propppropagation is harmful to network conver- gence especially when training a deeper model on a complex dataset. In this case the model suffers high accuracy loss.
It proposes SWAT, a sparse training algorithm that can train broad range of deep CNNs with minimal accuracy loss on complex datasets like CIFARIO, I0, CIFARIOO, and ImageNet. SWAT reduces the total number of operations during training by 50%-80%. also achieves 23%-37% activation and 50%-80% weight footprint reduction during the backward pass.
SWAT algorithm uses sparse weight both in the forward and backward passes, and therefore model learns sparse weights, i.e., a pruned architecture; If the model has been trained using SWAT with S% sparsity during training, then during inference, weight can be pruned to 9% without sacrificing any loss in accuracy.
We perform empirical studies to provide insight into why 'SWAT performs well; we showed that Top-K sparsification in general preserves direction in high-dimensional space.
2 SPARSITY INDUCED TRAINING
2.1 PRELIMINARIES
Let us consider a deep CNN with convolutional layers trained using mini-batch stochastic gradient descent, where the t layer maps the input activation (a-1) using function fi from ft computes C channel of output feature maps, each
+ R x of dimension RRixW, using Ct-1 channels of input feature maps of dimension R 1xW; for each of the N samples in the mini-batch. The 1th layer has weights wy E RCixC; xxH,xW, The forward pass of the tth layer can be defined as:
at = fi(ac-1, Wt)
During back-proyy proppappponatoon the 7th layer receives the gradient of the loss L w.rt its output activation (Va)). This used compute the gradient of the loss w.r.t its input activation (V and weight (Vw)). Thus, the backward pass for the 1th layer can be defined as:
Var = Fi(Var Wi)
(2)
V w = Fi(Va, 01-1)
(3)
MEPROP FORWARD PASS
SWAT
Law
FORWARD PASS
& Ay
a2
Wt. d:-1
al Wt. a1-1
01
a111
a1
W
Fe
V4
Frl M3)
BACKWARD PASS
BACKWARD PASS
Vw: Va, af
Vw: Van af
(b)
VWi
Va
Vw
Va
af11
file (n))
Val1 w. Va
Val.1 w/ Vaa
Sane Euckinadd
Va-1
W
Va
Va
w/
Va
Vaa. Badkward Pass
Figure I: meProp versus SWAT (a) Shows the forward and backward pass of MeProp and SWAT for a fully connected layer. (b) Computational flow of meProp for any layer I (c) Computational flow of SWAT for any layer l.
2

---

Under review as a conference paper at ICLR 2020
2.2 SPARSE WEIGHT ACTIVATION TRAINING
Our goal is to reduce the computations required during the training process. SWAT does this by effectively transforming small magnitude components of vectors into zero values. Since multiplica- tion of any number by zero results in zero the multiplication is not necessary. Such a sparsification process can be applied in a number of ways the context of the backpropagation algorithm. Ideally, the modified training algorithm will retain both model accuracy and rate of convergence.
We explore the sensitivity to applying this sparsification process at different points. We look the sensitivity of the model convergence to sparse input i.e. weight (wy) and input activation (at) for forward pass and weights (wy), input activation (at) and output activation gradient (Va,) for the backward pass shown in Equation
and 3
Figure 2a shows the result of our analysis of sparsification in the forward pass. Here we train ResNet- on CIFAR-100 after modifying the forward pass to use sparse inputs in Equation [ for weights (w)) and separately activations (at) while keeping the backward pass unchanged (i.c., only the forward pass computation is sparse). The results show that network convergence is more tolerant to sparse weights (wy) compared to sparse activations (at). Thus, as shown in Figure la and Tc SWAT uses sparse weights in the forward pass.
Figure 2b shows the result of our analysis of sparsification in the backward pass. We modify Equa- tion and Equation 3 to sparsify either output gradient (Va,), as in meProp (see Figure lb), or sparsify activations (ot) and weights (wy). The results show accuracy is extremely sensitive to spar- sification of output gradients. Such sparsity consistently results in networks converging to lower accuracy compared to using sparse activations and weights. Thus, as shown in Figure Ia] and [c] SWAT uses sparse weights and activations in the backward pass. The overall SWAT training algo- rithm is presented in Algorithm !
SWAT uses sparse computation in both the forward and the backward passes, while meProp (Sun et al. 2017) uses sparse computation only in the backward pass. SWAT uses sparse weights and ac- tivations in the backward pass allowing compression of weights and activations in the forward pas Effectively, reducing overall memory access overhead of fetching weights in the backward pass and activation storage overhead because only Top-K% K% are saved. This memory benefit is not present for meProp since dense weights and activations are needed in the backward pass, whereas there is no storage benefit sparsifying the output gradients since they are temporary values generated during back- sock-propagation
Training Surve
Validadon Curve
Validtioon Accurccy (N)
Palldaton Accuracyy (N)
50
50
Aocurssy
30
80
20
sparse W/
sparse W. and 3)
sparse 91-1
sparse VA
50
70
80
50
40
60 Sparsity
70
MeProp
SAWISparrity Basoline
Sparsity
SeProo 50%)
(b)
Figure 2: Convergence Analysis: (a) Sensitivity Analysis of ResNet18 for the Forward Pass on the CIFAR100 dataset. (b) Sensitivity Analysis of ResNet18 for the Backward Pass on the CIFAR100 dataset. (c) Shows the training curve of ResNet18 8 on ImageNet for meProp and SAW algorithm. Learning rate is reduced by 1 th 30th and thth epoch. 10
To compare SWAT's approach to that of meProp, the backward pass; we shall refer to this version of SWAT as SAW (Sparse Activation and Weight .oc-proppggtiion. We compare the performance of the meProp and SAW with deep networks, and SAW and meProp convergence of ResNet18 8 with the ImageNet dataset; compares the performance of meProp at 30% and 50% sparsity SAW 80% sparsity. As we can see, meProp converges to a good solution at sparsity of 30%. However, at 50% sparsity,
variant of SWAT that only sparsifies
complex d Tigure[e
mooe detail in section 3.3
"The performance of SWAT with deep networks and complex datasets in the result section.
3

---

Under review as a conference paper at ICLR 2020
meProp suffers from overfitting and fails to generalize (between epochs 5 to 30), and at the same time, it unable to reach an accuracy level above 45%. These results suggest that dropping output activation gradient (Vaz) is generally harmful during socc-propagation On the other hand, SAW succeeds to converge to an accuracy of 64% even at much higher sparsity of 80%.
Algorithm 1: Training an I. layer network using SWAT or SAW Algorithm
The data: A mini-batch of inputs & targets (an, a²) a²), training iteration I, previous weights w
learning rate 7).
The result: Update weights w
Step 1. Forward Computation;
end
for / = / do
== "Convolutionl.aye" or 'LinearLayer' then
Step 2. Backward Computation;
Compute the gradient of the output layer
if algorithm = 'SWAT' SSWT"" then
V (loss(ar..a)
w| fropk(Wff
das.
for |=L to I do
aj forward -)=
w/: a = save for backward,;
a = fIOPk 11
V --11
backward. (Va, wf):
else
// algorithm = SAW';
W
backward af11;;
a = forward(ww) (wi, a)
end
w = (wf)
a = (f
Step 3. Parameter Update; for /=1 do
end
save save.for.backwrrt += w/, a:11
w OptimiizerwWf Ve n)):
else
end
forwarddw ay)
save for backward, < w/. a;-1;
end
Top-K Selection: Given CNNs operate tensors with many dimensions, there several options for how to select which components are set to zero during sparsification. Our CNNs operate on fourth-order tensors, T E I RNNCCHXW Below we evaluate three variants of the Top-K opera- tion illustrated in the right side of Figure 3 We also compared against null hypothesis in which randomly selected components
tensor are set to zero.
Dataset: CIFAR100
75
Velletion Accuracy
70
0.2
08
03
SON
C9
3.8
+3
MDV
TOP
N
TOP
60
10%-
TOP 10%
100
55
CHW
50
RANDOM
TOPK-NCHT
TOPK-CHN
TOPK-HW
50
60
Sparsity
Figure 3: Different ways of performing top-k operation. 'N' denotes the #samples in the mini- batch or filters the layer, "C' denotes the #channels in the layer. 'H' and "W" denote the height and width of the filter/activation map in the layer. Color represent the selected activations/weights by the Top-K operation.
The first variant, labeled TOPK-NCHW in Figure 3 by considering the entire mini-batch. This variant performs Top-K operation over the entire tensor, STork A.N.C.H.W) (T), where the superscript represents the dimension along which the Top-K operation is performed. The second variant (TOPK-CHW) performs Top-K operation over the dimensions C, I and W i..., F(C.H.W) (T), i.e., selects % of input activations from every mini-batch sample and K% of weights from every filter in the layer. The third variant (TOPK-HW) is the strictest form of
selects activations and weights to set to zero
4

---

Under review as a conference paper at ICLR 2020
Top-K operation. It select K% of activations weights from all channels, and thereby performing the Top-K operation over the dimension H and W, i.e. fTOPk (TH,W)-
The left side of Figure 3 shows the accuracy achieved on ResNet-18 for CIFARI00 when using SAW configured with each of these Top-K variants along with a variant where a random subset of components is set to zero. The results show, first, that randomly selecting works only for low spar- sity. At high sparsity all variants of Top-K outperform random selection by a considerable margin. Second, they show that the more constrainted the Top-K K operation the less accuracy achieved. Con- straining Top-K K results in selecting some activations or weights which are quite small. Similarly, some essential activations and weights are discarded just satisfy the constraint.
3 RESULTS
In this section, we present our experimental results of SWAT algorithm on different architectures and datasets and we quantify the theoretical reduction in compute and memory bandwidth achievable using SWAT.
3.1 EXPERIMENTAL SETUP
We implement SWAT and SAW algorithms in PyTorch Framework (Paszke et al., 2017;; models are trained on three different datasets: CIFAR1O. CIFARI00 (Krizhevsky et al. (009) and Ima-
geNet ILSVRC2012 (Deng et al.) 2009) and are evaluated on four different architectures ResNet- 18, 34, 50, 101 (He et al., 2016). Wide Residual Networks (Zagoruyko & Komodakis 2016),
DenseNet-BC-12 (Huang et al. 2017). and VGG-16 (Simonyan & Zisserman (2014) with batch-
normalization (loffe& & Szegedy 2015). Natch-Normalization statisties are computed using the run-
ning average (with momentum 0.9). We use SGD with momentum as our optimization algorithm with initial learning rate of 0.1, momentum of 0.9 and weight decay À of 0.0001.
For CIFARIO and CIFARIOO dataset, ResNet, VGG, and DenseNet models are trained for 150 epochs, and learning rate are reduced by (1/10)th at the 50-th and the 100-th epoch whereas WRN is trained for 200 epochs and the learning rate annealed by a factor of (1/5)th at 60-th, 120-th and 160-th epoch. ResNet, VGG, and WRN are trained using batch-size of 128 whereas DenseNet is trained with a batch-size of 64. We run each experiment with three different seeds and use the average value for all the plots.
For training on ImageNet dataset, we use 224 x 224 random crops from the input images or its horizontal flip and the input image is normalized by the per-color mean and standard deviation. Networks are trained for 50 epochs with the mini batch-size of 256 samples, and the learning rate are reduced by (1/10)th after 30-th and 40-th epoch.
3.2 ACCURACY ANALYSIS
In this section, we provide a comprehensive analysis of SWAT and SAW algorithms and show the influence sparsity on validation accuracy; thereby showing the potential of reducing computation during training with negligible accuracy loss. We furthermore discuss the impact on rate of con- vergence and the robustness of the algorithm widths. Last, we provide an alternative to Top-K for efficient software/hardware implementation.
wide range of models with different depths and
Accuracy on CIFAR10 and CIFAR100: Figure 4 shows the accuracy of the SWAT and SAW algorithms at different sparsity budgets on CIFARIO and CIFAR 100 dataset. From the graph, we can conclude that models can be trained using SWAT and SAW algorithm up to 60% sparsity with almost zero accuracy loss and suffer only slight accuracy loss at 70% sparsity. For CIFAR10 dataset at 70% sparsity, VGG-16 16 and DenseNet- 121 have an accuracy loss of around 0.57% for SWAT (0. 20.26% for SAW) and 0.4% for SWAT (0.23% for SAW) whereas ResNet-18 gains an accuracy of 0.02% for SWAT (0.1% for SAW). For CIFAR 100 dataset at 70% sparsity. ResNet- 18, VGG-16 and DenseNet- BC-121 lose an accuracy of around 0.5%, 0.41% and 0.68% for SAW and 0.4%, 0.99% and 1.78% for SWAT respectively. At 80% sparsity the accuracy loss on CIFAR 10 and 100 is less than 1.8% for SAW and less than 2.5% for SWAT.
5

---

Under review as a conference paper at ICLR 2020
96
Dalaset: CIFAR10
Dataset: CIFAR10O
Validstion Accuracy
04-
Validation
92-
SAW
SAW SWAT
SWAT
90
0
10
20
30
40
Spansity
50
60 70 90
ResMet-18
Sparsity
VGG
DenseNet-121
(a)
(b)
Figure 4: Comprehensive analysis of sparsity vs accuracy trade-off: (a) Accuracy of SWAT and SAW algorithms CIFARIO dataset. (b) Accuracy of SWAT and SAW algorithms CIFAR100 dataset. The dashed line represents the baseline accuracy for the corresponding model. Datapoints for SAW algorithm are represented dots whereas for SWAT algorithm they represented as stars.
RN18
Dataset: ImageNet
ImageNet
VEG16
+725E
60
DN121
Validaaiion Accurscy
70-
71.5
26,46
60.
Validation
40--
se
56.10
$4.25
64
47
Daseline Spersity
SWAT
30
Sparsity 70%
60
50 55
60
65
Sparsity
0
5
30
20
25 Epoch
30
35
40
ResNet-18
VGG-16
DensoNet-121
(a)
(b)
Figure 5: Trend of SWAT algorithm on ImageNet dataset: (a) Validation curve of SWAT al- gorithm (b) Validation Accuracy of SWAT, SAW and DSG algorithms at different sparsity con- straints. Dotted line represents the baseline back-propagation algorithm. 'RN18' represent ResNet- 18, 'DN121' represent DenseNet-BC-121 and "DSG" denote the results reported by the Dynamic Sparse Graph(Liu et al. 2019) algorithm.
Accuracy on ImageNet: Figure 5a shows the validation curve of the SWAT algorithm on Ima- geNet dataset for three different architectures and Figure 55 shows the accuracy obtained by the SWAT and SAW algorithms. The result shows that the SWAT and SAW algorithms lose negligi- ble accuracy at 50% sparsity for all three architectures. The solution is within an accuracy loss of 1.26% compared to the baseline solution. For high sparsity of 70%, ResNet-18, VGG-16 and DenseNet-BC-12) lose only around 1.52%, 1.6% and 2.26% accuracy for the SWAT and 1.42%, 1.28% and 1.82% for the SAW algorithm respectively. Both the algorithms perform better than the DSG algorithm proposed by (Liu et al., 2019). which accelerates training by performing dimension- ality reduction search and performing the forward and backward passes in low dimensional space. The accuracy loss of DSG 50% sparsity around 2.83% for ResNet-18 and 1.54% for VGG-16 compared to the SWAT accuracy loss of 0.27% and 0.86% for ResNet-18 18 and VGG-16 respectively.
Impact on rate of Convergence: We define the rate of convergence is the number of epochs it takes to reach the saturation accuracy. Figure 5a shows the validation curve of SWAT algorithm when training ResNet-18. VGG-16 and NenseNNetBBC-12 on ImageNet dataset. As shown in Figure, when the learning rate is 0.1 (i.e. between epoch O) and 30) the SWAT algorithm reaches the satu- ration accuracy around the 15th epoch approximately the same epoch when the baseline algorithm also reaches saturation. Similarly, when the learning rate is 0.01 (i.e. between epoch 0-40th) both SWAT and the baseline saturate at epoch 35th. The experiment at 50% and 70% sparsity shows that SWAT algorithm converges with slight accuracy loss but at the same rate compared to the baseline algorithm.
Influence of Depth and Width: Network depth (#layers) and width (#chonnels) are two important design criteria. Previous studies (Lu et al.] 2017] Raghu et al., 2017 Sharir & Shashua (018)) have found that both depth and width affect network expressivity. Increasing network depth helps in

---

Under review as a conference paper at ICLR 2020
learning complex abstraction, whereas increasing width helps in learning more features. Ideally, we want SWAT and SAW algorithms to work with models of varying depth and width. Therefore, study the influence of depth and width the SWAT and SAW algorithms. Figure [ba] shows the accuracy of ResNet-50, ResNet- 101 and WRN-28-10 10 on CIFAR datasets at four different sparsities 0%, 50%, 70% and 80%. The result for deeper networks shows that enforcing sparsity up to 70% is beneficial for training as the ResNet-50 and ResNet-100 converged to an accuracy higher than the baseline training. At 80% sparsity, ResNet- 101 loses accuracy of mere 0.18% whereas ResNet-50 still has an accuracy advantage of O. 1919 over the baseline. WRN-28- 10 lose accuracy on training with SWAT and SAW algorithm, but the accuracy loss is only 0.67% for SAW and 0.49% for SWAT at 70% sparsity.
Datsee: CIFARIS6
We git Iwcl
Validrhin Acgurcyyn
60.
4890
6000
Act valion (n..))
Sounsiy
Restlet-50
WRN-26-10
Roshet-101
(a)
(b)
Figure 6: (a) Influence of Depth and Width: Accuracy of SAW and SWAT algorithms on CI- FAR 100 dataset for ResNet-50, 101 and WRN-28-10 (b) Threshold value (K-th largest values) in Top-K operation of different layers during training
Efficient Top-K Implementation: Top-K Joperation on I dimensional array of size n can be naively implemented using sorting. The computational complexity of a naive Top-K operation is O(n log n). The computational complexity can be reduced to (n). if the k-th largest element can be found in be implemented by threshold operation.
O(n) time, since for this case the Top-K operation The K-th largest element can be computed in (nz) average time using quickselect (Hoare 1961) or in ((n) time using BFPRT (Blum et al. 1973) introselect (Musser 1997). The computation can be further reduced since we found experimentally that for a given layer, the K-th largest elements almost constant during training as shown in Figure 6b So we don't need to compute the K-th largest elements during every training iteration and can be computed once in a while after multiple iterations.
3.3 COMPUTATIONAL AND MEMORY OVERHEAD REDUCTION DURING TRAINING
In this section, we will quantify the reduction in computational and memory overhead, using SWAT, over the baseline training algorithm.
Computation Reduction: sparsifying the computation in both the forward and the backward pass. During CNN training, most of the com- putation is in the convolution and fully connected layer. therefore sparsifying the computation in both of these lay-
SWAT algorithm is accelerating CNN training by
140
VGG19
forward backward
GFLOPS per example
VGG16
ers can result in linear speed-up dur-
100
ing training. Figure 7 shows the com- putational reduction possible by SWAT for three different architecture while training on ImageNet dataset. SWAT
80-
60
RN101
2.0k
DN161
20
RN50
DN121
2.0x
achieves a computation reduction of 2x, 3.3x, and 5x at 50%, 70%, and 80% sparsity respectively. Note that the over- all overhead of implementing efficient Top-K operation using BFRT/introselect + thresholding, as described in the pre- vious section, is only 1-2% additional computation during training. Another benefit of using SWAT
8x
50 70
80
5070 80
50 70
80
D 70
50 70
50 70 80
Sparsity
Figure 7: Computational reduction in SWAT at dif- ferent sparsity. "RN" denotes ResNet, "DN" denote DenseNet (Dataset: ImageNet)
Top-K operation is performed an absolute values.

---

Under review as a conference paper at ICLR 2020
is that the model learns sparse architecture and therefore, sparse weights are used during Inference. Thus, the same computational benefit of 2-5x is possible for Inference as well.
0.6
Cony-Parameter
BN-Activation
0.5
Lineer-Parameter
Memory Ovemead
per sample (GB)
0.23
Conv-Activation
DN161
Hemory Overnedd
044-
GG16
VGG19 L0V
0.15
0.10
RN50
RN101
VGG16
VGG19
DN121 1.x
9.2
RN101
RN50
DN161
0.05
DN121
80
70
30 Sparsity
50
70 80
50 707000
50 70
50
50 70
Sparsity
(a)
Figure 8: Reduction memory accesses during the backward pass. (a) Reduction in parameter
access (b) Reduction in activation access per sample (Dataset: ImageNet)
Memory Overhead Reduction: During training, most of the weights and activations are stored in DRAM and accessing DRAM consumes three orders of magnitude more energy consumption than computation (Horowitz). So reducing the memory access during training will directly reduce the energy consumption. SWAT algorithm uses sparse input activation (a-1) and weight (wx-1) in the backward, so input activation and weight can be compressed and stored in the memory in sparse for- mat thereby reducing the DRAM access in the backward pass. Figure shows the reduction of 2x, 3.3x, and 5x at sparsity of 50%, 70% and 80% in the parameters access the backward pass. The other sigmificant memory overhead is saving the activation, and this overhead is dependent not only on the model size but also the batch-size used during training. Figure 8b shows the activation memory overhead for different architectures for a mini-batch size of 1. The graph only shows the activation of batch-normalizationn and convolutional layer since memory overhead for the activations of the linear layer is negligible compared to them. Note that SWAT algorithm sparsifies the compu- tation of the linear, and convolutional layers, so full input activation of the batch aatch-normalization layer are saved for the backward pass. Thus, SWAT algorithm achieves activation compression of around 1.3x, 1.5x, and 1.7x sparsity of around 50%, 70%, and 80%. The activation of batch- nonch--ommaliztioon layer limits the overall activation compression.
4 EXPERIMENTAL ANALYSIS OF SWAT BEHAVIOUR
In this section, will give some experimental evidence which explains why the SWAT algorithm is working so well in practice. The experiment shows the general behavior of vector sparsification in high dimensional space. Let us first define two new terminologies which are going use in this section: "Top-K sparsification" and "Sparsification Angle Top-K K sparsification selects K% of the highest magnitude component and set the rest of the component to zero. Sparsifi- cation angle is the angle between the original and the Top-K sparsified instance of that vector.
a vector v
4.1 VECTOR SPARSIFICATION IN BIGGH-IMENSIONAL SPACE
A vector in high-dimensional space behaves differently from their low dimensional counterpart. is well known that in figh--imension. two independent isotropic vectors tend to be orthogonal. Re- cently Andersso & Berg (0077) extended the analysis of the geometry of high- nigh-dimensional vectors to binary vectors. They proved that the angle between any random vector, drawn from a rotationally invariant distribution, and its binarized version would be concentrated around 37 Thus, binariza- tion of high dimensional vector approximately preserves their direction. We apply similar kind of geometry analysis to high dimensional sparsified vectors. We show that sparsification indeed preserves direction in high-dimensional space, which is contrary our low-dimensional intuition.
The first question we need to answer is how should we sparsify a vector v E Rd such that the sparsification angle is minimum between the original and the sparsified instance of the original vector (has only K% non-zero component). We found that the cosine angle is minimum when the K2 non-zero component corresponds to the K% highest magnitude component, i.e., Top sparsification. The proof is in the appendix.
*The mathematical definition of both of these terms is in the appendix
8

---

Under review as a conference paper at ICLR 2020
We did experiment for analyzing how Top K sparsification affect angle as shown in Figure 9a Here we are showing the sparsification angle distribution for 1000 dimensional vector drawn from standard normal distribution at different sparsity. The peak of sparsification angle 90% sparsity concentrated around 48 which is much less than peak of random vectors which is concentrated around 90% Similarly, the peak up to 80% sparsity is concentrated an angle of 36.4 only. This suggest that deviation caused by sparsification is indeed small in high-dimension.
For our experiment shown in Figure 9b| we study how much vector of given dimen- sion, drawn from standard normal distribution, can be maximally sparsified such that the sparsi- fication angle less than 8. We can see that the percentage of Top-K components needed for 5 1200s 30", 40 is around only 43%. 29% and 18% respectively with variance less than 3% as shown Figure 9c Thus, these experiment suggest that a high-dimensional vector can spar- sified up to 70%, which will lead a deviation (5) of only 30 All the above experimental results are dependent on the distribution from which random vectors are drawn, in Figure 9e we calculate the sparsification angle during training ResNet18 on CIFARI00 dataset at 70% Sparsity. Here the sparsification angle for weight and activation is less than 36 for all the layer in the network. So the experiment suggests that the above analysis is applicable during training.
3,0
s=10%
5=60%
100.0%
$=20%
5=70%
Density Function
5 3030
5-80%
Top-K Components
80.0%
6= 10
2.0
40%
5-90%
8 30
60.0%
1.5
5 50%
random
6- 50
40.0%
1.0
0.5
26.0%
5 80
0
20
40
60
80
100
0
2000
6000
5000
8000
10000
cosine_and [
Dimension
(a)
(b)
Weight
300
dim-100
cosine (ngle)))
leyen13
8 20
dim=1000
Mayer
250
30
076
Top-K Components
dim=5000
0000
Variance
200
8 50"
40%
150
30%
Activation (8:-1)
100
50-
0%
2000
Dimension
4000
6000
8000
10000
20
cosine_angee (asine_angle(6)
40
60
80
25
90
Epach
75
100
125
150
(c)
(d)
(e)
Figure 9: Vector sparsification in high-dimension approximately preserves the (arection:(a)
Shows the sparsification angle distribution at different Top-K percentage for a 1000 dimensional vector in 10, 000 trials. Random represents the angle distribution between 2 random vectors. (b) and (c) Shows the percentage of Top-K components needed for sparsification angle to be within in 1000 trials and the variance in those trials. (d) Shows the relation between Top-K sparsification and the sparsification angle in 1000 trials. (e) Shows how the sparsification angle (at 70% sparsification) varies during training for ResNet-18 18 architecture CIFARI00 dataset.
5 RELATED WORK
We can classify most of the previous studies which focus on accelerating training or inference the following broad categories:
Pruning: Most of the pruning work focuses on inference optimization. Weight pruning can be
classified into two broad categories of structured and unstructured pruning. The idea of unstructured

---

Under review as a conference paper at ICLR 2020
pruning be traced back LeC al. [990); Hassibi & Stork (1993), which prune the network using the saliency of parameters derived from the second-order information of loss function. Ham
et al. (2015bla) a pruned network parameters using magnitude based method. There are several other unstructured pruning methods such as Moochanov et al. (2017): Louizos et al. (2017), but the drawback of all these methods is that it is difficult to extract parallelism on hardware. In contrast, x Luo et al. (2017); Wen al. (2016); MMochanov et al. (2016) removes entire channels or filters at time which preserves the inherent regular computation structure, and therefore it is easy to extract parallelism on hardware.
structured pruning such as Liu et al. (2017): [i et al. (201 He et al. (2017):
Quantized networks can be used to accelerate both training and inference since en- engy consumption in the hardware is directly proportional to bit-width of the operands. There are many works which focus on quantizing weights for efficient inference such as McDonnell (2018);
Quantization:
Wu et al. (): /hhu et al. (2016): [i et al. (2016a); Courbariaux et al. (2015) whereas much other work focuses on accelerating training as well, such as Banner et al. (2018) Choi et | (2018); Wang
ct al. (2018): ) Lin et al. (2017a); Zhou et al. Courbarraa al. (2016); Rastegari et al. (2016);
Gupta et al. (2013). Some the other work such as Zhao et al. 2019); McKinstry et al. (2018);
Zhou et al. (017)); Mellempudi et al.) (2017) shows that training from scratch is not necessary for finding the quantized model, but one can find a quantized model from pre-trained full precision models. Other work focuses discrete training and inference using Integers such Wu et al.
20188: Das et (2018); Jacob et al. (2018): Lin et al. (2016) since integer aadeed/multipier is more efficient than floating-poinn adder/multiplier. Few studies such as Louizos al. (2019): Jung et al. (2019): Zhang et al. 2018); Zhou et al. (2018): Hou & Kwok (2018): Hou et al. (2016) formulate the quantization as an optimization problem to minimize the accuracy loss due to quantization. Few other work such as Yang et al. (219 : De Sa et al. (2018) focus on improving the learning algorithm by proposing novel stochastic averaging of the low precision iterates or using SVRG to reduce the variance and by dynamically adjusting the precision representation using bit centering. Few works instead of quantizing the entire model to fixed bit- wit-width focus on per tensor or parameter quan- tization such as Sakr & Shanbhag (2019): Khoram & Li (2018). Compared to all these works, our work is orthogonal as we climinate the computation instead of reducing the computation precision.
There are few works on compressing the models by performing tensor decomposition or by learning compact structure. Alvarez & Salzmann (017) introduce regularizer that promotes the parameter matrix to have low rank. Thus the algo- rithm encourages the model to learn a compact structure by accounting for compression during the training itself. Novikov et al. (2015) showed that tensor decomposition could be used to compress a fully connected layer by using only a few parameters. Later, Garipov et al. (2016) extended it for the convolutional layer. The idea was to reshape the kernel into tensor of higher-order and then to factorize it.
Tensor Decomposition and Dimentionality Reduction:
Distributed Training: There are few works (Stich et al., 2018 Lin al. 2017b) which look at reducing the communication overhead in distributed training by transferring only sparse gradients during gradient aggregation step but these works are accumulating the rest of the gradients locally for subsequent iterations. Compared to all these works, our work objective is different as we are con- cerned with accelerating single node training, whereas their objective is minimizing communication during distributed training.
6 CONCLUSION
In this work, we propose SWAT, a robust training algorithm based on the insight that sparsifying weights and activation during training has little impact on convergence. SWAT sparsify both the for- ward and the backward passes, thereby eliminating lots of redundant computation such as addition and multiplication by zero. SWAT is a simpler technique and performs better than the recently pro- posed dimensionality reduction (DSG) technique for accelerating training. Our experiments over various benchmarks demonstrate significant computation reduction of up to 2-5x for training and inference and provides memory footprint reduction of activation by 1.3- and reduction in memory access overhead for weight by 2-5x in the backward pass.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jorge Albericio, Patrick Judd, Tayler Hetherington, Tor Aamodt, Natalic Enright Jerger, and Andreas Moshovos. Cnvlutin: neffectual-nnuron free deep neural network computing. ACM SIGARCH Computer Architecture News, 44(3):1-13, 2016.
Jose M Alvarez and Mathieu Salzmann. -oapression-aware training of deep networks. In Advances in Neural Information Processing Systems, pp. 856-867, 2017.
Alexander G Anderson and Cory P Berg. The high-dimensional geometry of binary neural networks arXiv preprint arXiv:1705.07191 2017.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. Advances in Neural Information Processing Systems, pp. 5145-5153, 2018.
Manuel Blum, Robert W. Floyd, Vaughan Pratt, Ronald L. Rivest, and Robert E. Tarjan. Time bounds for selection. Journal of Computer and System Sciences, 7(4):448 461, 1973. ISSN 0022-0000. doi: htt.a///c/....a.a..aa/////////0/
URL http:://www
sciencedirect com/
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural net- works (qnn). arXiv preprint 18X7.01807.066
2018.
Matthieu Courbariaux, Yoshua Bengio, and Jean- Pierre David. Binarycomnect: Training deep neural networks with binary weights during propagations. In Advances neural information processing systems, pp. 3123-3131, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint arXiv:1602. 02830, 2016.
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, Alexander Heinecke, Pradeep Dubey, Jesus Corbal, Nikita Shustrov, Roma Dubtsov, Evarist Fomenko, and Vadim Pirogov. Mixed precision training of convolutional neural networks ing integer operations. In International Conference on Learming Representations, 2018. URL
net wttp://wwww..ee//ooeoeed---- Foo.omiidddi35uut
Christopher De Sa, Megan Leszezynski, Jian Zhang, Alana Marzoev, Christopher R Aberger, Kunle Olukotun, and Christopher Ré. High-accuracy low-precision training. arXiv preprint a8Xiv: 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In European conference on computer vision, pp. 184-199. Springer, 2014.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: (00.:::80033333 2018.
Timur Garipov, Dmitry Podopríkhin, Alexander Novikov, and Dmitry Vetrov. Ultimate tensoriza- tion: compressing convolutional and fo layers alike. arXiv preprint arXiv:1611.03214 2016.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv: 151000019, 2015a.
11

---

Under review as a conference paper at ICLR 2020
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems, pp. 164-171, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
C. A. R. Hoare. Algorithm 65: Find. Commun, ACM, 4(7):321-322 July 1961. ISSN 0001-0782. doi: 10.1145/366222366644
Mark Horowitz. Energy table for 45mm process, Stanford VLSI wiki.
Lu Hou and James T. Kwok. Loss-aware weight quantization of deep networks. In International Conference on Learning Representations, 2018.
foormm?
id-BkrSv01A
Lu Hou, Quanming Yao, and James T Kwok. Loss-aware binarization of deep networks. arXiv preprint arXiv:1611.01660 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.0316; 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
efficient
2018.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4350 4359, 2019.
Soroosh Khoram and Jing Li. Adaptive quantization of neural networks. In International Confer- ence on Learning Representations, 2018. URL httpt:::/opeereeeeeeee net.
www.:/rrttoormmmmo wwrmmrrrrummmmmmmmmmm did=
Sy0K1Sg0M
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Fengfu Li. Bo Zhang, and Bin Liu. Temary weight networks. arXiv preprint arXiv:1605.04771 2016a.
Hao Li, Asim Kaday, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1608.0000. 2016b.
12

---

Under review as a conference paper at ICLR 2020
Darry Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolu- tional networks. In International Conference on Machine Learning, pp. 2849-2858, 2016.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pp. 345-353, 2017a.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compression: Re- ducing the communication bandwidth for distributed training. arXiv preprint arXiv:1112.01887, 2017b.
Liu Liu, Lei Deng, Xing Hu, Maohua Zhu, Guoqi Li, Yufei Ding, and Yuan Xie. Dynamic sparse graph for efficient deep learning. In International Conference on Learning Representations, 2019.
URL hattpss:/openreviiett www.eerwwwwwweeeeeeeeeeeeeeeeeee
net nttp://eee//ooooeeoeeo i go
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744 2017.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1. 0 regularization. arXiv preprint
2017.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Re- laxed quantization for discretized neural networks. In International Conference on Learning Representations, 2019. StRhtttps: htttttps:///pppreeeeeee net/ forumm Soweet//ooomll---xiieXeooo
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In Advances in neural information processing systems, pp. 6231-6239, 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference computer vision, pp. 5058-5066, 2017.
Mark D. McDonnell. Training wide residual networks for deployment using single bit for each weight. In International Conference on Learning Representations, 2018. URL https:
//openreview.
Jeffrey L McKinstry, Steven K Esser, Rathinakumar Appuswamy, Deepika Bablani, John V Arthur, Izzet B Yildiz, and Dharmendra S Modha. Discovering low-precision networks close to full- precision networks for efficient embedded inference. arXiv preprint arXiv: 011.888990999 2018
Naveen Mellempudi, Abhisek Kundu, Dheevatsa Mudigere, Dipankar Das, Bharat Kaul, and Pradeep Dubey. Ternary neural networks with fine-grained quantization. arXiv preprint 1rXiv:1705.01462 2017.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning-Voie Volume 70, pp. 2498-2507. JMLR. org, 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint urXiv:1611.06444 2016.
David R. Musser. Introspective sorting and selection algorithmss Software: Practice and Experience, 2788)99839993 1997. doi: 10. 100.00022/5C010007112XX119777719 URL https://ooLLneeLbrrrrr...... 00210000000//8800011109222223311 3.A.D-SPP 000.:333...003322-333
AID-SPEL17)3.0.cm /
com/ /oi/
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In Advances in neural information processing systems, pp. 442-450, 2015.
Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Stephen W Keckler, and William J Dally. Scnn: An accelerator for compressed-spar convolutional neural networks. In 2017 ACM/IEEE 44th Annual International Symposium Computer Architecture (ISCA), pp. 27-40. IEFE, 2017.
13

---

Under review as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein. On the expressive power of deep neural networks. In Proceedings of the 34th International Conference on Machine Learning-Voiie Volume 70, pp. 2847-2854. JMLR. org, 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, Lawrence, Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 91-99. Curran Associates, Inc., 2015.
Charbel Sakr and Naresh Shanbhag. Per-tensor fixed-point quantization of the back Back-propagation algorithm. In International Conference on Learning Representations, 2019. URL https:// openreview http:/eefffoooooo (ott///oooummiddxaaaaaaaaaa
Or Sharir and Amnon Shashua. On the expressive power of overlapping architectures of deep
learning. In International Conference on Learning Representations, 2018. URL https: //openzeview net/ www.:/ee//ooaae
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXivv1449.15535 2014.
Sebastian U Stich, lean-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. In Advances in Neural Information Processing Systems, pp. 4447-4458, 2018.
Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting. In Proceedings of the 34th International Conference on Machine Learning-Volune Volume 70, pp. 3299-3308. JMLR. org, 2017.
Christian Szegedy, Alexander Toshev, and Dumitru Erhan. Deep neural networks for ob- ject detection. In C. C. Burges, K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 2553-2561, Curran Associates, Inc., 2013. URL http:://pppers nips cc/paper/ for-ob wet--obbeetddetttttoo.....
Bottou, M. Welling, Ghahramani, and
5207-Gee-nnurra--ettooo
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Advances in neural information processing systems, pp. 7675-7684, 2018
Bingzhen Wei, Xu Sun, Xuancheng Ren, and Jingjing Xu. Minimal effort back propagation for 170.005844 2017.
convolutional neural networks. arXiv preprint
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Jiaxiang Wu, Cong Leng. Yuhang Wang. Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1820-4828, 2016.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. International Conference on Learning Representations, 2018. htRpst:s: d=HJG
//openrevi ew. wwt.:tt..oroo wtt.::rrrwm.m.................
14

---

Under review as a conference paper at ICLR 2020
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Chris De Sa. Swalp: Stochastic weight averaging in low precision training. In International Conference Machine Learning, pp. 7015-7024, 2019.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016.
Wide residual networks.
arXiv preprint
Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Le-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference Computer Vision (ECCV), pp. 365-382, 2018
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp.
2017.
Shijin Zhang, Zidong Du, Lei Zhang, Huiying Lan, Shaoli Liu, Ling Li, Qi Guo. Tianshi Chen, and Yunji Chen. Cambricon-x: An accelerator for sparse neural networks. In The 49th Annual IEEE/ACM International Symposium on Microarchitecttuee, pp. 20. IEEE Press, 2016.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, and Zhiru Zhang. Improving neural network quantization using outlier channel splitting. arXiv preprint arXiv:1901.09504 2019.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. arXiv preprint 1eXiv:/702.03044 2017.
Aojun Zhou, Anbang Yao, Kuan Wang, and Yurong Chen. Explicit loss-error-aware quantization for low-bit deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9426-9435. 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint N64::16066.61660. 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint 1rXiv:16/2.01064, 2016.
15

---

Under review as a conference paper at ICLR 2020
APPENDIX A PROOF SHOWING ToP-K IS THE BEST SPARSIFICATION FUNCTION
Definition 1 Sparsifying Function (fs): Given a parameter < k d, let fs: Rd Rd be function that selects k component of vector and sets the rest of the component to zero. For a vector e E RR (v) = μ4 @v) where 44 (v) is an indicator vector having non-zero values determined by input vector v.
Definition 2 Top-K Sparsifying Function (frOkk): ): Given a parameter 1 k let STOPK R + RR is a special sparsifying function that sets all but the k highest component of input vector absolute value zero. More precisely, for a vector where Topp is an indicator function, T(i E (* TT)), and is permutation of [d] such that [v| 2 |4/m for lori=l...... .....,d---
RR, frork (v) Itopk (v)
Definition Sparsification Angle (0): For a vector E RR, the deviation the direction caused by sparsification f.) is defined as the sparsification angle, i.e., it is the angle between the vector and sparse vector fu)
Lemma A.1. For any vector v E Red and of all the sparsifying function Ss, Top-K sparsifying function (STOPk) causes the minimum deviation in direction i.e. minimum sparsification angle.
Proof. Given a parameter k E [1, dJ. for a vector v = (0)..... CT) E Bd let fs(v) (m. "mavs)" E IRd such that m, [0,1] Vi, be the Top-K indicator mask i.e. my only tt component of e is selected by the sparsifying function.
d Σ
d Σ (m;z))²
(m;²²)
[s(v).v
cos(ss(v),,)
[[ssvv]llvv|
d
d
(m,
2
(m; )
Σ
2
d
m2,
[[ss(v]] [v|
(5)
Σ
In other words,
[vs[[
Sparsifying Angle(0) arccos
(6)
|
arccos is a strictly decreasing function, so to minimize 0, II (v)l must be maximized. Therefore Top-K component of the vector magnitude wise should be selected.
16

---

Under review as a conference paper at ICLR 2020
APPENDIX B PERIODIC TOP-K & EFFECT OF BATCH-NORMALIZATIOO
Periodic Top-K For Efficient Implementation: In section 3.2. we have shown there is little variation in the 'K-th' largest element during training, and remains approximately constant as training proceed. Therefore, the Top-K does need to be computed every iteration and can be periodically computed after some iterations. We define the number of iterations between computing the threshold for Top-K as the "Top-K period.
Top-K Period
70% Sparsity
90%
Sparsity
Default Top-K
76.41
73.81
10 Iteration
76.59
73.64
20 Iteration
76.03
73.45
50 Iteration
76.06
74.09
100 Iteration
76.52
73.29
We have empirically confirmed that the periodic Top-K performs equally well as the default Top-K implementation. The table above shows topl validation accuracy from single runs of ResNet- 18 on CIFAR 100 with different Top-K periods (i.e., Top-K is computed after every 10, 25, 50, and 100 iterations respectively). This data suggests the converged accuracy is indeed not significantly impacted when employing our proposed Periodic Top implementation.
90
90
Actvvtton Actvatton-Spariity Sparsity
Sparsity
80
Layer-1
Layer-1
70
20
40
60
80
100
120
140
0
20
40
60
80
100
120
140
92
Weight Sparsty
Sparsity
90
Activation Sparsty
Sativation-Sparsat
90
90
89
86
Layer-7
Layer-7
20
100
120
140
20
100
120
140
Foach
Fooch
Satration-Sparaty
Activation Sparstty
95
91
Weigh-Sparrtt
90
90
Layer-15
85
Layer-15
0
20
80
100
120
140
0
20
80
100
120
140
90
Weight Sparsity
Activattio Aativaation-Spaaity Sparsity
95
85
90
80
Layer-20
Layer-20
80
0
20
40
60
80
100
120
140
0
20
40
60
80
100
120
140
Epoch
Epach
Figure 10: Sparsity Variation using Periodic Top-K Implementation. Network: ResNet- 18, Dataset:
CIFAR100 100, Top-K period: 100 iterations, Target Sparsity: 90%
17

---

Under review as a conference paper at ICLR 2020
Since the periodic Top-K used the same threshold during the entire period, therefore, it is crucial to confirm that periodic Top-K implementation does not adversely affect the sparsity during training. We dumped the amount of sparsity obtained in weights and activation using periodic Top-K with period 100 iteration with target sparsity of 90%. FigureT0jshows the sparsity during training using periodic Top-K implementation is concentrated around our targeted sparsity, and the fluctuation decreases as training proceeds confirming our hypothesis that chosen Top-K parameter stabilizes i.c. the Top-K threshold converge
fixed value during the latter epochs.
The Top-K periods need not to be fixed throughout the training. The Top-K period can be increased as the training proceeds because the chosen Top-K parameters are unlikely change the later training iterations. To demonstrate this we perform new experiment. We train ResNet- 18 on CIFAR 100 for 150 epochs (learning rate decayed at epoch 50, 100) using a version of SWAT which computes Top-K periodically but with increasing Top-K K period for later epochs. The Top-K schedule used during training is shown below:
Scheduled Top-K
Top-K Implementation once per iteration
Top-1
Accuracy
Epoch 0-50
3 times per epoch
70% Sparsity
90% Sparsity
Epoch 50- 100
time per epoch
76.36
73.63
Epoch 100-150
time per 5 epoch
Scheduled Top-
76.41
73.81
Note: epoch has 392 iterations.
The activations and weights of BN layers are not sparsified in SWAT. Empirically, we found that sparsifying weights and activations are harmful to convergence. This is because the weight (gamma) of BN layers is scaling factor for an entire output channel, therefore, making even single BN weight (gamma) zero makes the entire output channel Similarly, dropping activations affects the mean and variance computed by BN. Empirically we found that the BN layer is extremely sensitive to changes in the per channel mean and variance. For example, when ResNet18 is trained on CIFAR 100 using SWAT with 70% sparsity and we sparsify the BN layer activations, accuracy is degraded by 4.9% compared to training with SWAT without sparsifying the BN layers. Therefore, the activations of batch- nithhnoomaalizaiion layer are not sparsified.
Sparsification of
Layer:
The parameters BN layer constitute less than 1.01% % the total parameters in the network and the total computation in the BN layer is less than 0.8% of the total computation in one forward and backward pass. Therefore, not sparsifying aatch-oormalization layers only affects the activation overhead in the backward pass.
Comparision with Lottery Ticket Hypothesis: The Lottery Ticket Hypothesis (Frankle & Carbin 2018) showed the difficulty of training with sparse architecture and that sparse training is very sensitive initial conditions. The Lottery Ticket showed if one could pick the right initial conditions for the weights, network without the need for oracle information about initialization values.
train with sparse network. SWAT is interesting in that does train a sparse
We believe the crucial difference that enables SWAT to work despite the observation in the Lottery Ticket Hypothesis paper is the following. SWAT updates which weights are part of the sparse net- work rather than attempting to train single unchanging sparse network. SWAT may work because it dynamically searches for the sparse architecture that will work with a given set of initial conditions.
18

---

Under review as a conference paper at ICLR 2020
APPENDIX C INDEXING OVERHEAD
To estimate the indexing overhead, we need to understand the generic architecture of the sparse CNN accelerator, the sparse format in which data is stored, and how the computations mapped to the accelerator.
Indoiing Unit
Veight Buffer
Actlvation Buller
PE5
PE6
PE7
PEB
Sahoouer Cantral Uni.
Ho
PES
PE9
PE10
PE11
PE12
Mclpplrr
PE11
PE13
PE14
PE15
PE16
Wo
PE1S
PE14
PE15
PE10
(b) Each PEs is responsible for creating portion of the output (Parashar (017)
(a) Generic Architecture of Sparse CNN Accelerator
Sparse Accelerator: At a bigh-level of abstraction, all the sparse accelerators (Parashar et al. 2017 Zhang et al. 2016;; Albericio et al. 2016) have 2D array of processing units (PEs) where each processing unit has an array of multipliers and have a dedicated weight, activation, and accumulation buffer. They have an indexing unit for enabling the sparse multiplication. The computations are spatially mapped and scheduled to these processing units by a control and scheduling logic. Each of the PE generates partial products which get accumulated compute the output values and finally stored in the DRAM.
Mapping Computations: Let us consider convolutional layer, which maps the input activations in (R Xx Hy xW1) to oul (R Rox xWo). The layer computes F channels of output feature maps, each of dimension RBo x Wo, using channel of input feature maps of dimension R x for each of the N samples in the mini-batch. The layer has parameter E RF xFxxCxHxXX
Algorithm 1: Dense Forward Pass Computation for single input sample (Assuming Stride=1)
The data: w,in The result: out for ho I to Ho do
for w. = to Wo do
for f= / to F do
for
I to C do
for h& I to H do
for Wkk
W k do
e" =
h- ho + hyi
w' = Wo + wk:
out [ut [ho] [wo] + [[/][c[//<[[[x]] x xin(c][[*][w*))
end
end
end
end end end
Thus, as shown in algorithm I] each activation is reused F x C x Hg x Wx times, each weight is reused N x C x xH WK times and the total computation as follow:
Dense Convolution FLOP = F Ho x Wo x HK x WK
The first three 'for' loops are independent and can be mapped independently to the PEs, whereas the inner three 'for' loop generate the partial products. The different sparse accelerators have different
19

---

Under review as a conference paper at ICLR 2020
ways of mapping the 'for' loops spatially over the PEs for maximizing and minimizing the data transfer to and from the DRAM.
From now onwards, SCNN architecture (Parashar et al. 2017). Each PEs is responsible for generating a chunk of output values, shown in Figure 11b The control unit is responsible for partitioning and transferring the corresponding imput activations to each PE, whereas each PE has the entire sparse weights. Depending on the buffer size, either the entire index offset or only block of index offset can be computed and distributed to the PEs.
assuming the mapping of the algorithm the PEs inspired by the
Sparse Storage Format: In the forward pass, only weights are sparse. Each filter weights independently stored in sparse format i.c. weights are sparsified along the dimension RCx Wik Figure 12a shows the storage of single filter in sparse format. The value stored vectors; data vector and index vector. The data vectors contain only the non-zero data values, whereas the index vector contains the number of zero values before the corresponding data values.
0.2
0.1
Hp
0.1
0
0.9
0
WF
Vector
0.2
0.1
0.3
Index Vector
(a) Storage format sparse weights
Storage Overhead for Index: Bits required for storing weights in dense format: (Assuming each weight value is stored in 32 bits)
Dense Weight Storage F C C 3
Let the weights have sparsity S. The number of non-zero values present in the data vector is (1-S)x F x C x H W, and the number of zero values present in the data vector Assuming the sparsity is uniformly distributed, the number of zero values between consecutive non-zero values is 3xPxCxHxX
F x
x W.
$ I-s- Therefore, the number of bits used for encoding the bits. Taking the maximum required when the ratio less than one. this case bit indicates whether the next value zero or not. The bits required for storing weights in the sparse format:
(Xx))xXXXXXXX
index vector is max flog, 1. 1.
(
data vector = (1 - = H 32 bit
(9)
S
inder vector x Fx5)xxxCxXxX max 1- 1,1)
(10)
S Sparse W Weight Storage (1 S) F [ max (lgg2 1.1)] (11)
Therefore, storage decreased by a factor of (1 (12 loga - 1.1)]
For 70% sparsity the
reduction in storage space is 3.13X.
20

---

Under review as a conference paper at ICLR 2020
Index Computation Overhead: The computation performed by each PE is shown in algorihmm
Algorithm 1: Sparse Forward Pass Computation for input sample (Assuumgg Stride=1)
The data: w, (in(in)
The result: (E,(out)
/STEP-1: Data-Structure Initialization Phase;
cl [ (eeeeiiee__eeeeee = 0;
F] (eeninnex_.vetto 0;
w (inexxxvivxe = 0;
//STP-22: Index Offset Calculation Phase; for F do
accumulated. inder 0:
for ind 1:len((ndex_vvetor do
accurralateddindez+++ =index
cif]]ind]. h[]][ind]] [in] compuctiicctcccmmuaaellllleelll
end end
//STEP-3: Computations Phase; for h. / to PF, (Ho) do
for w/ = 1 PE, (Wo) do
for f I F do
for ind = 1:len(index.vector do
e" i]]][ind];
tow + [ind; w w. + w[f]iind]
[hols[ data vector[ind] x [e](in)<""[["[[]]
end
end
end end
The indexing involved during the Computation Phase just base+offfst addition. The same kind of index computation is present in dense convolution algorithm calooutionallgorihm 1). Therefore, the only additional
indexing computation is during the Indexing Offset Calculation Phase. Let the weights have sparsity S.
Inder Computation OP x Fx HK x Wx
(12)
where a is the overhead of the computelndex function. computelndex function is accumulating the current index and calculating the offset (cif[[ind), h[f][ind] w[f][ind). Therefore, the value of will be approximately around 4.
Sparse Computation
F X X
(13)
Therefore,
Total Operation for Sparse Convolution S F C HK x WK x (a Ho x Wo) (14)
Since the Ho x Wo >> a, the indexing overhead minimal compared to the benefit obtained by performing sparse convolution.
Considering the chip area ransistoos available) is very critical when designing a custom accelerator since it decides the number of the computational units that can be fabricated on a given area. Generally, the number of transistors needed for dense computation unit is less than that of the sparse computation unit. This is because of the extra chip area required for the indexing, arbitration, and controlling logic. Based on the area value mentioned in the SCNN paper(Parashar et al. 2017). the scaling factor for adjusting the performance of the sparse CNN
Chip Area Consideration:
21

---

Under review as a conference paper at ICLR 2020
accelerator is 0.75 i.e., On given area, have almost 1.33x more dense computation units compared to sparse computation units.
Backward Pass: The computation in the backward pass is the deconvolution operation. Since deconvolution operation is very similar to the convolution operation therefore the overhead of index computation in the backward pass would be of the same order compared to the computation in the forward pass.
22


Reviews:
[{"review_id": "ryxGyqqk9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes SWAT as a training algorithm for sparse networks on different architectures. The paper claims being able to reach a level of sparsity with no drop in accuracy.  The goal is to minimize the computations during training time. To this end, SWAT sets to zero the vectors where necessary. Different from other approaches, SWAT uses sparse computation in the forward and backward passes. The intuition behind is that eliminating small components does not have an impact on the training process but can be used to minimize the computation required. \n\nSome Comments:\n\n- The paper is a bit on the empirical side with a decent number of experiments to demonstrate the effectiveness of the proposal. I am on the border between accepting and rejecting. \n\n\n- The top-K implementation is interesting. Page 7 suggests the top-K do not change during training which is reasonable as the update is limited to those components. Would it be possible to avoid completely that compute and quickly select K early in the training process? I would find that an interesting future direction. \n\n- In the experimental section, I missed actual numbers. At the moment, if I understand correctly, the paper is based on theoretical compute savings. How feasible is this considering the sparsity of the operation (assuming unstructured sparsity)? \n\n- In the case of structured sparsity, how this differs from the early pruning process of regularization based pruning algorithms? For instance, in the first reference (compression-aware training), the authors claim the model can be compressed in the early training. If that is the case, how different is SWAT from those type of methods? In those related works, the accuracy does not drop. Implementation wise, those algorithms do make the backward pass also sparse (setting to 0 the gradients). \n\n- At the moment, the algorithm is using a magnitude-based sorting. Would it be possible to have other sorting approaches?\n\n\nMinor things:\n\n- for clarity, I would summarize the algorithm in section 2.2 rather than in the appendix. \n\n- I am surprised by the imagenet training setting. Why only training for 50epochs? The standard training process is 90epochs changing the learning rate in the 30th and 60th. \n\n- I guess the S% sparsity contribution can be improved (rephrased). If the training algorithm sets to zero N parameters seems to me obvious that there will be no drop in accuracy compared to that training process. What is the drop in accuracy referring to?\n\n- check the references. While the list is quite comprehensive, some of them are not referred in the text. Please, add comments where appropriate.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SylCZRFtor", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am the emergency reviewer. Sorry for the late.\n\nThis paper studies a very interesting topic: eliminating small magnitude components of weight and activation vector instead of eliminating small magnitude components of gradients. A clear interpretation and definition towards the forward and backward propagation is presented. The difference between meProp versus SWAT is also plain. Based on some experiment results shown in Figure2, authors announced that accuracy is extremely sensitive to sparsification of output gradients. Thus algorithms SWAT and SAW are proposed to prune the model, which are respectively training with sparse weights and activations, and SWAT only sparsifies the backward pass. Top-K selection is implemented to select which components are set to zero during sparsification.\n\nStrengths:\n1. The writing logic ascends step by step.\n2. Authors showed the harmfulness of the sparsity of gradients by experiment results. Also the comparison between the sparsity of weights and activations are meaningful.\n3. Sufficient experiments are done to generalize SWAT to different models, and the results are fascinating on ImageNet.\n\nWeaknesses: \nIt's a borderline paper. \n1. lack of novelty. The paper has shown a lot experiment results on basic models, but the raising of Top-K algorithm is not novel. Why it is Top-K but not other metrics for selecting zero components? In this view, the paper is likely to be a project summary.\n2. Less comparison to other basic pruning models. More experiments should be done to compare SWAT with other sota pruning models. Then the results will be convincing.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xw9KbyKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies training neural networks with sparse weights and sparse activations (SWAT training). By using sparse weights in forward passes as well as sparse weights and activations in backward passes, SWAT can reduce the computation overhead and also reduce the training memory footprint. The primary contributions of the paper are in three folds: 1) The authors empirically compare the impact of (activation) gradient sparsity and weight + activation sparsity on the model performance---the comparison shows that the weight + activation sparsity has less influence on the model accuracy; 2) Across different models on CIFAR and ImageNet dataset, SWAP can reduce the training flops by 50% to 80% while using roughly 2 to 3x less training memory footprint saving (weight + activation); 3) The authors empirically study on why training using top-K based sparsification can attain strong model accuracy---the magnitude-based top-K approach can roughly preserve the directions of the vectors.\n\nI think the claimed contributions are well-validated in general. The design decisions of the approach are well supported by empirical observations and the components of the approach (different top-K methods) are studied properly. Additionally, I like the authors' synthetic-data studies to shed light on why top-K based sparsity can work well. Given the above reason, I give week accept and I am willing to raise the score if the following questions / concerns can be resolved in the rebuttal / future draft:\n\n1. In results such as in figure 4, we observe that using intermediate levels of sparsity can actually demonstrate better generalization performance than the dense baseline training approach. I was wondering if this is because the default hyperparameter produces better training loss in sparse training than in dense training, and consequently the sparse training test performance is also improved over dense training. Without showing this, it is not fully convincing that intermediate sparsity helps prevent overfitting and generalizes better (as the authors discussed in the text).\n\n2. For \"Impact on Convergence\" in section 3.2, it is not clear to me what the authors are using as a metric for the degree of convergence. Thus I can not evaluate the claims here.\n\n3. For \"Efficient Top K implementation\" in section 3.2, the authors suggest  only computing the K-th largest elements periodically to further improve efficiency. However the empirical evidence of whether this approach will significantly degrade the model performance at the end of training is not provided.\n\n4. For the GFLOPS comparison in Figure 7, could the authors elaborate what operations are included into the count? As the sparse operations requires additional indexing operations for computation, I was wondering whether the GFLOPS can realistically reflect the real latency / energy efficiency of the SWAT approach.\n\n5. How the memory access count calculated at the end of page 7? Is it counting the number of float point values (activations, activation gradients, weights) that needs to be fetched for forward and backward pass?\n\n6. At the first paragraph in page 8 (last paragraph above section 4), do the authors imply that the activations of BN layers is not sparsified? Could the authors provide a bit more evidence on how (any why) sparsification of BN activation impacts the model performance.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is proposed a rejection based on majority reviews.
Decision: Reject