Sparse Weight Activation Training

Training convolutional neural networks ( CNNs ) is time consuming . Prior work has explored how to reduce the computational demands of training by eliminating gradients with relatively small magnitude . We show that eliminating small magnitude components has limited impact on the direction of high-dimensional vectors . However , in the context of training a CNN , we find that eliminating small magnitude components of weight and activation vectors allows us to train deeper networks on more complex datasets versus eliminating small magnitude components of gradients . We propose Sparse Weight Activation Training ( SWAT ) , an algorithm that embodies these observations . SWAT reduces computations by 50 % to 80 % with better accuracy at a given level of sparsity versus the Dynamic Sparse Graph algorithm . SWAT also reduces memory footprint by 23 % to 37 % for activations and 50 % to 80 % for weights . 1 INTRODUCTION . The usage of convolutional neural networks ( CNNs ) has dominated a wide variety of complex computer vision tasks , such as object recognition ( Krizhevsky et al. , 2012 ; Szegedy et al. , 2015 ) , object detection ( Szegedy et al. , 2013 ; Ren et al. , 2015 ) , and image restoration ( Dong et al. , 2014 ; Zhang et al. , 2017 ) . However , CNNs are compute and memory intensive ; even a moderately sized CNN model , like ResNet-50 with tens of millions of parameters , requires billions of floating-point operations and consumes tens of gigabytes to store weights and activations during training . Previous works propose techniques for reducing computations and memory consumption during CNN training . Such techniques include quantization where every operation is quantized in lowprecision during training such a ( Zhou et al. , 2016 ; Choi et al. , 2018 ; Wu et al. , 2016 ; Wang et al. , 2018 ) , or , use fixed-point integers instead of floating-point numbers ( Wu et al. , 2018 ; Das et al. , 2018 ) . An orthogonal approach to reduce computations is sparsification , a process in which we eliminate computations involving small values . meProp ( Sun et al. , 2017 ; Wei et al. , 2017 ) sparsifies backpropagating by selecting a subset of output gradients in each layer . Using only the top 5 % of the gradients ( ranked by magnitude ) , meProp can train a CNN and MLP on MNIST dataset without accuracy loss . The computational flow of meProp is shown in Figure 1a and 1b . meProp does not modify the forward pass . In the backward pass meProp performs a “ Top-K ” operation on the output activation gradients which sets components not ranked in the Top-K by magnitude to zero . It then uses the sparsified output activation gradients to ( potentially more efficiently ) compute the input activation and weight gradients . Our experiments suggest meProp fails to converge on larger networks and datasets . Recently , Liu et al . ( 2019 ) proposed a method of reducing computation during training and inference by constructing a dynamic sparse graph ( DSG ) using random projection for dimensionality reduction . DSG loses accuracy on ImageNet dataset . In this work , we propose an alternative technique , Sparse Weight Activation Training ( SWAT ) , that can train deep CNNs on complex data sets like ImageNet . Compared to DSG , SWAT is a straightforward technique which uses less expensive Top-K operation , inspired by meProp , while achieving better accuracy than DSG on ImageNet . This paper provides the following contributions : • It shows that dropping gradients during back-propagation is harmful to network convergence especially when training a deeper model on a complex dataset . In this case the model suffers high accuracy loss . • It proposes SWAT , a sparse training algorithm that can train a broad range of deep CNNs with minimal accuracy loss on complex datasets like CIFAR10 , CIFAR100 , and ImageNet . SWAT reduces the total number of operations during training by 50 % –80 % . It also achieves 23 % –37 % activation and 50 % –80 % weight footprint reduction during the backward pass . • SWAT algorithm uses sparse weight both in the forward and backward passes , and therefore model learns sparse weights , i.e. , a pruned architecture ; If the model has been trained using SWAT with S % sparsity during training , then during inference , weight can be pruned to S % without sacrificing any loss in accuracy . • We perform empirical studies to provide insight into why ‘ SWAT performs well ; we showed that Top-K sparsification in general preserves direction in high-dimensional space . 2 SPARSITY INDUCED TRAINING . 2.1 PRELIMINARIES . Let us consider a deep CNN with L convolutional layers trained using mini-batch stochastic gradient descent , where the lth layer maps the input activation ( al−1 ) using function fl from RN×Cl−1×Hl−1×Wl−1 → RN×Cl×Hl×Wl . fl computes Cl channel of output feature maps , each of dimension RHl×Wl , using Cl−1 channels of input feature maps of dimension RHl−1×Wl−1 for each of the N samples in the mini-batch . The lth layer has weights wl ∈ RCl×Cl−1×Hf×Wf . The forward pass of the lth layer can be defined as : al = fl ( al−1 , wl ) ( 1 ) During back-propagation the lth layer receives the gradient of the loss L w.r.t its output activation ( 5al ) . This is used to compute the gradient of the loss w.r.t its input activation ( 5al−1 ) and weight ( 5wl ) . Thus , the backward pass for the lth layer can be defined as : 5al−1 = Fl ( 5al , wl ) ( 2 ) 5wl = Fl ( 5al , al−1 ) ( 3 ) 2.2 SPARSE WEIGHT ACTIVATION TRAINING . Our goal is to reduce the computations required during the training process . SWAT does this by effectively transforming small magnitude components of vectors into zero values . Since multiplication of any number by zero results in zero the multiplication is not necessary . Such a sparsification process can be applied in a number of ways in the context of the backpropagation algorithm . Ideally , the modified training algorithm will retain both model accuracy and rate of convergence . We explore the sensitivity to applying this sparsification process at different points . We look at the sensitivity of the model convergence to sparse input i.e . weight ( wl ) and input activation ( al ) for forward pass and weights ( wl ) , input activation ( al ) and output activation gradient ( 5al ) for the backward pass as shown in Equation 1 , 2 and 3 . Figure 2a shows the result of our analysis of sparsification in the forward pass . Here we train ResNet-18 on CIFAR-100 after modifying the forward pass to use sparse inputs in Equation 1 for weights ( wl ) and separately activations ( al ) while keeping the backward pass unchanged ( i.e. , only the forward pass computation is sparse ) . The results show that network convergence is more tolerant to sparse weights ( wl ) compared to sparse activations ( al ) . Thus , as shown in Figure 1a and 1c , SWAT uses sparse weights in the forward pass . Figure 2b shows the result of our analysis of sparsification in the backward pass . We modify Equation 2 and Equation 3 to sparsify either output gradient ( 5al ) , as in meProp ( see Figure 1b ) , or sparsify activations ( al ) and weights ( wl ) . The results show accuracy is extremely sensitive to sparsification of output gradients . Such sparsity consistently results in networks converging to lower accuracy compared to using sparse activations and weights . Thus , as shown in Figure 1a and 1c , SWAT uses sparse weights and activations in the backward pass . The overall SWAT training algorithm is presented in Algorithm 1 . SWAT uses sparse computation in both the forward and the backward passes , while meProp ( Sun et al. , 2017 ) uses sparse computation only in the backward pass . SWAT uses sparse weights and activations in the backward pass allowing compression of weights and activations in the forward pass1 . Effectively , reducing overall memory access overhead of fetching weights in the backward pass and activation storage overhead because only Top-K % are saved . This memory benefit is not present for meProp since dense weights and activations are needed in the backward pass , whereas there is no storage benefit of sparsifying the output gradients since they are temporary values generated during back-propagation . 40 50 60 70 80 90 Sparsity 10 20 30 40 50 60 70 80 Va lid at io n Ac cu ra cy ( % ) sparse wl sparse al 1 ( a ) 40 50 60 70 80 90 Sparsity 50 55 60 65 70 75 80 Va lid at io n Ac cu ra cy ( % ) sparse wl and al 1 sparse al ( b ) 0 5 10 15 20 25 30 35 40 45 Epoch 0 10 20 30 40 50 60 70 Ac cu ra cy ( % ) meProp ( Sparsity 30 % ) meProp ( Sparsity 50 % ) SAW ( Sparsity 80 % ) Baseline Training Curve Validation Curve ( c ) Figure 2 : Convergence Analysis : ( a ) Sensitivity Analysis of ResNet18 for the Forward Pass on the CIFAR100 dataset . ( b ) Sensitivity Analysis of ResNet18 for the Backward Pass on the CIFAR100 dataset . ( c ) Shows the training curve of ResNet18 on ImageNet for meProp and SAW algorithm . Learning rate is reduced by 110 th at 30th and 40th epoch . To compare SWAT ’ s approach to that of meProp , we use a variant of SWAT that only sparsifies the backward pass ; we shall refer to this version of SWAT as SAW ( Sparse Activation and Weight back-propagation ) . We compare the performance of the meProp and SAW with deep networks , and complex datasets2 . Figure 2c shows SAW and meProp convergence of ResNet18 with the ImageNet dataset ; it compares the performance of meProp at 30 % and 50 % sparsity to SAW 80 % sparsity . As we can see , meProp converges to a good solution at sparsity of 30 % . However , at 50 % sparsity , 1more detail in section 3.3 2The performance of SWAT with deep networks and complex datasets is in the result section . meProp suffers from overfitting and fails to generalize ( between epochs 5 to 30 ) , and at the same time , it is unable to reach an accuracy level above 45 % . These results suggest that dropping output activation gradient ( 5al ) is generally harmful during back-propagation . On the other hand , SAW succeeds to converge to an accuracy of 64 % even at a much higher sparsity of 80 % . Algorithm 1 : Training an L layer network using SWAT or SAW Algorithm The data : A mini-batch of inputs & targets ( a0 , a∗ ) , training iteration t , previous weights wt , learning rate η . The result : Update weights wt+1 . Step 1 . Forward Computation ; for l = 1 to L do if l == ‘ ConvolutionLayer ’ or ‘ LinearLayer ’ then if algorithm == ‘ SWAT ’ then wtl ⇐ fTOPK ( wtl ) ; atl ⇐ forward ( wtl , atl−1 ) ; atl−1 ⇐ fTOPK ( atl−1 ) ; else // algorithm == ‘ SAW ’ ; atl ⇐ forward ( wtl , atl−1 ) wtl ⇐ fTOPK ( wtl ) ; atl−1 ⇐ fTOPK ( atl−1 ) ; end save for backwardl ⇐ wtl , al−1 ; else al ⇐ forward ( wtl , al−1 ) ; save for backwardl ⇐ wtl , al−1 ; end end Step 2 . Backward Computation ; Compute the gradient of the output layer 5aL = ∂loss ( aL , a∗ ) ∂aL ; for l=L to 1 do wtl , al−1 ⇐ save for backwardl ; 5al−1 ⇐ backward input ( 5al , w t l ) ; 5wl−1 ⇐ backward weight ( 5al , a t l−1 ) ; end Step 3 . Parameter Update ; for l=1 to L do wt+1l ⇐ Optimizer ( wtl ,5wl , η ) ; end Top-K Selection : Given CNNs operate on tensors with many dimensions , there are several options for how to select which components are set to zero during sparsification . Our CNNs operate on fourth-order tensors , T ∈ RN×C×H×W . Below we evaluate three variants of the Top-K operation illustrated in the right side of Figure 3 . We also compared against a null hypothesis in which randomly selected components of a tensor are set to zero . The first variant , labeled TOPK-NCHW in Figure 3 , selects activations and weights to set to zero by considering the entire mini-batch . This variant performs Top-K operation over the entire tensor , f { N , C , H , W } TOPK ( T ) , where the superscript represents the dimension along which the Top-K operation is performed . The second variant ( TOPK-CHW ) performs Top-K operation over the dimensions C , H and W i.e. , f { C , H , W } TOPK ( T ) , i.e. , selects K % of input activations from every mini-batch sample and K % of weights from every filter in the layer . The third variant ( TOPK-HW ) is the strictest form of Top-K operation . It select K % of activations or weights from all channels , and thereby performing the Top-K operation over the dimension H and W , i.e. , f { H , W } TOPK ( TH , W ) . The left side of Figure 3 shows the accuracy achieved on ResNet-18 for CIFAR100 when using SAW configured with each of these Top-K variants along with a variant where a random subset of components is set to zero . The results show , first , that randomly selecting works only for low sparsity . At high sparsity all variants of Top-K outperform random selection by a considerable margin . Second , they show that the more constrainted the Top-K operation the less accuracy achieved . Constraining Top-K results in selecting some activations or weights which are quite small . Similarly , some essential activations and weights are discarded just to satisfy the constraint .

Summary: This paper studies training neural networks with sparse weights and sparse activations (SWAT training). By using sparse weights in forward passes as well as sparse weights and activations in backward passes, SWAT can reduce the computation overhead and also reduce the training memory footprint. The primary contributions of the paper are in three folds: 1) The authors empirically compare the impact of (activation) gradient sparsity and weight + activation sparsity on the model performance---the comparison shows that the weight + activation sparsity has less influence on the model accuracy; 2) Across different models on CIFAR and ImageNet dataset, SWAP can reduce the training flops by 50% to 80% while using roughly 2 to 3x less training memory footprint saving (weight + activation); 3) The authors empirically study on why training using top-K based sparsification can attain strong model accuracy---the magnitude-based top-K approach can roughly preserve the directions of the vectors.