DeepTwist: Learning Model Compression via Occasional Weight Distortion

1 INTRODUCTION . The number of parameters of deep neural networks ( DNNs ) is rapidly increasing in order to support various complex tasks associated with increasing amount of training data . For example , Deep Speech 2 has 10× more parameters compared with the previous version ( Narang et al. , 2017 ) . The Long Short-Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) model using PTB dataset ( Marcus et al. , 1993 ) requires exponential increase in number of parameters to achieve incremental improvement in test perplexity . Such a requirement of large DNN model size leads to not only long training time but also high latency and huge storage requirement for performing inference , creating a challenge for mobile applications to be deployed . Note that the memory access dominates the entire inference energy if data reuse is not high enough , since computation units demand relatively low amount of energy ( Han et al. , 2016 ) . Therefore , model compression is crucial to efficiently implementing DNN applications such as language models and speech recognition . As a result , model compression is actively being studied with the ideas such as low-rank matrix factorization ( N. Sainath et al. , 2013 ; Prabhavalkar et al. , 2016 ) , parameter pruning ( Han et al. , 2015 ) , quantization ( Xu et al. , 2018 ; Zhu et al. , 2017 ) , knowledge distillation ( Hinton et al. , 2015 ; Polino et al. , 2018 ) , low precision ( Zhou et al. , 2016 ) , and so on . Model compression is based on Bayesian inference and prior information on the model parameters ( Hinton & van Camp , 1993 ) . Depending on specific implementation methods , various ways to exploit prior information exist ( e.g. , pruning and low-rank approximation ) . Up to now , model compression techniques have been independently developed to fully utilize the unique compression principles of each compression technique . Consequently , different model compression techniques demand dedicated hyper-parameters and compression-aware training methods . We observe in recent studies that the number of required additional hyper-parameters is increasing . Training algorithms are also being modified intensively to take into account the model compression effects in order to enhance the compression rate given a target accuracy . Such efforts , however , lead to significantly increased training time due to computation overhead and substantial hyper-parameter search space . Desirable compression techniques exhibit the following characteristics : 1 ) Existing training method is not modified because of model compression , 2 ) The addition to the number of hyper-parameters exclusively dedicated to model compression is minimized , and 3 ) Nonetheless , high compression ratio should be achieved . In this paper , we propose a unified model compression framework for performing a variety of model compression techniques . Our proposed framework , called DeepTwist , involves only one extra hyperparameter regardless of the type of model compression while any modifications to the training algorithm is not necessary . For example , a pruning method based on DeepTwist does not need masking layers , which have often been entailed additionally to record individual pruning decision on each weight ( Han et al. , 2016 ; Guo et al. , 2016 ) . During the training process for model compression , weights are slightly and infrequently distorted in the intervals of training steps where weight distortion follows well-known basic model compression operations . In our proposed framework , weight distortion is considered as a weight-noise-injection process that has a potential to improve the accuracy . Because of no modifications required for the training algorithm and small computation overhead introduced from infrequent weight distortions , DeepTwist presents a practical and efficient model compression platform . We show that DeepTwist can provide high compression ratio for compression techniques including but not limited to weight pruning , quantization , and low-rank approximation . 2 RELATED WORKS . Parameter pruning is based on the observation that large number of parameters are redundant . Optimal brain damage finds redundant parameters using the second derivative of loss function at the cost of severe computation overhead ( LeCun et al. , 1990 ) . Deep compression ( Han et al. , 2015 ) suggests a fast pruning method using the magnitude of parameters to determine the importance of each parameter . Dynamic network surgery ( DNS ) allows weight splicing to correct wrongly pruned parameters at the expense of additional hyper-parameters ( Guo et al. , 2016 ) . Variational Dropout ( Molchanov et al. , 2017 ) introduces individual dropout rate for each parameter . Quantization techniques widely used for digital signal processing are useful for DNNs as well . Researchers have suggested aggressive binary quantization techniques dedicated to DNNs , where computations are replaced with simple logic units . BinaryConnect ( Courbariaux et al. , 2015 ) learns quantized weights where the forward propagation is performed using quantized weights while fullprecision weights are reserved for accumulating gradients . Scaling factors are stored additionally to compensate for the limited range of binary weights ( Rastegari et al. , 2016 ) . While binary quantization achieves impressive amount of compression , its accuracy in large models ( especially for RNNs ) is degraded seriously ( Xu et al. , 2018 ) . Thus , multi-bit quantization techniques to reduce mean squared error ( MSE ) between full-precision and quantized weights are introduced . For example , Alternating multi-bit quantization ( Xu et al. , 2018 ) extends the XNOR-Net architecture ( Rastegari et al. , 2016 ) to find a set of coefficients minimizing MSE , and recovers full precision accuracy with 3-4 bits per weight . Researchers have also successfully compressed weight matrices using low-rank matrix factorization . A matrix is decomposed into two smaller matrices by reducing the rank through Singular Value Decomposition ( SVD ) . Then , DNNs for acoustic modeling and language modeling can be reduced by 30-50 % without noticeable accuracy degradation ( N. Sainath et al. , 2013 ) . To further improve compression rate , two weight matrices inside an LSTM layer can share a projection layer to be commonly used for each decomposition ( Prabhavalkar et al. , 2016 ) . 3 DEEPTWIST : LEARNING MODEL COMPRESSION FRAMEWORK . Our proposed model compression framework , DeepTwist , mainly performs full-precision training as if model compression is not being considered . Occasionally , however , weights are distorted as shown in Figure 1 such that the distorted weights follow the compressed form . For instance , in the case of weight pruning , some weights are converted into zeros as a weight distortion function . Distortion step , SD , determines how frequently such distortions should be performed . As described in Figure 1 , weight distortion at N th batch replaces the original weights with distorted weights to be used for training at ( N + 1 ) th batch . For a successful model compression , DeepTwist is required to meet the following specifications : • Full-precision training should converge and achieve the full accuracy in the end , even though weights are distorted at every SD step . • The training should be finished right after the weights are distorted to hold the compression format of weights . • Accuracy drop through a weight distortion step should decrease during training . In other words , the loss function degradation caused by a weight distortion should be gradually diminished . From the above requirements of DeepTwist , one can notice that DeepTwist partly holds the characteristics of regularization techniques even though faster training or improving accuracy is not an immediate objective . The next section , indeed , shows that DeepTwist tends to enhance the accuracy via an appropriate amount of distortion . Accuracy of compressed models highly depends on the flatness of the loss surface ( Hochreiter & Schmidhuber , 1995 ) . The motivation of DeepTwist is derived from the goal of reaching flatter minima based on careful addition of noise to weights ( Hochreiter & Schmidhuber , 1995 ) . Weight distortion can be regarded as a weight-noise-injection method which has been known to regularize models associated with flatter minima ( Goodfellow et al. , 2016 ) . Flat minima reduce accuracy loss induced by model compression since DNNs become less sensitive to the variation of weights ( Courbariaux et al. , 2015 ; Hochreiter & Schmidhuber , 1995 ) . For high-dimensional DNNs , it is known that most of the local minima have similar loss function values to that of the global optimum ( LeCun et al. , 2015 ) . Nonetheless , local minima with similar DNN accuracy can represent vastly different amount of accuracy drop through compression depending on the flatness of the loss surface ( Choromanska et al. , 2015 ) . Correspondingly , searching for the local minima exhibiting the minimum compression loss is the key to maximizing the compression ratio . DeepTwist enables locating such local minima via occasional weight distortions . 4 DEEPTWIST APPLICATIONS . In this section , we describe how DeepTwist can be applied in practice to implement model compression with various DNN models . We present the effect of varying SD on compression ratio and the weight distortion methods tailored for selected compression techniques , specifically , pruning , quantization , and low-rank approximation . Note that DeepTwist is versatile to function with other model compression methods as well ( including even unknown ones yet ) provided that a weight distortion function implies an inference implementation form of the compressed weights . 4.1 PRUNING . The initial attempt of pruning weights was to locate redundant weights by computing the Hessian to calculate the sensitivity of weights to the loss function ( LeCun et al. , 1990 ) . However , such a technique has not been considered to be practical due to significant computation overhead for computing the Hessian . Magnitude-based pruning ( Han et al. , 2015 ) has become popular because one can quickly find redundant weights by simply measuring the magnitude of weights . Since then , numerous researchers have realized higher compression ratio largely by introducing Bayesian inference modeling of weights accompanying supplementary hyper-parameters . For example , DNS permits weight splicing when a separately stored full-precision weight becomes larger than a certain threshold ( Guo et al. , 2016 ) . Optimizing splicing threshold values , however , necessitates extensive search space exploration , and thus , longer training time . Variational dropout method ( Molchanov et al. , 2017 ) introduces an explicit Bayesian inference model for a prior distribution of weights , which also induces various hyper-parameters and increased computational complexity . Algorithm 1 : Weight distortion for pruning input : Weight matrix W , pruning rate p output : Distorted weight matrix Ŵ 1 : Ŵ ←W 2 : Sort the elements of Ŵ by magnitude 3 : The p % smallest elements of Ŵ ( by magnitude ) are set to zero Algorithm 1 presents the distortion function for pruning weights using DeepTwist . Since DeepTwist does not need any additional layers , masking layers do not exist in our proposed pruning method . As a result , even though weights are pruned and replaced with zero at a distortion step , pruned weights are still updated in full precision until the next distortion step . If the amount of updates of a pruned weight grows large enough between two consecutive distortion steps , then the weight pruned at last distortion may not be pruned at the next distortion step . Such a feature ( i.e. , pruning decisions are not fixed ) is also utilized for weight splicing in DNS ( Guo et al. , 2016 ) . Weight splicing in DNS relies on a hysteresis function ( demanding sophisticated fine-tuning process with associated hyperparameters ) to switch pruning decisions . Pruning decisions through DeepTwist , on the other hand , are newly determined at every SD step by Algorithm 1 . We present experimental results with LeNet-5 and LeNet-300-100 models on MNIST dataset which are also reported by Guo et al . ( 2016 ) ; Molchanov et al . ( 2017 ) . LeNet-5 consists of 2 convolutional layers and 2 fully connected layers while 3 fully connected layers construct LeNet-300-100 . We train both models for 20000 steps using Adam optimizer where batch size is 50 . All the layers are pruned at the same time and the pruning rate increases gradually following the equation introduced in Zhu & Gupta ( 2017 ) : pt = pf + ( pi − pf ) ( 1− t− ti tf − ti ) E , ( 1 ) where E is a constant , pf is the target pruning rate , pi is the initial pruning rate , t is the current step , and the pruning starts at training step ti and reaches pf at training step tf . After tf steps , pruning rate is maintained to be pf . For LeNet-5 and LeNet-300-100 , ti , pi , E are 8000 ( step ) , 25 ( % ) , and 7 , respectively . tf is 12000 ( step ) for LeNet-5 and 13000 ( step ) for LeNet-300-100 . Note that these choices are not highly sensitive to test accuracy as discussed in Zhu & Gupta ( 2017 ) . We exclude dropout to improve the accuracy of LeNet-300-100 and LeNet-5 since pruning already works as a regularizer ( Han et al. , 2015 ; Wan et al. , 2013 ) . We keep the original learning schedule and the total number of training steps ( no additional training time for model compression ) . Table 1 presents the comparison on pruning rates and test accuracy . Despite the simplicity of our pruning technique , DeepTwist produces higher pruning rate compared with DNS and similar compared with variational dropout technique which involves much higher computational complexity . For Table 1 , we use SD=10 for LeNet-5 and SD=5 for LeNet-300-100 . We investigate how sensitive SD is to the test accuracy when the other parameters ( such as pruning rates , learning rate , and total training time ) are fixed . As shown in Table 2 , for a wide range of SD , the test accuracy has negligible fluctuation1 . Too large SD would result in 1 ) too few weight distortion , 2 ) coarse-grained gradual pruning , and 3 ) unnecessarily large updates for correctly pruned weights . On the other hand , too small SD may yield excessive amount of weight distortion and reduce the opportunity for the pruned weights to recover . We apply DeepTwist-based pruning to an RNN model to verify the effectiveness of DeepTwist with various models . We choose an LSTM model ( Zaremba et al. , 2014 ) on the PTB dataset ( Marcus et al. , 1993 ) . Following the model structure given in Zaremba et al . ( 2014 ) , our model consists of 1Even though we can not show such a sensitivity study for all of the remaining experiments in this paper , SD has also shown low sensitivity to the accuracy even for other models and compression techniques . an embedding layer , 2 LSTM layers , and a softmax layer . The number of LSTM units in a layer can be 200 , 650 , or 1500 , depending on the model configurations ( referred as small , medium , and large model , respectively ) . The accuracy is measured by Perplexity Per Word ( PPW ) , denoted simply by perplexity in this paper . We apply gradual pruning with E = 3 , ti = 0 , pi = 0 , tf = 3rd epoch ( for medium ) or 5th epoch ( for large ) to the pre-trained PTB models . DeepTwist pruning for the PTB models is performed using SD = 100 and the initial learning rate is 2.0 for the medium model ( 1.0 for pre-training ) and 1.0 for the large model ( 1.0 for pre-training ) while the learning policy remains to be the same as in Zaremba et al . ( 2014 ) . For all of the pruning rates selected , Table 3 shows that DeepTwist improves perplexity better than the technique in Zhu & Gupta ( 2017 ) which is based on Han et al . ( 2015 ) . Interestingly , when the pruning rate is 80 % or 85 % , perplexity of the retrained model is even smaller than that of the pre-trained model because of the regularization effect of DeepTwist . The superiority of DeepTwistbased pruning is partly supported by the observation that non-zero weights successfully avoid to be small through retraining while the conventional pruning still keeps near-zero ( unmasked ) weights ( see Appendix for weight distribution difference after retraining ) .

Summary: The paper does not really propose a new way of compressing the model weights, but rather a way of applying existing weight compression techniques. Specifically, the proposed solution is to repeatedly apply weight compression and fine-tuning over the entire training process. Unlike the existing work, weight compression is applied as a form of weight distortion, i.e. the model has the full degree of freedom during fine-tuning (to recover potential compression errors).