Double Viterbi: Weight Encoding for High Compression Ratio and Fast On-Chip Reconstruction for Deep Neural Network

Weight pruning has been introduced as an efficient model compression technique . Even though pruning removes significant amount of weights in a network , memory requirement reduction was limited since conventional sparse matrix formats require significant amount of memory to store index-related information . Moreover , computations associated with such sparse matrix formats are slow because sequential sparse matrix decoding process does not utilize highly parallel computing systems efficiently . As an attempt to compress index information while keeping the decoding process parallelizable , Viterbi-based pruning was suggested . Decoding non-zero weights , however , is still sequential in Viterbi-based pruning . In this paper , we propose a new sparse matrix format in order to enable a highly parallel decoding process of the entire sparse matrix . The proposed sparse matrix is constructed by combining pruning and weight quantization . For the latest RNN models on PTB and WikiText-2 corpus , LSTM parameter storage requirement is compressed 19× using the proposed sparse matrix format compared to the baseline model . Compressed weight and indices can be reconstructed into a dense matrix fast using Viterbi encoders . Simulation results show that the proposed scheme can feed parameters to processing elements 20 % to 106 % faster than the case where the dense matrix values directly come from DRAM . 1 INTRODUCTION . Deep neural networks ( DNNs ) require significant amounts of memory and computation as the number of training data and the complexity of task increases ( Bengio & Lecun , 2007 ) . To reduce the memory burden , pruning and quantization have been actively studied . Pruning removes redundant connections of DNNs without accuracy degradation ( Han et al. , 2015 ) . The pruned results are usually stored in a sparse matrix format such as compressed sparse row ( CSR ) format or compressed sparse column ( CSC ) format , which consists of non-zero values and indices that represent the location of non-zeros . In the sparse matrix formats , the memory requirement for the indices is not negligible . Viterbi-based pruning ( Lee et al. , 2018 ) significantly reduces the memory footprint of sparse matrix format by compressing the indices of sparse matrices using the Viterbi algorithm ( Forney , 1973 ) . Although Viterbi-based pruning compresses the index component considerably , weight compression can be further improved in two directions . First , the non-zero values in the sparse matrix can be compressed with quantization . Second , sparse-to-dense matrix conversion in Viterbi-based pruning is relatively slow because assigning non-zero values to the corresponding indices requires sequential processes while indices can be reconstructed in parallel using a Viterbi Decompressor ( VD ) . Various quantization techniques can be applied to compress the non-zero values , but they still can not reconstruct the dense weight matrix quickly because it takes time to locate non-zero values to the corresponding locations in the dense matrix . These open questions motivate us to find a non-zero value compression method , which also allows parallel sparse-to-dense matrix construction . The contribution of this paper is as follows . ( a ) To reduce the memory footprint of neural networks further , we propose to combine the Viterbibased pruning ( Lee et al. , 2018 ) with a novel weight-encoding scheme , which also uses the Viterbi-based approach to encode the quantized non-zero values . ( b ) We suggest two main properties of the weight matrix that increase the probability of finding `` good '' Viterbi encoded weights . First , the weight matrix with equal composition ratio of ‘ 0 ’ and ‘ 1 ’ for each bit is desired . Second , using the pruned parameters as `` Don ’ t Care '' terms increases the probability of finding desired Viterbi weight encoding . ( c ) We demonstrate that the proposed method can be applied to Recurrent Neural Networks ( RNNs ) and Convolutional Neural Networks ( CNNs ) with various sizes and depths . ( d ) We show that using the same Viterbi-based approach to compress both indices and non-zero values allows us to build a highly parallel sparse-to-dense reconstruction architecture . Using a custom cycle-simulator , we demonstrate that the reconstruction can be done fast . 2 RELATED WORKS . DNNs have been growing bigger and deeper to solve complex nonlinear tasks . However , Denil et al . ( 2013 ) showed that most of the parameters in neural networks are redundant . To reduce the redundancy and minimize memory and computation overhead , several weight reduction methods have been suggested . Recently , magnitude-based pruning methods became popular due to its computational efficiency ( Han et al. , 2015 ) . Magnitude-based pruning methods remove weights according to weight magnitude only and retrain the pruned network to recover from accuracy loss . The method is scalable to large and deep neural networks because of its low computation overhead . Han et al . ( 2015 ) showed 9×-13× pruning rate on AlexNet and VGG-16 networks without accuracy loss on ImageNet dataset . Although the compression rate was high , reduction of actual memory requirement was not as high as the compression rate because conventional sparse matrix formats , such as CSR and CSC , must use large portion of memory to store the indices of surviving weights . Lee et al . ( 2018 ) succeeded in reducing the amount of index-related information using a Viterbi-algorithm based pruning method and corresponding custom sparse matrix format . Lee et al . ( 2018 ) demonstrated 38.1 % memory reduction compared to Han et al . ( 2015 ) with no accuracy loss . The memory reduction was limited , however , due to uncompressed non-zero values . Several weight quantization methods were also suggested to compress the parameters of neural networks . Courbariaux et al . ( 2016 ) ; Li et al . ( 2016 ) ; Rastegari et al . ( 2016 ) demonstrated that reducing the weights to binary or ternary was possible , but the accuracy loss of the binary neural networks was significant . Zhou et al . ( 2016 ) reduced the bit resolution of weights to binary , activations to 2 bits and gradients to 6 bits with 9.8 % top-1 accuracy loss on AlexNet for ImageNet task . Guo et al . ( 2017 ) demonstrated a binary-weight AlexNet with 2.0 % top-1 accuracy loss , achieving ∼10× compression rate . Xu et al . ( 2018 ) showed that RNNs can also be quantized to reduce the memory footprint . By quantizing the weight values to 3 bits with proposed method , the memory footprint of RNN models were reduced ∼10.5× with negligible performance degradation . Han et al . ( 2016b ) suggested to combine pruning with weight quantization to achieve higher compression rate . The results showed 35× increase in compression rate on AlexNet . However , the reduction was limited since the memory requirement of index-related information was only slightly improved with Huffman coding . Although several magnitude-based pruning methods showed high compression rate , computation time did not improve much , because it takes time to decode the sparse matrix formats that describe irregular weight indices of pruned networks . Han et al . ( 2016a ; 2017 ) suggested to use dedicated hardware , custom sparse matrix formats , and dedicated pruning methods to accelerate the computation even after pruning . Hanson & Pratt ( 1989 ) ; Yu et al . ( 2017 ) tried to accelerate the computation by limiting the irregularity of weight indices . By pruning neurons or feature maps , pruned weight matrices could maintain the dense format . These approaches successfully reduced the number of computation of neural networks , but the compression rate was limited due to additional pruning conditions . Although Lee et al . ( 2018 ) could use the Viterbi encoder to construct the index matrix fast , the process of pairing the non-zero weight values with the corresponding indices is still sequential , and thus relatively slow . 3 WEIGHT PRUNING AND QUANTIZATION USING DOUBLE-VITERBI APPROACH . Figure 1 illustrates the flowchart of the proposed compression method . Viterbi-based pruning ( Lee et al. , 2018 ) is applied first , and the pruned matrix is quantized using alternating multi-bit quantization method ( Xu et al. , 2018 ) . Quantized binary code matrices are then encoded using the Viterbi-based approach , which is similar to the one used in pruning . 3.1 VITERBI-BASED PRUNING FOR SPARSE MATRIX INDEX COMPRESSION . As the first step of the proposed weight encoding scheme , we compress the indices of the non-zero values in sparse weight matrix using the Viterbi-based pruning ( Figure 1 ) ( Lee et al. , 2018 ) . In this scheme , Viterbi algorithm is used to select a pruned index matrix which minimizes the accuracy degradation among many candidates which a Viterbi decompressor can generate . While the memory footprint of the index portion is significantly reduced by the Viterbi-based pruning , the remaining non-zero values after pruning still require non-negligible memory when high-precision bits are used . Hence , quantization of the non-zero values is required for further reduction of the memory requirement . Appendix A.1 explains the Viterbi-based pruning in detail . 3.2 MULTI-BIT QUANTIZATION AFTER VITERBI-BASED PRUNING . After Viterbi-based pruning is finished , the alternating multi-bit quantization ( Xu et al. , 2018 ) is applied to the sparse matrix ( Figure 1 ) . As suggested in Xu et al . ( 2018 ) , real-valued non-zero weights are quantized into multiple binary codes { bi } ki=1 ∈ { −1 , +1 } . Detailed algorithm is explained in Appendix A.2 . In addition to the high compression capabilities , another important reason we chose the alternating quantization is that the output distribution of the method is well suited to the Viterbi algorithm , which is used to encode the quantized non-zero values . Detailed explanation is given in Section 3.3 . 3.3 ENCODING BINARY WEIGHT CODES USING THE VITERBI ALGORITHM . A sparse matrix that is generated by Viterbi-based pruning and quantization can be represented using the Viterbi Compression Matrix ( VCM ) format ( Lee et al. , 2018 ) . A sparse matrix stored in VCM format requires much smaller amount of memory than the original dense weight matrix does . However , it is difficult to parallelize the process of reconstructing sparse matrix from the representation in VCM format , because assigning each non-zero value to its corresponding index requires a sequential process of counting ones in indices generated by the Viterbi encoder . To address this issue , we encode binary weight codes { bi } ki=1 as { b̂i } ki=1 in addition to the indices , based on the same Viterbi algorithm ( Forney , 1973 ) . By using similar VD structures ( Figure 2 ) to generate both { b̂i } ki=1 and indices , we can generate both { b̂i } ki=1 and corresponding indices at the same time ; thereby parallel sparse-to-dense matrix conversion becomes possible as shown in Figure 3 . While using VD structures to generate binary weight codes allows parallel sparse-to-dense matrix conversion , it requires the quantization method to satisfy a specific condition to minimize accuracy loss after Viterbi-based encoding . It is known that the VD structure acts as a random number generator ( Lee & Roy , 2012 ) , which produces ‘ 0 ’ and ‘ 1 ’ with 50 % probability each . Thus , generated binary weight codes will be closer to the target binary weight codes if the target binary weight code matrix also consists of equal number of ‘ 0 ’ and ‘ 1 ’ . Interestingly , the composition ratio of ‘ -1 ’ and ‘ +1 ’ in each bi , which was generated by the alternating quantization method , is 50 % each . It is because the weights in DNNs are generally initialized symmetrically with respect to ‘ 0 ’ ( Glorot & Bengio , 2010 ; He et al. , 2015 ) and the distribution is maintained even after training ( Lin et al. , 2016 ) . The preferable output distribution of the alternating quantization implies that the probability of finding an output matrix b̂i close to bi with the Viterbi algorithm is high . For comparison , we measured the accuracy differences before and after Viterbi encoding for several quantization methods such as linear quantization ( Lin et al. , 2016 ) , logarithmic quantization ( Miyashita et al. , 2016 ) , and alternating quantization ( Xu et al. , 2018 ) . When the Viterbi encoding is applied to the weight quantized by alternating quantization ( Xu et al. , 2018 ) , the validation accuracy degrades by only 2 % . However , accuracy degrades by 71 % when the Viterbi encoding is applied to the weight quantized using other methods ( Lin et al. , 2016 ; Miyashita et al. , 2016 ) . The accuracy difference mainly comes from the uneven weight distribution . Because weights of neural networks usually have normal distribution , the composition ratio of ’ 0 ’ and ’ 1 ’ is not equal when the linear or logarithmic quantization is applied to the weights unlike alternating quantization . Another important idea to increase the probability of finding `` good '' Viterbi encoded weight is to consider the pruned parameters in bi as `` Don ’ t Care '' terms ( Figure 4 ) . The `` Don ’ t Care '' elements can have any values when finding b̂i , because they will be masked by the zero values in the index matrix generated by the Viterbi pruning . Next , let us describe how we use the Viterbi algorithm for weight encoding . We select the b̂i that best matches with bi among all possible b̂i cases that the VD can generate , as follows . We first construct a trellis diagram as shown in Figure 5 . The trellis diagram is a state diagram represented with time index T . A state number is represented with Flip-Flop ( FF ) values , where the rightmost FF value is the most significant bit ( MSB ) .1 Each transition with a 1-bit input from a state generates the corresponding multiple output bits . A cost function for each transition using path and branch metrics is set and computed in the next step . The branch metric λi , jt is the cost of traveling along a transition from a state i to the successor state j at the time index t. The path metric is expressed as Γjt+1 = max ( Γi1t + λ i1 , j t , Γ i2 t + λ i2 , j t ) , ( 1 ) where i1 and i2 are two predecessor states of j . Equation 1 denotes that one of the two possible transitions is selected to maximize the accumulated value of branch metrics.2 The branch metric is defined as βi , j , mt = { 1 , if bi , j , mt = 2o i , j , m t − 1 0 , otherwise , λi , jt = No∑ m=1 βi , j , mt , ( 2 ) where bi , j , mt is the value of binary codes according to the m th VD output at time index t , oi , j , mt is the value of the mth VD output at time index t , and No is the number of outputs generated by the VD at each time step . { −1 , +1 } in the binary codes corresponds to { 0 , 1 } in the VD output in equation 2 . Equation 2 maximizes the number of VD outputs that exactly match with corresponding binary codes while ignoring the pruned parameters ( Figure 4 ) . When the last time index is reached , the state with the maximum path metric is selected . Previous states connected by surviving branches are traced while corresponding oi , j , mt of each branch is recorded as b̂ i , j , m t . Each binary weight code bi is encoded as b̂i with a compression ratio of 1/No using this scheme . 1If the VD has N FFs , then the number of states is 2N . 2max function is used instead of min function because the metric value is considered as a degree of ‘ reward ’ rather than ‘ cost ’ ( Lee et al. , 2018 ) .

Summary: This paper presents a new way to represent a dense matrix in a compact format. First, the method prunes a dense matrix based on the Viterbi-based pruning. Then, the pruned matrix is quantized with alternating multi-bit quantization. Finally, the binary vectors produced by the quantization algorithm are further compressed with the Viterbi-based algorithm. It spots the problem of each existing approach and solve the problems by combining each method. The combination is new and the result is encouraging.