Network Pruning for Low-Rank Binary Index

1 INTRODUCTION . Numerous parameter pruning techniques have been introduced based on the observation that significant amounts of connectivity can be removed without sacrificing model accuracy . Current active research strives to enhance pruning rate at the cost of additional computations ( Guo et al. , 2016 ; Molchanov et al. , 2017 ) , to reduce computational complexity of pruning procedures ( Zhu & Gupta , 2017 ; Han et al. , 2015 ) , and to find the underlying principles of pruning ( Liu et al. , 2019 ; Frankle & Carbin , 2019 ) , to name a few . Figure 1 shows a dense matrix after pruning redundant parameters and various masking index representations . Fine-grained parameter pruning ( i.e. , each parameter is evaluated to be pruned ) results in sparse matrices which can be represented by various formats depending on how indices of non-zero weights are described ( Lee et al. , 2018a ) . A desirable index format in sparse DNNs should produce 1 ) low memory footprint for index and 2 ) high parallelism for index storage accesses . Unfortunately widely used compressed sparse row ( CSR ) format involves two index values for each non-zero weight while the number of bits to represent indices increases as the parameter size increases . Even though the binary index form is a regular structure that can utilize full memory bandwidth , sparsity does not reduce memory footprint . A critical concern on those previous sparse matrix formats ( CSR format and binary index format ) is that the relative amount of index increases if the number of bits to present non-zero weight shrinks due to advanced quantization techniques . Table 1 describes index size divided by the amount of non-zero weights assuming that sparsity is S and non-zero weights are quantized to be Q bits . It is clear that reducing quantization bits aggravates the issue of too large index portion even if a relative indexing scheme ( Han et al. , 2016b ) is used for CSR format with 5 bits . If pruning is performed in a fine-grained manner to improve overall pruning rate , then each row/column or block exhibits different sparsity . Consequently , row/column-wise or block-wise computations require vastly different computation latency leading to significantly degraded parallelism ( Han et al. , 2016a ) . Thus , some of recent pruning techniques suggest removing connectivity in a well-structured form , ( Wen et al. , 2016 ; He et al. , 2017 ) , or in a block-level ( Yu et al. , 2017 ; Narang et al. , 2017 ) . In this paper , we propose a new fine-grained pruning method to find an efficient sparse matrix representation based on binary index-matrix factorization . As shown in Figure 1 , our proposed index compression scheme decomposes a binary index matrix into two small binary matrices in order to reduce index storage and maintain regular structure . Binary matrix multiplication is inherently parallelizable and sparse non-zero weights can be decoded with high parallelism using recent approaches ( Ahn et al. , 2019 ) . Thus , decoding sparse matrices can be performed with high performance . In order to accomplish such a new indexing form , we propose an algorithm that finds a particular finegrained pruning result by generating a low-rank binary index matrix . Since factorization may not exactly reproduce the original binary index matrix , we investigate whether a low-rank binary index matrix produces a pruning result while maintaining allowable model accuracy . Our proposed binary matrix factorization technique significantly reduces the amount of index data compared to using CSR , as demonstrated in the next sections . We also introduce a tiling technique to alleviate on-chip memory size and the burden of on-chip binary matrix multiplication , and to further improve compression ratio . Recently , a regular-structured index compression method has been proposed using the Viterbi algorithm ( Lee et al. , 2018a ) , which explores sequences to be used as pruning mask bits , decompressed by Viterbi encoders . Even though compression rate improves over CSR , for every one input bit , a large number of XOR gates and delay units is required . In comparison , our proposed decompression relies on simple binary matrix multiplications and achieves even higher compression . 2 BINARY MATRIX FACTORIZATION FOR PRUNING INDEX . Suppose that a 5× 5 weight matrix W is given as W = −0.1 0.9 1.2 −0.2 −0.6 1.8 0.2 −0.7 −1.6 0.6 −0.1 −1.7 0.1 −0.3 1.2 −0.4 1.4 −0.9 0.6 1.4 −1.1 0.5 1.0 1.0 −0.3 . ( 1 ) Following the magnitude-based pruning method ( Han et al. , 2015 ) , all weights with magnitude smaller than a certain threshold value are pruned to zero . For example , for a threshold of 0.7 , we obtain the following pruning index ( i.e. , binary masking layer ) matrix I as I = 0 1 1 0 0 1 0 1 1 0 0 1 0 0 1 0 1 1 0 1 1 0 1 1 0 . ( 2 ) It is important to note that magnitude-based pruning is not an optimal solution ( Lee et al. , 2018a ; LeCun et al. , 1990 ; Han et al. , 2016b ) to maximize the pruning rate , i.e. , a variety of masking layers exist for the same pruning rate . Binary matrix factorization ( BMF ) ( Zhang et al. , 2007 ) is a reasonable approach to compress I . For I ∈ { 0 , 1 } m×n , BMF tries to find the best Im×kp and Ik×nz to approximate I as I ≈ Ip ⊗ Iz = Ia , where k corresponds to the binary matrix factorization rank and ⊗ stands for binary matrix multiplication . A binary product of Ip and Iz is then defined as ( Ia ) i , j = k∨ l=1 ( Ip ) i , l ∧ ( Iz ) l , j . ( 3 ) While BMF should minimize the number of mismatched bits between I and Ia , such an optimization is NP-hard . Hence , several heuristic methods for BMF have been proposed in the literature ( Zhang et al. , 2007 ) . Moreover , BMF using I ( without referring to W ) lacks weight magnitude information . Yet , in the context of magnitude-based pruning , weights of small magnitude should be pruned with higher probability ( i.e. , lower importance ) . Therefore , we explore a method that preserves the importance of each weight when it is not possible to exactly reproduce I through matrix factorization . 2.1 BINARY MATRIX FACTORIZATION BASED ON NON-NEGATIVE MATRIX FACTORIZATION . Non-negative matrix factorization ( NMF ) factorizes a real-valued matrix H into two real-valued matrices H1 and H2 under the constraint that all three matrices consist of non-negative elements . Similar to singular-value decomposition ( SVD ) , NMF attempts to minimize ||H−H1H2||2F , where ||H||F denotes the Frobenius norm of the matrix H . The property of non-negativity in NMF is useful for analysis of various multivariate data . Numerous numerical approximations of NMF have been suggested since an exact solution of NMF is not generally available ( Lee & Seung , 1999 ; Zhang et al. , 2007 ) . In our proposed technique , we take the magnitude of each element of W to generate M ( i.e. , Mi , j = |Wi , j | ) . and M is factorized by an NMF library ( e.g. , ( Zitnik & Zupan , 2012 ) ) into two matrices Mp and Mz . For example , a magnitude matrix M of the matrix W of Eq . ( 1 ) can be factorized into Mp = 0.2 0.5 1.3 0.0 0.0 0.9 0.3 0.8 0.8 0.2 , Mz = [ 1.3 0.1 0.7 1.2 0.3 0.0 1.8 0.7 0.2 1.3 ] , ( 4 ) where rank k is 2 . The next step is to convert Mp and Mz into two binary matrices Ip and Iz using threshold values Tp and Tz ( i.e. , ( Ip ) i , j = 1 if ( Mp ) i , j ≥ Tp , or 0 otherwise ) . The sparsity of Ip and Iz can each be controlled by Tp and Tz . Our goal is to achieve similar sparsity between I and Ip ⊗ Iz . Suppose that Tp = 0.5 and Tz = 0.6 are carefully chosen to produce similar sparsity as I . We obtain Ip = 0 1 1 0 0 1 0 1 1 0 , Iz = [ 1 0 1 1 0 0 1 1 0 1 ] ( 5 ) and the binary product of Ip and Iz becomes Ia = Ip ⊗ Iz = 0 1 1 0 1 1 0 1 1 0 0 1 1 0 1 0 1 1 0 1 1 0 1 1 0 . ( 6 ) Compared with the pruning-index matrix I in Eq . ( 2 ) , there are 2 mismatched elements ( underlined in Eq . ( 6 ) ) . Algorithm 1 : Binary pruning-index-data matrix factorization input : W ∈ Rm×n , rank k , target sparsity S output : Ip ∈ { 0 , 1 } m×k , Iz ∈ { 0 , 1 } k×n 1 : Generate the magnitude matrix M using W 2 : Mp , Mz = NMF ( M , k ) 3 : Costmin ←∞ , Sminp ← 0.0 , Sminz ← 0.0 4 : for Sp = 0.0 to 1.0 do 5 : Compute Sz using Eq . ( 7 ) 6 : repeat 7 : Convert ( Mp , Mz ) into ( Ip , Iz ) w/ ( Sp , Sz ) 8 : Adjust Sz depending on ( Sa − S ) 9 : until Sa ≈ S 10 : Cost← ∑ Ii , j=1 , ( Ia ) i , j=0 Mi , j 11 : if Costmin > Cost then 12 : Costmin ← Cost , Sminp ← Sp , Sminz ← Sz 13 : end if 14 : end for 15 : Convert ( Mp , Mz ) into ( Ip , Iz ) w/ ( Sminp , S min z ) 16 : Return Ip , Iz The rationale behind this approach is as follows : 1 ) If Mi , j is large , then its corresponding k components of Mp and Mz ( i.e. , ( Mp ) i , : and ( Mz ) : ,j ) will also be large with high probability and , correspondingly , 2 ) Binary matrix conversion using Tp and Tz would yield a high probability of being ‘ 1 ’ within Ip and Iz if the corresponding Mi , j is large . Note that in order to match ‘ OR ’ operations in Eq . ( 3 ) , negative values in Mp or Mz are not allowed . Thus , SVD is not applicable for our proposed scheme . Let S , Sa , Sp , and Sz be the sparsity of I , Ia , Ip , and Iz , respectively . From the dot product operation in Eq . ( 6 ) , the expression for pruning rate S becomes S = ( ( 1− ( 1− Sp ) ( 1− Sz ) ) k , ( 7 ) assuming that the probability of a bit being ‘ 0 ’ in Ip and Iz follows Sp and Sz . Then , Sz = ( S1/k − Sp ) / ( 1 − Sp ) , which needs to be fine-tuned in practice . If Tp and associated Sp are given , then Tz and Sz are automatically determined by the target pruning rate . Subsequently , given W and rank k , it is necessary to find a certain Sp that produces the best masking layer for pruning . In order to optimize Sp , we define the cost function for pruning-index compression to be ∑ Mi , j , where ( I ) i , j = 1 and ( Ia ) i , j = 0 ( i.e. , a sum of all unintentionally pruned weights ’ magnitude by binary matrix decomposition ) . Algorithm 1 describes a method to find the best Sp by sweeping Sp and monitoringCost . Given the linear relationship between Sa and Sz , the algorithm can use binary search to expedite adjustment of Sz .

Summary: This paper proposed a new network pruning method that generates a low-rank binary index matrix to compress index data, and a tile-based factorization technique to save memory. The binary index can achieve larger compression ratio than the CSR index, and the  low-rank binary index can further reduce memory usage. The results for various networks, including DNN, CNN and LSTM, have shown the effectiveness of the propsoed method. The paper is well-written and easy to follow.