Network Pruning for Low-Rank Binary Index

1 INTRODUCTION . Numerous parameter pruning techniques have been introduced based on the observation that significant amounts of connectivity can be removed without sacrificing model accuracy . Current active research strives to enhance pruning rate at the cost of additional computations ( Guo et al. , 2016 ; Molchanov et al. , 2017 ) , to reduce computational complexity of pruning procedures ( Zhu & Gupta , 2017 ; Han et al. , 2015 ) , and to find the underlying principles of pruning ( Liu et al. , 2019 ; Frankle & Carbin , 2019 ) , to name a few . Figure 1 shows a dense matrix after pruning redundant parameters and various masking index representations . Fine-grained parameter pruning ( i.e. , each parameter is evaluated to be pruned ) results in sparse matrices which can be represented by various formats depending on how indices of non-zero weights are described ( Lee et al. , 2018a ) . A desirable index format in sparse DNNs should produce 1 ) low memory footprint for index and 2 ) high parallelism for index storage accesses . Unfortunately widely used compressed sparse row ( CSR ) format involves two index values for each non-zero weight while the number of bits to represent indices increases as the parameter size increases . Even though the binary index form is a regular structure that can utilize full memory bandwidth , sparsity does not reduce memory footprint . A critical concern on those previous sparse matrix formats ( CSR format and binary index format ) is that the relative amount of index increases if the number of bits to present non-zero weight shrinks due to advanced quantization techniques . Table 1 describes index size divided by the amount of non-zero weights assuming that sparsity is S and non-zero weights are quantized to be Q bits . It is clear that reducing quantization bits aggravates the issue of too large index portion even if a relative indexing scheme ( Han et al. , 2016b ) is used for CSR format with 5 bits . If pruning is performed in a fine-grained manner to improve overall pruning rate , then each row/column or block exhibits different sparsity . Consequently , row/column-wise or block-wise computations require vastly different computation latency leading to significantly degraded parallelism ( Han et al. , 2016a ) . Thus , some of recent pruning techniques suggest removing connectivity in a well-structured form , ( Wen et al. , 2016 ; He et al. , 2017 ) , or in a block-level ( Yu et al. , 2017 ; Narang et al. , 2017 ) . In this paper , we propose a new fine-grained pruning method to find an efficient sparse matrix representation based on binary index-matrix factorization . As shown in Figure 1 , our proposed index compression scheme decomposes a binary index matrix into two small binary matrices in order to reduce index storage and maintain regular structure . Binary matrix multiplication is inherently parallelizable and sparse non-zero weights can be decoded with high parallelism using recent approaches ( Ahn et al. , 2019 ) . Thus , decoding sparse matrices can be performed with high performance . In order to accomplish such a new indexing form , we propose an algorithm that finds a particular finegrained pruning result by generating a low-rank binary index matrix . Since factorization may not exactly reproduce the original binary index matrix , we investigate whether a low-rank binary index matrix produces a pruning result while maintaining allowable model accuracy . Our proposed binary matrix factorization technique significantly reduces the amount of index data compared to using CSR , as demonstrated in the next sections . We also introduce a tiling technique to alleviate on-chip memory size and the burden of on-chip binary matrix multiplication , and to further improve compression ratio . Recently , a regular-structured index compression method has been proposed using the Viterbi algorithm ( Lee et al. , 2018a ) , which explores sequences to be used as pruning mask bits , decompressed by Viterbi encoders . Even though compression rate improves over CSR , for every one input bit , a large number of XOR gates and delay units is required . In comparison , our proposed decompression relies on simple binary matrix multiplications and achieves even higher compression . 2 BINARY MATRIX FACTORIZATION FOR PRUNING INDEX . Suppose that a 5× 5 weight matrix W is given as W = −0.1 0.9 1.2 −0.2 −0.6 1.8 0.2 −0.7 −1.6 0.6 −0.1 −1.7 0.1 −0.3 1.2 −0.4 1.4 −0.9 0.6 1.4 −1.1 0.5 1.0 1.0 −0.3 . ( 1 ) Following the magnitude-based pruning method ( Han et al. , 2015 ) , all weights with magnitude smaller than a certain threshold value are pruned to zero . For example , for a threshold of 0.7 , we obtain the following pruning index ( i.e. , binary masking layer ) matrix I as I = 0 1 1 0 0 1 0 1 1 0 0 1 0 0 1 0 1 1 0 1 1 0 1 1 0 . ( 2 ) It is important to note that magnitude-based pruning is not an optimal solution ( Lee et al. , 2018a ; LeCun et al. , 1990 ; Han et al. , 2016b ) to maximize the pruning rate , i.e. , a variety of masking layers exist for the same pruning rate . Binary matrix factorization ( BMF ) ( Zhang et al. , 2007 ) is a reasonable approach to compress I . For I ∈ { 0 , 1 } m×n , BMF tries to find the best Im×kp and Ik×nz to approximate I as I ≈ Ip ⊗ Iz = Ia , where k corresponds to the binary matrix factorization rank and ⊗ stands for binary matrix multiplication . A binary product of Ip and Iz is then defined as ( Ia ) i , j = k∨ l=1 ( Ip ) i , l ∧ ( Iz ) l , j . ( 3 ) While BMF should minimize the number of mismatched bits between I and Ia , such an optimization is NP-hard . Hence , several heuristic methods for BMF have been proposed in the literature ( Zhang et al. , 2007 ) . Moreover , BMF using I ( without referring to W ) lacks weight magnitude information . Yet , in the context of magnitude-based pruning , weights of small magnitude should be pruned with higher probability ( i.e. , lower importance ) . Therefore , we explore a method that preserves the importance of each weight when it is not possible to exactly reproduce I through matrix factorization . 2.1 BINARY MATRIX FACTORIZATION BASED ON NON-NEGATIVE MATRIX FACTORIZATION . Non-negative matrix factorization ( NMF ) factorizes a real-valued matrix H into two real-valued matrices H1 and H2 under the constraint that all three matrices consist of non-negative elements . Similar to singular-value decomposition ( SVD ) , NMF attempts to minimize ||H−H1H2||2F , where ||H||F denotes the Frobenius norm of the matrix H . The property of non-negativity in NMF is useful for analysis of various multivariate data . Numerous numerical approximations of NMF have been suggested since an exact solution of NMF is not generally available ( Lee & Seung , 1999 ; Zhang et al. , 2007 ) . In our proposed technique , we take the magnitude of each element of W to generate M ( i.e. , Mi , j = |Wi , j | ) . and M is factorized by an NMF library ( e.g. , ( Zitnik & Zupan , 2012 ) ) into two matrices Mp and Mz . For example , a magnitude matrix M of the matrix W of Eq . ( 1 ) can be factorized into Mp = 0.2 0.5 1.3 0.0 0.0 0.9 0.3 0.8 0.8 0.2 , Mz = [ 1.3 0.1 0.7 1.2 0.3 0.0 1.8 0.7 0.2 1.3 ] , ( 4 ) where rank k is 2 . The next step is to convert Mp and Mz into two binary matrices Ip and Iz using threshold values Tp and Tz ( i.e. , ( Ip ) i , j = 1 if ( Mp ) i , j ≥ Tp , or 0 otherwise ) . The sparsity of Ip and Iz can each be controlled by Tp and Tz . Our goal is to achieve similar sparsity between I and Ip ⊗ Iz . Suppose that Tp = 0.5 and Tz = 0.6 are carefully chosen to produce similar sparsity as I . We obtain Ip = 0 1 1 0 0 1 0 1 1 0 , Iz = [ 1 0 1 1 0 0 1 1 0 1 ] ( 5 ) and the binary product of Ip and Iz becomes Ia = Ip ⊗ Iz = 0 1 1 0 1 1 0 1 1 0 0 1 1 0 1 0 1 1 0 1 1 0 1 1 0 . ( 6 ) Compared with the pruning-index matrix I in Eq . ( 2 ) , there are 2 mismatched elements ( underlined in Eq . ( 6 ) ) . Algorithm 1 : Binary pruning-index-data matrix factorization input : W ∈ Rm×n , rank k , target sparsity S output : Ip ∈ { 0 , 1 } m×k , Iz ∈ { 0 , 1 } k×n 1 : Generate the magnitude matrix M using W 2 : Mp , Mz = NMF ( M , k ) 3 : Costmin ←∞ , Sminp ← 0.0 , Sminz ← 0.0 4 : for Sp = 0.0 to 1.0 do 5 : Compute Sz using Eq . ( 7 ) 6 : repeat 7 : Convert ( Mp , Mz ) into ( Ip , Iz ) w/ ( Sp , Sz ) 8 : Adjust Sz depending on ( Sa − S ) 9 : until Sa ≈ S 10 : Cost← ∑ Ii , j=1 , ( Ia ) i , j=0 Mi , j 11 : if Costmin > Cost then 12 : Costmin ← Cost , Sminp ← Sp , Sminz ← Sz 13 : end if 14 : end for 15 : Convert ( Mp , Mz ) into ( Ip , Iz ) w/ ( Sminp , S min z ) 16 : Return Ip , Iz The rationale behind this approach is as follows : 1 ) If Mi , j is large , then its corresponding k components of Mp and Mz ( i.e. , ( Mp ) i , : and ( Mz ) : ,j ) will also be large with high probability and , correspondingly , 2 ) Binary matrix conversion using Tp and Tz would yield a high probability of being ‘ 1 ’ within Ip and Iz if the corresponding Mi , j is large . Note that in order to match ‘ OR ’ operations in Eq . ( 3 ) , negative values in Mp or Mz are not allowed . Thus , SVD is not applicable for our proposed scheme . Let S , Sa , Sp , and Sz be the sparsity of I , Ia , Ip , and Iz , respectively . From the dot product operation in Eq . ( 6 ) , the expression for pruning rate S becomes S = ( ( 1− ( 1− Sp ) ( 1− Sz ) ) k , ( 7 ) assuming that the probability of a bit being ‘ 0 ’ in Ip and Iz follows Sp and Sz . Then , Sz = ( S1/k − Sp ) / ( 1 − Sp ) , which needs to be fine-tuned in practice . If Tp and associated Sp are given , then Tz and Sz are automatically determined by the target pruning rate . Subsequently , given W and rank k , it is necessary to find a certain Sp that produces the best masking layer for pruning . In order to optimize Sp , we define the cost function for pruning-index compression to be ∑ Mi , j , where ( I ) i , j = 1 and ( Ia ) i , j = 0 ( i.e. , a sum of all unintentionally pruned weights ’ magnitude by binary matrix decomposition ) . Algorithm 1 describes a method to find the best Sp by sweeping Sp and monitoringCost . Given the linear relationship between Sa and Sz , the algorithm can use binary search to expedite adjustment of Sz .

Summary: The paper addresses the problem of reducing the computational complexity of neural network pruning. Main idea is to compute a low-rank approximation of the binary index matrix used to represent the structure of the pruned network. In the considered setup, the binary index matrix is the (sparse) boolean matrix associated with the nonzero network's weights. As low-rank decomposition of binary matrices is a hard problem, the authors propose a method to approximate the solution by computing a more standard non-negative matrix factorization.