Encoding Weights of Irregular Sparsity for Fixed-to-Fixed Model Compression

1 INTRODUCTION . As one of the efficient compression methods , pruning reduces the number of parameters by replacing model parameters of low importance with zeros ( LeCun et al. , 1990 ) . Since magnitude-based pruning has shown that pruning can be conducted with low computational complexity ( Han et al. , 2015 ) , various practical pruning methods have been studied to achieve higher compression ratio ( Zhu and Gupta , 2017 ; Molchanov et al. , 2017 ; Louizos et al. , 2018 ; Gale et al. , 2019 ) . Recently , pruning has been extended to a deeper understanding of weight initialization . Based on the Lottery Winning Ticket hypothesis ( Frankle and Carbin , 2018 ) , ( Renda et al. , 2020 ) suggests a weight-rewinding method to explore sub-networks from full-trained models . Furthermore , pruning methods at initialization steps without pre-trained models have also been proposed ( Lee et al. , 2019b ; Wang et al. , 2020 ) . Despite a high compression ratio , fine-grained pruning that eliminates each parameter individually has practical issues to be employed in parallel computing platforms . One of the popular formats to represent sparse matrices after pruning is the Compressed Sparse Row ( CSR ) whose structures are irregular . For parallel computing , such irregular formats degrade inference performance that is dominated by matrix multiplications ( Gale et al. , 2020 ) . Algorithm 1 presents a conventional sparse matrix-vector multiplication ( SpMV ) algorithm using CSR format which involves irregular and data-dependent memory accesses . Correspondingly , performance gain using a sparse matrix multiplication ( based on CSR ) is a lot smaller than the compression ratio of pruning ( Yu et al. , 2017 ) . Structured pruning methods ( e.g , block-based pruning ( Narang et al. , 2017 ; Zhou et al. , 2021 ) , filter-based pruning ( Li et al. , 2017 ) , and channel-based pruning ( He et al. , 2017 ; Liu et al. , 2017 ) ) have been suggested to enhance parallelism by restricting the locations of pruned weights . Those methods , however , induce degraded accuracy and/or lower pruning rate than fine-grained pruning . In this paper , as an efficient method to compress sparse NNs pruned by fine-grained pruning , we consider weight encoding techniques . As shown in Algorithm 2 , encoded weights are multiplied by a fixed binary matrix M⊕ to reconstruct the original weights . We propose an encoding method and M⊕ design methodology to compress sparse weights in a regular format . It should be noted that a sparse matrix multiplication can be even slower than a dense matrix multiplication unless pruning rate is high enough ( Yu et al. , 2017 ; Gale et al. , 2020 ) that does not happen with Algorithm ∗Equal contribution . 2 for memory-intensive workloads . We study the maximum compression ratio of such encoding-based compression using entropy and propose a sequential fixed-to-fixed scheme that keeps high parallelism after fine-grained pruning . We show that by our proposed fixed-to-fixed scheme , a compression ratio can approach the maximum ( estimated by entropy ) even under the variation of the unpruned weights in a block . Algorithm 1 : SpMV ( CSR format ) In : Dense vector x , CSR vectors dat , row , col Out : Dense vector y for i← 1 to n do for j ← rowi to rowi+1 do yi ← yi + dat [ j ] × x [ col [ j ] ] / * Irregular , data-dependent access * / Algorithm 2 : Proposed SpMV ( using encoded weights ) In : Dense vector x∈Rm , Encoded vectors we1 .. n∈Rk Fixed matrix M⊕ ∈ { 0 , 1 } k×m , Mask * * ( k m ) Out : Dense vector y for i← 1 to n do Wi ← wei ×M⊕ over GF ( 2 ) yi = Wi · x with Mask ( for zero skipping ) / * Decoding m elements using wei ( Regular access ) * / 2 FIXED-TO-FIXED SPARSITY ENCODING . Data compression is a process of encoding original data into a smaller size . If a fixed number of original bits are encoded into a fixed number of ( smaller ) bits , such a case is categorized into a “ fixed-to-fixed ” compression scheme . Similarly , “ fixed-to-variable ” , “ variable-to-fixed ” , and “ variableto-variable ” categories are available while variable lengths of original and/or encoded bits allow higher compression ratio than fixed ones ( e.g. , Huffman codes ( Huffman , 1952 ) as fixed-to-variable scheme , Lempel-Ziv ( LZ ) -based coding ( Ziv and Lempel , 2006 ) as variable-to-fixed scheme , and Golomb codes ( Golomb , 1966 ) as variable-to-variable scheme ) . Among those 4 categories , “ fixed-to-fixed ” compression is the best for NNs that rely on parallel computing systems . Fixed-to-fixed compression schemes are , however , challenging when fine-grained pruning is employed in NNs because the number of unpruned weights in a fixed-size block varies . Accordingly , most previous sparsity representations ( such as CSR format ) translate a fixed-size weight block into a variable-size block while such a translation would demand non-uniform memory accesses that lead to significantly degraded memory bandwidth utilization as shown in Figure 1 . Specifically , in the case of fixed-to-variable sparsity format ( e.g. , CSR ) in Figure 1 , we observe that memory bandwidth is low because fine-grained pruning induces a variable number of pruned weights for a certain block or row while memory is designed to access a fixed amount of consecutive data . Since higher sparsity leads to higher relative standard deviation ( i.e. , coefficient of variation ) on pruned weights in a block , low memory bandwidth is a significant issue even though the amount of data to be stored is reduced ( see Appendix A ) . As a result , for fixed-to-variable sparsity format , it is difficult to implement fine-grained pruning with parallel computing systems that require high memory bandwidth ( Yu et al. , 2017 ) . On the other hand , fixed-to-fixed compression schemes in Figure 1 can maintain the same memory bandwidth regardless of sparsity . In this work , we propose a “ fixed-to-fixed ” compression scheme as shown in Figure 2 when the number of pruned weights in a block can vary . A successful fixed-to-fixed compression of sparse NNs should consider the followings : • ( High compression ratio ) The maximum compression ratio is limited by the minimum entropy ( that can be obtained by a fixed-to-variable scheme as we discuss in Appendix D ) . Suppose that a block to be encoded contains ( fixed ) nb bits among which ( fixed ) nu bits are unpruned . A fixed-to-fixed encoding scheme is required to support high compression close to ( nb/nu ) ( estimated by entropy ) . Fixed-to-fixed decoding , then , accepts ( fixed ) Nin bits as an input and produces Nout bits as an output while Nout/Nin = nb/nu . • ( Variation tolerance ) For a fine-grained pruning , nu is given as a random variable whose distribution is affected by pruning rate , nb size , a particular pruning method , and so on . Our goal is to maintain a fixed-to-fixed scheme with a high compression ratio even under Var [ nu ] 6= 0 . In Figure 2 , for example , 5 blocks of original data have various nu values while the size of an encoded block is fixed to be 4 ( =E [ nu ] ) . We will discuss how to design a variation tolerant encoding . 3 RANDOM NUMBER GENERATOR . Before we discuss compression schemes , let us assume that a binary masking matrix is given to represent which weights are pruned or not ( note such a binary masking matrix can be compressed significantly ( Lee et al. , 2019a ) ) . Then , a pruned weight can be described as a don ’ t care value ( 5 ) that is to be masked . We also assume that 1 ) pruning each weight is performed independently with pruning rate S and 2 ) unpruned weight is assigned to 0 or 1 with equal probability ( such assumptions are not necessary when we demonstrate our experimental results in Section 5 ) . To obtain both “ high compression ratio ” and “ variation tolerance ” while a fixed-to-fixed compression scheme is considered , we adopt random number generator schemes that enable encoding/decoding of weights . A random number generator accepts a fixed number of inputs and produces nb bits so as to implement a fixed-to-fixed compression scheme . As shown in Figure 3 , a weight block is compared with every output of a random number generator . If there is an output vector matching original ( partially masked ) weights , then a corresponding input vector of a random number generator can be an encoded input vector . As an effort to increase the Hamming distance between any two outputs ( i.e. , the number of bit positions in which two bits are different ) , 2nu outputs out of 2nb possible candidates need to be randomly selected . Note that random encoding has already been suggested by Claude Shannon to introduce channel capacity that is the fundamental theory in digital communication ( Morelos-Zaragoza , 2006 ) . Since then , practical error correction coding techniques have been proposed to implement random-like coding by taking into account efficient decoding ( instead of using a large look-up table ) . Similar to error correction coding that usually depends on linear operations over Galois Field with two elements , or GF ( 2 ) ( Morelos-Zaragoza , 2006 ) , for simple encoding of original data , recently , two compression techniques for sparse NNs have been proposed . An XOR-gate decoder produces ( a large number of ) binary outputs using ( a relatively much smaller number of ) binary inputs while outputs are quantized weights ( Kwon et al. , 2020 ) . Another example is to adopt a Viterbi encoding/decoding scheme ( Forney , 1973 ) to generate multiple bits using a single bit as an input ( Ahn et al. , 2019 ) . For a block that can not be encoded into a compressed one by a random number generator , we can attach patch data to fix unmatched bits ( Kwon et al. , 2020 ) or re-train the model to improve the accuracy ( Ahn et al. , 2019 ) . To compare the random number generation capability of various block-level compression schemes , we introduce ‘ encoding efficiency ’ given as a percentage . E ( Encoding Efficiency ) = # of correctly matched bits # of unpruned bits × 100 ( % ) ( 1 ) Let S be pruning rate ( 0 ≤ S ≤ 1 ) . To measure encoding efficiency ( E ) , we assume that the compression ratio of a random number generator ( =the number of output bits / the number of input bits ) is 1/ ( 1 − S ) . We generate numerous randomly pruned ( binary ) weight blocks , and for each block , we investigate all of the possible outputs that a random number generator can produce . If there is a block missing a matching output of a generator , then the maximum number of correctly matched bits is recorded for each block . We repeat such an experiment for all of the blocks . Note that E can not be higher than 100 % for any generators . 3.1 FIXED PRUNING RATE IN A BLOCK . For simplicity , we assume that nu in a block is a fixed number . Let us study E when nu is fixed using an XOR-gate decoder introduced in ( Kwon et al. , 2020 ) . For an XOR-gate decoder , when Nout is the number of output bits and Nin is the number of input bits , a matrix M⊕ ∈ { 0 , 1 } Nout×Nin presents connectivity information between an input vector wx ( ∈ { 0 , 1 } Nin ) and an output vector wy ( ∈ { 0 , 1 } Nout ) such that we have wy = M⊕ ·wx over GF ( 2 ) . For example , if the second row of M⊕ ( with Nin = 4 and Nout = 8 ) is given as [ 1 0 1 1 ] , then w y 2 = w x 1 ⊕wx3 ⊕wx4 ( ‘ ⊕ ’ indicates a binary XOR operation or an addition over GF ( 2 ) equivalently ) . An element of M⊕ is randomly filled with 0 or 1 as a simple random number generator design technique ( Kwon et al. , 2020 ) . To measure E , let Nin/Nout ≈ 1− S such that Nout = bNin · ( 1/ ( 1−S ) ) c. Correspondingly , for a certain S , a block size ( =Nout ) increases as Nin increases . When nb = Nout and nu = Nin , Figure 4a describes statistics of E when random M⊕ matrices are associated with random blocks . From Figure 4a , it is clear that increasing Nin is the key to improving encoding efficiency . Note that , however , increasing Nin and Nout complicates the decoding complexity ( due to large M⊕ ) and the encoding complexity as well ( due to an exponentially large search space ) .

Summary: This paper tackles an interesting and important question which is how can we make the pruned network regular such that it can run quickly on GPU / common hardwares. The problem essentially originates from the fact that typical computational hardware can be slow when accessing in-contiguous memory address. This paper proposes a sequential encoding-decoding shemes and can compress irregular pruned weights to a regular (block-wise) structure stored in memory and shows experimental results in large-scale models / datasets.