Post-Training Sparsity-Aware Quantization

1 Introduction . Deep neural networks ( DNNs ) are at the heart of numerous applications , such as image classification and object detection [ 8 ] , image synthesis [ 30 ] , and recommendation systems [ 7 ] . DNNs , however , require abundant computations , as , for example , billions of multiply-and-accumulate ( MAC ) operations are required to assign a 224×224 colored image from the ImageNet dataset to one of its thousand possible classes . Limited computational resources , such as those in edge devices , latency constraints , and higher input resolutions , are all catalysts for development of methods that increase the ratio between DNN execution performance to hardware area , with as minimal impact on model accuracy as possible . One common method of doing so is quantization . Quantization is commonly used to map the 32-bit floating-point ( FP32 ) activations and weights in convolutional neural networks ( CNNs ) to 8-bit integers ( INT8 ) , which is known to result in minor or no degradation in model accuracy while easing hardware implementation [ 14 ] . Going below 8 bits , however , is not trivial , as quantization noise leads to a noticeable decrease in model accuracy . Quantization-aware training ( QAT ) methods employ training for quantization , to decrease quantization noise and recoup model accuracy [ 3 , 25 , 42 ] . Nevertheless , it is not always possible to employ training , for reasons such as lack of hardware resources , time , power , energy , dataset availability , or skilled manpower . Post-training quantization ( PTQ ) methods circumvent these issues [ 1 , 5 , 6 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . PTQ methods , basically , search for the optimal tensor clipping values to minimize quantization noise [ 1 , 5 ] . They usually employ uniform quantization , since computing a dot product ( DP ) of evenly-spaced integer values can be implemented efficiently in hardware . DNN tensor distributions , however , are known to follow a bell-shaped distribution , such as Gaussian or Laplacian , i.e. , the uniform quantization that is , on one hand , hardware-friendly , may not be , on the other hand , the best choice for minimizing the noise induced by the quantization process . To solve this mismatch , to some extent , PTQ methods that break tensor distributions into different quantization regions were proposed [ 6 , 12 , 24 ] . Computing a DP comprising values from different quantizations is not trivial though , since each activation-weight multiplication result may correspond to a different scaling factor , i.e. , it will induce a multiplication by a different FP value per quantization region . In this paper , we propose sparsity-aware quantization ( SPARQ ) , which leverages the inherent and dynamic activation sparsity from granularities of entire integer 8-bit values ( vSPARQ ) , down to INT8 representation zero-value bits ( bSPARQ ) . With bSPARQ , instead of quantizing every activation to , for example , 4 bits according to a predetermined scaling factor , activations are first quantized to 8 bits and then dynamically quantized to 4 bits by choosing the most significant consecutive 4 bits while skipping leading zero bits ( Figure 1 ) . bSPARQ effectively achieves a number of quantization ranges while still enabling a practical hardware implementation . Moreover , inspired by [ 32 ] , we also leverage the entire 8-bit activation sparsity with vSPARQ , for additional mitigation of quantization noise . Instead of quantizing activation-by-activation to 4 bits , activations are quantized to 4 bits in pairs . If one activation is zero , then the other can span its bits across the first , and thereby still be represented by 8 bits to avoid additional quantization noise . If , however , both activations are non-zero , both are quantized to 4 bits by bSPARQ . We experiment with vSPARQ and bSPARQ in configurations of 4 , 3 , and 2 data bits . This paper makes the following contributions : • Sparsity-aware quantization ( SPARQ ) . We present a sparsity-aware quantization method , in which n-bit quantization takes place by picking the most significant n bits from the 8-bit value representation , while skipping leading zero-value bits . Moreover , since many activations are zero-value , we consider pairs of activations in the quantization process . If one activation is zero , the other can use the entire 2n-bit budget . We experiment with a number of bit-group selection options and activation bit-widths that demonstrates the trade-off between model accuracy and hardware overhead . • Practical hardware implementation . We implement SPARQ on top of a systolic array ( SA ) , inspired by Google TPUs , and on top of a Tensor Core ( TC ) DP unit , inspired by NVIDIA GPUs , and show that SPARQ is practical in terms of area overheads . In addition , we also discuss SPARQ implementation on top of NVIDIA Sparse TCs ( STCs ) , thus leveraging activation sparsity on top of weight sparsity . • Comprehensive evaluation . We evaluate our method on a variety of image classification models , with numerous configurations and activation bit-widths , and compare it with previous PTQ works . 2 Related Work . PTQ methods are the most relevant works that are related to this work . ACIQ [ 1 ] analytically extracts the optimal quantization clipping values from the tensors ’ distributions and uses per-channel bit-allocation and per-channel quantization of activations . LBQ [ 5 ] formulates a minimum MSE optimization problem that is then solved numerically per layer , and employs additional low-precision tensors to sensitive layers . AdaQuant [ 10 ] and AdaRound [ 21 ] optimize the common round-to-nearest rounding scheme to reduce quantization noise . BRECQ [ 16 ] analyzes the second-order error and optimizes the quantization at block granularity . Conceptually , both vSPARQ and bSPARQ can be employed on top of any of the above quantizations ( for simplicity ’ s sake , we use a simple 8b-8b min-max symmetric quantization , as we also describe in Section 5 ) . Other works , such as OLAccel [ 24 ] , PWLQ [ 6 ] , and BiScaled-DNN [ 12 ] , divide the tensor distribution into two regions . OLAccel divides the tensor distribution into a low-precision region that contains the majority of data , and a high-precision region that contains a small portion of the data ( e.g. , 3 % ) , which they define as outliers . PWLQ and BiScaled-DNN , on the other hand , divide the tensor distribution into two regions with the same bit-width . BiScaled-DNN uses different scale factors on overlapping regions and implements a ratio heuristic to set the breakpoint between the regions , whereas PWLQ picks the appropriate breakpoint via minimization of the quantization error . Interestingly , PWLQ is capable of breaking the distribution into more than two regions ; however , the authors state that from a hardware perspective , this may not be feasible . Following OLAccel , OverQ [ 41 ] leverages activation sparsity to avoid the dedicated outlier datapath used in OLAccel . In this work , we employ a simple rounding mechanism and bit-level sparsity to mitigate noise in the occasion a zero-value does not exist , and we propose a parallel implementation rather than a serial one . SySMT [ 32 ] leverages sparsity in quantization of both activations and weights to 4 bits . Their method incurs relatively high area overheads , since the quantization logic has to be scaled with the number of processing units . Moreover , SySMT incurs relatively high degradation in accuracy , since quantization to 4 bits is implemented by trimming either the 4-bit most significant bits ( MSBs ) or the 4-bit least significant bits ( LSBs ) . These two options are not optimal , since we find that , for example , with ResNet-18 and ILSVRC-2012 , 67 % of the non-zero-value activation values have at least one of the 4-bit MSBs toggled ( i.e. , equal to one ) , even though 90 % of the time , the two MSBs are not toggled . That is , the two MSBs are most likely not toggled when the 4-bit MSBs are chosen . 3 The Basic Principle of SPARQ . SPARQ comprises two orthogonal techniques : bSPARQ and vSPARQ . The former leverages zerovalue bits to trim an 8-bit value to an n-bit value ; and the latter leverages zero-value activations . Below , we describe both in detail . Throughout this work , we focus on quantizing the activations and leveraging only their sparsity , i.e. , no correlation is made with the weight values , unless otherwise stated . 3.1 bSPARQ : Leveraging Bit Sparsity . Consider an already quantized 8-bit activation , x , and quantization to 4 bits ( i.e. , n = 4 ) . bSPARQ trims the activation from 8 bits to 4 bits by inspecting the activation bits and choosing the most significant consecutive 4 bits within it , which , in practice , is achieved by searching for the first most significant toggled bit . The motivation behind bSPARQ is twofold : first , activations usually follow a bell-shaped distribution , meaning that the MSBs are usually equal to zero and , therefore , can be skipped ; and second , if the MSBs are toggled , the LSBs ’ contribution to the entire value is insignificant . For example , given the value 000110112 ( 2710 ) , the 4-bit window will be positioned at bits [ 4:1 ] ( 000110112 ) , thus achieving the approximated value 2610 . Notice that since there are five window position options , the 4-bit window is accompanied by a 3-bit identifier that corresponds to the window position—that is , how much shift-left is required on top of the four trimmed bits . In addition , to further reduce the dynamic quantization noise , we round the value within the chosen window according to the residual LSBs . bSPARQ is visually demonstrated in Figure 1 . Supporting five window options requires additional circuitry compared with , for example , three window options , since additional placement options require additional hardware support by the shift-left unit . The trade-off is , however , improved accuracy , since additional placement options introduce less quantization noise . We experiment with five , three , and two placement options , denoted as 5opt , 3opt , and 2opt , respectively . With the 3opt configuration , [ 7:4 ] , [ 5:2 ] , or [ 3:0 ] are chosen , and with the 2opt configuration , either [ 7:4 ] or [ 3:0 ] are chosen ( we leave the analysis of asymmetrical configurations for future work ) . For example , given the previous value , 000110112 , 3opt will choose bits [ 5:2 ] ( 000110112 ) , whereas 2opt will choose bits [ 7:4 ] ( 000110112 ) . Relation to piecewise linear quantization . To mitigate quantization errors , previous works suggest dividing the tensor distributions into different quantization regions , each with a scaling factor of its own [ 6 , 12 , 24 ] . In a sense , bSPARQ is somewhat similar to those . First , each activation is assigned to a quantization range according to its value ; however , we break the distributions into hardware-oriented regions of power of two . For example , for the 5opt case , the regions are [ 0 , 21 − 1 ] , [ 21 , 22 − 1 ] , and so on . As a result , values are mapped to their appropriate range by simply counting the leading zero bits . In addition , we avoid any need for preprocessing that searches for the distribution breakpoints to minimize the quantization noise . Second , each region has an individual scaling factor ; however , each region scaling factor is a product of a base scaling factor with the corresponding power of two . For example , in the 5opt configuration , the scaling factor of the decimal number 3310 = 001000012 is the original scaling factor times 22 . This enables a relatively simple implementation with up to five regions when considering 4-bit activations , and even six and seven regions when considering 3- and 2-bit activations , respectively—as opposed to the two quantization regions used by previous works .

Summary: This paper proposes a post-training sparsity-aware quantization algorithm, SPARQ, for neural networks. SPARQ leverages the bitwise sparsity by skipping leading zero-value during quantization. SPARQ also leverages the elementwise sparsity by dynamically increase the quantization precision if one in the activation pair is zero. This paper also presents a practical hardware implementation for supporting the SPARQ. The experiments show a 0.18% accuracy drop in 4-bit quantized ResNet50 on ImageNet.