Post-Training Sparsity-Aware Quantization

1 Introduction . Deep neural networks ( DNNs ) are at the heart of numerous applications , such as image classification and object detection [ 8 ] , image synthesis [ 30 ] , and recommendation systems [ 7 ] . DNNs , however , require abundant computations , as , for example , billions of multiply-and-accumulate ( MAC ) operations are required to assign a 224×224 colored image from the ImageNet dataset to one of its thousand possible classes . Limited computational resources , such as those in edge devices , latency constraints , and higher input resolutions , are all catalysts for development of methods that increase the ratio between DNN execution performance to hardware area , with as minimal impact on model accuracy as possible . One common method of doing so is quantization . Quantization is commonly used to map the 32-bit floating-point ( FP32 ) activations and weights in convolutional neural networks ( CNNs ) to 8-bit integers ( INT8 ) , which is known to result in minor or no degradation in model accuracy while easing hardware implementation [ 14 ] . Going below 8 bits , however , is not trivial , as quantization noise leads to a noticeable decrease in model accuracy . Quantization-aware training ( QAT ) methods employ training for quantization , to decrease quantization noise and recoup model accuracy [ 3 , 25 , 42 ] . Nevertheless , it is not always possible to employ training , for reasons such as lack of hardware resources , time , power , energy , dataset availability , or skilled manpower . Post-training quantization ( PTQ ) methods circumvent these issues [ 1 , 5 , 6 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . PTQ methods , basically , search for the optimal tensor clipping values to minimize quantization noise [ 1 , 5 ] . They usually employ uniform quantization , since computing a dot product ( DP ) of evenly-spaced integer values can be implemented efficiently in hardware . DNN tensor distributions , however , are known to follow a bell-shaped distribution , such as Gaussian or Laplacian , i.e. , the uniform quantization that is , on one hand , hardware-friendly , may not be , on the other hand , the best choice for minimizing the noise induced by the quantization process . To solve this mismatch , to some extent , PTQ methods that break tensor distributions into different quantization regions were proposed [ 6 , 12 , 24 ] . Computing a DP comprising values from different quantizations is not trivial though , since each activation-weight multiplication result may correspond to a different scaling factor , i.e. , it will induce a multiplication by a different FP value per quantization region . In this paper , we propose sparsity-aware quantization ( SPARQ ) , which leverages the inherent and dynamic activation sparsity from granularities of entire integer 8-bit values ( vSPARQ ) , down to INT8 representation zero-value bits ( bSPARQ ) . With bSPARQ , instead of quantizing every activation to , for example , 4 bits according to a predetermined scaling factor , activations are first quantized to 8 bits and then dynamically quantized to 4 bits by choosing the most significant consecutive 4 bits while skipping leading zero bits ( Figure 1 ) . bSPARQ effectively achieves a number of quantization ranges while still enabling a practical hardware implementation . Moreover , inspired by [ 32 ] , we also leverage the entire 8-bit activation sparsity with vSPARQ , for additional mitigation of quantization noise . Instead of quantizing activation-by-activation to 4 bits , activations are quantized to 4 bits in pairs . If one activation is zero , then the other can span its bits across the first , and thereby still be represented by 8 bits to avoid additional quantization noise . If , however , both activations are non-zero , both are quantized to 4 bits by bSPARQ . We experiment with vSPARQ and bSPARQ in configurations of 4 , 3 , and 2 data bits . This paper makes the following contributions : • Sparsity-aware quantization ( SPARQ ) . We present a sparsity-aware quantization method , in which n-bit quantization takes place by picking the most significant n bits from the 8-bit value representation , while skipping leading zero-value bits . Moreover , since many activations are zero-value , we consider pairs of activations in the quantization process . If one activation is zero , the other can use the entire 2n-bit budget . We experiment with a number of bit-group selection options and activation bit-widths that demonstrates the trade-off between model accuracy and hardware overhead . • Practical hardware implementation . We implement SPARQ on top of a systolic array ( SA ) , inspired by Google TPUs , and on top of a Tensor Core ( TC ) DP unit , inspired by NVIDIA GPUs , and show that SPARQ is practical in terms of area overheads . In addition , we also discuss SPARQ implementation on top of NVIDIA Sparse TCs ( STCs ) , thus leveraging activation sparsity on top of weight sparsity . • Comprehensive evaluation . We evaluate our method on a variety of image classification models , with numerous configurations and activation bit-widths , and compare it with previous PTQ works . 2 Related Work . PTQ methods are the most relevant works that are related to this work . ACIQ [ 1 ] analytically extracts the optimal quantization clipping values from the tensors ’ distributions and uses per-channel bit-allocation and per-channel quantization of activations . LBQ [ 5 ] formulates a minimum MSE optimization problem that is then solved numerically per layer , and employs additional low-precision tensors to sensitive layers . AdaQuant [ 10 ] and AdaRound [ 21 ] optimize the common round-to-nearest rounding scheme to reduce quantization noise . BRECQ [ 16 ] analyzes the second-order error and optimizes the quantization at block granularity . Conceptually , both vSPARQ and bSPARQ can be employed on top of any of the above quantizations ( for simplicity ’ s sake , we use a simple 8b-8b min-max symmetric quantization , as we also describe in Section 5 ) . Other works , such as OLAccel [ 24 ] , PWLQ [ 6 ] , and BiScaled-DNN [ 12 ] , divide the tensor distribution into two regions . OLAccel divides the tensor distribution into a low-precision region that contains the majority of data , and a high-precision region that contains a small portion of the data ( e.g. , 3 % ) , which they define as outliers . PWLQ and BiScaled-DNN , on the other hand , divide the tensor distribution into two regions with the same bit-width . BiScaled-DNN uses different scale factors on overlapping regions and implements a ratio heuristic to set the breakpoint between the regions , whereas PWLQ picks the appropriate breakpoint via minimization of the quantization error . Interestingly , PWLQ is capable of breaking the distribution into more than two regions ; however , the authors state that from a hardware perspective , this may not be feasible . Following OLAccel , OverQ [ 41 ] leverages activation sparsity to avoid the dedicated outlier datapath used in OLAccel . In this work , we employ a simple rounding mechanism and bit-level sparsity to mitigate noise in the occasion a zero-value does not exist , and we propose a parallel implementation rather than a serial one . SySMT [ 32 ] leverages sparsity in quantization of both activations and weights to 4 bits . Their method incurs relatively high area overheads , since the quantization logic has to be scaled with the number of processing units . Moreover , SySMT incurs relatively high degradation in accuracy , since quantization to 4 bits is implemented by trimming either the 4-bit most significant bits ( MSBs ) or the 4-bit least significant bits ( LSBs ) . These two options are not optimal , since we find that , for example , with ResNet-18 and ILSVRC-2012 , 67 % of the non-zero-value activation values have at least one of the 4-bit MSBs toggled ( i.e. , equal to one ) , even though 90 % of the time , the two MSBs are not toggled . That is , the two MSBs are most likely not toggled when the 4-bit MSBs are chosen . 3 The Basic Principle of SPARQ . SPARQ comprises two orthogonal techniques : bSPARQ and vSPARQ . The former leverages zerovalue bits to trim an 8-bit value to an n-bit value ; and the latter leverages zero-value activations . Below , we describe both in detail . Throughout this work , we focus on quantizing the activations and leveraging only their sparsity , i.e. , no correlation is made with the weight values , unless otherwise stated . 3.1 bSPARQ : Leveraging Bit Sparsity . Consider an already quantized 8-bit activation , x , and quantization to 4 bits ( i.e. , n = 4 ) . bSPARQ trims the activation from 8 bits to 4 bits by inspecting the activation bits and choosing the most significant consecutive 4 bits within it , which , in practice , is achieved by searching for the first most significant toggled bit . The motivation behind bSPARQ is twofold : first , activations usually follow a bell-shaped distribution , meaning that the MSBs are usually equal to zero and , therefore , can be skipped ; and second , if the MSBs are toggled , the LSBs ’ contribution to the entire value is insignificant . For example , given the value 000110112 ( 2710 ) , the 4-bit window will be positioned at bits [ 4:1 ] ( 000110112 ) , thus achieving the approximated value 2610 . Notice that since there are five window position options , the 4-bit window is accompanied by a 3-bit identifier that corresponds to the window position—that is , how much shift-left is required on top of the four trimmed bits . In addition , to further reduce the dynamic quantization noise , we round the value within the chosen window according to the residual LSBs . bSPARQ is visually demonstrated in Figure 1 . Supporting five window options requires additional circuitry compared with , for example , three window options , since additional placement options require additional hardware support by the shift-left unit . The trade-off is , however , improved accuracy , since additional placement options introduce less quantization noise . We experiment with five , three , and two placement options , denoted as 5opt , 3opt , and 2opt , respectively . With the 3opt configuration , [ 7:4 ] , [ 5:2 ] , or [ 3:0 ] are chosen , and with the 2opt configuration , either [ 7:4 ] or [ 3:0 ] are chosen ( we leave the analysis of asymmetrical configurations for future work ) . For example , given the previous value , 000110112 , 3opt will choose bits [ 5:2 ] ( 000110112 ) , whereas 2opt will choose bits [ 7:4 ] ( 000110112 ) . Relation to piecewise linear quantization . To mitigate quantization errors , previous works suggest dividing the tensor distributions into different quantization regions , each with a scaling factor of its own [ 6 , 12 , 24 ] . In a sense , bSPARQ is somewhat similar to those . First , each activation is assigned to a quantization range according to its value ; however , we break the distributions into hardware-oriented regions of power of two . For example , for the 5opt case , the regions are [ 0 , 21 − 1 ] , [ 21 , 22 − 1 ] , and so on . As a result , values are mapped to their appropriate range by simply counting the leading zero bits . In addition , we avoid any need for preprocessing that searches for the distribution breakpoints to minimize the quantization noise . Second , each region has an individual scaling factor ; however , each region scaling factor is a product of a base scaling factor with the corresponding power of two . For example , in the 5opt configuration , the scaling factor of the decimal number 3310 = 001000012 is the original scaling factor times 22 . This enables a relatively simple implementation with up to five regions when considering 4-bit activations , and even six and seven regions when considering 3- and 2-bit activations , respectively—as opposed to the two quantization regions used by previous works .

Summary: The authors propose a post-training quantization scheme comprising two orthogonal ideas: bSPARQ and vSPARQ.  bSPARQ a dynamic quantization technique. Instead of rounding values to the required bitwidth (equivalent to keeping the MSBs), the technique keeps the most significant consecutive non-zero bits (i.e. the sequence of bits starting from the first non-zero bit). This is equivalent to making a power-of-two adjustment to the scale factor for each individual value.  vSPARQ is a dynamic sparsity technique meant for activations. First, adjacent pairs of activatons are grouped (e.g. [x1, x2, x3, x4] -> [x1x2, x3x4]). In each pair, if one of the two activations is zero, the other can make use of its compute resources. The idea is detailed in Equation (2) and can be realized in hardware with the architecture in Figure 2.  The authors demonstrate SOTA accuracy results on CNN image classification for very low activation bitwidths (4/3/2-bit activations and 8-bit weights). However, the technique also incurs significant area overhead: 22% for the 3opt design point.