Post-Training Sparsity-Aware Quantization
Authors: Gil Shomron
Freddy Gabbay
Samer Kurzum
Uri Weiser
Venue: NeurIPS 2021 Poster (2021)
Keywords: quantization
post-training
hardware
systolic array
tensor core
performance
sparsity

Abstract: Quantization is a technique used in deep neural networks (DNNs) to increase execution performance and hardware efficiency. Uniform post-training quantization (PTQ) methods are common, since they can be implemented efficiently in hardware and do not require extensive hardware resources or a training set. Mapping FP32 models to INT8 using uniform PTQ yields models with negligible accuracy degradation; however, reducing precision below 8 bits with PTQ is challenging, as accuracy degradation becomes noticeable, due to the increase in quantization noise. In this paper, we propose a sparsity-aware quantization (SPARQ) method, in which the unstructured and dynamic activation sparsity is leveraged in different representation granularities. 4-bit quantization, for example, is employed by dynamically examining the bits of 8-bit values and choosing a window of 4 bits, while first skipping zero-value bits. Moreover, instead of quantizing activation-by-activation to 4 bits, we focus on pairs of 8-bit activations and examine whether one of the two is equal to zero. If one is equal to zero, the second can opportunistically use the other's 4-bit budget; if both do not equal zero, then each is dynamically quantized to 4 bits, as described. SPARQ achieves minor accuracy degradation and a practical hardware implementation.

Sparsity-Aware Quantization
Gil Shomron
Freddy Gabbay
Samer Kurzum
Uri Weiser
Technion - Israel Institute Technology, Haifa, Israel
Ruppin Academic Center, Emek Hefer, Israel
{gilsho@campue, ssamer15@campus, wri.weisereee.... wwwhnion.cc........
freddygōruppin.ac.o
Abstract
Quantization technique used in deep neural networks (DNNs) increase cution performance and hardware efficiency. Uniform post-training quantization (PTQ) methods common, since they be implemented efficiently in hard- ware and do not require extensive hardware resources or a training set. Mapping FP32 models to INT8 using uniform PTQ yields models with negligible accuracy degradation; however, reducing precision below 8 bits with PTQ is challenging, as accuracy degradation becomes noticeable, due the increase in quantization noise. In this paper. propose sparsity- aware quantization (SPARQ) method, in which the unstructured and dynamic activation sparsity is leveraged in differ- ent representation granularities. 4-bit quantization, for example, is employed by dynamically examining the bits of 8-bit values and choosing a window of 4 bits, while first skipping zero-value bits. Moreover, instead of quantizing activation-by. activation to 4 bits, we focus on pairs of 8-bit activations and examine whether one of the two is equal to zero. If one is equal to zero, the second can oppor- tunistically use the other's 4-bit budget; if both not equal zero, then each dynamically quantized to 4 bits, described. SPARQ achieves minor accuracy degradation and a practical hardware implementation. The code available at https://githin com/ htte/glsshie ggilshm/sst /sparq
Introduction
Deep neural networks (DNNs) are at the heart of numerous applications, such as image classification and object detection (8. image synthesis [30], and recommendation systems (7]. DNNs, however, require abundant computations, as, for example, billions of multppy--ad--acumull erations are required to assign 224 224 colored image from the ImageNet dataset to one of its thousand possible classes. Limited computational resources, such as those in edge devices, latency constraints, and higher input resolutions, are all catalysts for development of methods that increase the ratio between DNN execution performance to hardware area, with as minimal impact on model accuracy as possible. One common method of doing so is quantization.
(MAC) op-
Quantization commonly used to map the 32-bit doating-poim (FP32) activations and weights in convolutional neural networks (CNNs) to 8-bit integers (INT8), which is known to result in minor or no degradation in model accuracy while easing hardware implementation [14] Going below 8 bits. however, is not trivial, as quantization noise leads to noticeable decrease in model accuracy. Quantization-awa
training (QAT) methods employ training for quantization, to decrease quantization noise and recoup model accuracy to employ training, for reasons such as lack of hardware resources, time, power, energy, dataset availability, skilled manpower. Post-training quantization (PTQ) methods circumvent these issues 5 6.
3 25 42 Nevertheless, it is not always possible
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

PTQ methods, basically, search for the optimal tensor clipping values to minimize quantization noise 5.. They usually employ uniform quantization, since computing dot product (DP) of evenly-spaced integer values can be implemented efficiently in hardware. DNN tensor distributions, however, are known to follow a bell-shaped distribution, such as Gaussian or Laplacian, i.e., the uniform quantization that is, on one hand, hardware- fardeare-friendly, may not be, on the other hand, the best choice for minimizing the noise induced by the quantization process. To solve this mismatch, to some extent, PTQ methods that break tensor distributions into different quantization regions were proposed Computing a DP comprising values from different quantizations is not trivial though, since each activation-wett weight multiplication result may correspond to different scaling factor, i.e., it will induce a multiplication by a different FP value per quantization region.
12
In this paper, we propose sparsity aware quantization (SPARQ), which leverages the inherent and dynamic activation sparsity from granularities of entire integer 8-bit values (ySPARQ), down to INT8 representation zero-value bits (bSPARQ). With bSPARQ, instead of quantizing every activation to, for example, 4 bits according to a predetermined scaling factor, activations are first quantized 8 bits and then dynamically quantized to 4 bits by choosing the most significant consecutive 4 bits while skipping leading zero bits (Figure D. bSPARQ effectively achieves a number of quantization ranges while still enabling practical hardware implementation.
Moreover, inspired by [32], we also leverage the entire 8-bit activation sparsity with vSPARQ for additional mitigation of quantization noise. Instead of quantizing activation cetivation-by-actiaatte by-activation to 4 bits, activations are quantized to 4 bits in pairs. If one activation zero, then the other can span its bits across the first, and thereby still be represented by 8 bits to avoid additional quantization noise. If, however, both activations are non-zero, both are quantized to 4 bits by bSPARQ. We experiment with vSPARQ and bSPARQ in configurations of 4, 3, and 2 data bits.
This paper makes the following contributions:
Sparsity-aware quantization (SPARQ). We present sparsity- awarr quantization method, in which n-bit quantization takes place by picking the most significant n bits from the 8-bit value representation, while skipping leading zero-value bits. Moreover, since many activations are zero-value, we consider pairs of activations the quantization process. If one activation is zero, the other can use the entire 2n-bit budget. We experiment with a number of bit-group selection options and activation bit-widths that demonstrates the trade-off between model accuracy and hardware overhead.
Practical hardware implementation. We implement SPARQ on top of a systolic array (SA), inspired by Google TPUs. and on top of a Tensor Core (TC) DP unit, inspired by NVIDIA GPUs, and show that SPARQ is practical in terms of area overheads. In addition, we also discuss SPARQ implementation on top of NVIDIA Sparse TCs (STCs), thus leveraging activation sparsity top of weight sparsity.
Comprehensive evaluation. We evaluate our method on variety of image classification models, with numerous configurations and activation bit-widths, and compare with previous PTQ works.
2 Related Work
PTQ methods are the most relevant works that are related to this work. ACIQ [1] analytically extracts the optimal quantization clipping values from the tensors' distributions and uses per-channel bit-allocation and per-channel quantization of activations. LBQ [5] formulates minimum MSE optimization problem that is then solved numerically per layer, and employs additional low- tensors to sensitive layers. AdaQuant [10] and AdaRound [21] optimize the common lowpicccsioon rounding scheme to reduce quantization noise. BRECQ [6] analyzes the second-order error and optimizes the quantization at block granularity. Conceptually, both vSPARQ and bSPARQ can be employed on top of any of the above quantizations (for simplicity's 's sake, we use a simple 8b-8b min-max symmetric quantization, as we also describe in Section 5)..
Other works, such as OLAccel [24), PWLQ (6) and BiScaled-DNN 12], divide the tensor distribution into two regions. OLAccel divides the tensor distribution into low-precision region that contains the majority of data, and a high-precision region that contains a small portion of the data (e.g.. 3%), which they define as outliers. PWLQ and BiScaled-DNN. on the other hand, divide the tensor distribution

---

into two regions with the same wit-width. BiScaled-DNi uses different scale factors on overlapping regions and implements a ratio heuristic set the breakpoint between the regions, whereas PWLQ picks the appropriate breakpoint via minimization of the quantization error. Interestingly, PWI.Q is capable of breaking the distribution into more than two regions; however, the authors state that from a hardware perspective, this may not be feasible.
Following OLAccel, OverQ [4 leverages activation sparsity to avoid the dedicated outlier datapath used in OLAccel. In this work, we employ a simple rounding mechanism and bit-level sparsity to mitigate noise in the occasion a zero-value does not exist, and we propose parallel implementation rather than a serial one.
SySMT 32 leverages sparsity in quantization of both activations and weights to 4 bits. Their method incurs relatively high area overheads, since the quantization logic has to be scaled with the number of processing units. Moreover, SySMT incurs relatively high degradation in accuracy, since quantization to 4 bits is implemented by trimming either the 4-bit most significant bits (MSBs) or the 4-bit least significant bits (LSBs). These two options are not optimal, since we find that, for example, with ResNet-18 18 and ILSVRC-2012, 67% of the non-zero-value activation values have at least one of the 4-bit MSBs toggled (i.e., equal to one), even though 90% of the time, the two MSBs are not toggled. That is, the two MSBs are most likely not toggled when the 4-bit MSBs are chosen.
3 The Basic Principle of SPARQ
SPARQ comprises two orthogonal techniques: bSPARQ and vSPARQ. The former leverages zero- value bits trim an 8-bit value to an n-bit value; and the latter leverages zero-value activations. Below, we describe both in detail. Throughout this work, we focus on quantizing the activations and leveraging only their sparsity, i.e., no correlation is made with the weight values, unless otherwise stated.
3.1 BSPARQ: Leveraging Bit Sparsity
Consider an already quantized 8-bit activation, x, and quantization to 4 bits (i.e., 4). BSPARQ trims the activation from 8 bits to 4 bits by inspecting the activation bits and choosing the most significant consecutive bits within it, which, in practice, achieved by searching for the first most significant toggled bit. The motivation behind bSPARQ is twofold: first, activations usually follow bell-shaped distribution, meaning that the MSBs are usually equal to zero and, therefore, can be skipped; and second, if the MSBs are toggled, the LSBs' contribution to the entire value is insignificant. For example, given the value 0001 1011, (2),)), the 4-bit window will be positioned at bits [4:1] (000110112), thus achieving the approximated value 2610- Notice that since there are five window position options, the 4-bit window accompanied by a 3-bit identifier that corresponds to the window position that is, how much shift-left is required top of the four trimmed bits. In addition, to further reduce the dynamic quantization noise, we round the value within the chosen window according to the residual LSBs. BSPARQ is visually demonstrated in Figure D1
Supporting five window options requires additional circuitry compared with, for example, three window options, since additional placement options require additional hardware support by the shift-left unit. The trade-off is. however, improved accuracy, since additional placement options introduce less quantization noise. We experiment with five, three, and two placement options, denoted as Sopt, 3opt, and 2opt, respectively. With the 3opt configuration, [7:4], [5:2], or [3:0] are chosen, and with the 2opt configuration, either [7:4] or [3:0] are chosen (we leave the analysis of asymmetrical configurations for future work). For example, given the previous value, 00011011g, 3opt will choose bits [5:2] (000110112), whereas 20pt will choose bits [7:4] (000110112).
Relation to piecewise linear quantization. To mitigate quantization errors, previous works suggest dividing the tensor distributions into different quantization regions, each with a scaling factor of its own 6 12 [24]. In sense, bSPARQ somewhat similar to those. First, each activation assigned to a quantization range according to its value; however, we break the distributions into hurdware-orient
regions of power of two. For example, for the 5opt case, the regions are [0, 2 - 1]. [2], 2² - 1). and so on. As result, values are mapped to their appropriate range by simply counting the leading zero bits. In addition, we avoid any need for preprocessing that searches for the distribution breakpoints to minimize the quantization noise. Second, each region has individual scaling factor; however, each
3

---

Options
ption
Options
[7:4]
[6:3]
[5:2]
[5:2]
[4:1]
[3:0]
[3:0]
[3:0]
0
0
0
Example
Example
+Round
0
0 0
0 0
5opt
30pt
(c) 2opt
Figure 1: Demonstration of SPARQ 8b-to-4b quantization. More window placement options (e.g., Sopt) decrease the quantization noise; however, additional hardware is needed to support many placement options.
region scaling factor is product of a base scaling factor with the corresponding power of two. For example, in the Sopt configuration, the scaling factor of the decimal number 3310 00100001, is the original scaling factor times 22. This enables a relatively simple implementation with up to five regions when considering 4-bit activations, and even six and seven regions when considering and 2-bit activations, respectively-as opposed to the two quantization regions used by previous works.
3.2 vSPARQ: Leveraging Sparsity with Pairs of Activations
Consider an 8-bit unsigned activation vector, X (I,, LL), and an 8-bit signed weight vector, W = (Wi..... .. wr), both of length L. Also, consider a single MAC unit that computes a single activation-weight multiplication per cycle. vSPARQ, similar to [3 34 41, groups activations in pairs, to leverage the dynamic and unstructured activation sparsity. That is, the DP calculations can be formulated as:
X
Σ T200 + ceven
y
(1)
where y is the DP scalar result, and in our context, an output activation. For some i, if x; = 0, then 1 can be used with 8-bit representation, and vice versa. If, however, both x; A and 0;+1 A and given that, for example, bSPARQ is employed, then the precision of both æ; and #++1 is reduced bits. For certain the vSPARQ operation can also be formulated as:
K;WW;;
a+++ = 0 if x = otherwise
T2W0 + T4410/11
1141W;11+
(2)
bSPARQ(2:)us bSPARQ(
Notice that the first statements correspond 8b-8b computation, whereas the last statement corresponds to two 4b-8b computations. The latter case is possible, since two 4b-8b single 8b-8b multiplication, as we describe next.
multiplications logically equivalent
8b-8b = 2x4b-8h, Given an 8-bit unsigned activation, x, and an 8-bit signed weight, w, the activation- weight multiplication can be formulated as
Σ
2' Wp:0
Σ 2 2441
Σ
2'
I[7:0] W
(3)
2 1(7-4) W(7:0
+ T[3.0] W[7:0]
where the [b a) notation represents the b-to-a range bits, the two soavaationweigght multiplications 4b-8b wide, and the 24 equivalent 4-bit shift-left operation.
By considering an additional weight input as well as dynamic shift-left operations, we can reuse the multipliers and achieve multiplier capable of either 8b-8b multiplication two independent

---

Rucy
Activation Buffer
4b-8b Multiplier
4b-8b
Multiplier
PE
PE
PE
8b-8b Mulliplier
Weight Buffer
PE
PE
P Partial Sum
PE
PE
Figure 2: Equation
hardware
Figure 3: Illustration of conventional 8b-8b output stationary systolic array.
implementation.
4b-8b multiplications with dynamic range:
29h Tinl,4b Win1,86
200 2m2244 Win2.8b
where the activation and weight inputs are 4 bits and 8 bits long, respectively. Equation resembles a FP representation; however, the "opt" configurations are not necessarily continuous, as in 3opt and 2opt. Figure Tigurr@iilustrattes iillustrates how Equation (4) is mapped to hardware. The two 4b-8b multipliers correspond to Timl Winl and Tin2 Winz, and the two shift-left units (<<) correspond to 29h and 20H², The adder corresponds to the addition of the two groups, and the multiplexers, which are not explicitly formulated in Equation {4) are used to choose dynamically between Wint, Win2, or select both, during execution. We this multiplier instead of the conventional used in well-known hardware structures.
4 Case Studies
In this section, we examine SPARQ on top of two well-known matrix multiplication accelerator implementations: systolic arrays (SAs) and Tensor Cores (TCs). These accelerators commonly used for CNNs, since is standard practice to map the convolution operation to matrix multiplication 2 [8 39]. Our focus here is the processing engines (PEs) comprising each of these structures and that are responsible for single DPs. Both implementations are fully equivalent from a mathematical point of view.
Systolic arrays. SAs consist of a large monolithic network of PEs designed for fast and efficient processing of systematic algorithms that execute the same computations with different data at different time instances [15). The topology of SAs, illustrated in Figure 3 consists
homogeneous network of tightly coupled PEs, each performing MAC operation. PEs work in tandem: each PE in the SA receives data from its upstream neighbors, performs a MAC operation, and forwards the data downstream. In our PE design, also known output-statie stationart SA, each PE will eventually hold the result of a DP; and the entire SA will comprise a tile from a result matrix. Google's 's TPUv2 and TPUv3, for example, consist of 128x x 128 SA arrays [22]. To deploy SPARQ, the conventional multiplier in each PE is replaced with the one presented in Figure 2 the weight bandwidth is doubled, and the activation bandwidth does not change.
Tensor cores. TCs were first introduced in NVIDIA's Volta architecture to accelerate matrix oper- ations 4 13 19 TCs multiply two 4x4 matrices and add additional one to the multiplication result. The specific implementation details of TCs are not publicly disclosed; however, a proposed architecture that fits the original TC performance is suggested in 27 In the proposed TC architecture, there are a number of DP units. Each DP unit performs four parallel accumulating them in an adder tree together with an additional third value. In this work, focus on the architecture of a single DP, as presented in Figure 4 To enable SPARQ, the multipliers are replaced and the weight bandwidth is doubled, similar the SA.
ceivattionweighhe multiplications,
NVIDIA also recently introduced weight sparsity acceleration in its Ampere microarchitecture 20 23
The Sparse TC (STC) hardware achieves 2x speedup over the original TC by essentially
5

---

Activations
8b-8b Multipier
8b-8b Multiplier
8b-8b Multiplier
8b-8b Multiplier
MUX 4:2
MUX 4:2
Weights
2d
Idx
48
DP Unit
Figure 4: Illustration of a conventional 8b-8b DP unit comprising TC 27
Figure 5:
Conventional STC microarchitec-
ture 23
skipping 50% of the computations (Figure 5). STC requires 50% weight structured pruning at granularity of four elements, i... every four adjacent weights must have two zero-value weights. Only the non-zero-value weights are stored with additional coordinates. In Figure 5 the two leftmost weights and two rightmost weights correspond to the four leftmost activations and rightmost activations, respectively. The stored coordinates indicate which activations are picked, since they to be multiplied by non-zero-value weights. After filtering the activations, they are passed with the weights to the DP unit for further processing. Notice, however, that activation sparsity may still exist even after the selection process.
Experiments
We evaluate the impact on model accuracy using PyTorch (26). the ILSVRC-2012 dataset 28 and various CNN models 88 9 [1 37 37 38) (see Table I). All models are quantized using a simple uniform min-max quantization, employing symmetric unsigned per-layer quantization for activations and symmetric signed per-kernel quantization for weights. The min-max statistics are gathered during a quick preprocessing stage on 2K randomly picked images from the training set. In addition, during preprocessing, we recalibrate the BatchNorm layers' running mean and running variance statistics In all models, the first convolution layer is left intact, since its input activations, which correspond to the image pixels, do not include many zero values, if any. Quantization is, therefore, performed on all convolution layers, with the exception of the first layer. We present the quantization results in T. Throughout this section, we use SPARQ top of the 8-bit models (A8W8) and report the accuracy degradation relative to the corresponding FP32 model. A4W8 and A8W4 presented in Table references to the worse-case accuracy.
29
33 35 36).
Table ILSVRC-2012 CNN top-1 accuracies, given different quantization precisions. Throughout this work, SPARQ is used on top of the A8W8 representation.
Model
FP32
A8W8
A4W8
A8W4
ResNet-18
69.76%
69.80%
67.70%
67.49%
ResNet-34
73.31
73.39%
71.47%
72.01%
ResNet-50
76.13%
76.22%
72.79%
75.03%
ResNet-101
77.37%
77.38%
73.74%
76.41%
GoogLeNet
69.78%
69.67%
65.38%
65.81%
Inception-v3
77.49%
77.50%
73.91%
74.22%
DenseNet-121 74.69%
74.68%
72.57%
72.89%
SqueezeNet 58.09%
57.81%
28.12%
34.14%
In Section 5.3
we experiment with a 2:4 structured pruning 23 To achieve the sparse model with
the baseline accuracy, we prune the network based on its pretrained weights and retrain the model from scratch for epochs with learning rate starting from 0.1 and divided by 10 at epochs and 60. Weight decay and momentum set to 0.0001 and 0.9, respectively.

---

The different designs are implemented using System Verilog and synthesized using Synopsys Compiler and Virage (now Synopsys) 65nm standard cell library. We use a frequency of 500MHz at slow and fast corners for setup and hold timing closure, respectively. Area estimates were extracted after place-and-route using C InnovusTM, We assume that the overall hardware overhead related to activation trimming and rounding is relatively negligible with respect to the SA and TC, since (1) the trimming and rounding unit involves a simple hardware scheme; and (2) is performed at significantly lower processing rate. We validated our multiplier against our PyTorch CUDA implementation with cycle-accurate testbenches to verify calculation integrity.
@ Design
5.1 Accuracy Results
In Table 2 we present our method's results for the Sopt, 30pt, and 2opt configurations, with and
without rounding (=R), as described in Section [.1, and without vSPARQ (-vS). ). As expected, we observe that (1) better accuracy is achieved with the increase of window placement options; (2) overall, rounding further reduces quantization noise, which leads to smaller accuracy degradation: and (3) vSPARQ contribution is noticeable mainly in configurations with relatively high quantization noise. In addition, we observe a large impact on accuracy in the transition from 2opt to 3opt, since there high probability that least one of the 4-bit MSBs will be toggled. For example, given the non-zero- non-zero-valued activations in ResNet- 18 with the ILSVRC-2012 dataset, we measure that bits 7, 6, 5, and 4 are toggled 0.5%, 9.2%, 33.8%, and 44.8% of the time, respectively. Assuming the bit values are statistically independent, the probability of at least one toggled bit is 67%. Notice that there is clear redundancy in the 2opt configuration that picks the 4-bit MSBs, even though 10% of the time the two MSBs are toggled.
Table 2: SPARQ accuracy results using the ILSVRC-2012 dataset, without rounding (-R), with rounding (+R), and with rounding but without vSPARQ (+R-vS).
5opt +R
3opt +R
2opt +R
Model
Trim
+R-bS
Trim
+R-hS
Trim
+R-bS
ResNct-18
-0.11% 11%
-0.07%
-0.11%
-0.22%
-0.14%
-0.48%
-2.87%
-1.37%
-2.02%
ResNet-34
-0.00%
+0.04%
-0.05%
-0.25%
-0.14%
-0.25%
-2.38%
-1.10%
-1.75%
ResNet-50
-0.03%
-0.05%
-0.02%
-0.41%
-0.18%
-0.31%
-4.18%
-2.18%
-2.83%
ResNet-1110
-0.22%
-0.25%
-0.19%
-0.67%
-0.59%
-0.60%
-3.31%
-1.64%
-2.82%
GoogLeNet
-0.83%
-0.68%
-0.77%
-1.59%
-0.75%
-0.99%
-5.14%
-2.55%
-4.11%
Inception-v3
-0.73%
-0.62%
-0.95%
-1.51%
-1,21%
-1.68%
-3.98%
-1.86%
-3.30%
DenseNet-121
+0.10%
+0.09%
+0.05%
-0.16%
+0.05%
-0.02%
-2.39%
-0.57%
-1.10%
SqueezeNet -1.63%
-0.80%
-0.90%
-3.73%
-1.05%
-1.26%
-54.5%
-8.24%
-11.6%
Computationally, SPARQ may be considered as dynamic 4b-8b PTQ, in which quantization to 4 bits from 8 bits conducted occasionally in the event of two adjacent non-zero-valu activations. The upside of conventional PTQ methods, however, is the reduction in memory footprint, where the dynamic method falls short, due the additional metadata. For example, the 3opt configuration requires additional 3-bit metadata per 4-bit activation data (2-bit ShiftCtrl and 1-bit MuxCtrl). Still, the memory footprint may be reduced by grouping the metadata for several activations, which we leave for future exploration. In Table 33] we present our results compared with previous related works 5 6, 31). We would like to point out that SySMT is similar to the 2opt configuration. The slightly different results are due to the different BatchNorm calibrations and the slightly different 8-bit quantized models. Regarding ResNet-50, SySMT quantizes its weights, whereas SPARQ focuses on quantizing activations.
Reducing the bit width: bits and 2 bits. To further challenge SPARQ efficiency, we experiment with 3-bit and 2-bit configurations. The lower budget leads to increased quantization noise even when one of the activations within the activation pair has a zero value, since the total window sizes are and bits for the 3-bit and 2-bit configurations, respectively. In Table results compared with other methods that reported sub-4b quantization results. As opposed to Table we observe that v vPARQ impact is more significant in lower bit-widths.
4 we present SPARQ accuracy
7

---

Table Relative top-1 accuracy degradation (relative FP32) of SPARQ versus different quantiza- tion methods used for 4b-88 8b quantization (the best out of 4-bit activations or weights).
SPARQ 3opt
Model
5opt
2opt
SySMT
PWLQ
ACIQ
LBQ
KURE
ResNet-18 18
-0.07%
-0.14%
-1.37%
-1.29%
-2.01%
-1.20%
-2.44%
ResNet-34
+0.04%
-0.14%
-1.10%
ResNet-50
-0.03%
-0.18%
-2.18%
-0.43%
-0.67%
-1.05%
-1.66%
-0.92%
ResNet-101
-0.22%
-0.59%
-1.64%
-0.52%
-118%
GoogLeNet
-0.68%
-0.75%
-2.55%
-2.5%%
Inception-v3
-0.62%
-1.21%
-1.86%
-1.44
-2.72%
-1.8%
DenseNet-122
+0.10%
+0.05%
-0.57%
-0.39%
-.17%
SqueezeNet -0.80%
-1.05%
-8.24%
-2.66%
Table 4: Relative top-1 accuracy degradation (relative to FP32) for 3-bit and 2-bit SPARQ (with 8-bit weights) in 60pt and 7opt configurations, respectively, also with and without vSPARQ (-vS).
SPARQ
KURE
ACIQ
Model
3b
2b
3b (-vS)
2b (-vS)
3b
2b
3b
ResNet-18
-0.21%
-1.64%
-0.51%
-2.57%
-10.9%
-42.8%
-17.1%
ResNet-34
-0.18%
-1.19%
-0.37%
-1.66%
ResNet-50
-0.59%
-2.34%
-0.73%
-3.53%
-3.53%
-15.9%
-1.4%
ResNet-101
-0.66%
-2.64%
-1.06%
-3.73%
-6.08%
GoogLeNet
-1.32%
-6.47%
-1.91%
-9.16%
Inception-v3
-1.70%
-5.60%
-2.45%
-9.29%
-26.4%
DenseNet- 121
-0.07%
-0.86%
-0.25%
-1.73%
SqueezeNet
-1.63%
-10.4%
-2.32%
-15.0%
5.2 Hardware Evaluation
Table 5 summarrzze the area overhead normalized to the MAC throughput of SPARQ for both SA and TC use cases. The SA and TC baselines are conventional 8b-8b SA and TC PEs, respectively. Memory, such as SRAMs, are not considered percentages). The 2 4b-8b design presented as reference implementation in the case of 4b-8b quantized values with equivalent throughput to the design Figure For the sake of fair comparison, there is single psum in the 2x 4b-8b design.
analysis (which could decrease the area overhead
With respect to the SA, the 2 4b-8- PE requires approximately half the area per MAC operation than the 8b-8b PE. On the one hand, the total multipliers' area of the 4b-bb PE is significantly smaller; however, the 2 4b-888 PE employs 3-input adder. The shift-left logic the main contributor the increasing area overhead of opt2 through opt5. As the number of shift-left options increases, the shift logic becomes more complex and utilizes a bigger logic area. Regarding 6opt (3 bits) and 7opt (2 bits) configurations, even though they require additional window placement options, the overall area decreases, since the area of the multipliers, registers, and multiplexers within the shift-left units reduced. Also, our 2opt scheme introduces a significantly smaller overhead compared with SySMT, due to the fact that SySMT required the trimming and rounding hardware to operate at the same high throughput rate as the SA. Regarding TC, the 2 4b-8b implementation requires half the area (normalized) of the TC 8b-8b baseline PE. Similar to the SA use case, the 4b-b-8b PE multipliers are smaller; however, this time the 2 4b-88Bb PE adder tree grows.
Interestingly, the relative area of 5opt no-vSPARQ (-vS) is only slightly higher than the "full" 3opt SPARQ implementation. Given the accuracy differences between the two configurations (Table 2 the 3opt SPARQ operating point presented in this work may not be good trade-off between accuracy and area.

---

Table Relative hardware area (normalized MAC operation throughput) of different SA and TC implementations.
SPARQ
SPARQ (-vS)
Type
8b-8b
2x4b-8b
7opt
6opt
5opt
3opl
2opt
5opt
3opt
SySMT
Systolic Array PE
1.00
0.50
0.59
0.66
0.72
0.61
0.57
0.62
0.59
0.72
Tensor Core PE
1.00
0.50
0.58
0.63
0.72
0.66
0.61
0.67
0.61
Table Accuracy results of SPARQ simulated on top of an STC with 2:4 structured pruned models.
4-bit
3-bit
2-bit
Model
FP32
A8W8
5opt
3opt
2opt
6opt
7opt
ResNet-18
69.77%
69.79%
-0.13%
-0.34%
-1.59%
-0.41%
-1.92%
ResNet-50
76.16%
76.10%
-0.24%
-0.57%
-2.59%
-0.85%
-3.18%
ResNet-101
77.38%
77.34%
-0.28%
-0.39%
-2.06%
-0.79%
-2.94%
5.3
Leveraging Activation Sparsity on Top of Sparse Tensor Cores
We simulate SPARQ on top an STC with models pruned with 2:4 structured pruning. As presented in Figure activations are first filtered through the multiplexers according the non-zero-value weight coordinates. Then, vSPARQ comes into play, inspecting pairs of activations, as described in Section 3 Since STC the trimming and rounding logic should be replicated for each DP unit, implemented and synthesized the trimming and rounding unit to estimate its area overhead. The unit area, relative the conventional TC (Figure 4.. 17%, 12%, and 9% for the Sopt, 3opt, and 2opt configurations. respectively. The relative area may be even smaller if we consider the entire STC design (Figure 5). SPARQ is, therefore, beneficial in terms of performance- an STC.
when attached to
In Table report the pruned models' FP32 and A8W8 quantized accuracies, and repeat all experiments described thus far. Interestingly, the relative accuracy degradation of the pruned models slightly higher than that of the unpruned models in Table 3 [17 40). Nevertheless, SPARQ still achieves less than 1 % relative degradation in accuracy with 4-bit Sopt and 3opt, and 3-bit 6opt.
Limitations and Societal Impacts
SPARQ has two main limitations: (1) does not achieve the memory footprint decrease as native 4b-8b quantization methods do, because of the additional metadata that accompanies each value, as discussed in Section The memory footprint may be decreased by giving up vSPARQ or sharing ShifiCtrl for number of activations. We leave these research directions for future work. (2) From hardware perspective, SPARQ requires hardware support, i.e., it cannot run on today's 's commodity hardware. In addition, compared with native 4b-8b quantizations, our hardware implementation incurs some overhead, as described in Section 5.2
As for the societal impacts, quantization methods, in general, increase the effective amount of available computing resources, since the execution requirements of quantized models are lower. The effective increase in computing power may be targeted towards negative use, such as surveillance and fake profile generation.
7 Conclusion
We present SPARQ, sparsity-aware quantization method that dynamically leverages sparsity in different granularities- -frm the entire 8-bit value to the individual bits. Thanks to the inherent activation sparsity, quantization to bits occurs only occasionally. When quantization to bits does occur, bit-level sparsity leveraged by trimming leading zero bits and picking the most significant consecutive bits. SPARQ induces minor accuracy degradation and is hardware-friendll, friendly.

---

Acknowledgements
We thank the anonymous reviewers for their comments and suggestions. We also thank Moran Shkolnik, Mario Shalabi, and Michael Behar for their valuable feedback.
References
[1] R. Banner, Y. Nahshan, and D. Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In Advances in Neural Information Processing Systems (NIPS), pages 7948-7956, 2019.
[2] S. Chetlur, C. Woolley, P. Vandermersch, Cohen, Tran, B. Catanzaro, and E. Shelhamer. cuDNN: Efficient primitives for deep learning. arXiv preprint arXiv: 1410.0759, 2014.
[3] Choi, Z. Wang, S. Venkataramani, I.JJ. Chuang. Srinivasan, and K. Gopalakrishnan. PACT: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805. 00000805500 2018.
J. Choquette, O. Giroux, and D. Foley. Volta: Performance and programmability. IEEE Micro, 38(2): 42-52, 2018.
[5] Y. Choukroun, E. Kravchik, F. Yang, and P. Kisilev, Low-bit bit quantization of neural networks for efficient inference. International Conference on Computer Vision (ICCV) Workshops, pages 3009-3018, 2019.
[6] J. Fang. A. Shafiee, H. Abdel-Aziz. D. Thorsley, G. Georgiadis, and J. H. Hassoun. Post-training piecewise lincar quantization for deep neural networks. In European Conference on Computer Vision (ECCV), pages 69-8c 86, 2020.
[7] U. Gupta, C.-J. Wu, X. Wang. M. Naumov, B. Reagen, D. Brooks, B. Cottel, K. Hazelwood, M. Hempstead, B. Jia, et al. The architectural implications of facebook's ' DNN-based personalized recommendation. In International Symposium on High Performance Computer Architecture (HPCA), pages 488-501, 2020.
[8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778. 2016.
[91 G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Conference Computer Vision and Pattern Recognition (CVPR), pages 4700-4708, 2017.
[10] Hubara, Y. Nahshan, Y. Hanami, R. Banner, and Soudry. Accurate post training quantization with small calibration sets. In International Conference on Machine Learning (ICML), pages 4466-4475. PMLR, 2021.
[11] F. landola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer. Squeezenet: Alexnet-leved accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07366 2016.
[12] S. Jain, S. Venkataramani, Srinivasan, Choi, Gopalakrishnan, and L. Chang. BiScaled-DNNY Quantizing long-tailed datastructures with two scale factors for deep neural networks. In Design Automation Conference (DAC), pages 1 1-6. IEEE, 2019.
[13] Z. Jia, M. Maggioni, B. Staiger, and D. P. Scarpazza, Dissecting the NVIDIA Volta GPU architecture via microbenchmarking. arXiv preprint / 2018.
[14] R. Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: whitepaper. arXiv preprint /806 2018.
[15] H. Kung and C. E. Leiserson. Systolic arrays (for VLSI). In Sparse Matrix Proceedings 1978, pages 356-282. Society for Industrial and Applied Mathematies, 1979.
[16] Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021.
[17] L. Liebenwein, C. Baykal, B. Carter, D. Gifford, and D. Rus. Lost in pruning: The effects of pruning neural networks beyond test accuracy. In Conference Machine Learning and Systems (MLSys), 2021.
[18] Z.-G. Liu, P. N. Whatmough, and M. Mattina. Sparse systolic tensor array for efficient CNN hardware acceleration. arXiv preprint arXiv: 200,000000003881, 2020.
[19] S. Markidis, S. W. Der Chien, E. Laure, I. B. Peng. and J. S. Vetter. NVIDIA tensor core programmability. performance & precision. In International Parallel and Distributed Processing Symposium Workshops (IPDPSW). pages 922-531. IEEE, 2018
10

---

[20] A. Mishra, J. A. Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu, and P. Micikevicius. Accelerat- ing sparse deep neural networks. arXiv preprini arXiv:2104.08378 2021.
[21] M. Nagel, A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort. Up down? adaptive rounding for post-training quantization. International Conference on Machine Learning (ICML), pages 7197-7206. PMLR, 2020.
[22] T. Norrie, N. Patil, D. H. Yoon, G. Kurian, S. Laudon, C. Young, N. Jouppi, and D. Patterson. The design process for Google's training chips: TPUv2 and TPUv3. IEEE Micro, 41(2):56-63 63, 2021.
[23] NVIDIA. NVIDIA A100 tensor core GPU architecture. httpss//www..viddaa.c.cc/cootttt cattiinnnnDaaa-CCeeeeiiiiil aamprired-aaa-e---eeeeee
-mpere-archiiteeurre architecture archiiecture-whittppppppp. whitepaper.pdf.
[24] E. Park, D. Kim, and S. Yoo. Energy-efficieen neural network accelerator based on outlier-aware low- precision computation. In International Symposium on Computer Architecture (ISCA), pages 688-698, 2018.
[25] E. Park, S. Yoo, and P. Vajda. Value-aware quantization for training and inference of neural networks. In European Conference Computer Vision (ECCV), pages 580-595 000,5555222 2018.
[26] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury. G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf. E. Yang, Z. DeVito, M. Raison, A. Tejani. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Py PyTorch: An imperative style, high- pererrerformacce deep learning library. In Advancey in Neural Information Processing Systems (NIPS), pages 8024-8035, 2019.
[27] M. A. Raihan, N. Goli, and T. M. Aamodt. Modeling deep learning accelerator enabled GPUs. In International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 79-92, 2019.
[28] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Sathcesh, S. Ma, 7. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision (IJCV). 1 11153))21111222 1-222, 2015.
[29] S. Schneider, E. Rusak, L. Eck, O, Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in Neural Information Processing Systems (NIPS), pages 1539-11151,, 2020.
[30] T. R. Shaham, T. Dekel, and Michaeli. SinGAN: Learning a generative model from single natural image. In International Conference on Computer Vision (ICCV), pages 4570-4580, 2019.
[31] M. Shkolnik, B. Chmiel, R. Banner, G. Shomron, Y. Nahshan, A. Bronstein, and U. Weiser Robust quantization: One model to rule them all. In Advances in Neural Information Processing Systems (NIPS), volume 33, pages 5308 5317, 2020.
[32] G. Shomron and U. Weiser. Non-blocking simultaneous multithreading: Embracing the resiliency of deep neural networks. International Symposium on Microurchitecture (MICRO), pages 256-269, 2020.
[33] G. Shomron and U. Weiser. Post-training BatchNorm recalibration. arXiv preprint arXiv:2010.05625. 2020.
[34] G. Shomron, T. Horowitz, and U. Weiser. SMT-SA: Simultancous multithreading in systolic arrays. Computer Architecture Letters (CAL.), 18(2):99-102, 2019.
[35] G. Shomron, R. Banner, M. Shkolnik, and U. Weiser. Thanks for nothing: Predicting zero-valued activations with lightweight convolutional neural networks. In European Conference on Computer Vision (ECCV), 2020.
[36] X. Sun, J. Choi, C.-Y. Chen, N. Wang. S. Venkataramani, V. V. Srinivasan, X. Cui, W. Zhang, and K. Gopalakrishnan. Hybrid 8-hit floating point (HFP8) training and inference for deep neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 4900 2019.
[37] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
[38] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna Rethinking the inception architecture for computer vision. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818 2826. 2016.
11

---

[39] A. Vasudevan, A. Anderson. and D. Gregg, Parallel multi channel convolution using general matrix multiplication. In International Conference implication-Specffe Systems, Archilectures and Processors (ASAP), pages 19-24, 2017.
[40] R. Yazdani, M. Riera, J.-M. Amau, and A. González. The dark side of DNN pruning. In International Symposium on Computer Architecture (ISCA), pages 790-801. IEEE, 2018.
[41] R. Zhao, J. Dotzel, Z. Hu, P. Ivanov, C. D. Sa, and Z. Zhang. OverQ: Opportunistic outlier quantization for neural network accelerators. arXiv preprint arXiv:
2021.
[42] S. Zhou, Y. Wu, Z. Ni, X. Zhou. H. Wen, and Y. Zou. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint urXiv:1606.06160. 2016.
12


Reviews:
[{"review_id": "z-xE8rJgF6", "reviewer": "Reviewer_8ZWQ", "summary": "This paper presents a sparsity-aware post-training quantization approach, which considers the sparsity on both the digit representation level and the activation level. The algorithm thus can select the most significant bits from the original 8-bit value representation, and dynamically adjust the bit budget allocation for the activation pairs during multiplication. The authors also discuss the hardware implementation for the proposed approach based on systolic array and Tensor Core DP unit. Experimental results show the effectiveness of the proposed method on various network architectures. Nonetheless, the paper still has some unclear issues with the experimental results and hardware implementations.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n\n* The paper proposes a novel solution that leverages the sparsity of bit representations and activations to enhance the quantized representation dynamically.\n\n* Comprehensive empirical evaluations are provided to verify the proposed approach.\n\n* The paper is overall easy to follow.\n\nWeakness:\n\n* The paper has missing some recent efforts on post-training quantization listed below [1,2,3,4]. These works largely follow uniform quantization and are therefore friendly to general hardwares. \n\n  [1] Hubara I, Nahshan Y, Hanani Y, et al. Improving post training neural quantization: Layer-wise calibration and integer programming. ICML, 2021.\n\n  [2] Li Y, Gong R, Tan X, et al. Brecq: Pushing the limit of post-training quantization by block reconstruction. ICLR 2021.\n\n  [3] Nagel M, Amjad R A, Van Baalen M, et al. Up or down? adaptive rounding for post-training quantization. ICML, 2020.\n\n  [4] Nahshan Y, Chmiel B, Baskin C, et al. Loss aware post-training quantization. arXiv preprint arXiv:1911.07190, 2019.\n\n* The hardware implementation and evaluation are not clear enough:\n\n  - In terms of the hardware evaluation, please introduce more details on how to calculate the hardware area for the proposed method, which would be clear to the general audience.\n\n  - It is still unclear why the proposed method enjoys 2x speedup. Is it due to the design of STC (L195)? Does the performance refer to acceleration in L251? What about the time consumption on hardware to search the most significant bits in bSPARQ (L111)? Do you measure the practical hardware speed up for your proposed algorithm?\n\n* More issues for the experiments:\n\n  - What is the absolute value accuracy of the quantized models? For now the results are mostly based on the accuracy degradation, however, the original model might have different full-precision accuracies in the first place.\n\n  - It would be more comprehensive to compare with the missing recent works [1,2,3,4]? Especially, BRECQ[2] also considers the low bit post-training quantization on ResNet-18 and ResNet-50. It would be also convenient to compare with these approaches with absolute accuracy values reported.\n\n  - Missing ablations studies for results without bSPRAC or vSPARC. You may present the ablation for low bit quantization (e.g. Table 5), so that the gain of bSPARC and vSPARC can be presented more clearly.\n\n\n\nDetailed comments:\n\n* It would be more inspiring to show the statistics of activation sparsity for different layers or network architectures. This illustrates how much we can benefit from vSPARQ (Equation 2).\n\n* Is the proposed method only designed for activation quantization? What are the potential issues if applied for weight quantization?\n\n* The bit sparsity under sparsity-aware post-training quantization seems kind of misleading to me, which simply refers to the zeros in the binary digits of 8-bit representation.\n\n* For bSPARQ, what about trimming the bit-width with non-consecutive but equally spanned digits, e.g., 4-bit at 1,3,5,7 or 2,4,6,8 digits from the 8-bit representation?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "sUitlsjaBd3", "reviewer": "Reviewer_J1hE", "summary": "The authors propose a post-training quantization scheme comprising two orthogonal ideas: bSPARQ and vSPARQ.\n\nbSPARQ a dynamic quantization technique. Instead of rounding values to the required bitwidth (equivalent to keeping the MSBs), the technique keeps the most significant consecutive non-zero bits (i.e. the sequence of bits starting from the first non-zero bit). This is equivalent to making a power-of-two adjustment to the scale factor for each individual value.\n\nvSPARQ is a dynamic sparsity technique meant for activations. First, adjacent pairs of activatons are grouped (e.g. [x1, x2, x3, x4] -> [x1x2, x3x4]). In each pair, if one of the two activations is zero, the other can make use of its compute resources. The idea is detailed in Equation (2) and can be realized in hardware with the architecture in Figure 2.\n\nThe authors demonstrate SOTA accuracy results on CNN image classification for very low activation bitwidths (4/3/2-bit activations and 8-bit weights). However, the technique also incurs significant area overhead: 22% for the 3opt design point.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very interesting paper that at first glance combines two unrelated techniques: bSPARQ shifts each value and vSPARQ exploits dynamic sparsity. However, both techniques can make use of a shifter after the multiplier (see Figure 2) which is what makes combining them a compelling idea. Overall a strong paper with solid results on both accuracy and hardware.\n\nSome comments/critiques of the paper:\n1. The overhead of the control bits is significant (3 bits per 4-bit activations with the 3-opt configuration). The area results on the multiplier unit shows that this is not an issue for compute, but for routing and memory it feels like a big problem. In particular, the paper doesn't describe the datapath at the output of the conv/matmul unit, which must re-quantize the wide accumulator outputs back to 4-bit for the next layer. This datapath will likely be significantly larger due to the overhead bits and the logic to generate them. I think this is what may kill the idea in practice.\n\n2. The accuracy comparison in Table 2 is a bit misleading. ACIQ, for example, incurs no area overhead. This should be pointed out in Table 3 (which specifically only mentions SySMT to make the comparison look better).\n\n3. The authors should cite https://arxiv.org/abs/1910.06909 which proposes a very similar idea to vSPARQ, with more analysis.\n\n4. The hardware evaluation methodology section in the supplements should be moved to main paper to assure readers that a solid hardware evaluation was done.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "cozGlIlzAHo", "reviewer": "Reviewer_f1rB", "summary": "This paper proposes a post-training sparsity-aware quantization algorithm, SPARQ, for neural networks. SPARQ leverages the bitwise sparsity by skipping leading zero-value during quantization. SPARQ also leverages the elementwise sparsity by dynamically increase the quantization precision if one in the activation pair is zero. This paper also presents a practical hardware implementation for supporting the SPARQ. The experiments show a 0.18% accuracy drop in 4-bit quantized ResNet50 on ImageNet.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The writing is clear. The paper is well structured and easy to follow. \n- The paper presents the corresponding hardware design and analyzes the extra area cost for supporting the proposed quantization method.\n\nWeakness:\n- The evaluation is mainly conducted on large-scale neural networks such as ResNet and GoogleNet. It is unclear how SPARQ performs on lightweight networks such as SqueezeNet and MobileNets.\n- The evaluation lacks an ablation study. It is unclear how much improvement comes from bSPARQ and how much comes from vSPARQ. \n- Each bSPARQ quantized value requires additional index bits. For example, 4-bit bSPARQ with 5opt is actually 7-bit quantization, and 2-bit with 7opt is 5-bit quantization. It remains questionable whether it is fair to compare against other 4-bit quantization methods. Moreover, it is unclear how speedup is calculated. According to the roofline model, most neural network computation is memory-bounded. Since the memory footprint does not decrease much, what is the real benefit of SPARQ in terms of both latency and energy? Even though it has been stated in the limitation section in the paper, it is a severe problem of the proposed method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "SMPBrYyUkc", "reviewer": "Reviewer_DrG4", "summary": "This paper proposes a sparsity-aware quantization (SPARQ) method, in which n-bit quantization takes place by picking the most significant n bits from the 8-bit value representation. Also, SPARQ is implemented on a systolic array(SA) or a tensor core(TC) DP unit. Finally, compared with the previous PTQ work, this paper evaluates the quantization effect of this method for various image classification models, and achieves the most advanced results.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- According to the description of the paper, is the hardware implementation only at the algorithm level?\n- Please describe the process of 8b-4b in Figure 1 with words and figures more clearly.\n- The results shown in Table 1,2,4 are not clear enough. Can the full precision results be reflected in tables or words?\n- Compared with other methods, the accuracy of the results presented in this paper is greatly improved, and the description of hardware implementation is surprising.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- According to the description of the paper, is the hardware implementation only at the algorithm level?\n- Please describe the process of 8b-4b in Figure 1 with words and figures more clearly.\n- The results shown in Table 1,2,4 are not clear enough. Can the full precision results be reflected in tables or words?\n- Compared with other methods, the accuracy of the results presented in this paper is greatly improved, and the description of hardware implementation is surprising.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "11"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the activation sparsity at both bit-level and numerical-level for 8-bit post-training quantization. For bit-level, a dynamic quantization technique is proposed to dynamically examine and trim leading zero-bits, which enabling a multi-scale quantization and a 2x4bit-8bit MAC unit. For numerical level, the paper proposed to skip a pair of activations when one of them is zero. The techniques are evaluated on popular CNN vision models and dataset, achieving baseline accuracy and potentially 2x speedup with some reasonable overhead.

Overall, the paper is strong. The techniques introduced are interesting and novel. The experiments are sound, and results are solid. However, it would have been helpful if the author could describe more clearly on the hardware benefits, which seems to be confusing to some reviewers. In addition, the hardware evaluation methodology in appendix should be moved to the main paper as suggested by reviewers, plus more detailed explanation on the overhead of control logic.

In short, this paper is recommended for acceptance.

Decision: Accept (Poster)