Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a post-training quantization scheme comprising two orthogonal ideas: bSPARQ and vSPARQ.

bSPARQ a dynamic quantization technique. Instead of rounding values to the required bitwidth (equivalent to keeping the MSBs), the technique keeps the most significant consecutive non-zero bits (i.e. the sequence of bits starting from the first non-zero bit). This is equivalent to making a power-of-two adjustment to the scale factor for each individual value.

vSPARQ is a dynamic sparsity technique meant for activations. First, adjacent pairs of activatons are grouped (e.g. [x1, x2, x3, x4] -> [x1x2, x3x4]). In each pair, if one of the two activations is zero, the other can make use of its compute resources. The idea is detailed in Equation (2) and can be realized in hardware with the architecture in Figure 2.

The authors demonstrate SOTA accuracy results on CNN image classification for very low activation bitwidths (4/3/2-bit activations and 8-bit weights). However, the technique also incurs significant area overhead: 22% for the 3opt design point.
main_review: This is a very interesting paper that at first glance combines two unrelated techniques: bSPARQ shifts each value and vSPARQ exploits dynamic sparsity. However, both techniques can make use of a shifter after the multiplier (see Figure 2) which is what makes combining them a compelling idea. Overall a strong paper with solid results on both accuracy and hardware.

Some comments/critiques of the paper:
1. The overhead of the control bits is significant (3 bits per 4-bit activations with the 3-opt configuration). The area results on the multiplier unit shows that this is not an issue for compute, but for routing and memory it feels like a big problem. In particular, the paper doesn't describe the datapath at the output of the conv/matmul unit, which must re-quantize the wide accumulator outputs back to 4-bit for the next layer. This datapath will likely be significantly larger due to the overhead bits and the logic to generate them. I think this is what may kill the idea in practice.

2. The accuracy comparison in Table 2 is a bit misleading. ACIQ, for example, incurs no area overhead. This should be pointed out in Table 3 (which specifically only mentions SySMT to make the comparison look better).

3. The authors should cite https://arxiv.org/abs/1910.06909 which proposes a very similar idea to vSPARQ, with more analysis.

4. The hardware evaluation methodology section in the supplements should be moved to main paper to assure readers that a solid hardware evaluation was done.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a post-training sparsity-aware quantization algorithm, SPARQ, for neural networks. SPARQ leverages the bitwise sparsity by skipping leading zero-value during quantization. SPARQ also leverages the elementwise sparsity by dynamically increase the quantization precision if one in the activation pair is zero. This paper also presents a practical hardware implementation for supporting the SPARQ. The experiments show a 0.18% accuracy drop in 4-bit quantized ResNet50 on ImageNet.
main_review: Strength:
- The writing is clear. The paper is well structured and easy to follow. 
- The paper presents the corresponding hardware design and analyzes the extra area cost for supporting the proposed quantization method.

Weakness:
- The evaluation is mainly conducted on large-scale neural networks such as ResNet and GoogleNet. It is unclear how SPARQ performs on lightweight networks such as SqueezeNet and MobileNets.
- The evaluation lacks an ablation study. It is unclear how much improvement comes from bSPARQ and how much comes from vSPARQ. 
- Each bSPARQ quantized value requires additional index bits. For example, 4-bit bSPARQ with 5opt is actually 7-bit quantization, and 2-bit with 7opt is 5-bit quantization. It remains questionable whether it is fair to compare against other 4-bit quantization methods. Moreover, it is unclear how speedup is calculated. According to the roofline model, most neural network computation is memory-bounded. Since the memory footprint does not decrease much, what is the real benefit of SPARQ in terms of both latency and energy? Even though it has been stated in the limitation section in the paper, it is a severe problem of the proposed method.
limitations_and_societal_impact: The authors have adequately addressed the limitations and societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a sparsity-aware post-training quantization approach, which considers the sparsity on both the digit representation level and the activation level. The algorithm thus can select the most significant bits from the original 8-bit value representation, and dynamically adjust the bit budget allocation for the activation pairs during multiplication. The authors also discuss the hardware implementation for the proposed approach based on systolic array and Tensor Core DP unit. Experimental results show the effectiveness of the proposed method on various network architectures. Nonetheless, the paper still has some unclear issues with the experimental results and hardware implementations.
main_review: 
Strengths:

* The paper proposes a novel solution that leverages the sparsity of bit representations and activations to enhance the quantized representation dynamically.

* Comprehensive empirical evaluations are provided to verify the proposed approach.

* The paper is overall easy to follow.

Weakness:

* The paper has missing some recent efforts on post-training quantization listed below [1,2,3,4]. These works largely follow uniform quantization and are therefore friendly to general hardwares. 

  [1] Hubara I, Nahshan Y, Hanani Y, et al. Improving post training neural quantization: Layer-wise calibration and integer programming. ICML, 2021.

  [2] Li Y, Gong R, Tan X, et al. Brecq: Pushing the limit of post-training quantization by block reconstruction. ICLR 2021.

  [3] Nagel M, Amjad R A, Van Baalen M, et al. Up or down? adaptive rounding for post-training quantization. ICML, 2020.

  [4] Nahshan Y, Chmiel B, Baskin C, et al. Loss aware post-training quantization. arXiv preprint arXiv:1911.07190, 2019.

* The hardware implementation and evaluation are not clear enough:

  - In terms of the hardware evaluation, please introduce more details on how to calculate the hardware area for the proposed method, which would be clear to the general audience.

  - It is still unclear why the proposed method enjoys 2x speedup. Is it due to the design of STC (L195)? Does the performance refer to acceleration in L251? What about the time consumption on hardware to search the most significant bits in bSPARQ (L111)? Do you measure the practical hardware speed up for your proposed algorithm?

* More issues for the experiments:

  - What is the absolute value accuracy of the quantized models? For now the results are mostly based on the accuracy degradation, however, the original model might have different full-precision accuracies in the first place.

  - It would be more comprehensive to compare with the missing recent works [1,2,3,4]? Especially, BRECQ[2] also considers the low bit post-training quantization on ResNet-18 and ResNet-50. It would be also convenient to compare with these approaches with absolute accuracy values reported.

  - Missing ablations studies for results without bSPRAC or vSPARC. You may present the ablation for low bit quantization (e.g. Table 5), so that the gain of bSPARC and vSPARC can be presented more clearly.



Detailed comments:

* It would be more inspiring to show the statistics of activation sparsity for different layers or network architectures. This illustrates how much we can benefit from vSPARQ (Equation 2).

* Is the proposed method only designed for activation quantization? What are the potential issues if applied for weight quantization?

* The bit sparsity under sparsity-aware post-training quantization seems kind of misleading to me, which simply refers to the zeros in the binary digits of 8-bit representation.

* For bSPARQ, what about trimming the bit-width with non-consecutive but equally spanned digits, e.g., 4-bit at 1,3,5,7 or 2,4,6,8 digits from the 8-bit representation?

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a sparsity-aware quantization (SPARQ) method, in which n-bit quantization takes place by picking the most significant n bits from the 8-bit value representation. Also, SPARQ is implemented on a systolic array(SA) or a tensor core(TC) DP unit. Finally, compared with the previous PTQ work, this paper evaluates the quantization effect of this method for various image classification models, and achieves the most advanced results.
main_review: - According to the description of the paper, is the hardware implementation only at the algorithm level?
- Please describe the process of 8b-4b in Figure 1 with words and figures more clearly.
- The results shown in Table 1,2,4 are not clear enough. Can the full precision results be reflected in tables or words?
- Compared with other methods, the accuracy of the results presented in this paper is greatly improved, and the description of hardware implementation is surprising.

limitations_and_societal_impact: - According to the description of the paper, is the hardware implementation only at the algorithm level?
- Please describe the process of 8b-4b in Figure 1 with words and figures more clearly.
- The results shown in Table 1,2,4 are not clear enough. Can the full precision results be reflected in tables or words?
- Compared with other methods, the accuracy of the results presented in this paper is greatly improved, and the description of hardware implementation is surprising.
review_title: null

Meta-review: This paper explores the activation sparsity at both bit-level and numerical-level for 8-bit post-training quantization. For bit-level, a dynamic quantization technique is proposed to dynamically examine and trim leading zero-bits, which enabling a multi-scale quantization and a 2x4bit-8bit MAC unit. For numerical level, the paper proposed to skip a pair of activations when one of them is zero. The techniques are evaluated on popular CNN vision models and dataset, achieving baseline accuracy and potentially 2x speedup with some reasonable overhead.

Overall, the paper is strong. The techniques introduced are interesting and novel. The experiments are sound, and results are solid. However, it would have been helpful if the author could describe more clearly on the hardware benefits, which seems to be confusing to some reviewers. In addition, the hardware evaluation methodology in appendix should be moved to the main paper as suggested by reviewers, plus more detailed explanation on the overhead of control logic.

In short, this paper is recommended for acceptance.

Decision: Accept (Poster)