Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution

Abstract: Model quantization is to discretize weights and activations of a deep neural network (DNN).
      Unlike previous methods that manually defined the quantization hyperparameters such as precision (\ie bitwidth), dynamic range (\ie minimum and maximum discrete values) and stepsize (\ie interval between discrete values),
      this work proposes a novel approach to differentiably learn all of them, named Differentiable Dynamic Quantization (DDQ), which possesses several appealing benefits. (1) Unlike previous works that applied the rounding operation to discretize values, DDQ provides a unified perspective by formulating discretization as a matrix-vector product, where different values of the matrix and vector represent different quantization methods such as mixed precision and soft quantization, and their values can be learned differentiably from training data, making different hidden layers in a DNN used different quantization methods. 
      (2) DDQ is hardware-friendly, where all variables can be computed by using low-precision matrix-vector multiplication, making it capable in wide spectrum of hardwares.
      (3) The matrix variable is carefully reparameterized to reduce its number of parameters from O(2^{b^2}) to O(\log2^b), where b is the bit width.
      Extensive experiments show that DDQ outperforms prior arts on various advanced networks and benchmarks. For instance, compared to the full-precision models, MobileNetv2 trained with DDQ achieves comparable top1 accuracy on ImageNet (71.7% vs 71.9%), while ResNet18 trained with DDQ increases accuracy by 0.5%.
      These results relatively improve recent state-of-the-art quantization methods by 70% and 140% compared to the full-precision models.
      

Reviews and author responses:

The paper proposes an approach for quantization-aware training, named Differentiable Dynamic Quantization(DDQ) to automatically learn all the quantization parameters. DDQ represents a wide variety of quantizers by formulating quantization as matrix-vector product in a unified framework instead of using traditional rounding function, causing to be hardware-friendly. The authors show that DDQ outperforms the SOTA methods in MobileNetv2, even under memory constraints. 

Strengths of the paper: 

- The approach to provide a unified perspective by formulating discretization as a matrix-vector product seems to be novel. 
- The paper is clearly structured and provides a unifying view of wide range of quantizers, especially effective in non-uniform quantizers with different weight distributions in the challenging MobileNet.
- THe proposed scheme can be easily implemented using the GEMM over wide spectrum of hardwares. 

Weaknesses of the paper: 

- Experiments seems to be too conservative to get a noticeable improvement in memory footprints. Further experiments with more aggressive setting might show the trade-off between accuracy and memory in low-precision bitwidth. 
- Some of ablation studies are a bit predictable. Most of results seems to be already obvious. Different ablation study on more memory constraints might show less obvious results.  

Detailed comments: 

- Generally well-written, easy to follow, and represents a valuable contribution. 
- While there are a few glitches in experimental evaluation, they do not hinder the overall novelty of the proposed approach.  
- The authors may want to consider the following minor comments:

(1) Number representation in uniform/non-uniform quantization has a duplicate 0(-0 and _0). Does this hurt accuracy in the very low precision setting? 2's complement representation could use the full range of representing number. Any impact on accuracy loss in the quantized model?

(2) In eq.(8), what are differences between the first one and second one? A bit confusing to follow in the corresponding phrase. 

(3) Any reasoning of better accuracy than FP32 in ResNet18 with DDQ training in Table 1?

(4) Any reasoning of insignificant memory footprint reduction in Table 2? Per-channel or hybrid quantization might be preferable instead of using per-layer quantization in DDQ(mixed)?

(5) Minor typos: 
- O(w^(2b)) instead of O(w^(b^2)) in abstract
- making different layers of kernels 'have' in page 5


In this paper, the authors proposed a nonuniform quantization method to automatically adjust the quantization levels to adapt the distribution of the weights and activations for quantization. The observations and motivation are interesting and useful.

However, there are some problems in the paper as follows:
1. Figure 1(a) is incorrect. Specifically, the center of the dense region is 0 for power-of-two quantization.
2. The basic idea of U is to combine the quantization levels, but such an idea has been proposed by "Additive powers-of-two quantization: A non-uniform discretization for neural networks". Unfortunately, the authors purposely avoid to discuss with it when comparing with power-of-two quantizations.
3. The size of U is a big problem. Although it could be alleviated by using Kronecker products, the operator will increase the complexity of the quantizers. Indeed, such problems will make the hardware design more complex and increase the cost for edge devices.
4. The necessity of U is confusing. In fact, the U can be removed from Eqn. (3) to simplify the computation and reduce the memory complexity. However, there is no discussion and experiment study to evaluate the reasonability of U.
5. The computation complexity of Eqn. (3) is too higher than the prior quantizations, such as uniform and power-of-two quantizations. Hence, the latency experiment should compare the power-of-two quantization that is an important baseline in the paper.
7. The authors emphasize their method improves state-of-the-art quantization methods by very high percentages. However, from their citations, the authors seem to purposely omit some better results, such as Additive power-of-two quantization. To my knowledge, comparing with the state-of-the-art methods, the improvement is not as good as stated even comparing with uniform quantization methods, such as Neural Network Quantization with Scale-Adjusted Training.
8. In conclusion, the basic idea of the paper is to learn the quantization levels for each quantizer, which is a trivial and direct motivation. However, the proposed solution is inefficient and not elegant. Especially, the proposed quantization is not friendly for the resource-constraint platforms, such as edge devices. The experiments are not as good as they stated.

This paper proposes a differentiable dynamic quantization (DDQ), which automatically learns all the quantization policies including arbitrary bitwidths, quantization levels, and dynamic ranges for different layers. In this way, DDQ is able to represent a wide range of quantizers for model quantization. Extensive experiments on ImageNet and CIFAR-10 demonstrate that DDQ outperforms prior methods with much less training epochs. My detailed comments are as follows.

Positive points: 
1. Instead of using the rounding function, DDQ proposes to formulate quantization as a matrix-vector product, where different values of the matrix and vector represent different quantization methods.

2. DDQ is hardware-friendly and can be easily implemented using a low-precision matrix-vector multiplication.

3. Extensive experiments on ImageNet and CIFAR-10 demonstrate the effectiveness of the proposed DDQ on various advanced networks, such as ResNet18 and MobileNetV2.

Negative points:
1. In Section 3.2, why imposing $ l_2 $ regularization into the loss function has no effect of gradient correction when using STE? More explanations are required.

2. In Section 2.3, how to ensure that $g$ is in order during training? Please provide more detailed discussions about that.

3. The experiments are insufficient in Section 4. In Table 1, the compared mixed-precision methods are not state-of-the-art. Please compare the DDQ with the latest mixed-precision methods, such as EdMIPS [1] and HMQ [2]. 

4. Some important details regarding the hyper-parameters are missing. For example,  how to set the value of the hyper-parameters $\alpha$ and $\lambda$? It would be better to explain more about that.

5. Some important details are missing. In Eq. (3), how to initialize $q$? Please provide more explanations.

6. Bit operations (BitOps) [1][3] is an important metric to measure the computational costs of a quantized network. More results in terms of BitOps are required.

7. In Section A.4.1, the authors state that “all activations are uniformly quantized for fair comparison”. It is possible to perform non-uniform quantization on activations using the proposed DDQ?

8. The meaning of “adaptive resolution” in Section 4 is confusing. What does the “adaptive resolution” denote for? More explanations and analyses are required.


Minor issues:
1. In Section A.4.1, “all activations are uniformly quantized for fair comparison” should be “all activations are uniformly quantized for a fair comparison”.


References

[1] Cai Z, Vasconcelos N. Rethinking Differentiable Search for Mixed-Precision Neural Networks[C]. In CVPR, pages 2349-2358, 2020.

[2] Habi, Hai Victor, Roy H. Jennings, and Arnon Netzer. HMQ: Hardware Friendly Mixed Precision Quantization Block for CNNs. In ECCV, 2020.

[3] Baskin C, Schwartz E, et al. Uniq: Uniform Noise Injection for Non-uniform Quantization of Neural Networks. In arXiv, 2018.



This work proposed a method to decompose the quantizer operation into the matrix-vector product based on bit-width, quantization levels, and the dynamic ranges, which are parameterized for updates via stochastic gradient descent. The authors addressed the issues of the increased number of parameters and the corresponding overheads by exploiting the regular patterns of the block-diagonal matrix. It is also worth noting that the proposed method is implemented in mobile DSP to convince that there's still meaningful latency reduction compared to the floating-point baseline. 

Despite all these merits, the experimental results shown in the paper might not be very compelling compared to the state-of-the-art (e.g., [Esser et al., 2020]). Note that [Esser et al., ICLR2020] uses a uniform quantization with the gradient correction similar to what is proposed in Sec 3.1. But it can be noted from the paper that [Esser et al., ICLR2020] achieves superior accuracy for ResNet18-ImageNet (= 71.1). This counters the authors' claim that the uniform quantization is a serious limitation. In the case of MobileNetV2, PROFIT [Park and Yoo, ECCV 2020] (which also uses the uniform quantization with a few other tricks) already achieved the 4-bit accuracy even higher than the full-precision baseline. Noting that the performance comparison in Table 1 is a little out-dated, the merits of the proposed method (with all the decomposition tricks and the parameter updates) might diminish.

In this regard, it would be very interesting to push the proposed algorithm further to the lower-bit (2-bit mixed) quantization on more diverse neural network models, like MobileNet V3 and EfficientNet. 

Also, the authors noted that the proposed technique requires only 30 epochs of retraining. This sounds quite counterintuitive since more instability is often expected when more quantization parameters are optimized together with the weight parameters; this is partially due to the fact that the more hyper-parameters are involved in the search, like the initial values for those bit-width, etc. It would be appreciated if the authors provide in-depth reasoning behind this. 









Meta-review: Two referees support accept and two indicate reject. Despite the author's rebuttal, reviewers determined through subsequent private discussions that the paper was insufficient to satisfy the high standards of ICLR due to the lack of diverse evaluations on various models/datasets  and increased computational overhead. Even the two positive reviewers agree on the weakness of the paper in terms of experimental evaluations and do not have a strong opinion on the acceptance. 

Decision: withdrawn-rejected-submissions