Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: When training quantized neural networks, one typically first fixes the desired bitwidth $b$ (of weights and activations). While training, one maintains and updates full-bitwidth weights during backprop and "cheats" by using $b$-quantized versions of these full-precision weights during forward propagation. The quantization scheme used may vary.

What if we wished instead to *learn* $b$? This is especially attractive, for instance, if we wish a distinct $b_i$ for each layer $i$. We could introduce a variable $\beta_i$ to represent $b_i$ during training. We would probably add a regularizer to minimize $\Sigma \beta_i$ to encourage training toward small bitwidths. However, this still leaves the problem that the $\beta_i$ would be real, not integer, as required for quantization. We could cheat here by using $\lceil \beta_i \rceil$, but note that quantization error $\beta_i - \lceil \beta_i \rceil$ may be quite large relative to the small integer values $\beta$ is expected to take. Is there a way to ensure that training automatically produces $\beta$s that are close to integer values?

This is where the current paper introduces a neat trick. By embedding $\beta$ inside a (suitably scaled) sinusoidal loss function, they ensure that the $\beta$s tend to have integer values (that correspond to the minima of the sinusoid).

I haven't seen this trick before, and it seems like an elegant approach to the problem of favoring integer values within the global optimization process. The goal of producing distinct bitwidths for each layer is definitely a valid one, so this technique does also address a practical problem.

The technique's practical impact seems limited. In particular, it seems that the main effect is to reduce the average bitwidth of SOTA quantized models from 4 to >= 3.6 at roughly the same accuracy. This should correspond roughly to a 10% gain in performance (runtime), which is quite good but not great. I have no idea how to to interpret the power numbers, so I will ignore those for now.

One question I have though is how the technique compares to a decent heuristic variable-bitwidth baseline. The authors do compare to a  "decrement the bitiwidth of a single layer" baseline, which is a good start. However, what if you do something more informed than that as baseline? Can you tell us what bitwidths your techniques chose for Resnet, for instance? Is there a simple pattern there (e.g. high bitwidth for early layers, and lower later on)? Based on this, what is the "best" non-learned bitwidth selection you can come up with?

In any case, the sinusoid trick for learning integer-like values is a good one for people to know, and it seems that the authors have shown that it can be implemented to have practical impact. Even if bitwidth reduction is currently modest, I can imagine follow-up work to increase it. So I support acceptance of the paper.

review_title: How to include bitwidths in your training objective

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a regularization term to control the bit-width and encourage the DNN weights moving to the quantization intervals. The key in such regularization is the Sinusoidal function, where the penalty is maximized in the middle of quantization levels and minimized at the quantization points. The sinusoidal period is regarded as the continuous representation of the bit-width.

Pros:
1. The paper is well written and easy to understand. Illustration figures help a lot to present the proposed method. E.g., Figure 2 clearly shows the key idea of this paper.

2. The idea of using the sinusoidal period as a continuous representation is novel and it makes it natural to use gradient descent methods to optimize the bit-width.

3. Experiments on different datasets and tasks show that the proposed method can bring improvement on accuracy by using better regularization and / or better bit-width allocation.

Cons:
1. The quantization regularization can only be used on weight quantization, while activation quantization is very important and sometimes more sensitive to the accuracy drop. On this point, the proposed method cannot be used to allocate the bit-width of activation, while activations can actually have different bit-width to the weights.

2. Some related works are missing.
For quantization regularization, it is used in ProxQuant[1] as well, where the weights are regularized with a norm-based quantization regularization. It also discussed the similar observation that the regularization can help quantization-aware training.
For bit-width allocation, [2] formulated the bit-width allocation as a knapsack problem and used ADMM to iteratively optimize the NN compression. [3] used Bayesian optimization to allocate the bit-width for different layers.

3. The procedure to tune the regularization strengths is a little complicated. Although the authors proposed a detailed procedure for tuning these hyper-parameters, it's questionable that the same procedure can be widely applied in different networks/optimizers/datasets. For $\lambda_\beta$, it actually determines the number of total bits for the entire network, so it should depends on the expected compression rate of the quantized DNN.

4. Using the sinusoidal period to decide the bit-width has an assumption that the range of the weights is fixed: otherwise one can reduce both the period and value range to keep the bit-width unchanged. However, the range of weights is usually not fixed in the training.

5. The experiments on bit-width allocation may be not enough to show the effectiveness of the proposed method as a bit-width allocation method. It's better to compare with some other bit-width allocation methods.

In general, my rating is borderline. I hope the authors can give some response to the cons listed above.

Reference:

[1] Bai, Y., Wang, Y.X. and Liberty, E., 2018. Proxquant: Quantized neural networks via proximal operators. arXiv preprint arXiv:1810.00861.

[2] Yang, H., Gui, S., Zhu, Y. and Liu, J., 2020. Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-Based Approach. In CVPR 2020.

[3] Tung, F. and Mori, G., 2018. Clip-q: Deep network compression learning by in-parallel pruning-quantization. In CVPR 2018.
review_title: Borderline

final_score: rating: 7: Good paper, accept
review_content: The paper proposes using a sinusoidal regularizer for neural network quantization. The regularizer “WaveQ” (sin^2) pushes floating-point parameters towards quantized values. Because the period of the function is highly related to the required bit-width, it can be used to determine the bit-width while keeping good characteristics - continuous and trainable. The authors provide experiments on both CNN and Transformers. The proposed method is widely adaptable and easy-to-use with quite promising results.

Major questions/suggestions:

1.	In equation 2.1, the period (=quantization step) is s= 1/ (2^beta – 1), so the regularizer tries to move ‘w’ towards the multiple of ‘s’. As I understand, the desired range of ‘w’ is (-1, 1), to properly match the actual quantized value. But there is no clipping nor normalization applied to ‘w’. Is there something I missed?

2.	There is a missing work [Nyuyen, 2020] which suggests a similar regularizer (|cos|). However, this paper is still valuable and does not hurt novelty.
[Ngugen, 2020] Quantization Aware Training with Absolute-Cosine Regularization for Automatic Speech Recognition

3.	It seems that WaveQ is also used for activation quantization (Section 4.1.), but there are not enough details about applying WaveQ for activations.

Minor issues:

1.	Equation B.2 is frequently used in Section 2. Also, the ‘alpha’ term needs more explanation when it first appears. (‘scaling factor’ is not enough) Consider moving the equation B.1, B.2. to the front.

2.	Many recent quantization papers use w / max(|w|) to normalize weight. Is there any advantage to use tanh(w) / max(tanh(w)) in Equation B.1?

3.	Typo: ‘bitwidh’ in Sec.4.1, ‘citep’ in Sec.5, ‘ofcitepp’ in Sec.6, 

4.	I believe the theoretic part is OK but am not sure about the exactness and its importance.

review_title: Good paper, suggests a useful method for quantization-aware training

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes WaveQ which proposes a sinusoidal regularization approach for quantizing Neural Networks. The motivation is to enable mixed-precision quantization, since homogeneously quantizing the model to low precision can lead to accuracy degradation.

The main problem with heterogeneous/mixed-precision bit-width is that its search space is exponentially large. To address this the authors propose to automatically learn this bit-precision by adding a sinusoidal regularization where the period learn able.

While the approach is interesting but the theoretical and empirical results are not satisfactory and as such it is not clear how the proposed method is superior to other methods proposed in the literature. In particular:


- The theoretical analysis provided does not apply to the proposed method. This is because the theory requires that the regularization function vanish after many iterations, whereas in the experiments the opposite approach is used.

- The accuracy results provided in the empirical section incur significant degradation as compared to the baseline. Furthermore, the comparison is performed with old quantization methods. Newer mixed-precision quantization results using for example HAQ leads to much better quantization. As such, it is not clear what is the advantage of the proposed method?

- How is this approach different than Achterhold et al., 2018?
review_title: Interesting Motivation but Unfortunately  Unsatisfactory Empirical and Theoretical Results

Meta-review: This paper proposed a regularization term to control the bit-width and encourage the DNN weights moving to the quantization intervals. The paper is well-written and the idea of using the sinusoidal period as a continuous representation is novel. However, the theoretical analysis provided are not consistent with the proposed method.

As for the experimental results, the proposed method incurs significant degradation as compared to the baseline, and comparison with recent quantization methods is lacking.
Decision: Reject