Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a method to implement each layer with different precision (mixed-precision quantization). The method employed is referred to as bit-level sparsity quantization whereby each bit of the parameter set is treated as a trainable parameter. A differential bit sparsity regularizer enables a smooth trade-off between accuracy and complexity/compression. 

1) results are a slight improvement over SOTA methods. This is to be expected given the maturity of this topic.
2) typo in Table 2 ("wight precision")

3) It will be good to relate this work to [1] that also studies mixed-precision quantization using a pre-trained floating point network.

[1] Sakr et al., An analytical method to determine minimum per-layer precision of deep neural networks.

Overall a well-written paper with a solid reasoning behind the work. The results improve marginally over SOTA methods. 
review_title: The paper proposes a method to implement each layer with different precision (mixed-precision quantization). The method employed is referred to as bit-level sparsity quantization whereby each bit of the parameter set is treated as a trainable parameter. Overall a well-written paper with a solid reasoning behind the work. The results improve marginally over SOTA methods. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a new method to quantize neural networks in a differentiable manner. Proposed method applies the group lasso on the bit-planes of the weight parameters to let certain LSBs in each layer to be zero-ed out. STE is used to train the binary representation of each bit-plane and the sign of weights during the training. Results demonstrate that the proposed method can achieve higher accuracy and compression ratio compared to previous studies.

I think that the idea of introducing group lasso to prune the entire bit-plane is very interesting and the paper is well written, but some additional analysis will be helpful.

1. I think the result must be compared with more recent papers, such as LSQ (Esser, Steven K., et al., 2020). For example, LSQ demonstrates that it acheives 75.8% top-1 and 92.7% top-5 accuracy with 3/3-bit model (weight/activation) on ResNet-50. However, according to the appendix C, proposed method seems to achieve only 92.16% when the activation is quantized to 3-bit.
2. This is more of a question than suggestion: after the requantization, would the batch-normalization layers function correctly? It seems like the parameters of batch normalization layers are kept the same after the requantization, while the requantization will impact the distribution of activations. Analysis on the difference between the distribution of the activation before the batch normalization layer before and afther the requantization will be helpful to see if the distribution of the activations really do not differ that much, or the batch-norm layer will just adapt to the occasion.
review_title: Good paper, more analysis required

final_score: rating: 6: Marginally above acceptance threshold
review_content: Quantization of weights in DNNs is a very effective way to reduce the computational and storage costs which can enable deployment of deep learning at the edge. However, determining suitable layer-wise bit-widths while training is a difficult task due to the discrete nature of the optimization problem. This paper proposes to utilize bit-level sparsity as a proxy for bit-width and employ regularization techniques to formulate the problem so that precision can be reduced while training the model.

The method proposed by the authors is sound. It leverages insights that have been employed in a neighboring area (pruning via regularization) and re-purposes those to the problem of quantization. The empirical evaluation is robust as well. 

One issue I have with the proposed method is that the parameter space is expanded by a large amount. Since for every scalar weight, we end up with a collection of binary weights. Doesn't this make training more difficult? It would be nice to discuss this issue. And more importantly how does the extra effort compare to other approaches (such as Dorefanet and others).

Regarding the proposed regularization technique. Lasso (least absolute shrinkage and selection operator) is, as far as I am aware, an optimizer that regularizes the L_1 norm of the parameters. Why is the regularizer in eq. (4) using the L_2 norm? Maybe I am missing something and/or there is an inconsistency is the notation/wording.  

The authors do a good job comparing with related works. However, one of the main early claims is that all works trying to find per-layer precision do so manually. This is not true, there have been some works that have done exactly that. One example is [1] which analytically determines precisions at all layers using a noise gain concept. It would be nice to contrast with such works as well.

Minor issue:
'comp x' is used in the results (tables) without being defined. It appears to indicate 'compression ratio'. This has to be explicitly stated at least once (maybe in the captions).

references:
[1] Sakr, Charbel, and Naresh R. Shanbhag. "Per-tensor fixed-point quantization of the back-propagation algorithm." ICLR 2019.
review_title: A technique to serialize multi-bit computations and focus on generating high accuracy binary networks using sparsity

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper basically proposed to learn the quantization bits (precision) in each layer. Specially, weights are constructed with binary representation as $W_s = \[W_s^1,...,W_s^b\]$. During training, $W_s^i$ is relaxed to $ \in \[0, 2\]$. And a group sparsity is imposed to all $W_s^i$ for all weights in a layer, leading to certain $W_s^i \to 0$, thus cancelling the bit allocation in $i$-th. Experimental results is promising.

Pros:
1. It is interesting to see that weights are represented in binary format, while each bit is trained in a full-precision scheme. 

Cons:
1. Training process is intricated: one has to tune the penalty in group sparsity. Also, training is separated in several steps: training and post-training finetuning.

Questions:
1. After determining the quantization bit in ("fake") quantization training (although $W_q$ is quantized but $W_s^i$ is not exactly binary, which is the exactly weight we want) using Eq.5. Author mention in "Re-quantization and precision adjustment" that $W_q^{'}$ is converted to binary value. But how to deal with the precision loss here? i.e. from $W_s^i \in \[0,2\]$ to $\{0, 1\}$
2. Author mentioned that DoReFa-Net is adopted to finetune the trained model. Since DoReFa-Net use tanh to constrain value to $[0,1]$. it seems there is no connection to the proposed quantization scheme (Eq.6). How to exactly finetune ?
3. Why is necessary for $W_s$ to be separated into postive and negative part ($W_p$, $W_n$) in processing ?
4. Since $W_s^i$ is float and trainable, is it necessary to incorporate a trainable $s$ ?
review_title: Learnable Quantization Bits

Meta-review: The paper explores a solution for mixed precision quantization. The authors view the weights in their binary format, and suggest to prune the bits in a structured way. Namely, all weights in the same layer should have the same precision, and the bits should be pruned from the least significant to most significant. This point of view allows the authors to exploit techniques used for weight pruning, such as L1 and group lasso regularization.

Although the field of quantization and model compression/acceleration is quite mature by now and has a large body of works, this paper is novel in its approach. Although the improvements provided over SoTA results are not very large, I believe that the novelty of the approach would make this paper a welcome addition to ICLR.

There are a few issues to be dealt with pointed out by the reviewers such as confusing terminology or required clarifications, but these are minor revisions that I trust the authors will be able to add to their paper.

Decision: Accept (Poster)