Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors propose learning a quantizer for mixed precision DNNs. They do so using a suitable parameterization on the quantizer's step size and dynamic range using gradient descent, and where the quantizer's bitwidth are inferred from the former two rather than also learned jointly.

As a non-expert in the field, I found the paper well-written and interesting in their analysis of their proposed parameterization. They explain well how quantizers work, and the intuition and relationships of the parameters behind two popular types of quantizers: uniform and power-of-two. Equation (3) is especially explicit in understanding how the choice of 2 of the 3 parameters makes an impact on the choice of gradients. My understanding is that this is the core contribution.

Novelty-wise, I don't have enough background to tell if this is much of a leap from related work that has already proposed learning certain parameters of quantizers (but different parameters, or not the exact 2 proposed by the authors). I do like the discussion of related quantizer literature noted in the introduction.

I don't know if there is already previous work in the paper's follow-up section of learning quantized DNN under a constraint involving maximum total memory, total activation memory, and maximum activation memory. The solution of a Laplace multiplier seems fairly naive and hard to work in practice as it is not a hard constraint. As a naive question, how does the scale of these values compare to the original loss function? For example, if we think of the original loss function as a negative log-likelihood which computes bits/example, does it make sense to add a constraint penalty in kB as in the experiments, which is a completely different unit scale? Do you also backpropagate through the constraint function g?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The work studies differentiable quantization of deep neural networks with straight-through gradient (Bengio et. al., 2013). The authors find that a proper parametrization of the quantizer is critical to stable training and good quantization performance and demonstrated their findings to obtain mixed precision DNNs on two datasets,  i.e., CIFAR-10 and Imagenet.

The paper is clearly written and easy to follow. The idea proposed is fairly straight-forward. Although the argument the authors used to support the finding is not very rigorous, the finding itself is still worth noting. 

One of the arguments that the authors used to support the specific form of parametrization is that it leads to diagonal Hessian. From optimization perspective, what matters is the condition number, i.e., max/min of the eigenvalues of the Hessian. Could this explain the small difference between the three different parametrization forms with uniform quantization and the big difference for power-of-two quantization? 

The penalty method used to address the memory constraints will not necessarily lead to solutions that satisfy the constraints. The authors noted that the algorithm is not sensitive to the choice of the penalty parameters. Have the authors tried to tackle problems of hard memory constraints?


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper considers the problem of training mixed-precision models. 
Since quantization involves non-differentiable operations, this paper discusses how to use the straight-through estimator to estimate the gradients,  and how different parameterizations of the quantized DNN affect the optimization process. The authors conclude that using the parameterization wrt the stepsize d and quantization range q_max has the best performance.

In the discussion for the three parameterization choices in section 2.1. 
It is not clear how the range for U2 is obtained. Given d an integer, the gradient wrt b is also bounded. In this case, why is case U3 better than U2? In Table 1, it is shown that U2 also has good performance for uniform quantization.

Indeed, the gradient of any of the three parameters (stepsize, bitwidth and quantization range) can be derived by using chain rule given the gradients of the other two. It is not clear to me why some of them can be unbounded while others do not. In addition, It is not clear to me why having different gradient scales is a big problem. Adaptive learning rate methods like Adam should be able to help deal with the different scale of the gradients for three parameters. Can the authors compare the three parameterizations using Adam and see if similar empirical results can still be observed.

At the end of Section 2.1, the authors said that "similar considerations can be made for power-of-two quantization".  However, from table 1, these three parameterizations indeed have quite different performances for uniform and power-of-two quantization.  E.g., for uniform quantization, U2 and U3 perform significantly better than U1, while for power-of-two quantization, U1 and U3 perform significantly better than U2. Can the authors elaborate more on the difference?

Is the proposed differential quantization method used for both weight and activation? If so, how are the gradients w.r.t. the weights propagated through the quantized activations?

---------- post-rebuttal comment -------------
I thank the authors for their detailed response. It has solved most of my concerns and I accordingly raised my score.
---------------------------------------------------------
review_title: Official Blind Review #1

Meta-review: The reviewers uniformly vote to accept this paper. Please take comments into account when revising for the camera ready. I was also very impressed by the authors' responsiveness to reviewer comments, putting in additional work after submission.
Decision: Accept (Poster)