Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper shows that if we add L1 regularization on gradient in the training phase, the obtained model can achieve better post-training quantization performance since it is more robust to Linf perturbation on the weights. I like the intuition of the paper but there are several weaknesses: 

1. The main concern is that the proposed method cannot outperform quantization-aware fine-tuning. This probably limits the application of the method --- it will only be used when there's not enough time budget for quantization-aware fine tuning for each specific choice of #bits. It will be good if the authors can discuss in what practical scenario their algorithm can be applied. 

2. The method is only tested under uniform symmetric quantization. I believe to demonstrate that the L1 regularized models are indeed easier to be quantized, we need to test it on several different kinds of quantizations. 

3. I have concerns about the hyper-parameter selection for lambda. The authors mentioned that lambda is chosen by grid-search, but what's the grid search criteria? In other words, are the hyper-parameters trying to minimize the validation error of the "unquantized model", or they are minimizing the validation error of the "post-quantized model"? 

4. Some minor suggestions: 

- The current paper uses boldfaced n as perturbation which is quite confusing (since small n is the dimension). I would suggest to replace it by something else, e.g, \Delta. 

- Section 2.3 seems redundant. It's clearly that L1 regularization is better given it's the dual norm of Linf, so clearly it's better than L2 norm. You have proved L2 is not good anyway in experiments. 

===========

After seeing the rebuttal, my concerns about the parameters have been well addressed. Also, I agree with the authors that there are use cases for post quantization, and personally I think post quantization is much easier to do in practice than quantization-aware training. However, this is quite subjective so the fact that the proposed method doesn't outperform quantization-aware training is still a weakness of the paper. 

I would like to slightly raise the score to borderline/weak-accept. I hope the authors can have some experiments on non-uniform quantization if the paper is being accepted; I really think that will demonstrate the strength of the method. People will likely to use this method if it can consistently improve many different kinds of post quantization. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper models the quantization errors of weights and activations as additive l_inf bounded perturbations and uses first-order approximation of loss function to derive a gradient norm penalty regularization that encourage the network's robustness to any bit-width quantization. The authors claim that this method is better than previous quantization-aware methods because those methods are dedicated to one specific quantization configuration.

The derivation of the proposed method is not complex but I like the idea that models quantization error as additive perturbation in this context and how it eventually connects with gradient penalty that's widely used in GAN training and adversarial robustness.

Questions:

1. What is the capital N in the time complexity of gradient computation in Sec. 4.1? The authors should discuss in details the time complexity of the proposed regularization well because this is an essential problem of the regularization, which involves double back-propagation and should be computationally heavy. For the same reason, I'd like to see the training time comparison, and more results with deeper networks.

2. Compared to STE, one of the quantization-aware methods, the proposed method is not very competitive even in the setting when a STE network, which is specially trained for 6,6 bits but quantized to 4,4 bits, can outperforms the proposed method. This contradicts with the claimed strength of the proposed method. Will it be better when we regularize more, if we want the model to perform well when quantized to 4,4 bits? It would be better if there is a set of experiments of different regularization hyperparameters.

***********************

Update: I'd like to keep my score after reading the authors' response to all reviewers. I think the authors do address some questions but the paper still has some weakness in terms of performance.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary: the authors propose a regularization scheme that is applied during regular training to ease the pose-training quantization of a network. Modeling the quantization noise as an additive perturbation bounded in \ell_\inf norm, they bound from above the first-order term of the perturbations applied to the network by the \ell_1 norm of the gradients. Their claims are also supported by experiments and qualitative illustrations.

Strengths of the paper:
- The paper is clearly written and easy to follow. In particular, section 2.1 clearly motivates the formulation of the regularization term from a theoretical point of view (reminiscent of the formulation of adversarial examples) and Figures 1 and 2 motivate the regularization term from a practical point of view. I found Figure 5 particularly enlightening (the regularization term "expands" the decision cells). 
- The method is clearly positioned with respect to previous work (in particular using \ell_2 regularization of the gradients) 
- Experiments demonstrate the effectiveness fo the method. 

Weaknesses of the paper:
- The link between the proposed objective and the sparsity could be made clearer: does this objective enforce sparsity of the gradients, the weights, and how does this affect training?


Justification of rating:
The paper clearly presents a regularization method to improve post-training quantization. The approach is motivated both from a theoretical point of view and from a practical point of view. The latter aspect is of particular interest for the community. The claims are validated by a limited set of experiments that are seem nonetheless well executed.


review_title: Official Blind Review #2

Meta-review: Reviewers uniformly suggest acceptance. Please take their comments into account in the camera-ready. Congratulations!
Decision: Accept (Poster)