Title: Relaxed Quantization for Discretized Neural Networks
Authors: Christos Louizos
Matthias Reisser
Tijmen Blankevoort
Efstratios Gavves
Max Welling
Venue: ICLR 2019
Keywords: Quantization
Compression
Neural Networks
Efficiency

Abstract: Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.

TL;DR: We introduce a technique that allows for gradient based training of quantized neural networks.

Review discussion:
Summary
This paper introduces a method for learning neural networks with quantized weights and activations. The main idea is to stochastically – rather than deterministically – quantize values, and to replace the resulting categorical distribution over quantized values with a continuous relaxation (the "concrete distribution" or "Gumbel-Softax distribution"; Maddison et al., 2016; Jang et al., 2016). Good empirical performance is demonstrated for LeNet-5 applied to MNIST, VGG applied to CIFAR-10, and MobileNet and ResNet-18 applied to ImageNet.
Review
Relevance:
Training non-differentiable neural networks is a challenging and important problem for several applications and a frequent topic at ICLR.
Novelty:
Conceptually, the proposed approach seems like a straight-forward application/extension of existing methods, but I'm unaware of any paper which uses the concrete distribution for the express purpose of improved efficiency as in this paper. There is a thorough discussion of related work, although I was missing Williams (1992), who used stochastic rounding before Gupta et al. (2015), and Soudry et al. (2014), who introduced a Bayesian approach to deal with discrete weights and activations.
Results:
The empirical work is thorough, achieving state-of-the-art results in several classification benchmarks. It would be interesting to see how well these methods perform in other tasks (e.g., compression or even regression), even though the literature on quantization seems to focus on classification.
Clarity:
The paper is well written and clear.
Dear Reviewer 2,
Thank you for your review and comments for approval.
We will make sure to update the related work section with the work of Soudry et al. (2014).
As for Williams (1992); to our understanding the focus of that paper was to introduce the unbiased score function estimator REINFORCE for the gradient of an expectation of a non-differentiable function. In this sense, Williams (1992) is more of a related work to the concrete / Gumbel-softmax approaches, rather than the stochastic rounding of Gupta et al. (2015). We will update the submission to include a brief discussion between the REINFORCE and concrete / Gumbel-softmax as choices for the fourth element of RQ.
Regarding experiments on different tasks; we agree that it would be interesting to check performance on tasks that require more “precision”, such as regression. We chose classification for this submission, as this provides a large amount of literature to compare against, and leave the exploration of different tasks for future work.
Quality:
The work is well done. Experiments cover a range of problems and a range of quantization resolutions. Related work section in, particular, I thought was very nicely done. Empirical results are strong.
In section 2.2, it bothers me that the amount of bias introduced by using the local grid approximation is never really assessed. How much probability mass is left out by truncating the Gumbel-softmax, in practice?
Clarity:
Well presented. I believe I'd be able to implement this, as a practitioner.
Originality:
Nice to see the concrete approximation having an impact in the quantization space.
Significance:
Quantization has obvious practical interest. The regularization aspect is striking (quantization yielded slightly improved test error on CIFAR-10; is that w/in the error bars?). A recent work [https://arxiv.org/abs/1804.05862] links model compressibility to generalization;
while this work is more focused on activations, there is no reason that it couldn't be used for weights as well.
Nits:
top of pg 6 'reduced execution speeds' -> times, or increased exec speeds
'sparcity' misspelled
Dear Reviewer 1,
Thank you for your review and comments for approval.
Regarding the bias of the local grid approximation; we mentioned in the main text that the local grid is constructed such that points that are within \delta standard deviations from the mean are always part of it. For all of our experiments, we set \delta = 3, which means that, roughly, only 2% of the probability mass of the logistic distribution is truncated. Unfortunately, due to lack of space we moved these experimental details about hyperparameters in the appendix.
Regarding the regularization aspect;
indeed we observed that for VGG, quantizing to 8/8 bits resulted in consistent improved test errors. We are definitely aware of [https://arxiv.org/abs/1804.05862] and believe that further research in this direction is a fruitful direction.
The authors proposes a unified and general way of training neural network with reduced precision quantized synaptic weights and activations. The use case where such a quantization can be of use is the deployment of neural network models on resource constrained devices, such as mobile phones and embedded devices.
The paper is very well organized and systematically illustrates and motivates the ingredients that allows the authors to achieve their goal: a quantization grid with learnable position and range, stochastic quantization due to noise, and relaxing the hard categorical quantization assignment to a concrete distribution.
The authors then validate their method on several architectures (LeNet-5, VGG7, Resnet and mobilnet) on several datasets (MNIST, CIFAR10 and ImageNet) demonstrating competitive results both in terms of precision reduction and accuracy.
Minor comments:
- It would be interesting to know whether training with the proposed relaxed quantization method is slower than with full-precision activations and weights. It would have been informative to show learning curves comparing learning speed in the two cases.
- It seems that this work could be generalized in a relatively straight-forward way to a case in which the quantization grid is not uniform, but instead all quantization interval are being optimized independently. It would have been interesting if the authors discussed this scenario, or at least motivated why they only considered quantization on a regular grid.
Dear Reviewer 3,
Thank you for your review and comments for approval.
Addressing the first point of training speed: training a neural network with the proposed method indeed imposes an additional burden in computing and sampling the categorical probabilities over the local grid for every weight and activation in the network.
As such, this method introduces an overhead which is not present in methods that rely on deterministic rounding and the straight-through estimator for gradients.
As for convergence speed, we will include an exemplary learning curve for the 32/32, 8/8 and 2/2 bit VGG in the appendix.
Addressing your second point about non-uniform grids: as you have stated, this method can be easily extended to non-uniform grids. Doing so would only require evaluating the CDF of the continuous signal at different points on the real line. We have mentioned this possibility of non-uniform grids in the conclusion to our work. The reason for why we consider uniform grids only lies in that non-uniform grids, although more powerful, generally do not allow for a  straightforward implementation in today’s low-bit hardware. We mention that we explicitly focus on uniform grids for this specific reason of hardware suitability.
Dear reviewers and commenters,
We have updated the submission to include all of the discussed points, except for the learning curves for VGG as we are currently rerunning the experiments in order to track them. We will perform another update as soon as that is finished.
Please also note that we have updated Figure 4 that contains the Imagenet results. We updated our BOP count implementation to correctly take into account the 8bit input of the models that have a full precision first layer. This resulted in a lower BOP count for these models.
Nevertheless we still observe that the RQ models lie on the Pareto frontier, hence the conclusions do not change.
EDIT: We have uploaded a new version of the paper that contains the VGG learning curves in the appendix.

Meta-review:
role: AC
data: metareview: This paper proposes an effective method to train neural networks with quantized reduced precision. It's fairly straight-forward idea and achieved good results and solid empirical work. reviewers have a consensus on acceptance. 

Decision: Accept (Poster)