Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
Summary of the paper
-------------------------------

The authors propose a new way to perform multi-bit quantization based on greedy approximation and binary search tree for RNNs. They first show how this method, applied to the parameters only, performs on pre-trained networks and show great performances compared to other existing techniques on PTB. Then they present results with the method applied to both parameters and activations during training on 3 NLP datasets, showing again great performances compared to existing technique.

Clarity, Significance and Correctness
--------------------------------------------------

Clarity: The paper is clearly written.

Significance: I'm not familiar with the quantization literature, so I'll let more knowledgeable reviewers evaluate this point.

Correctness: The paper is technically correct.

Questions
--------------

1. It would be nice to have those memory and speed gains for training as well. Is it possible to use those quantization methods to train networks from scratch, i.e. without using a pre-train model?

Pros
------

1. The paper defines clear goals and contributions.
2. Existing methods (and their differences) are clearly and concisely presented.
3. The proposed method is well explained.
4. The experimental setup shows clear results compared to the non-quantized baselines and other quantization techniques.

Cons
-------

1. It would be nice to have another experiment not based on text (speech recognition / synthesis, audio, biological signals, ...) to see how it generalizes to other kind of data (although I can't see why it wouldn't).

Typos
--------

1. abstract: "gate recurrent unit" -> "gated recurrent unit"
2. equation (6): remove parenthesis in c_(t-1)
3. section 4, paragraph 1: "For the weight matrices, instead of on the whole, we quantize them row by row." -> "We don't apply quantization on the full matrices but rather row by row."
4. section 4, paragraph 2: Which W matrix is it? W_h? (2x)

Note
-------

Since I'm not familiar with the quantization literature, I'm flexible with my evaluation based on what other reviewers with more expertise have to say.
review_title: Nicely written paper

final_score: rating: 7: Good paper, accept
review_content: Revision:

The authors have addressed my concerns around the achievable speedup. I am increasing my score to 7.

Original Review:

The paper proposes a technique for quantizing neural network weight matrices by representing columns of weight matrices as linear combinations of binary (+1/-1) vectors. Given a weight vector, the paper proposes an alternating optimization procedure to estimate the set of k binary vectors and coefficients that best represent (in terms of MSE) the original vector. This yields a k-bit quantization. First, the coefficients/binary weights are initialized using a greedy procedure proposed in prior work. Then, the binary weights are updated using a clever binary search procedure, followed by updates to the coefficients. Experiments are conducted in an RNN context for some language modeling tasks.

The paper is relatively easy to read, and the technique is clearly explained. The technique is as far as I can tell novel, and does seem to represent an improvement over existing approaches for similar multi-bit quantization strategies.

I have a few questions/concerns. First, I am quite skeptical of many of the speedup calculations: These are rather delicate to do properly, and depend on the specific instructions available, SIMD widths, the number of ALUs present in a core, etc. All of these can easily shift numbers around by a factor of 2-8x. Without an implementation in hand, comparing against a well-optimized reference GEMM for full floating point, it's not clear how much faster this approach really would be in practice. Also, the online quantization of activations doesn't seem to be factored into the speedup calculations, and no benchmarks are provided demonstrating how fast the quantization is (unless I'm missing something). This is concerning since the claimed speedups aren't possible without the online quantization of actiations.

It would have been nice to have more discussion of/comparison with other approaches capable of 2-4 bit quantization, such as some of the recent work on ternary quantization, product quantization approaches, or at least scalar (per-dimension) k-means (non-uniform quantization).

Finally, the experiments are reasonable, but the choice of RNN setting isn't clear to me. It would have been easier to compare to prior work if the experiments also included some standard image classification tasks (e.g., CIFAR10).

Overall though, I think the paper does just enough to warrant acceptance.
review_title: Nice approach; skeptical about speed claims

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: I have read the comments and clarifications from the authors. They have added extra experiments, and clarified the speed-ups concern raised by others. I keep my original rating of the paper.

---------------
ORIGINAL REVIEW:

This paper introduces a multi-bit quantization method for recurrent neural networks, which is built on alternating the minimization formulated by Guo et al. 2017 by first fixing the \alpha values and then finding the optimal binary codes b_i with a BST, to then estimate \alpha with the refined approximation by Guo et al. 2017, iteratively. The observation that the optimal binary code can be computed with a BST is simple and elegant.

The paper is easy to follow and the topic of reducing memory and speeding up computations for RNN and DNN is interesting and relevant to the community.

The overall contribution on model quantization is based on existing methods, which makes the novelty of the paper suffer a bit. Said that, applying it to RNN is a convincing and a strong motivation. Also, in the paper it is shown how the matrix multiplications of the quantized model can be speeded up using 64 bits operation in CPU. This is, not only saves memory storage and usage, but also on runtime calculation using CPU, which is an important characteristic when there are limited computational resources.

Results on language models show that the models with quantized weights with 3 bits obtain the same or even slightly better performance on the tested datasets with impressive speed-ups and memory savings.

For completeness, it would be interesting, and I would strongly encourage to add a discussion or even an experiment using feedforward DNN with a simple dataset as MNIST, as most of previous work discussed in the paper report experiments on DNN that are feedforward. Would the speed-ups and memory savings obtained for RNN hold also for feedforward networks?





review_title: multi-bit quantization method for recurrent neural networks

Meta-review: The reviewers unanimously agree that this paper is worth publication at ICLR. Please address the feedback of the reviewers and discuss exactly how the potential speed up rates are computed in the appendix. I speed up rates to be different for different devices.
Decision: Accept (Poster)