Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper studies in depth the idea of quantizing down convolutional layers to 3 bits, with a different positive and negative per-layer scale. It goes on to provide an exhaustive analysis of performance (essentially no loss) on real benchmarks (this paper is remarkably MNIST-free).

The relevance of this paper is that it likely provides a lower bound on quantization approaches that don't sacrifice any performance, and hence can plausibly become the approach of choice for resource-constrained inference, and might suggest new hardware designs to take advantage of the proposed structure.

Furthermore, the paper provides power measurements, which is really the main metric that anyone working seriously in that space cares about. (Nit: I don't see measurements for the full-precision baseline).

I would have loved to see a SOTA result on ImageNet and a result on a strong LSTM baseline to be fully convinced.
I would have also liked to see discussion of the wall time to result using this training procedure.
review_title: Simple premise, well argued, useful result, exhaustive analysis.

final_score: rating: null
review_content: Have you experimented with training ResNet from scratch (on either CIFAR-10 or ImageNet)?
review_title: ResNet Training

final_score: rating: 3: Clear rejection
review_content: The paper shows a different approach to a ternary quantization of weights.
Strengths:
1. The paper shows performance improvements over existing solutions
2. The idea of learning the quantization instead of using pre-defined human-made algorithm is nice and very much in the spirit of modern machine learning.

Weaknesses:
1. The paper is very incremental.
2. The paper is addressed to a very narrow audience. The paper very clearly assumes that the reader is familiar with the previous work on the ternary quantization. It is "what is new in the topic" update, not really a standalone paper. The description of the main algorithm is very concise, to say the least, and is probably clear to those who read some of the previous work on this narrow subject, but is unsuitable for a broader deep learning audience.
3. There is no convincing motivation for the work. What is presented is an engineering gimmick, that would be cool and valuable if it really is used in production, but is that really needed for anything? Are there any practical applications that require this refinement? I do not find the motivation "it is related to mobile, therefore it is cool" sufficient.

This paper is a small step further in a niche research, as long as the authors do not provide a sufficient practical motivation for pursuing this particular topic with the next step on a long list of small refinements, I do not think it belongs in ICLR with a broad and diversified audience.

Also - the code was not released is my understanding.
review_title: Somewhat interesting, but incremental work lacking sufficient practical motivation

final_score: rating: 3: Clear rejection
review_content: This paper presents a ternary quantization method for convolutional neural networks. All weights are represented by ternary values multiplied by two scaling coefficients. Both ternary weights and the scaling coefficients are updated using back-propagation. This is useful for CNN model compression. Experiments on AlexNet show that the proposed method is superior Ternary-Weight-Networks (TWN) and DoReFa-Net. This work has the following strengths and weaknesses.

Strengths:
(1). Good results are shown on CIFAR-10 dataset.

(2). Massive energy saving of the ternary weights comparing to 32-bit weights.

(3). It is well written, and easy to understand.

Weaknesses:

(1). It seems that this work is an incremental improvement on the existing works. The main difference from Binary-Weight-Networks (BWN) proposed in XNOR-net[1] is using ternary weights instead of binary weights, while ternary weights have been used by many previous works. Both BWN and the proposed method in this paper learn the scaling factors during training. Comparing to Ternary-Weight-Networks (TWN), the main difference is that two independent quantization factors are used for positive and negative weights, while TWN utilizes the same scaling factor for all weights. 

(2). In the experiment, the authors did not process the first conv layer and the last fully-connected layer. The results of processing all layers should be given for fair comparison. 

(3). In the experiment, comparison with BWN is not reported. In [1], the top-1 and top-5 error of AlexNet on ImageNet of BWN is 43.2% and 20.6%, which is comparable with the method proposed in this paper (42.5% and 20.3%). However, the BWN only uses binary weights and all layers are binarized including the first conv layer and the last fully-connected layer. 

(4). For the baseline method of full precision alexnet (with BN), the reported accuracy seems to be too low (44.1% top-1 error). Commonly, using batch normalization can boost the accuracy, while the reported accuracy are much lower than alexnet without batch normalization. On the other hand, the error rates of pre-trained model of alexnet (with BN) reported by the official MatConvNet [2] are 41.8% and 19.2%. 

(5). The proposed method should be evaluated on the original AlexNet, whose accuracy is publicly available for almost all deep learning frameworks like caffe. Moreover, more experiments should be added on ImageNet, like VGG-S, VGG-16, GoogleNet or ResNet.

(6). In previous methods such as XNOR-net and TWN, most of the 32-bit multiply operation can be replaced by addition by using binary or ternary weights. However, the proposed method in this paper utilizes independent scaling factors for positive and negative weights. Thus it seems that the multiply operation can not be replaced by addition. 


References:
[1] Mohammad R, Vicente O, Joseph R, Ali F. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. ECCV 2016
[2] http://www.vlfeat.org/matconvnet/pretrained/#imagenet-ilsvrc-classification

review_title: Incremental improvement, experimental issues

final_score: rating: 7: Good paper, accept
review_content: This work presents a novel ternary weight quantization approach which quantizes weights to either 0 or one of two layer specific learned values. Unlike past work, these quantized values are separate and learned stochastically alongside all other network parameters. This approach achieves impressive quantization results while retaining or surpassing corresponding full-precision networks on CIFAR10 and ImageNet.

Strengths:

- Overall well written and algorithm is presented clearly.
- Approach appears to work well in the experiments, resulting in good compression without loss (and sometimes gain!) of performance.
- I enjoyed the analysis of sparsity (and how it changes) over the course of training, though it is uncertain if any useful conclusion can be drawn from it.

Some points:

- The energy analysis in Table 3 assumes dense activations due to the unpredictability of sparse activations. Can the authors provide average activation sparsity for each network to help verify this assumption. Even if the assumption does not hold, relatively close values for average activation between the networks would make the comparison more convincing.

- In section 5.1.1, the authors suggest having a fixed t (threshold parameter set at 0.05) for all layers allows for varying sparsity (owed to the relative magnitude of different layer weights with respect to the maximum). In Section 5.1.2 paragraph 2, this is further developed by suggesting additional sparsity can be achieved by allowing each layer a different values of t. How are these values set? Does this multiple threshold style network appear in any of the tables or figures? Can it be added?

- The authors claim "ii) Quantized weights play the role of "learning rate multipliers" during back propagation." as a benefit of using trained quantization factors. Why is this a benefit? 

- Figure and table captions are not very descriptive.

Preliminary Rating:
I think this is an interesting paper with convincing results but is somewhat lacking in novelty. 

Minor notes:
- Table 3 lists FLOPS rather than Energy for the full precision model. Why?
- Section 5 'speeding up'
- 5.1.1 figure reference error last line

review_title: Review

final_score: rating: null
review_content: Any plans to release the source code to reproduce these experiments?
review_title: Source code?

final_score: rating: 7: Good paper, accept
review_content: This paper presents new way for compressing CNN weights. In particular this paper uses a new neural network quantization method that compresses network weights to ternary values.
The group has recently published multiple paper on this topic, and this one offers possibly the lowest returns I have seen. Only a fraction of percentage in ImageNet. Results on AlexNet are of very little interest now, given the group already showed this kind of older style-network can be compressed by large amounts. 
I also would have liked to see this group release code for the compression, and also report data on the amount of effort required to compress: flops, time, number of passes, required original dataset, etc. This data is important to decide if a compression is worth the effort.
review_title: Trained Ternary Quantization

Meta-review: The paper describes a method for training neural networks with ternary weights. The results are solid and have a potential for high impact on how networks for high-speed and/or low-power inference are trained.
Decision: Accept (Poster)