Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a method to train neural networks with low precision. However, it is not clear if this work obtains significant improvements over previous works. 

Note that:
1)	Working with 16bit, one can train neural networks with little to no reduction in performance. For example, on ImageNet with AlexNet one gets 45.11% top-1 error if we don’t do anything else, and 42.34% (similar to the 32-bit result) if we additionally adjust the loss scale (e.g., see Boris Ginsburg, Sergei Nikolaev, and Paulius Micikevicius. “Training of deep networks with halfprecision float.” NVidia GPU Technology Conference, 2017). 
2)	ImageNet with AlexNet top-1 error (53.5%) in this paper seems rather high in comparison to previous works. Specifically, DoReFA and QNN, which used mostly lower precision (k_W=1, k_A=2 and k_E=6, k_G=32)  one can get much lower performance (47% and 49%, respectively). So, the main innovation here, in comparison, is k_G=12.
3)	Comparison using other datasets is made with different architectures then previous works, so it is hard to quantify what is the contribution of the proposed method. For example, on MNIST, the authors use a convolutional neural network, while BC and BNN used a fully connected neural network (the so called “permutation invariant mnist” problem).
4)	Cifar performance is good, but may seem less remarkable, given that “Gated XNOR Networks: Deep Neural Networks with Ternary Weights and Activations under a Unified Discretization Framework” already showed that k_G=k_W=k_A=2, k_E=32 is sufficient to get 7.5% error on CIFAR. So the main novelty, in comparison, is that k_E=12.

Taking all the above into account, it hard to be sure whether the proposed methods meaningfully improve existing methods. Moreover, I am not sure if decreasing the precision from 16bit to 12bit (as was done on ImageNet) is very useful for hardware applications, especially if there is such a degradation in accuracy. If, for example, the authors would have demonstrated all-8bit training on all datasets with little performance degradation, this would seem much more useful.

Minor: there are some typos that should be corrected, e.g.: “Empirically, We demonstrates” in abstract.

%%% Following the authors response %%%
The authors have improved their results and have addressed my concerns. I therefore raised my scores.


review_title: It is not clear if this work obtains significant improvements in comparison with previous works

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose WAGE, which discretized weights, activations, gradients, and errors at both training and testing time. By quantization and shifting, SGD training without momentum, and removing the softmax at output layer as well, the model managed to remove all cumbersome computations from every aspect of the model, thus eliminating the need for a floating point unit completely. Moreover, by keeping up to 8-bit accuracy, the model performs even better than previously proposed models. I am eager to see a hardware realization for this method because of its promising results. 

The model makes a unified discretization scheme for 4 different kinds of components, and the accuracy for each of the kind becomes independently adjustable. This makes the method quite flexible and has the potential to extend to more complicated networks, such as attention or memory. 

One caveat is that there seem to be some conflictions in the results shown in Table 1, especially ImageNet. Given the number of bits each of the WAGE components asked for, a 28.5% top 5 error rate seems even lower than XNOR. I suspect it is due to the fact that gradients and errors need higher accuracy for real-valued input, but if that is the case, accuracies on SVHN and CIFAR-10 should also reflect that. Or, maybe it is due to hyperparameter setting or insufficient training time?

Also, dropout seems not conflicting with the discretization. If there are no other reasons, it would make sense to preserve the dropout in the network as well.

In general, the paper was written in good quality and in detail, I would recommend a clear accept.

review_title: a thorough and flexible approach towards discretizing neural networks

final_score: rating: 7: Good paper, accept
review_content: The authors describe a method called WAGE, which quantize all operands and operators in a neural network, specifically, the weights (W), activations (A), gradients (G), and errors (E) . The idea is using quantizers with clipping (denoted in the paper with Q(x,k)) and some additional operators like shift (denoted with shift(x)) and stochastic rounding. The main motivation of the authors in this work is to reduce the number of bits for representation in a network for all the WAGE operations and operands which influences the power consumption and silicon area in hardware implementations.

After introducing the idea and related work, the authors in Section 3 give details about how to perform the quantization. They introduce the additional operators needed for training in such network. Since quantization may loss some information, the authors need to quantize the signals in the network around the dynamic range in order not to "kill" the signal. The authors describe how to do that. Afterward, as in other techniques for quantization, they describe how to initialize the network values. Also, they argue that batch normalization in this network is replaced with the shift-quantize operations, and what is matter in this case is (1) the relative values (“orientations”) and not the absolute values and (2) small values in errors are negligible.

Afterward, the authors conduct experiments on MNIST, SVHN, CIFAR10, and ILSVRC12 datasets, where they show promising results compared to the errors provided by previous works. The WAGE parameters (i.e., the quantized no. of bits used) are 2-8-8-8, respectively. For understand more the WAGE, the authors compare on CIFAR10 the test error rate with vanilla CNN and show is small loss in using their network. The authors investigate mainly the bitwidth of errors and gradients.

In overall, this paper is an accept since it shows good performance on standard problems and invent some nice tricks to implement NN in hardware, for *both* training and inference. For inference only, other works has more to offer but this is a promising technique for learning. The things that are still missing in this work are some power reduction estimates as well as area reduction estimations. This will give the hardware community a clear vision of how such methods may be implemented both in data centers as well as on end portable devices. 

review_title: The paper describe a method for how to train and make inference in a network using only integer values.

Meta-review: High quality paper, appreciated by reviewers, likely to be of substantial interest to the community. It's worth an oral to facilitate a group discussion.
Decision: Accept (Oral)