Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper is about quantization, and how to represent values as the finite number of states in a low bit width, using discretization. Particularly, they propose an approach to tackle the problems associated with previous ternarization which quantize weights to three values. Their approach is a hybrid weight representation method, which uses a network to output two weight types: ternary weight and sparse-large weights. For the ternary weight, they need 3 states to be stored with 2 bits. The one remaining state is used to indicate the sparse-large weight. They also propose an approach to centralize the weights towards ternary values. Their experiments show that their approach outperforms other compressed modeling approaches, and show an increase in AlexNet performance on the CIFAR-100 while increasing model size by 1.13%. 

- Overall, this is an interesting paper, offering a novel solution to tackle the degradation in accuracy occuring in ternary quantization techniques because of the number of quantization steps.  Their method seems technically sound, however I am not familiar with this area, so I would trust more the opinion of other reviewers - subject experts in the matter.
- Their results on AlexNet and ResNet do show an improvement in terms of model accuracy, with only a slight increase in model size. They have also provided extensive experiments studying the benefits of their quantization method, the tradeoff among accuracy and model size. 
- I find the abbreviations, being used too often, to be confusing
- The paper is targeted for a very focused audience, and does not give enough background for readers not familiar with "ternarizations".  Even the abstract could benefit from a motivation/ problem statement sentence, as well as less abbreviations being used. 

I would vote for acceptance of this paper, although it does seem a too-targeted paper on a specific audience. I would urge the authors to revise writing to make it broader accessible.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper works on weight quantization in deep networks. The authors propose to utilize the extra state in 2-bit ternary representation to encode large weight values. The authors also propose to use a weighted ridge regularizer which contains a "part of L1" term to make the weights with large values sparse.

The idea is simple and straightforward. However, the paper is not written very well with some typos and some terms defined unclearly. For instance, in the basic quantization method in Section 3.1, 1. What does RELU1 in the first paragraph mean? 

Clarification in the experiment section can be further improved. How are the activations for TTQ in section 4.2 quantized? The original TTQ paper also has results for ImageNet, how does the proposed method perform when compared with TTQ on ImageNet?

One major concern is that some popular recent quantization methods are not compared. For instance, [1] also quantized both weights and activations. Can the proposed method outperform it? More comparison with these methods can better illustrate the efficacy of the proposed method. 

Another concern is that, though the proposed method has accuracy gain compared with the full-precision baseline and TTQ, the quantization becomes much more complex due to the usage of SLW, does the proposed quantization method cause extra burden to memory access and inference time?

Others:
1. In Tables 3 and 4, "Top-1 Error" => "Top-1 Accuracy"?

[1]. Choi, Jungwook, et al. "Pact: Parameterized clipping activation for quantized neural networks." arXiv preprint arXiv:1805.06085 (2018).

---------- post-rebuttal comments-----------
I really appreciate the authors for their detailed response and additional experiments. However, from the revised manuscript, the proposed method performs significantly worse than some recent quantization methods like PACT and QIL. Thus I keep my rating unchanged.
--------------------------------------------------------

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:

The paper proposes a hybrid weights representation method where the weights of the neural network is split into two portions: a major portion of ternary weights and a minor portion of weights that are represented with different number of bits. The two portions of weights are differentiated by using the previous unused state of a typical ternary neural network since only three states are used out of the four states given 2-bit representation. The experiments are solid based on the selected baseline model on CIFAR-100 and Imagenet dataset.

Pros:

•	The idea of using the previous unused state in ternary neural network is interesting
•	Overall, the paper is well written. The proposed method is presented clearly with proper graph illustration.
Cons:
•	The idea of using mixed bit width for neural network quantization is not new. However, the experiments in the paper only compare with basic quantization method which makes the comparison not fair enough. For example, in ABC-net[1], a few full precision coefficients are used to binarize the network. With 3 bit for both weights and activations, it achieves 61% top 1 classification on ImageNet dataset with ResNet-18 as backbone model. This is around 3% higher than the paper’s proposed method with 2/4 bits for weights and 4 bits for activations. 
•	In the paper, it claims that the proposed weight ridge method “can obtain better accuracy than L2 weights decay”. However, there are no experiments or any theoretical supports for it.
•	After utilizing the forth state of a ternary neural network, it implies that all four states provided by 2 bit representation are used. Hence, the comparison with a quantized neural network of 2 bits should be given in the experiments also.

[1] Lin, Xiaofan, Cong Zhao, and Wei Pan. "Towards accurate binary convolutional neural network." Advances in Neural Information Processing Systems. 2017.

review_title: Official Blind Review #3

Meta-review: The paper proposes a hybrid weighs representation method in deep networks. The authors propose to utilize the extra state in 2-bit ternary representation to encode large weight values. The idea is simple and straightforward. The main concern is on the experimental results. The use of mixed bit width for neural network quantization is not new, but the authors only compare with basic quantization method in the original submission. In the revised version of the paper, the proposed method performs significantly worse than recent quantization methods such as PACT and QIL. Moreover, writing can be improved, and parts of the paper need to be clarified.
Decision: Reject