Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes to quantize the weights of neural networks that can minimize the L_2 loss between the quantized values and the full-precision ones. The authors propose solutions for optimal 1-bit/ternary/2-bit quantization, as well as a greedy algorithm to approximate the optimal k-bit quantization.  Experiments are performed on image classification data set ImageNet using ResNet-18.

First of all, the authors should explicitly state in the paper that the optimality is in terms of what?  Indeed the authors obtain the solution of (1) quantization with a scaling parameter, and (2) in terms of minimizing quantization error (L_2 loss between the quantized value and the full-precision one), which is quite restricted to be a universally optimal one.

Since the number of weights and activations are limited in the network, it is not appropriate to formulate quantization error the weights and activations in the format of continuous distribution in (4) and (8). 

It is also not clear to me why this paper begins with rank-1 quantization but ends up with scaled quantization. What kind of assumption are used in this approximation, and can the optimality still be guaranteed?

One of my main concerns is the novelty of this paper. Many of the solutions in the paper have already been discovered in literature. For instance, the optimal 1-bit solution in (5) was already obtained in Binary-Weight-Network [1] in 2016. The optimal ternary solution (i.e., the ternarization threshold should be 1/2 of the scaling parameter) in (12)  was also already obtained in Corollary 3.1 in  [2] in 2018, as a special case when curvature information is dropped.

Yet another concern is about the experiments. Since the proposed optimal binarization has the same solution as BWN, where does the performance gain in Table 2 come from? Moreover, some of the recent quantization methods are not compared. For instance, in PACT [3], 2-bit weight&acitvation quantization already achieves 64.4% top-1 accuracy of Resnet18 on ImageNet, while the proposed method achieves the same accuracy with full-precision activation and 2-bit weight (Table 2). In addition, according to Table 2, the proposed method also can not beat LQ-Net. 


[1] Rastegari, Mohammad, et al. "Xnor-net: Imagenet classification using binary convolutional neural networks." ECCV, 2016.
[2] Hou, Lu, and James T. Kwok. "Loss-aware weight quantization of deep networks." ICLR 2018.
[3] Choi, Jungwook, et al. "Pact: Parameterized clipping activation for quantized neural networks." arXiv preprint arXiv:1805.06085 (2018).
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This work proves seval binary quantization guarantee for the 1 or 2 bit cases, and shows some empirical error analysis. 

I am not an expert on neural network compression so I am not quite sure how the proposed method compares with the state-of-the-art algorithms. On the other hand, I checked several proofs provided by the authors for the 1-bit and 2-bit quantization cases. The proofs look good to me.

Some minor comments:
1. For the definition (9) can the authors make it clear that it is for all v_j s.t. v_1>= v_2>=.... instead of \exits v_j?
2. Can authors provide some explanation why in (8) we want to have v1>=v2>=vk in the constraint? I understand we need that in the proof, but is there any reason this is also the case in empirical evaluation? To me we may also have cases such that v1 < v2, is there any guarantee for those cases?
3. Feels like the draft can be compressed into 8 pages, even if the work looks nice.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new family of quantized neural networks, where the weights are quantized based on fixed precisions. The authors proposed the optimal 1-bit/2-bit/ternary quantization schemes based on minimizing the L_2 loss. The authors proposed a greedy algorithm to approximate the optimal k-bit quantization.  The authors showed through extensive experiments on real DNNs that the proposed optimal quantization (Opt in the tables) can give better generalization as compared to several state-of-the-art alternative methods.

The reviewer appreciates that this quantized DNN-paper has a theory. This theory is based on minimizing the L_2 loss between the original data and quantized data. It unifies several quantization schemes into one framework. This is in contract with technical papers in the area which propose a single quantization method. The proposed quantization can further be implemented efficiently using XNOR operations and bit-counting.

The writing is of good quality. The length is a bit larger than the recommended length of 8 pages.

The reviewer has the following concerns,

- This paper motivates from rank-k quantization but implemented as a scaled quantization. At the end of page 3, the authors showed that the scaled quantization can somehow approximate the rank-k quantization. However, the approximation is loose without any guarantee. Overall, I don't understand how the low-rank quantization in section 2 is related to the proposed method and fits in the overall picture. Ideally, in section 2, the authors can have some theoretical statements to state the optimality of the rank-k quantization. Then, the authors can say that, because of practical difficulties to implement the SVD, the use the scaled binary quantization instead. Or, the authors can simply remove section 2, and add a paragraph to introduce rank-k quantization, which is in contract to the scaled quantization they used in the paper.

Somewhere in the text, the authors have to explain the optimality is with respect to the L_2 loss. There can be alternative quantization based on different losses.

Figure 4, could you explain why the angle trends up as the layer index increases?

After equation(4): mention the choice of p(x)

In conclusion, analyz -> analyze, introduc->introduce
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: this paper looks at different quantization schemes (replace values of vector in \R to, basically, plus or minus v, for well chosen v).

Different schemes are considered, namely the low rank binary quantization (a matrix is approximated by the componentwise product of a low rank matrix and a +/-1 matrix) and k-bit binary  quantization.
Finding the best quantization ends up in solving a program, which is convex and quite straightforward fo k=1 and 2, and unfortunately (apparently) non-convex for k>2. So the authors suggest a greedy approach for k>2.

The motivation of this work is DNN, arguing that quantized vectors should improve computations cost, hence some experiments are provide on DNN, yet they only illustrate the fact that quantization does not deteriorate too much the learning & test error.

The main criticisms is therefore the lack of concrete evidences that those schemes are actually helpful and, on the other hand, the relative simplicity (so the theoretical part of the paper are not sufficient by itself)
review_title: Official Blind Review #3

Meta-review: This paper proposes to quantize the weights of neural networks that can minimize the L_2 loss between the quantized values and the full-precision ones. The paper has limited novelty, as many of the solutions presented in the paper have already been discovered in the literature. During the discussion, the reviewers agree that it is an incremental contribution. Parts of the paper can also be clarified, particularly on the optimality of the solution, assumptions used in the approximation, and some of the experimental results. Experimental results can also be made more convincing by adding comparision with the more recent quantization methods.
Decision: Reject