Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Contribution:
1. Using quantization to make the deployment of CACIM feasible is straightforward and interesting. 

2. The proposed method was validated on CIFAR-10 and ImageNet with different models. 


Cons:
1. Combining quantization and CACIM is interesting. However, I am concerning the novelty of methods used in this paper. The 'general quantization' used in this paper is quite similar to Product quantization. [Stock, Pierre, et al. "And the bit goes down: Revisiting the quantization of neural networks." arXiv preprint arXiv:1907.05686 (2019).] Quantization by grouping and k-means is well-known in this community. 

2. In the abstract, the authors claim that 'The analog weight has its unique advantages when doing quantization. Because there is no encoding and decoding process, the range of quantization function will not affect the computing process.' When we do quantization for the digital device, we quantize (almost) continual (high precision) numbers into discrete numbers. The range of quantizer is telling us which range we want to pay more attention to during quantization. Although I agree with the claim that 'there is no encoding and decoding.', I didn't follow the sentence 'the range of quantization function will not affect the computing process'. More explain should be provided about 'what is the definition of the range of quantizer and why it will impact quantization in the digital system and it will not impact quantization in the analogy system.'
review_title: Interesting paper but Limited Novelty

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 
This paper proposes a method to train weight-quantized neural networks.  The authors propose to directly calculate the endpoints that minimize the quantization error according to the weight distribution of each layer. Empirical results on image classification tasks and object detection tasks show that the proposed method outperforms other compared weight quantization methods under the same number of bits.

Strengths: 
- The paper is clearly written and the proposed method is simple.
- Experiments are performed on image classification tasks CIFAR-10 and ImageNet, with extensive comparisons with other methods. 

Weaknesses: 
- Since the Floyd algorithm requires alternating minimization between q and e, one concern is that this may be costly. How often is the Floyd algorithm called in Algorithm 1? What is the training time compared with other methods?

- Comparison with one important reference [1] is missing. [1] also learns the quantized values, but (i) through the step size instead of the endpoints,  and (ii) uses uniform distributed quantized values instead of non-uniformly distributed ones. However, the reported results in [1] show that their 3-bit, 4-bit and 8-bit quantized models on ImageNet have  smaller accuracy gaps with the full-precision baseline compared to the proposed method. 

- One other concern is that while the authors claim that this method is proposed for compute-in-memory (CIM) system, and discussed in section 5 that the property of the CIM does not restrict the quantized values to be uniformly strictly, compared to digital systems, there is no empirical comparison of the efficiency (e.g. storage, latency) of the proposed method and other quantization methods in CIM. Do they just perform similarly in terms of computation and memory efficiency? If so, when the uniform quantization in [1] already has good accuracy, using generalized quantization as in the proposed method does not seem quite well-motivated?

[1] Esser, Steven K., et al. "LEARNED STEP SIZE QUANTIZATION." International Conference on Learning Representations. 2019.
review_title: A new non-uniform weight quantization method. Conenction to Compute-in-Memory system not strong. Comparison with one similar method is missing. 

final_score: rating: 3: Clear rejection
review_content: In order to improve robustness of analog written weights to circuit variation, quantization is used. My first question is why is that the case? The claim that quantization reduces analog noise does not seem to be correct. As far as I can tell, this setup leads to quantization noise on top of analog noise. This is a very serious point since the authors just assume this claim to be true (first para of page 3) and that is the whole motivation. The claim should be justified, or at least a reference should be provided. P.S., I do not think the claim is correct; two noise sources (quantization + analog) is worse than one noise source (analog only).

Second issue: what is the contribution of this paper? As far as I can tell, two old methods (LLoyd max and dithering) are used and that's it. Sure, the experimental section is massive but this paper does not claim to be a pure empirical study. In fact, the claim is that for the first time, a general quantizer is used. Those methods have been around for decades!

The experiments only use quantization. One would expect that CACIM would be emulated somehow and we would see that indeed the proposed method improves CACIM's accuracy (which I don't think it would as discussed above). Even the conclusion talks about digital systems. So CACIM was (erroneously) used to motivate the work but then completely forgotten. 

Post Rebuttal Comments:

I thank the authors for their feedback. I have no modification to make to my original review.
review_title: Quantization to enable DNN deployment onto analog crossbars

final_score: rating: 5: Marginally below acceptance threshold
review_content: Starting from the property of CACIM, this paper found that weight quantization fits CACIM's analog calculation process and proposed a general quantization method. Specially, it use Lloyd;s quantizer to quantize weights and followed the commonly used quantization training method (STE) to update the full-precision weights.

Pros:
1. Find a good usage of model quantization in a new hardware system.

Cons:
1. The biggest problem from my perspective is that: the proposed quantization method is disentangled with CACIM: It only considers that quantized weight has its unique advantages in analog computation. But CACIM is not involoved in how to acchieve the quantized weight. Other quantization method can also be applied to CACIM. Although experiments show outperformance over certain baselines, it only shows that the proposed method can achieve satisfying quantization. Overall, property of CACIM does not contribute to the design of quantization method.
2. Besides, the proposed quantization method is not innovative. Lloyd's formula has been used in quantization before [1][2]. Not to mention the noise-aware training and STE.
3. This paper's focus is on CACIM, but experiments are not conducted on this hardware. It will be much convincing if the proposed method is applied on CACIM.
4. Is $|\frac{\bar{W}_i W_i}{\bar{W}_i \bar{W}_i} - 1| > T$ in Algorithm 1 typo?

[1] https://github.com/spencerkent/generalized-lloyd-quantization
[2] https://openreview.net/forum?id=rJ8uNptgl
review_title: Less Innovative Quantization Method on a new Hardware System

Meta-review: This work develops a weight-quantization method for deep neural networks that is suitable for a type of analog hardware system known as crossbar-enabled analog computing-in-memory (CACIM).  The goal of this work is to train models on GPUs in such a way that they retain their predictive accuracy during inference when deployed on the analogue hardware system.

Pros:
* Good adaptation of quantization methods to the CACIM system
* Simple method
* Validation of the proposed method on multiple datasets and models

Cons:
* Lack of novelty: the proposed method is a simple combination of two popular methods, LLoyd's quantization and noise-aware training

All reviewers appreciate the simplicity of the method and the good fit to the hardware.  The authors responded to all reviews and two reviewers acknowledged the authors' response.  The authors acknowledge some reviewer observations (motivation of quantization as reducing analogue noise, lack of experiments on the actual CACIM system), and the authors added an experimental evaluation on the actual physical CACIM system showing that their method performs well.

Overall the work is well-executed and the proposed method is a good fit to the CACIM system.  However, the proposed quantization method is a straightforward adaptation of popular quantization methods.
Decision: Reject