Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: They propose a pseudo-Lagrange multiplier-based learning algorithm to quantize weights. Their contributions include:

1. They propose a pseudo-LMM framework that is capable of addressing constraints on the weights.

2. They introduce sawtooth-like constraint functions with gradually vanishing unconstrained-weight windows, which eases the difficulties of co-optimization.

3. They propose a guiding principle to optimize weights and Lagrange multipliers, and their results outperform the previous state-of-the-art results.
main_review: The authors propose a very interesting method to improve the standard post-training weight quantization by solving a constraint optimization problem with a pseudo-Lagrange multiplier-based learning algorithm. Compared to the previous work [1], I think this work has great originality from many perspectives. Also, the paper is mostly well-written with minor typos. The proposed sawtooth-like constraint functions with gradually vanishing unconstrained-weight windows (controlled by g) are useful for the mentioned constraint optimization problem. Although I am concerned about the complexity of the proposed learning algorithm, I believe overall this is a good submission above the acceptance threshold.

My questions and suggestions are listed below:

1. The authors summarize the learning kinetics with BDMM as a major contribution but they merely mention it once in the main text. It's better to at least discuss and conclude the results of the learning kinetics in the main text as well. 

2. In line 141, the authors mention the algorithm excludes those nondifferentiable points in the function. How are they excluded? or those points are not occurred during training most likely?

3. Why use SGD and Adam to optimize W and \lamda separately? What if using the same optimizer? 

4. I am concerned about the cost of parameter selections including learning rate, learning rate schedule, and stopping criteria for p and g. It's good to explain those costs in the appendix. 

5. The experiment results look great overall. How about the computational complexity? Are these algorithms compared based on the same number of training epochs? I think more training details are needed.

Typos:
1. In line 180, missing reference.



[1] C. Leng, Z. Dou, H. Li, S. Zhu, and R. Jin, “Extremely low bit neural network: Squeeze the last bit out with admm,” in Proceedings of the AAAI Conference on Artiﬁcial Intelligence, vol. 32, no. 1, 2018.
limitations_and_societal_impact: Although the authors do not mention their societal impact in the paper, I do think they have positive impacts by producing high accuracy and low-precision models. I think the limitations include:

1. As mentioned above, the cost of parameter selections is unknown, and probably those parameters are sensitive to different learning tasks.

2. Although it's good to see its improvement for the post-training quantization problems, I think a broader impact would be using the proposed method on low-precision training problems where the precision of weights is limited.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a simple and effective way of applying constraints on weight parameters during training. The method is basically an application of pseudo-Lagrange multiplier method to the deep neural networks weight constraint problem. Heuristic methods such as Lagrange multiplier update scheduling and the concept of unconstrained weight window are proposed together to achieve better accuracy. The proposed method achieved the state-of-the-art accuracy on ImageNet classification tasks with multiple network architectures.
main_review: **Strengths**
1. The proposed method is simple and effective. 
2. The proposed method is theoretically grounded.
3. Experiments were performed thoroughly with ablation studies.

**Weakness**

The proposed method requires additional training process after pre-training using conventional backprop. However, as authors mentioned in the paper, it may not be considered as a significant overhead if the efficiency in the inference stage is more important than the efficiency in the training stage.

**Questions**
1. What exactly does the "Initialization;" in Algorithm 1 refer to?
2. Based on my understanding, the weight parameters would be almost converged after the pre-training process is finished and the introduction of the constraints will penalize if the weights moves far away from their nearest quantization points. In such situation, how could the weights are updated significantly so that the different quantization points are assigned in inference stage?
3. Do you expect this method will also be effective on networks that use binary activation function?

**Additional Comments**

It seems there are more previous works that also suggests novel methods for network training weight constraints. Please check following works and add them to the performance comparison table if they are relevant. 
[1] Gong, Ruihao, et al. "Differentiable soft quantization: Bridging full-precision and low-bit neural networks." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
[2] Yang, Jiwei, et al. "Quantization networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
limitations_and_societal_impact: The paper properly included the expected limitation, which is the retraining overhead.

Although the societal impact was not mentioned, I could not come up with direct societal impact of this work either.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes the constrained back-propagation algorithm based on a pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. Experimental results suggest promising performance.

main_review: This paper proposes a novel method that can incorporate given constraints into back-propagation with a Lagrangian function as the objective function. The writing quality is good and well-organised for readers to follow. Related works are covered and discussed. The proposed method is introduced with clear algorithm pseudocode and questions to support. Experiments are conducted on a variety of datasets with many well-known constraints to demonstrate performance. 

limitations_and_societal_impact: This paper discusses limitation but no negative impact. 
Questions:
1. Can we also see its performance on more datasets, other than image related?
2. Can it be applied on to other network architecture (for other types of tasks other than image classification)? Such as network with recurrent units.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposed a new approach to learn the quantized weight by adding a constraint on backpropagation using the Lagrange multiplier method. To solve the nondifferentiable issue of the constraint function, the pseudo-LLM approach is proposed. Moreover, for stable co-optimization weights and Lagrange multipliers, the optimal (sawtooth-like) constraint functions are proposed. The new approach is compared with the state-of-the-art approaches and shows a slight improvement in accuracy on the average of benchmarks. 

main_review: Strength:

1- The idea behind the constraint (weight precision) backpropagation (CBP) with pseudo-LLM is novel and interesting. 

2- The paper is well organized. The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.

3- Most of the previous work are adequately cited and discussed

Weakness:

1- The approach is evaluated in limited models (2 models). It would be suggested the approach is tested with various models such as MobileNet V2, EfficientNet. 

2- The proposed approach did not compare with [1]. It would be suggested the comparison with this work discussed in the paper.

3- The result section is not complete. The author does not explain the reason why the CBP approach is shown to perform better compared to other approaches. The discussion for figure 1 and all figures in Appendix B is not complete.

4- The overhead compared to the previous approach and the limitation of the CBP algorithm are not discussed in the paper.
  
References:

[1] Pouransari, Hadi, Zhucheng Tu, and Oncel Tuzel. "Least squares binary quantization of neural networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.

Post Rebuttal =====
Thanks for the additional results in terms of GoogleNet and overhead complexity. I think the response resolved most of my comments on the clarity and quality part and I raised my score from 5 to 6. Still realizing a way to show the reasoning behind the advantage of CBP over others' approaches is highly recommended. 

limitations_and_societal_impact: 
The limitations and potential negative societal impact of their work are not addressed by the author. For instance, the author can discuss the order of practical applicability there approach [1]. 

[1] Nagel, Markus, et al. "Data-free quantization through weight equalization and bias correction." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

Post Rebuttal =====
Clarification: I suggested looking at the Levels of quantization solutions section in [1] and discuss in the paper, CBP approach is categorized in the which level. You do not need to compare your approach with [1].
review_title: null

Meta-review: The reviewers generally felt that the method proposed in this paper was interesting, well-presented, and applicable. Multiple reviewers raised the question of computational complexity, but the authors addressed this adequately in their response (and will address it in a revision). Overall, this seems like a legitimate contribution.
Decision: Accept (Poster)