Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method to train neural networks with 4-bit matrix multiplications. The authors propose logarithm unbiased quantization (LUQ), which combines stochastic rounding and logarithm quantization, both of which are known to be required for extreme low-bit training. The proposed training algorithm involves 4-bit deterministic integer quantization for weight and activation and 4-bit LUQ for gradients. The authors further propose some optional methods to improve the accuracy, including averaging the quantized gradient (SMP) and high-precision fine-tuning (FNT). On various computer vision and natural language processing benchmarks, the proposed approach can train with 4-bit numerical formats, with superior accuracy than previous FP4 training. 
strength_and_weaknesses: Strength:
- The proposed approach is simple. 
- The proposed approach achieves good accuracy, the result is impressive.
- Training with low-bit arthemetic is a major problem. The paper can be significant.
- The experiments cover a wider range of applications than other papers in the field.

Weaknesses:
- The novelty is somewhat thin: Until the second half of page 5, the paper is mostly presenting existing backgrounds. The novelty mainly falls in Sec. 4. But the LUQ itself is rather straightforward to design, once the goal of designing logarithmic and unbiased quantizer is clear. The approaches in Sec. 5 are also rather standard and to some extent explored in previous literature. I'd say the main contribution of this paper is   showing that such a simple combination of existing techniques is sufficient to achieve (surpringly good) accuracy, rather than proposing novel techniques. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is mostly clearly written, but it might require slightly more training details. For example, in the training phase, is just SAWB applied without any modification? (Training from scratch is more challenging than training from converged full-precision model used in commly quantization-aware training setting, and the result looks somewhat too good to me.) Is any other tricks such as chunk-based accumulation adopted?

Quality: The technical is good.

Novelty: The novelty is somewhat thin, as pointed out before.

Reproducibility: I didn't run the code, but the code looks to be reproducible to me. 
summary_of_the_review: Though the novelty is somewhat thin. I still think this is a good paper due to 
1. Its simplicity and effectiveness;
2. Its reproducibility. At least the official code is provided, which I suspect to be the first open-sourced recipe for 4-bit training.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a 4-bit quantization method (LUQ) for matrix multiplication in training deep learning models. This method uses 4-bit integer quantization for weight and activation quantization, and use the proposed logarithmic unbiased quantization (LUQ) for gradient back  propagation. The major contributions are the following:

1. A study showing that at the 4-bit quantization level for matmul, stochastic quantization is preferred for backward pass while the nearest rounding presents better accuracy when applied in forward pass.

2. The first training pipeline with fully 4-bit matmul using standard format. With the LUQ design, this pipeline does not require special techniques like Radix-4 or double pass rounding [Sun et al. 2020]. Yet, it gives the SoTA 4-bit training accuracy with small accuracy degradations to single precision training on several NLP and vision tasks.

3. Two practical techniques to further improve the accuracy of 4-bit training 1) Variance reduction with resampling for stochastic rounding 2) high precision continuous training.


Reference: 
[Sun et al. 2020] Ultra-Low Precision 4-bit Training of Deep Neural Networks
strength_and_weaknesses: Strength

1. The effort and design towards the simplicity is impressive. The key of the practical use of low precision training is the simplicity which could be efficiently implemented and embraced by the HW/SW practitioners. Previous methods needs very special hardware design or algorithmic pieces that could trigger more compute overhead than the savings of going to 4-bit. The LUQ method uses standard Radix-2 and just standard way of stochastic rounding. By choosing the right rounding method for forward and backward pass. This simple method could achieve stronger accuracy than previous method.

2. The two optional methods to improve 4-bit accuracy is practical and potentially lead to impact on future hardware designs to accommodate these considerations.


Weakness

Some of the technical and writing aspects need some more clarity to be fully evaluated.

1. Around Equ 13 and 14, the authors discussed a method to correct the inherent bias in directly rounding the exponent. Is the intention to discuss an exact or approximated approach to correct the bias yet still preserve the hardware efficiency of directly rounding the exponents? I ask this question mostly because that converting back to linear space from logarithmic space and then do the stochastic rounding might trigger additional computation overhead. If this discussion intends to tell how rounding on exponent can achieve bias correction, further elaborations are needed to clarify how the bias correction is achieved.

2. The value of \alpha in Equ 18 determines the exact interval end point values of LUQ. What is the way of determining \alpha in practice? Seems from the beginning of page 6, \alpha is determined by the gradient value range. Does this need dataset / model specific values in practice? If yes, a discussion on the method to determination in the main text or appendix would would be useful. This is important for understanding the practical overhead of enabling the 4-bit training using LUQ. 

3. Around Equ 15, the authors claimed the quantized gradient is unbiased. It seems this is under the assumption that the activation is unbiased. If this is the case the assumption should be elaborated to avoid conflations with the fact that LUQ suggest biased forward activation quantization.

clarity,_quality,_novelty_and_reproducibility: The overall writing is relatively easy to follow. It would help improve the clarify and quality by resolving the a few technical clarifications discussed in strength and weakness.
summary_of_the_review: Overall I recommended marginally-above-acceptance. I think the LUQ method provides ways to achieve 4-bit matmul in forward and backward pass using standard hardware and software techniques, which opens the door towards system and hardware practical 4-bit training. The writing and technical discussion could be further improved by resolving clarity comments above. I would highly recommend some centralized discussion about practical way to determining the threshold (\alpha) in LUQ method. If there is a simple yet effective way to achieve this, the impact will be much stronger.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes techniques to quantize the gradients in the back-propagation algorithm down to 4 bits. To do so, the authors propose the LUQ format, as well as bias removal and variance reduction techniques. In order to achieve SOTA accuracy, the method requires an additional stage of high precision fine-tuning.
strength_and_weaknesses: While this paper is interesting and presents promising results, I did have a few questions listed hereafter:
Section 3: In the calculation of rounding variance and MSE, why is the input x considered deterministic?

In Section 3's conclusion, the authors claim that the quantization MSE has to be minimized in the forward pass, which is a reasonable claim. Some works have shown how to provably do that for INT4 [1]. Can this work be compared to the proposed technique?
[1] Sakr, Charbel, et al. "Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training." International Conference on Machine Learning. PMLR, 2022.

clarity,_quality,_novelty_and_reproducibility: The experimental results, and crucially, the implementation details in the Appendix look good.

Finally, there are some typos and grammatical errors throughout the paper. I urge the authors to perform a spell check.

summary_of_the_review: The paper is interesting, tackles an important problem, and presents promising results. I raised some questions and hope to receive responses from the authors.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper shows how to perform 4-bit quantization training for deep learning models. The method uses 4-bit integer quantization for weight *and* activation quantization, and use logarithmic unbiased quantization (LUQ) and stochastic rounding for gradient back propagation. LUQ and stochastic rounding are not new, and the main novelty of the paper is to combine them and perform experimental validations. The methods improves over the state of the art for full training quantization by using a small high precision fine-tuning step, which can be seem as a remaining minor flaw. The paper is clear, the experiments (ResNets and Transformers, ImageNet/WMT/Squad datasets and tasks) support the claims adequately. 
justification_for_why_not_higher_score: The scope is limited to model quantization, the improvement is real but not extraordinary. There are some minor flaws which make it a "not totally seamless method" to deploy / change our training loops with.
justification_for_why_not_lower_score: This is a good paper, giving it more visibility may help follow-up research. The flaws are not deal-breakers.
Decision: Accept: poster