Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes low-precision model memory (LPMM) as a quantization training concept. In LPMM, all parameters, including weights, activation, gradient, and momentums are quantized to low-bit, which achieves theoretical low-bit training that saves significant memory footprints on the hardware. 
strength_and_weaknesses: Strengths

1. This paper proposes the low-bit version SGD optimizer, which is a good approach towards low-bit training on the device that support low precision computation.

2. The batch size experiments in figure 3 is very interesting. The author also provide convergence analysis on the training algorithm.

3. The paper organization is clear, which makes it easy to read. 


Weaknesses

1. There are many existing work on all low-bit training. And the author didn’t compare the proposed method to them. Such as WAGEUBN (Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers), DoReFa-Net (Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients), FXPNet (Fxpnet: Training a deep convolutional neural network in fixed-point representation), GradScale (Ultra-Low Precision 4-bit Training of Deep Neural Networks) and (Towards Unified INT8 Training for Convolutional Neural Network). They are very important research on all low-bit training and has been published in top conference/journals. Although some of them are cited in related works, but the author didn’t discuss nor compare with them.

2. Because there are many similar works, then the novelty of this work is not that significant. And I don’t see the author discuss about the problems of existing works, which makes the motivation of this work relatively weak.

3. One important part is missing, which is how batch-norm layers are used. During training, batch-norm is keeping the training, but it will change the low-bit number into fp number. Could  the author discuss this problem? 
  

clarity,_quality,_novelty_and_reproducibility: Please refer to strength and weaknesses
summary_of_the_review: To sum up, the clarity and quality of this paper need to be improved. The author of the paper did some interesting works on model quantization but fails to demonstrate them with thorough discussion and experiments. Please refer to strengths and weaknesses for more information.

I think this paper needs major revise, both on the technical contribution and experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose low-precision model memory (LPMM), which quantizes a model's parameters, gradients, and optimizer state to low precision. Prior works have focused only on params/gradients or optimizer state separately. The main challenge is that params, gradients, and momentum may have very different dynamic range, leading to underflow (typically of the gradient updates). LPMM addresses this via unbiased stochastic rounding and increasing the batch size to increase the magnitude of the updates. The paper uses the term "microbatching" but this is just accumulating multiple batches of gradients before computing a weight update (some frameworks call this gradient accumulation).

The authors experiment with ResNet-18 and ResNet-50 on ImageNet, showing that LPMM with 8-bits for params, grads, and momentum can get within 2% of the full precision accuracy. If the params are kept in 12 bits then they can match full precision.

EDIT: raised score from 3 to 5 after rebuttal.
strength_and_weaknesses: Strengths:
 - Appears to be the first paper to quantize params, gradients, and optimizer state jointly
 
Weaknesses
 - Experiment section is weak. ImageNet is no longer a SOTA benchmark and ResNet is an 8-year old model that is known to be easy to quantize and train. Experiments focus only on SGD with momentum, but currently ADAM is necessary for Transformers.
 - The techniques applied are well known. For example, the Deepmind Gopher paper [1] used stochastic rounding during parameter updates. Microbatching is already implemented as a toggle option in HuggingFace and Megatron Transformer frameworks. These techniques are not really used in a novel or insightful way.
 - The technique seems difficult to implement and may incur run time overhead. The authors had to rely on non-linear quantization for the momentum, which is quite complex. Unbiased stochastic quantization requires generating random numbers and comparing them to a high-precision rounding probability for each element. These two issues may cause run time overhead on GPU and present difficulties for dedicated hardware. I would like the authors to discuss the actual memory saved on GPU (not just bitwidth) and whether LPMM reduces wall-clock time for training.

[1] https://arxiv.org/pdf/2112.11446.pdf
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well-written and clear.

Originality: the work mostly applies well-known quantization techniques

Reproducibility: there should be sufficient detail to reproduce the results.
summary_of_the_review: The paper applies stochastic quantization and microbatching to quantize parameters, gradients, and optimizer state. These techniques are well known in DNN quantization and thus the paper is a bit lacking in novelty. The experiment section is weak with only results for ResNets on ImageNet. Lastly, the techniques may be difficult to implement and/or incur run time overhead, which is currently not discussed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the issue of large model memory as a bottleneck to model scaling. It proposes a framework to compress model parameters, gradient accumulations, and momentum, during training. Achieves iso-accuracy on ResNet/Imagenet models down to 12/8/8 bits, respectively, and minimal degradation down to 8/8/8 bits. Underflow is addressed by combining stochastic rounding with microbatching (accumulation of subsequent gradients). Convergence is mathematically guaranteed in the presence of stochastic quantization.
strength_and_weaknesses: Strengths:
- combination of stochastic rounding (SR) and microbatching achieves remarkable model memory compression with no (or limited) accuracy degradation
- good theoretical support on LPMM-SGDM convergence

Weaknesses:
- the paper heavily relies on known techniques (SR and microbatching) which are applied as-is to the particular case study
- the main insight is claimed to be identification of underflow as the core issue to be solved. However, the impact of delays in the parameters update (or lack thereof) is a well known quantization-related issue, both in the case of having access or not to a full precision copy of the quantized parameters (see for example the already cited [1], which discusses the conceptually-similar swamping issue as well as SR, or [2], which discusses mitigation strategies for updates delayed by quantization). So, it's hard to accept the claim that it has been "discovered" in this particular scenario
- demonstration is limited to two CNN models so it's unclear if findings may generalize (e.g., is this compression level suitable for other models?)
- what is the runtime (or training time overhead) of this algorithm? how practical is it to apply this training technique beyond the two examples shown?
- there is a mention of a training algorithm implementation at the level of CUDA kernel in PyTorch but no details about it

Other corrections:
- section 3.2: part of sentence missing: "...is far from the Extremely, there could be"
- algorithm 1: should the quantization in line 5 be carried out _after_ all the micro-batch gradients have been accumulated? Doesn't applying the quantization at every step defeats the purpose of accumulation if gradients are individually small?

[1] N. Wang et al. "Training Deep Neural Networks with 8-bit Floating Point Numbers" (section 3.2)
[2] M. Nagel et al. “Overcoming Oscillations in QAT”
clarity,_quality,_novelty_and_reproducibility: The paper is well structured, clear, and easy to follow.
While the techniques used to quantize the model parameters and address underflow are known, there is novelty in how they are jointly applied to this case study (CNNs) and how convergence is theoretically demonstrated.
Implementation is explained in details so results should be reproducible.


summary_of_the_review: The paper applies known techniques to successfully compress model memory (i.e., model parameters, gradient accumulations, and momentum) and achieve strong compression and iso-accuracy in two CNN models. I would like to see more details about the algorithm implementation and its runtime (possibly an additional appendix).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose low-precision training techniques to lower the number of bits to represent parameters, gradients, and moments. SGDM optimizer is assumed while low-precision training can improve the model accuracy by using two main principles: unbiased stochastic quantization and microbatching. Overall, memory consumption can be reduced by over 70% when ResNet-18 and ResNet-50 (on ImageNet) are trained.
strength_and_weaknesses: *Strength
- Convergence of the proposed low-precision training is analyzed.
- The impact of microbatching is investigated with details.

*Weakness
- Stochastic rounding has been introduced by lots of previous works. What is the major difference compared to those related works? For example, the paper titled "Training deep neural networks with 8-bit floating point numbers" already introduced while the first paper introducing stochastic rounding is probably "Deep Learning with Limited Numerical Precision" which is missing in the reference.
- Properties of microbatching seem to be very similar to chunk-based additions of the above reference.
- Comparisons with previous low-precision training are missing in the experiments.

clarity,_quality,_novelty_and_reproducibility: - Two major ideas of the manuscript seem to have been already introduced previously. What is new compared to those previous works?
- Improvements and novelty are incremental at best.
- Only two models (ResNet-18 and ResNet-50) are analyzed. A lot more models and dataset need to be investigated to claim the generality of the proposed techniques.
summary_of_the_review: - Ideas are incremental (or not new at all)
- Experiments need to be more extensive. Only two models are not enough to prove the claim.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a training quantization method called low-precision model memory (LPMM).  In LPMM, model parameter / activation / gradient and optimizer states are all quantized to low-bit which could provide significant memory footprint saving on the hardware. 

Main challenge is to deal with various dynamic ranges of variables. 

Authors show that in ResNet trained on ImageNet, LPMM with 8-bits could get 2% of the full precision accuracy. When parameters are kept in 12 bits authors show full precision could be achieved. 

Strength
- Considered problem is important problem in model quantization
- First paper to quantize parameter, gradients and optimizer state jointly, where previous work focused on them separately. 
- Good theoretical support on LPMM-SGDM convergence

Weakness
- While there are other existing low-bit training works, the paper does not compare to them. Reviewer wG9u pointed out a few examples of low-bit training methods, some of which are cited but not compared. 
- Related novelty of the work maybe limited and lack of comparison to existing method makes motivation of this work to be weak
- Details on how to do batch-norm is missing
- Main claim of benefit of the method is not sufficiently supported
- Proposed technical solution: stochastic rounding and micro batching is already widely used.
- Complicated to implement (non-linear quantization) and issues with run time overhead. 

Overall Clarity and Quality needs to be improved

justification_for_why_not_higher_score: While one reviewer (`Dafx`) evaluated the paper as marginally above the acceptance threshold, all other reviewers are recommending rejection. Two of the reviewers recommending rejection are highly confident about the recommendation. 

To improve, the paper requires major revision and requires another round of full review. Thus AC recommends rejection following the suggestion by the reviewers.
Decision: Reject