Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a combination of three hardware-accelerated techniques (Gecko, Quantum Mantissa, and BitChop) for optimizing the memory footprint of model training through low-precision floating point tensors. The approach is inspired by the observation that in some scenarios, the value distribution of training state, such as weights and activations, could be represented with fewer bits without harming the model performance. In particular, the paper proposes both dynamic and adaptive techniques for lossy and lossless quantization of the mantissa and exponents of training tensors.  The evaluation results show memory savings and negligible convergence impact for some image models. 
strength_and_weaknesses: Strengths
1. Scaling model size is the current approach for improving quality, and so the memory footprint of model training is an important problem for training efficiency and costs. 
2. Dynamic and adaptive techniques are likely most effective because of high variability of models, hardware, and training phases. 
3. Hardware-acceleration is probably the most efficient way to realize these optimization techniques. 

Weaknesses
1. I think the evaluation is greatly weakened by only considering image models. In particular, it would have been useful to evaluate the effectiveness on NLP models, which have driven most of the interest in mixed-precision training. 
2. The paper does not discuss the handling of overflows/underflows during training. In my experience, these are important issues for mixed-precision training in practice. 
3. The opportunity for low-precision representation of training tensors because of their value distribution is not a novel observation and is well-studied. I feel that new insights to the community could be in more efficient exploitation approaches (hardware acceleration is certainly a good direction) and demonstrating generality over a range of model architectures (e.g., transformers) and tasks (e.g., NLP, multi-modal). 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to read. 
summary_of_the_review: Overall, I feel the paper is lacking in novelty and evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper dynamically adjusts the size and format of the floating point activations and weights using gradient-based approaches to improve the training energy efficiency and execution time of ML models. It also implements encoders and decoders that guided by these approaches reduce the off-chip memory accesses, further boosting the energy efficiency.
strength_and_weaknesses: Strengths:

1. The paper is generally well-motivated and the problem chosen is timely, given the adverse environmental impact of training of large-scale ML models.

Weaknesses:

1. The paper is hard to read at times. For example, in the abstract, the authors pose Schrodinger's FP as one of the contributions, but it is only described in the supplementary materials. It is very unclear to me what is the encoding/decoding scheme, how does the bit-length reduction results in the memory footprint reduction, etc.
2. The paper's contributions look incremental. For example, the stochastic quantization scheme employed for Quantum mantissa seems similar to a wide range of works [1-2]. The loss function to learn the optimal bit-length is not new either [3]. In fact, there are also works that progressively reduce the bit-width of the network during training [4].
3. The authors did not compare their results with integer (fixed point) quantized neural networks which can even further reduce the memory footprint. I am curious what might be the accuracy drop compared to the proposed quantized FP16 approach.

[1] Dong et al., "Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization", BMVC 2017

[2] Gholami et al., "A Survey of Quantization Methods for Efficient Neural Network Inference", 2021

[3] Yang et al., "BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization", ICLR 2021

[4] Datta et al., "ACE-SNN: Algorithm-Hardware Co-design of Energy-Efficient & Low-Latency Deep Spiking Neural Networks for 3D Image Recognition", Frontiers in Neuroscience 2022
clarity,_quality,_novelty_and_reproducibility: The paper's clarity and quality can be significantly improved as mentioned above. The novelty is also marginal. The authors did not open-source their code yet for reproducibility but that did not affect my rating.
summary_of_the_review: See weaknesses above. I am giving a score of 3, but I can reconsider my score based on the authors' rebuttal and discussions with the other reviewers.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed reduced-precision quantization methods for efficient deep neural network training. One of the methods mainly proposed in this paper parameterized the number of mantissa bits so that the number of mantissa bits can be automatically adjusted during training (Quantum-Mantissa). The authors also proposed another (more computationally simple yet less effective) mantissa bit decision algorithm based on the difference in the loss (BitChop). Lastly, the authors proposed a lossless exponent bit encoding (Gecko). The authors claim that the proposed method can reduce memory transfer while maintaining original model accuracy after training.
strength_and_weaknesses: (Strengths)
- The concept of customizing mantissa and exponent bits would be critical for improving the efficiency of challenging DNN training.

(Weaknesses)
- Although the proposed Quantum-Mantissa method is based on parameterization of bit-width for the mantissa, there is no in-depth analysis of how well this parameter is optimized. Also, there is no discussion on the impact of these techniques on the training convergence 

- The proposed methods are mainly applied to forward-pass computation, and thus the scope of computational savings is limited. Also, the authors presented the energy savings based on their hardware implementation, but it is hard to thoroughly verify the claimed advantages given the limited information provided by the discussion.

- The evaluation is very limited; only covers two CNNs with typical types. So, it is hard to generalize the proposed methods.
clarity,_quality,_novelty_and_reproducibility: Although combined together, the proposed three quantization techniques seem to be separately proposed with incremental ideas, limiting the novelty of this paper. Also, the hardware benefits are hard to verify given the limited discussion about the proposed hardware architecture and experiment settings.
summary_of_the_review: The authors proposed a set of bit-reduction methods for floating point format, but without sufficient theoretical analysis and empirical justification. It would be highly appreciated if the authors add more in-depth discussion on the algorithmic success of the proposed methods and if they demonstrate this benefit throughout more variety of applications.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents two novel methods for performing dynamic quantization of the activations and weights used during training. That is the authors suggest two ways to dynamically calculate the number of bits used to compress the activations and weights during training. The first method, Quantum Mantissa, adds an additional term to the loss, so the model is able to learn the required quantization for each layer during training. The authors show that this approach can reduce the footprint significantly without sacrificing validation accuracy while incurring only modest overheads. The second method, BitChop, is a less intrusive method that greedily attempts to decrease the network-wide bitlength while the loss keeps decreasing and increases the bitlength when the loss increases. They show that BitChop also decreases the footprint significantly incurring even smaller overheads. The authors also introduce a lossless delta encoding scheme to encode the exponent bits by taking advantage of the non-uniformity of the exponent values. They call this approach Gecko. By combining the mantissa encoding schemes and Gecko the authors evaluate the entire model, which they call Schrodinger FP, on ResNet18 and MibileNet V3 Small on Imagenet. They show that Schrodinger FP has similar performance to the full model but with a significantly lower footprint that leads to a more performant and energy-efficient training scheme.
strength_and_weaknesses: # Strengths
- The paper addresses an interesting problem that seems to suggest a significant opportunity for making training large-scale models more efficient and performant.
- The results are very compelling.
- The techniques seem simple to implement.
- The authors have done a great job providing references and situating their work within the rest of the contemporary research in the community.

# Weaknesses
- The paper was not very polished and difficult to read.
- The description for quantization could be improved. I still don't understand how it works. Perhaps an example would help.
- What does it mean to fix the bitlengths for some training time to fine-tune the network to this state?
- Error bars and standard deviations are not calculated.
- I would like to see a fair comparison of Quantum Mantissa used on a single quantization level for the entire network with BitChop.
- I would like to see more ablations on the experiments tried for the decisions of BitChop.
- Fig 2c suggests that each network layer should have a different quantization level, but it seems like BtChop only needs a single quantization level for the entire network. The reason for this is not clearly explained.
- For Gecko, since the distribution is already available why not use an optimal value-based lossless encoding like a Huffman encoding scheme
clarity,_quality,_novelty_and_reproducibility: - The paper was not very clearly written, there were several sections that were difficult to parse.
- This is a medium-quality paper with lots of references to contemporary work in the area. The authors do a great job situating their work within the rest of the research in the community. However, the paper still needs a little polishing.
- To my knowledge this approach is novel.
- I have not tried to reproduce this work.

# Minor Points
- Is equation (3) supposed to be $m_i$ instead of $n_i$ and what is the sum over?
- In Figure 2b the x-axis is clipped.
- Is Fig 4a the distribution or the CDF?
summary_of_the_review: While the paper tackles a very interesting problem and has very compelling results, I don't think the paper is ready for publication in its current form. The prose needs some work, and there need to be more experiments as well as standard error measurements before the paper is ready for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is about utilizing  reduced-precision quantization methods for efficient deep neural network training. The authors propose a combination of Gecko, Quantum Mantissa and BitChop hardware-accelerated techniques for optimizing the memory footprint of neural network model training through low-precision floating point tensors. In evaluation, authors show memory saving and negligible accuracy drop for image classification models. 

Here quoting the strengths and weakness highlighted by the reviewers

Strengths

- Problem is well-motivated and the problem chosen is timely since scaling model size is a recent trend in deep learning. The memory footprint of model training is an important problem for training efficiency and costs
- Dynamic and adaptive techniques are likely most effective because of high variability of models, hardware, and training phases.
- Hardware-acceleration is probably the most efficient way to realize these optimization techniques.

Weaknesses

- Only consideration of image models based on CNNs. Interest in mixed-precision training is mostly driven by NLP models which are lacking.
- The paper does not discuss the handling of overflows/underflows during training, which are important issues for mixed-precision training in practice.
- The opportunity for low-precision representation of training tensors because of their value distribution is not a novel observation and is well-studied. The paper's contributions may be considered incremental rather than significant. 
- No comparison to integer quantized neural networks
- No code for reproducibility
- In depth analysis is lacking in terms of training convergence and optimization of bit-width for the mantissa. 
- Proposed three quantization techniques seem to be separately proposed with incremental ideas, limiting the novelty of this paper
- Claimed advantages of the proposed hardware are hard to validate, since there is no careful analysis of the proposed architecture and the performance comparison with the state-of-the-art accelerators.

There are mixed evaluations on readability of the paper but the majority had issues on clarity.  

justification_for_why_not_higher_score: Paper is lacking in novelty and evaluation. While reviewers include NLP fine tuning results during discussion phase and the reviewer saw it a step in the right direction, but it is still insufficient to convince that author's approach is ready to replace existing mixed-precision approach for NLP pretraining. 

After reviewer discussion with authors, some concerns were addressed but still many raised weakness weren't fully addressed. In the end,  no reviewer saw the submission meeting the bar for ICLR publication.
Decision: Reject