Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This manuscript describes a new low-bit training framework as well as a new low-bit format and shows promising accuracy-precision trade-offs and better energy efficiency. The proposed method achieves no accuracy drop with 3-bit training on CIFAR dataset and 1% accuracy drop with 6-bit training on ImageNet dataset. Although the results are comparable to state-of-art works like "Hybrid 8-bit Floating Point", this paper has several drawbacks.

1. Section 3 provides detailed description of the MLS design. However, it is also very important to show a logic justification. Previous works will usually adopt metrics like mismatch probability across layers to show advantages of their design. Therefore, I would suggest authors to move section 5.1.1 (or Appendix C) to section 3 as a justification for MLS design. 

2. Figure 2 looks like a typical quantization-aware training flow and I did not see any potential difference on the figure caused by MLS scheme. Therefore I wonder if it is right or necessary to put Figure 2 in this section.

3. It is not clear to me that how MLS format avoids 32-bit FP multiplication. As the first level scaling factor is still FP32, I suppose every activation still requires FP32 mul?
review_title: Comparable results to state of art. Writing needs improvement.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a new tensor format referred to as multi-level scaling (MLS) for training deep neural networks in low precision. Dynamic quantization and MLS tensor arithmetic are used to enhance the effectiveness of MLS. Hardware energy efficiency gains are listed as 6.8X over full-precision and 1.2X over previous low-precision methods.

Comments and concerns are listed below:

1. This paper is a difficult read because of the disconnect between the figures used and the text. For example, the notation in Fig. 1 does not match the notation in the text describing Fig. 1 in 3.1 and 3.2. The symbol 'E' is used for both activation gradient and for the number of exponent bits. Typo ('quantizaiton') in line 7 of section 4. Algorithm 1 is too busy to be useful. Please abstract it out so the flow is clearer. The paper could do with a complete rewrite to tighten it up.

2. The main results in Table 2 shows that the improvement w.r.t. (Sun, 2019) is incremental. This seems consistent with the 1.2X energy efficiency gain mentioned elsewhere. Also, since this paper purports to contribute to low-bit training, why isn't there a comparison with the references (Dillon, Koester, Gysel, and Wang)? Instead Banner is included even though it falls in the category of post-training quantization.

3. It was not clear from Fig. 2 if the high-bit data and computations are in FP-32 or equivalent? Not sure what is meant.

Overall, this was a difficult paper to read and understand. The results seem incremental.


review_title: The paper proposes a new tensor format referred to as multi-level scaling (MLS) for training deep neural networks in low precision. Overall, this was a difficult paper to read and understand. The results seem incremental.

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
Summary:
Efficient training is becoming a crucial research topic as deep learning models get deeper and more complex to improve model accuracy. The authors propose a low-bit floating point quantization method to reduce energy and time consumption during training. To enhance training efficiency, the authors suggest multi-level scaling (MLS) tensor format that enables low-bit training. MLS is designed to compute complex operations with lower-cost operators (e.g., shift, add) or lower-bit operators that can be hardware-friendly. This reviewer considers the proposed hardware-friendly MLS format as the main contribution of the manuscript.

This reviewer raises the following serious concerns.
 
- I have no idea whether this work can help efficient training in practice. Since there is no prior information on how many bits are required for training without serious accuracy degradation, any efficient training method needs to be robust to various learning settings and dataset configurations. This work, however, studies the optimal formats for particular models, such as ResNet models. It would be required to suggest a general data format and demonstrate that such a format can be applied to various DNN models.
- The authors argue that the proposed method achieves over 6.8x higher energy efficiency than training with floating-point (FP) arithmetic units. Unlike training with FP, the proposed method requires dynamic quantization for every batch. The authors need to present the computational overhead of the dynamic quantization with a sufficiently thorough analysis. If the cost is negligible, the ground should also be provided. Also, it would be better if the authors suggest the expected time reduction and space complexity compared to full-precision training. Even though Section 5.2 briefly addresses such overhead of dynamic quantization (as to be comparable with that of a batch normalization), this reviewer cannot estimate the overall benefit of the proposed method compared with the previous works.

Overall, this reviewer votes for rejection. An efficient training algorithm needs to be supported by detailed experimental results on various types of DNN models while any computational overhead should be reasonably addressed.

Minor comments:
- Section 2 needs to be focused on training while the comparisons with previous works can be elaborated in the experimental results.
- Table 1 is not relevant unless the authors want to suggest ResNet-specific accelerator designs.
- Similarly, Table 2 and 3, and Figure 3 are not necessary if the authors claim that the flow in Figure 2 can be general.
review_title: Official review comments #1

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper investigated the low-bit training problem and proposed a novel method, which can reduce the element-wise bit-width to simplify floating-point computations to nearly fixed-point. The major contribution can be summarized as follows: a multi-level scaling (MLS) tensor format, a dynamic quantization and the low-bit tensor convolution arithmetic. Experiments show the proposed method could strike a good trade-off between the accuracy and bit-width, while taking the hardware efficiency into consideration.

----Strengths:
- Experiment shows the proposed method can outperform the state-of-the-art methods
- There is abundant model analysis in this paper.
----Weaknesses: 
- The paper is written in a very high-level manner. The paper could have benefitted with mode detailed explanations throughout, to aid comprehension.
- Many details of the method are unclear. For example: The definition of {I, j, k, l} in Formula 1 are unknown. Besides, in figure2, the exhibition of the proposed low-bit training computation flow is also confusing.
- On the algorithm side, The MLS format appears to be just a new combination of quantitative activation and multiple gain items. it is not clear how or why the proposed model should be more efficiency to reduce the element-wise bit-width compared with other methods? Is it just an observation/interpretation of the results?


review_title: Borderline Paper

Meta-review: The authors propose a low-bit floating point quantization method to reduce energy and time consumption for deep learning training. Dynamic quantization and MLS tensor arithmetic are used to enhance the effectiveness of MLS. The motivation is clear and the efficient training is an important problem to address. However, the effectiveness of proposed method is not well justified and experimental results are less convincing.  In addition, the clarify of paper still needs to be further improved. 
Decision: Reject