Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigated a proper bit-precision for a block floating-point format for deep neural network training and revealed that training with a small number of mantissa bits can be compensated if the training iterations with a large mantissa bit are followed. The authors also investigate the interplay between the block size and the mantissa bits for the block floating-point performance. The authors provided the experimental results to empirically support their claims.
strength_and_weaknesses: (Strengths)
- an interesting observation that the accuracy degradation by aggressively reduced mantissa bits can be recovered by the training with the increased number of mantissa bits.  

(Weaknesses)
- This paper is mainly about the presentation of empirical findings, and many of them are already well-known (e.g., full precision for the first/last layers.)

- There are no deeper insights into the interplay between the block size and the mantissa bits.


clarity,_quality,_novelty_and_reproducibility: It seems that many lessons suggested in this paper are incremental or well-known; e.g., full precision for the first/last layers, training with larger block size incurs more accuracy degradation, etc.). Therefore, the novelty of this paper is quite limited.

summary_of_the_review: In this paper, the authors investigated the characteristics of block floating point format. Although some of the empirical findings in this paper are interesting, many lessons derived from these observations seem to be incremental, limiting the novelty of this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Block floating point format is a HW implementation technique for deep neural networks (DNNs), wherein floating point (FP) matrix multiply (Matmul) operations are simplified to fixed point multiply operations by sharing the exponent bits for a block of input values. Further, the mantissa bits for the fixed-point format are truncated to the minimum required for achieving the same or better accuracy for a given ML task than FP baseline. 

This paper presents a design space exploration over mantissa bit-width and block size for a range of DNNs targeting different applications. Based on this exploration the authors identify that 6-bit mantissa format (HBFP6) achieve little or no loss of accuracy for significant HW savings. 

Further, the authors propose the approach Accuracy Boosters, which uses the insight that the epochs in a model training have varying importance for final accuracy, e.g., last few epochs are critical and suffer the largest accuracy impact due to lower mantissa bit-width. Therefore, by selectively applying higher bit-width mantissa config (e.g., 6) to the last few epochs authors demonstrate a significant accuracy improvement even if majority of the cycles use lower mantissa bit-width (e.g., 4)

Authors show the accuracy results for a few DNNs targeting different applications and claim ~17x improvement in their figure of merit (FOM) for hardware cost, active Silicon area used. 

strength_and_weaknesses: Authors provide a good prior art section describing a range of papers that attempted the problem of developing new encoding formats to reduce the hardware cost of training high accuracy DNNs. 

The paper tries to estimate hardware cost by estimating the approximate area of a design (adding the area for components misses glue logic) that trains DNNs for a given mantissa/exponent bit-width and block size. As a first order metric this is helpful to understand the savings achieved by reducing the mantissa bit-width or changing the block size. But a real hardware design would require the author to either choose a fixed HBFP format (e.g., HBFP5 only) or a few formats (e.g., HBFP4 and HBFP6) or to choose a reconfigurable hardware which will keep switching the size of HBFP mantissa bit-width used. Note that, the added cost of using reconfigurable hardware can be a limiting factor in its usage. 

If the authors commit to having HW available to support a few formats, they can then simply compare the energy efficiency of using mixed HBFP format as a figure of merit. 

Using the chosen FOM, approximate hardware area, Figure 1 illustrates the improvements achieved over FP32. To improve the readability of the figure, please add horizontal lines showing when the design achieves 95% of the hardware benefit. Also consider adding lines that link the points with same/similar accuracy for a given benchmark, e.g., 2% accuracy loss point for the different HBFP bit widths.
Authors highlight that HBFP6 achieves good accuracy for a range of benchmarks, this is a new empirical result not explored in prior art. 

Section 4.1 explores the results of minimizing HBFP bit widths. This is a minor issue, but the authors claim that HBFP6 achieves accuracy within 2% of FP32 for all block sizes but ResNet50 trained for CIFAR-100 with block size 576 fails this condition. Some of the following discussion in this section is confusing, e.g., “for the smallest model ResNet20, decreasing the block size of HBFP5 to 25 leads to 1.5% accuracy degradation but 1.1× hardware gain in return”, it’s not clear what the baseline here is. Decreasing the block size generally helps improve accuracy and reduce the hardware gain. Also, it would be helpful for improving the readability if the authors added the area improvement ratios discussed in this section as a column in Table 1. It’s not easy to confirm these area improvement numbers from Figure 1. 

Also, Table 1 highlights how using FP32 format for first and last layers helps improve the accuracy of training despite using HBFP format on other layers, but this has been done in previous papers. And the requirement of FP32 format even for a single layer means that Si area associated will be required, hence energy saving would be a better metric for comparing the efficacy of this technique. 

Accuracy boosters is a useful technique which minimizes the HBFP mantissa bit width for epochs that are least impacted by it. The accuracy improvement achieved using this technique is remarkable. It enables HBFP4 to train ResNet50/74 to higher than FP32 accuracies which wasn’t feasible by default as show in Table 1. It is unclear however, that why the authors didn’t fully explore the benefit of using Accuracy boosters? How would it fare for HBFP3/2/1? They could potentially come with their tradeoff of how many epochs need a bit width boost. 

Minor issue, the Conclusion section states, “Keeping the first and last layers in FP32 enables HBFP5 training with block sizes of 49 and higher for all models.”, it’s unclear what the authors mean here, cause for larger block sizes accuracy numbers drop regardless of the technique used. 

clarity,_quality,_novelty_and_reproducibility: Comments related to clarity are addressed in Strengths and weakness section.
summary_of_the_review: The authors provide two new insights, 1) HBFP6 can achieve high accuracy for a range of networks and 2) Accuracy boosters is a reliable way to get higher accuracy while spending lesser energy. However, the figure of merit used by the authors, Si area, is not very useful as even a single usage of a unique data format requires associated hardware which if custom designed will come with its own area. Energy efficiency would have been a more useful metric. Also, the analysis of Accuracy boosters is limited and could use more experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces "accuracy boosters" that aim to train DNNs using HBFP4 for most of the epochs while switching to HBFP6 for the last few epochs. Thy show that mixed precision training is applicable under HBFP setting and achieve similar accuracy with better hardware utilization.

strength_and_weaknesses: Strengths:

* Good background section that clearly explains the use of HBFP
* Intuitive explanation of mixed precision training in the context of HBFP with proper experimental backing

Weaknesses:

* It is unclear why training at the very end using HBFP6 allows you to recover the accuracy loss, more explanation would have been better
clarity,_quality,_novelty_and_reproducibility: Novelty: I am not an expert in fixed point formats. The method seems novel (HBFP mid precision training). However, there was no automatic technique to find how the best mixed precision training schedule.

Clarity: I found the paper was well written with a good background section.
summary_of_the_review: I enjoyed reading the paper. I am not an expert in different fixed point formats. However, the background section was fairly detailed for me to appreciate the use of HBFP.

The technique seems fairly intuitive. The authors adapt mixed precision training in the context of HBFP. The results show that by training most of the time with HBFP4 and switching to HBFP6 at the end allows the authors to achieve comparable accuracy at a lower hardware cost.

That said the technique for finding this schedule was mainly through manual experimentation. Therefore, I am not sure of the novelty of the work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors empirically study hybrid block floating-point (HBFP). HBFP uses a single shared exponent with a block of multiple mantissas. The authors run training experiments on CIFAR-10/100 using ResNets and DeseNet40 with HBFP. Mantissa bits of 6, 5, and 4 are tested with block size ranging from 16 to 576. Both forward and backward pass is quantized to HBFP. The authors conclude that HBFP5 with the first and last layers in FP32 is sufficient to match FP32 training accuracy.

The authors also experiment with FracTrain, where most epochs of training is done in low precision and the final one or few epochs in higher precision. For CNNs, training in HBFP4 with last epoch in HBFP6 recovers FP32 accuracy. However, for Transformer there was still a gap between this setup and FP32.
strength_and_weaknesses: Strengths:
1. Paper was easy to understand
2. HBFP seems like a interesting data format with potential

Weaknesses:
1. The paper is entirely empirical. All conclusions are drawn largely from experiments on CIFAR-10/100 using ResNets and DenseNet40. One 1 table had data on transformers. This is a very weak set of benchmarks and models for an empirical paper.
2. The results are not too exciting. For CNNs, the first and last layers must be kept in FP32 - so HBFP is only used for the "easy to quantize" layers. For the Accuracy Booster technique, it doesn't seem to work well on transformer.
3. The paper combines two existing techniques: HBFP and FracTrain. The techniques are mostly orthogonal and not combined in a novel or insightful way. I didn't get much out of the paper besides "certain configurations of HBFP4/5/6 can be used to train CNNs", which isn't all that surprising.
clarity,_quality,_novelty_and_reproducibility: Questions:
1. Which axis of the CNN tensors is used for exponent sharing? I assume it's input channels but I would like a confirmation.
2. Why was a 10-bit exponent used for HBFP? Why not an 8-bit exponent like FP32 and bfloat16? This would make it easier for both software and hardware to operate between these different formats.
3. Why does using unnecessarily large block size hurt the model accuracy? I assume tensors would be padded with zeros to meet the block size requirement, which shouldn't affect accuracy at all.
4. Why use non-power of two block sizes like 49? A power of two block size makes sense since doing matrix multiplies usually requires an adder tree in hardware. This tree is naturally power of two sized.

Table 3 should say "Epochs using HBFP6".
summary_of_the_review: A purely empirical paper that studies the HBFP format for DNN training. The paper does not have much novelty and the experimental section is weak, with data mostly coming from ResNets trained on CIFAR-10/100. The main takeaway is that training with HBFP5 works fine for CNNs. If training with HBFP4, the last few epochs should be done in higher precision like HBFP6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the HBFP parameter search space to further improve efficiency and density in hardware accelerators.
They propose training on HBFP6, after experiments making the trade off: lower block size gives more accuracy, but is less HW efficient.
They show accuracy booster method that switch number of mantissa bits in different epochs in the training stage. They evaluate their experiments on CNN and transformers for cifar10/100 and wmt16.
strength_and_weaknesses: The paper is well written and clear. The paper shows comparison between various HBFP configurations and presents accuracy booster method


Few points that could improve the paper:

It would be interesting to see trade-off accuracy and hardware efficiency when the variable block size is across different layers.

The paper should also evaluate on bigger dataset: Imagenet and updated model: ViT

Table 1: should have HBFP8 for comparison 

HBFP6 with block size 1, is same as having 16bit for every value, wouldn't FP16 with mixed-precision training be better or equivalent in terms of accuracy? 

In table1, HBFP6 doesnt achieve same accuracy of FP32 model. Thus, the intro's claim should be revisited: "We show that HBFP6 is the smallest HBFP format achieving FP32 accuracy".

In figure2, should add some form to show power savings while using accuracy booster.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and the work is well explained. 
The paper presents original incremental work upon previous research on HBFP.
Code was provided and the experiments can be reproduced.  
Some improvements on figure2, could make the papers better: figure2 should highlight the last 10 epochs to show accuracy benefit from accuracy booster.
Table 1 should add flop to train fp32 for ease of comparison.

summary_of_the_review: Overall the paper is well written and clear to understand.
The paper should show additional data or experiments that will make the claims and proposed methods even stronger.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:

This paper investigated a proper bit-precision for a block floating-point format for deep neural network training and revealed that training with a small number of mantissa bits can be compensated if the training iterations with a large mantissa bit are followed. The authors also investigate the interplay between the block size and the mantissa bits for the block floating-point performance. The authors provided the experimental results to empirically support their claims.

Strengths:

- an interesting observation that the accuracy degradation by aggressively reduced mantissa bits can be recovered by the training with the increased number of mantissa bits.
- a good prior art section 
- The authors provide two new insights, 1) HBFP6 can achieve high accuracy for a range of networks and 2) Accuracy boosters is a reliable way to get higher accuracy while spending lesser energy. 
- Good background section that clearly explains the use of HBFP
- Intuitive explanation of mixed precision training in the context of HBFP with proper experimental backing
- Paper was easy to understand
- HBFP seems like a interesting data format with potential

Weaknesses:

- This paper is mainly about the presentation of empirical findings, and many of them are already well-known (e.g., full precision for the first/last layers.)
- There are no deeper insights into the interplay between the block size and the mantissa bits.
- the figure of merit used by the authors, Si area, is not very useful as even a single usage of a unique data format requires associated hardware which if custom designed will come with its own area. Energy efficiency would have been a more useful metric. 
- the analysis of Accuracy boosters is limited and could use more experiments.
- It is unclear why training at the very end using HBFP6 allows you to recover the accuracy loss, more explanation would have been better
- The paper is entirely empirical. All conclusions are drawn largely from experiments on CIFAR-10/100 using ResNets and DenseNet40. One 1 table had data on transformers. This is a very weak set of benchmarks and models for an empirical paper.
- The results are not too exciting. For CNNs, the first and last layers must be kept in FP32 - so HBFP is only used for the "easy to quantize" layers. For the Accuracy Booster technique, it doesn't seem to work well on transformer.
- The paper combines two existing techniques: HBFP and FracTrain. The techniques are mostly orthogonal and not combined in a novel or insightful way.

Recommendation:

A majority of reviewers vote for rejection. I, therefore, recommend to reject the paper. I encourage the authors to use the feedback provided to improve the paper and resubmit to another venue.

justification_for_why_not_higher_score: Two reviewers voted strongly for rejection.
Decision: Reject