Title: Rethinking Numerical Representations for Deep Neural Networks
Authors: Parker Hill
Babak Zamirai
Shengshuo Lu
Yu-Wei Chao
Michael Laurenzano
Mehrzad Samadi
Marios Papaefthymiou
Scott Mahlke
Thomas Wenisch
Jia Deng
Lingjia Tang
Jason Mars
Venue: ICLR 2017
Keywords: Deep learning

Abstract: With ever-increasing computational demand for deep learning, it is critical to investigate the implications of the numeric representation and precision of DNN model weights and activations on computational efficiency. In this work, we explore unconventional narrow-precision floating-point representations as it relates to inference accuracy and efficiency to steer the improved design of future DNN platforms.  We show that inference using these custom numeric representations on production-grade DNNs, including GoogLeNet and VGG, achieves an average speedup of 7.6x with less than 1% degradation in inference accuracy relative to a state-of-the-art baseline platform representing the most sophisticated hardware using single-precision floating point.  To facilitate the use of such customized precision, we also present a novel technique that drastically reduces the time required to derive the optimal precision configuration.

TL;DR: We find that the optimal numerical representation for large-scale DNNs is very different than the small-scale ones that are used in current DNN hardware research.

Review discussion:
Did you try training these networks using these customized precisions too ? It would be a bonus if the same hardware could be used for training.
We briefly explored training DNNs with customized precision and found that training requires more precision than inference, since training makes minute adjustments to DNN weights until they converge to their final values. A complete sweep of the customized-precision design space for inference already requires a large amount of time (i.e. the product of several DNNs, hundreds of customized-precision design points, millions of input images, and orders-of-magnitude slowdown from hardware emulation). The same study for training would require further order-of-magnitude increase in experiment runtime, since training is an iterative process.
Could you please elaborate on the link with other architecture such as XNOR-NET by Rastegari and other related works (binarizing or quantizing, such as Gong et al.), aiming at compressing networks weights or activations?
===Comparison with binary DNNs===
Customized-precision DNNs provide transparent, flexible benefits to existing DNN models by removing inconsequential bits. XNOR-Net[1] and similar works[2,3] operate on binary computational primitives that require changes to the DNN architecture and are inflexible (i.e. either on or off). These problems are manifested when applied to large DNNs.
For example, the state-of-the-art XNOR-Net technique applied to GoogleNet does not converge (XNOR-Net[1], table 2), while our customized-precision approach yields over 5x speedup with less than 1% loss in accuracy.
===Comparison with DNN compression===
Our customized-precision DNNs use less precision for computation and storage, while previous works that evaluate compressed DNNs[4,5,6] use less precision for storage *without* optimizing the compute units. Computation precision tuning allows quadratic improvement in throughput due to smaller hardware designs (more parallelism) and shorter circuit delay (higher frequency). Smaller numeric representations in storage are limited to a linear increase in throughput, since storage and bandwidth scale linearly with the size of the data.
[1] XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. ECCV'16. Rastegari, et al.
[2] BinaryConnect: Training Deep Neural Networks with Binary Weights during Propagations. NIPS'15. Courbariaux, et al.
[3] Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation.  arXiv'15. Cheng, et al.
[4] Compressing Deep Convolutional Networks using Vector Quantization. arXiv'14. Gong, et al.
[5] EIE: Efficient Inference Engine on Compressed Deep Neural Network. ISCA'16. Han, et al.
[6] Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. ICLR'16. Han, et al.
The paper provides a first study of customized precision hardware for large convolutional networks, namely alexnet, vgg and googlenet. It shows that it is possible to achieve larger speed-ups using floating-point precision (up to 7x) when using fewer bits, and better than using fixed-point representations.
The paper also explores predicting custom floating-point precision parameters directly from the neural network activations, avoiding exhaustive search, but i could not follow this part. Only the activations of the last layer are evaluated, but on what data ? On all the validation set ? Why would this be faster than computing the classification accuracy ?
The results should be useful for hardware manufacturers, but with a catch. All popular convolutional networks now use batch normalization, while none of the evaluated ones do. It may well be that the conclusions of this study will be completely different on batch normalization networks, and fixed-point representations are best there, but that remains to be seen. It seems like something worth exploring.
Overall there is not a great deal of novelty other than being a useful study on numerical precision trade-offs at neural network test time. Training time is also something of interest. There are a lot more researchers trying to train new networks fast than trying to evaluate old ones fast.
I am also no expert in digital logic design, but my educated guess is that this paper is marginally below the acceptance threshold.
We thank you for your valuable time and comments.
===Search method clarifications
Our search model only requires a small subset (evaluation uses 10) of the validation inputs to predict accuracy by leveraging all of the activations in the last layer, rather than observing only the top-1 (or top-5) results.
Last layer activations require fewer inputs compared to top-1 accuracy, since each DNN input produces many scalar activations rather than a single binary correctness value.
For example, 10 inputs using top-1 accuracy provides 10 binary values (i.e. each DNN output being either correct or incorrect), while the activations for 10 ImageNet inputs provides 10,000 scalar values (i.e. 10 DNN inputs * 1000 scalar output activations per ImageNet input).
===Batch normalization
Batch normalization primarily impacts the training process of DNNs rather than inference, so we expect to arrive at very similar conclusions with or without it. Batch normalization during inference applies a fixed linear transformation to each activation, which requires few hardware resources and does not change the shape of the activation distribution.
===Importance of inference performance
Reducing the computational requirements of DNN inference is relevant to the machine learning community. A number of papers on this topic have been published at venues such as ICLR, ICML, and NIPS[1,2,3,4].
[1] Compressing Neural Networks with the Hashing Trick. ICML 2015. Chen, at al.
[2] Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation. NIPS 2014. Denton, et al.
[3] Learning both Weights and Connections for Efficient Neural Networks. NIPS 2015. Han, et al.
[4] Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. ICLR 2016. Han, et al.
This paper explores the performance-area-energy-model accuracy tradeoff encountered in designing custom number representations for deep learning inference. Common image-based benchmarks: VGG, Googlenet etc are used to demonstrate that fewer than1 6 bits in a custom floating point representation can lead to improvement in runtime performance and energy efficiency with only a small loss in model accuracy.
Questions:
1. Does the custom floating point number representation take into account support for de-normal numbers?
2. Is the custom floating point unit clocked at the same frequency as the baseline 32-bit floating point unit? If not, what are the different frequencies used and how would this impact the overall system design in terms of feeding the data to the floating point units from the memory
Comments:
1. I would recommend using the IEEE half-precision floating point (1bit sign, 5bit exponent, and 10bit mantissa) as a baseline for comparison. At this point, it is well known in both the ML and the HW communities that 32-bit floats are an overkill for DNN inference and major HW vendors already include support for IEEE half-precision floats.
2. In my opinion, the claim that switching to custom floating point  lead to a YY.ZZ x savings in energy is misleading. It might be true that the floating-point unit itself might consume less energy due to smaller bit-width of the operands, however a large fraction of the total energy is spent in data movement to/from the memories.
As a result, reducing the floating point unit’s energy consumption by a certain factor will not translate to the same reduction in the total energy. A reader not familiar with such nuances (for example a typical member of the ML community), may be mislead by such claims.
3. On a similar note as comment 2, the authors should explicitly mention that the claimed speedup is that of the floating point unit only, and it will not translate to the overall workload speedup.
Although the speedup of the compute unit is roughly quadratic in the bit-width, the bandwidth requirements scale linearly with bit-width.
As a result, it is possible that these custom floating point units may be starved on memory bandwidth, in which case the claims of speedup and energy savings need to be revisited.
4. The authors should also comment on the complexities and overheads introduced in data accesses, designing the various system buses/ data paths when the number representation is not byte-aligned. Moving to a custom 14-bit number representation (for example) can improve the performance and energy-efficiency of the floating point unit, but these gains can be partially eroded due to the additional overhead in supporting non-byte aligned memory accesses.
We thank you for your valuable time and comments.
===Denormal floating-point units
Our evaluated floating-point ALUs do not operate on denormal floating-point representations. Aside from minor differences in empirical results that depend on denormal floating-point arithmetic support, we expect that our conclusions hold. A different design that operates on denormal floating-point representations will, ideally, allow one less bit in the floating-point exponent for the same accuracy, but will be disadvantaged by increased hardware area, circuit delay, and power.
===Frequency scaling of customized-precision units
We scale the ALU frequency with the inverse of the hardware circuit delay, so each customized-precision design can have a different frequency.
===Frequency of compute compared to storage
Our DRAM memory and compute clocks (frequencies) are different, which is common across almost all hardware designs (e.g., modern CPU/GPU designs). DRAM memory fabrication is primarily optimized for higher transistor density (i.e. memory capacity) rather than higher frequency, which is opposite of computational units.
===Validity of MAC results
The MAC operations capture the majority of the DNN performance and power breakdown [1]. Other units will scale with the customized-precision design as well, so we expect that a full-system implementation would show benefits very similar to our results.
For example, using 84% [1] as the power used by compute, we find our reported 3.4x savings in energy would become 3.15x (time = 0.84/3.4 + 0.16/(32-bit/14-bit from linear scaling) = 0.31 => speedup = 1/0.31 = 3.15x).
===Compute throughput as DNN bottleneck
The DRAM memory bandwidth requirements for DNNs is much lower than its computational requirements [2]. This is due to matrix multiplication, the central DNN computational kernel, requiring roughly N^2 memory operations (i.e. from loop tiling [3]) compared to N^3 arithmetic operations.
===Customized-precision memory access
DRAM memory access can be left unchanged when using customized-precision compute units. Similar to how GPUs do 32-bit computation efficiently when using a 128- to 512-bit memory bus, a single memory access is distributed to multiple compute units.
For example, a 128-bit bus provides data to nine 14-bit compute units.
[1] ShiDianNao: Shifting Vision Processing Closer to the Sensor. ISCA 2015. Zidong Du, et al.
[2] DjiNN and Tonic: DNN as a Service and Its Implications for Future Warehouse Scale Computers. ISCA 2015. Hauswald, et al.
[3] Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks. FPGA 2015.
The paper studies the impact of using customized number representations on accuracy, speed, and energy consumption of neural network inference. Several standard computer vision architectures including VGG and GoogleNet are considered for the experiments, and it is concluded that floating point representations are preferred over fixed point representations, and floating point numbers with about 14 bits are sufficient for the considered architectures resulting in a small loss in accuracy.
The paper provides a nice overview of floating and fixed point representations and focuses on an important aspect of deep learning that is not well studied. There are several aspects of the paper that could be improved, but overall, I am leaned toward weak accept assuming that the authors address the issues below.
1- The paper is not clear that it is only focusing on neural network inference. Please include the word "inference" in the title / abstract to clarify this point and mention that the findings of the paper do not necessarily apply to neural network training as training dynamics could be different.
2- The paper does not discuss the possibility of adopting quantization tricks during training, which may result in the use of fewer bits at inference.
3- The paper is not clear whether in computing the running time and power consumption, it includes all of the modules or only multiply-accumulate units?
Also, how accurate are these numbers given different possible designs and the potential difference between simulation and production? Please elaborate on the details of simulation in the paper.
4- The whole discussion about "efficient customized precision search" seem unimportant to me.
When such important hardware considerations are concerned, even spending 20x simulation time is not that important. The exhaustive search process could be easily parallelized and one may rather spend more time at simulation at the cost of finding the exact best configuration rather than an approximation.
That said, weak configurations could be easily filtered after evaluating just a few examples.
5- Nvidia's Pascal GP100 GPU supports FP16. This should be discussed in the paper and relevant Nvidia papers / documents should be cited.
More comments:
- Parts of the paper discussing "efficient customized precision search" are not clear to me.
- As future work, the impact of number representations on batch normalization and recurrent neural networks could be studied.
We thank you for your valuable time and comments.
===Performance and power methodology
We evaluate our results based on the multiply-accumulate (MAC) operations, the key building block of any hardware implementation of a DNN. The MAC operations capture the majority of the DNN performance and power breakdown [1]. Other units will scale with the customized-precision design as well, so we expect that a full-system implementation would show benefits very similar to our results.
For example, using 84% [1] as the power used by compute, we find our reported 3.4x savings in energy would become 3.15x (time = 0.84/3.4 + 0.16/(32-bit/14-bit from linear scaling) = 0.31 => speedup = 1/0.31 = 3.15x).
Similarly, full-system performance is dictated by MAC performance.
===ASIC design tool details
Our comparisons are made using designs synthesized with Synopsys Design Compiler and are further verified using Synopsys PrimeTime and SPICE simulations. The specific cell library, provided by the manufacturer, which produces area, power, and timing information, cannot be disclosed. These cell libraries are used for timing verification for chip tape out, hence, they must (and do) accurately model real hardware.
===Importance of search
Our search method’s 170x speedup mitigates the time-intensive process of emulating customized precision operations, allowing researchers to iteratively adjust DNN topology for optimized hardware efficiency.
===Suggestions for future work and clarifications
Thank you for the suggestions. We will integrate this feedback into future versions of our work.
[1] ShiDianNao: Shifting Vision Processing Closer to the Sensor. ISCA 2015. Zidong Du, et al.

Meta-review:
role: PC
data: comment: The reviewers feel that this is a well written paper on floating and fixed point representations for inference with several state of the art deep learning architectures. At the same time, in order for results to be more convincing, they recommend using 16-bit floats as a more proper baseline for comparison, and to analyze tradeoffs in overall workload speedup, i.e broader system-level issues surrounding the implementation of custom floating point units.

Decision: Reject