Title: Bit-Pragmatic Deep Neural Network Computing
Authors: Jorge Albericio
Patrick Judd
Alberto Delmas
Sayeh Sharify
Andreas Moshovos
Venue: ICLR 2017
Keywords: Deep learning
Applications

Abstract: We quantify a source of ineffectual computations when processing the multiplications of the convolutional layers in Deep Neural Networks (DNNs) and propose Pragrmatic (PRA), an architecture that exploits it improving performance and energy efficiency. 
The source of these ineffectual computations is best understood in the context of conventional multipliers which generate internally multiple terms, that is, products of the multiplicand and powers of two, which added together produce the final product. At runtime, many of these terms are zero as they are generated when the multiplicand is combined with the zero-bits of the multiplicator. While conventional bit-parallel multipliers calculate all terms in parallel to reduce individual product latency, PRA calculates only the non-zero terms resulting in a design whose execution time for convolutional layers is ideally proportional to the number of activation bits that are 1. Measurements demonstrate that for the convolutional layers on Convolutional Neural Networks and during inference, PRA improves performance by 4.3x over the DaDiaNao (DaDN) accelerator and by 4.5x when DaDN uses an 8-bit quantized representation. DaDN was reported to be 300x faster than commodity graphics processors. 



TL;DR: A hardware accelerator for DNNs whose execution time for convolutional layers is proportional to the number of activation *bits* that are 1.

Review discussion:
We added an appending with the essential bit distributions but I mislabeled the graphs in that revision. We updated the original PDF with the correct labels and explanation. The main paper body remains as it was in Nov 4.
An interesting idea, and seems reasonably justified and well-explored in the paper, though this reviewer is no expert in this area, and not familiar with the prior work.
Paper is fairly clear. Performance evaluation (in simulation) is on a reasonable range of recent image conv-nets, and seems thorough enough.
Rather specialized application area may have limited appeal to ICLR audience. (hence the "below threshold rating", I don't have any fundamental structural / methodological criticism for this paper.)
Improve your bibliography citation style - differentiate between parenthetical citations and inline citations where only the date is in parentheses.
Thank you for your comment identifying this as interesting work. We believe that this work opens up new opportunities for the ML community and we do demonstrate one such opportunity in the paper: trim precision and boost performance even further and beyond what was previously possible (see Stripes reference). Other opportunities for investigation exist such as a adjusting activation values and considering the implications for training. To the best of our knowledge, there is no other accelerator whose performance depends primarily on the 1 bit content of activations.
Regarding your recommendation and given that the CFP includes " Implementation issues, parallelization, software platforms, ***hardware***" could you please reconsider whether the best way to communicate your opinion on whether this fits with the conference is to rate it below acceptance? Wouldn't it best to rank the paper first on its technical merit and then have a discussion on what is the interpretation.vision behind the inclusion of the term "hardware" in the ICLR call for papers for the future of ICLR?
IMHO collaboration between the ML and the computer hardware community is important for further innovation on both domains. Computing hardware performance is not going to improve anymore as it used to and specialized designs seem to be the only viable way forward from a hardware perspective. One way to foster this cross-discipline work is by having works such as pragmatic appear and become known to the ICLR community. Pragmatic may be a victim of its own success: it offers nearly 4x speedup over the faster previously proposed accelerator (which itself was claimed to be 300x faster than commodity GPUs) without requiring any changes to the network.
Interesting and timely paper. Lots of new neural network accelerators popping up.
I'm not an expert in this domain and to familiarize myself with the topic, I browsed through related work and skimmed the DaDianNao paper.
My main question is about the choice of technology. What struck me is that your paper contains very few implementation details except for the technology (PRA 65nm vs DaDianNao 28nm).
Combined with the fact that the main improvement of your work appears to be performance rather than energy efficiency, I was wondering about the maximum clock estimated frequency of the PRA implementation due to the added complexity? Based on the explanation in the methodology section, I assume that the performance comparison is based on number of clock cycles. Do you have any numbers/estimates about the performance in practices (taking into account clock frequency)?
Thank you for your comment and question, we will revise to clarify: In short, both designs were synthesized to operate at 980Mhz as limited by the access latency of the eDRAM blocks implementing the SB and the NM.
Accordingly, a comparison in cycles is equivalent to a comparison in time. The key contribution here is the idea of processing only the essential bits in a practical manner.
Further optimizations at the circuit level should be possible.
Longer version: Pragmatic compared to DaDN omits the multipliers thurs reducing cost per output term.
However, since it processes a bit a time, it needs more parallelism.
Hence the need for more of these simpler units. The key challenge as far as operating frequency is concerned with Pragmatic is the communication cost over the relatively longer wires. Fortunately, since there is lots of parallelism even within the computations of each output activation, the design can be pipelined avoiding any increase in clock cycle. It is for this reason why Pragmatic can match the latency of DaDN.
We used the best technology that was available to us (65nm). We fully expect the design to be synthesizable in a better logic technology (e.g., 28nm). The synthesis tools can easily pipeline away any delays given the lack of cross-cycle dependencies and similarly a designer can also pipeline the design at various levels.
For example, fetching the next set of weights while processing the current one, or even fetching two sets of weights ahead if necessary.
The paper presents a hardware accelerator architecture for deep neural network inference, and a simulated performance evaluation thereof. The central idea of the proposed architecture (PRA) revolves around the fact that the regular (parallel) MACC operations waste a considerable amount of area/power to perform multiplications with zero bits. Since in the DNN scenario, one of the multiplicands (the weight) is known in advance, the multiplications by the zero digits can be eliminated without affecting the calculation and lowest non-zero bits can be further dropped at the expense of precision. The paper proposes an architecture exploiting this simple idea implementing bit-serial evaluation of multiplications with throughput depending on the number of non-zero bits in each weight.
While the idea is in general healthy, it is limited to fixed point arithmetics. Nowadays, DNNs trained on regular graphics hardware have been shown to work well in floating point down to single (32bit) and even half-precision (16bit) in many cases with little or no additional adjustments.
However, this is generally not true for 16bit (not mentioning 8bit) fixed point. Since it is not trivial to quantize a network to 16 or 8 bits using standard learning, recent efforts have shown successful incorporation of quantization into the training process. One of the extreme cases showed quanitzation to 1bit weights with negligible loss in performance (arXiv:1602.02830). 1-bit DNNs involve no multiplication at all;
moreover, the proposed multiplier-dependent representation of multiplication discussed in the present paper can be implemented as a 1-bit DNN. I think it would be very helpful if the authors could address the advantages their architecture brings to the evaluation of 1-bit DNNs.
To summarize, I believe that immediately useful hardware DNN accelerators still need to operate in floating point (a good example are Movidius chips -- nowadays Intel). Fixed point architectures promise additional efficiency and are important in low-power applications, but they depend very much on what has been done at training. In view of this -- and this is my own extreme opinion -- it makes sense to build an architecture for 1-bit DNNs. I have the impression that the proposed architecture could be very suitable for this, but the devil is in the details and currently evidence is missing to make such claims.
Thank you for your encouraging comments and all the best for 2017. Please consider the following:
1. The approach presented could be valuable for floating-point as well.
In fact, our next step is to try to apply it to training and FP. Here's why it can be used for FP as well: When doing multiplication in FP, the exponents are added and the mantissas are multiplied. Even for single-precision FP, that would be a 23bx23b multiplication which would take much longer than adding  the exponents. Our approach would reduce the time needed to perform these operations. A proper study would be needed to determine the performance and energy characteristics of such a design.
2. Most work on acceleration for inference uses 16-bit fixed point at present, and many end-users will be performing primarily classification and in many cases on mobile and embedded platforms.
So, while our design target is primarily these systems, the user base will be very large.
3. As you appropriately point out, there is work that suggests that even smaller precision and at the extreme, single-bit arithmetic may be usable. In our work we show that the bit-pragmatic approach offers great benefits for two commonly used platforms: 16-bit fixed-point and 8-bit quantized tensorflow-like. The approach itself combines judicious use of parallelism to maintain wide memory references, 2-level shifting to reduce area costs, and on-the-fly recoding to save computations. We would like to think that these concepts can be applicable to other architectures and thus believe that it would be valuable to have the paper presented.
Moreover, the proposed architecture opens up new opportunities for network design where the precision and the values can be tuned to improved performance and energy efficiency.
We are reluctant to complete the reviewer ratings as stated as we think we would mischaracterize the reviews. For the most part, the reviews were concerned with the fit to ICLR (this is for you to interpret what the inclusion of "hardware" in the CFP means). The technical comments we addressed below.
The main take away for the ML community here is that it is possible to improve performance and energy efficiency by taking advantage of the distribution of zero bits in the activation stream. This opens up additional opportunities for exploration such as adjusting the precision to zero out portion of the activations, or adjusting the activation function to avoid numbers with many bits that are 1. We presented one application where the precisions were adjusted to get an additional boost in performance. We hope that this work will serve as motivation for followup work in hardware (better encoding) and in network design.
The technique was presented and evaluated for fixed-point hardware but it is applicable to other representations such as floating point (which would require further investigation).
At the hardware level this is a unique design that exploits value content to reduce computations. We do show that there is a heavy bias toward zero bits. We also present a wide parallel engine that exploits shift-and-add units in a practical manner getting much of the benefit that is possible. The key challenge in hardware is that the straightforward application of shift-and-add results in an impractically large, power inefficient design. We proposed techniques to overcome these. To the best of our knowledge this is the first design of a wide data-parallel engine whose performance varies with the zero bit content of the inputs.
The area and energy efficiency results are based on  post-synthesis layout and were collected using actual activity measurements.
My apologies for the confusion. I posted a comment meant for a different submission here. I deleted the comment and restored the paper.
We updated the paper to present a new enhanced encoding that further boosts performance at almost no hardware cost. We would like to clarify that the baseline designs evaluated do include the dispatcher cost.

Meta-review:
role: PC
data: comment: Unfortunately, none of the reviewers, nor the AC, have strongly supported for the acceptance of this paper. The fact that fixed-point arithmetic is the focus of this work, while floating-point arithmetic is much more common, is also a concern. The PCs thus don't think this work can be accepted to ICLR. However, we are happy to invite the authors to present their work at the Workshop Track.

Decision: Invite to Workshop Track