Title: Tartan: Accelerating Fully-Connected and Convolutional Layers in Deep Learning Networks by Exploiting Numerical Precision Variability
Authors: Alberto Delmás Lascorz
Sayeh Sharify
Patrick Judd
Andreas Moshovos
Venue: ICLR 2017
Keywords: Deep learning
Applications

Abstract: Tartan {TRT} a hardware accelerator for inference with Deep Neural Networks (DNNs) is presented and evaluated on Convolutional Neural Networks. TRT exploits the variable per layer precision requirements of DNNs to deliver execution time that is proportional to the precision p in bits used per layer for convolutional and fully-connected layers. Prior art has demonstrated an accelerator with the same execution performance only for convolutional layers. Experiments on image classification CNNs show that on average across all networks studied,  TRT outperforms a state-of-the-art bit-parallel accelerator by 1.90x without any loss in accuracy while it is 1.17x more energy efficient. TRT requires no network retraining while it enables trading off accuracy for additional improvements in execution performance and energy efficiency. For example, if a 1% relative loss in accuracy is acceptable, TRT is on average 2.04x faster and 1.25x more energy efficient than the bit-parallel accelerator.
This revision includes post-layout results and a better configuration that processes 2bits at time resulting in better efficiency and lower area overhead.

TL;DR: A hardware accelerator whose execution time for Fully-Connected and Convolutional Layers  in CNNs vary inversely proportional with the number of bits used to represent the input activations and/or weights.

Review discussion:
The paper provides insights into hardware and circuit design. My question is what is the take-away for the machine learning community? How to train such low precision networks that work well on the specialized hardware? Is there any plan to open source the modified caffe framework?
On the hardware perspective, how to justify the 1.40× increase in area to trade-off for 1.90× speedup and 1.17× energy efficiency? The energy efficiency improvement looks marginal. Does this gain come from low precision model or come from specailzied hardware?
Thank you for your question, our answer follows but please do follow up with any additional clarification questions especially if we misinterpreted your questions:
Summary:
* Speedup vs. a commodity GPU is in the 600x range and energy efficiency in the 250x range.
* There is no need to train explicitly for TARTAN. To improve performance, we used existing pre-trained networks and found per layer precisions that maintained accuracy. For this we exploited the observation that in practice the precision required: 1) is lower than the precision supported by commodity hardware, and 2) varies across the network (we used per layer precisions here).
* TARTAN can run unmodified networks at par with DaDianNao.
* TARTAN provides a new degree of freedom and a new direction that we hope the ML community can further exploit when designing networks to boost performance. Below we enumerate several directions for further investigation by the ML community. The paper demonstrates one such possibility.
* The area vs. performance and energy trade-off is very favorable as typically in hardware this trade-off is sublinear, i.e., less than 2x performance for 2x area and 2x energy.
* Performance comes from exploiting the combination of specialized hardware and of a deep neural network property: the precision needed by DNNs varies significantly per layer and at a fine granularity (e.g., 13b or 9b or 5b). This is not possible with commodity hardware which only supports a few datatypes (e.g., 8b, 16b, or 32b).
* Some tools we have already available online, others will follow. See below.
In detail:
**** Take-away for the ML community:
TARTAN can translate even minute reductions in precision to performance: Commodity hardware offers only a few precisions (e.g., 8b, 16b or 32b) whereas TARTAN naturally supports *all* precisions at a bit granularity, e.g., 1b to 16b in this work.
For the ML community the question is whether and how to exploit this new flexibility.  We investigated one possibility: We took *out of the box* networks that were pre-trained on commodity floating-point hardware and found precision configurations (see https://arxiv.org/abs/1511.05236 for the base method -- here we trim both activations and weights together) that maintained top-1 accuracy vs. floating-point while running over fixed-point hardware with a maximum precision of 16 bits. We did not retrain for TARTAN. Even without retraining the average speedup was 1.9x over DaDianNao.
Several directions for further investigation by the ML community exist. With the understanding that these are presently just directions for further investigation here are a few:
1. Train or retrain for TARTAN to get further performance.
For example, we already have configurations that lead to better performance at a loss of accuracy. We suspect that retraining for these precision configurations, would reduce or even eliminate the accuracy loss. Past experience has shown that training for specific hardware features  can preserve accuracy, e.g., https://arxiv.org/abs/1602.02830, https://arxiv.org/abs/1511.00363, https://arxiv.org/abs/1510.00149.
2. It may be possible to further boost performance by rethinking network design knowing that even small differences in precision will lead to performance improvements.
3. Finer granularity for precision control: We assumed per layer precisions.
However, the hardware is capable of adjusting precision on a per activation/weight group. Could we then adjust precision at a finer granularity? Or could we adjust it on-the-fly?
We hope that you will find this a worthwhile direction to investigate further and thus see the value in making the ML community aware of the TARTAN design.
**** Where do the gains come from:
A combination of hardware design and of exploiting the precision tolerance/requirements of networks. Co-designing networks and hardware is commonly explored. TARTAN is less invasive as it does not require retraining or redesigning of the networks but can still benefit from doing so. Commodity hardware does not benefit from fine precision control.
**** Is a 1.90x performance gain justified given the 1.40x area cost?
Absolutely: performance usually scales sublinearly with area.
For example a 16-core system will typically not be 2x faster than an 8-core one for most applications even when there is lots of parallelism. Tartan offers superlinear speedup vs. area instead.
****  Performance and Energy Efficiency:
TARTAN is 1.90x faster than DaDianNao which itself was estimated to be ~300x faster than a GPU.
Thus, we are looking at roughly a 670x speedup over commodity hardware.  Energy efficiency is also very favorable: roughly speaking, the Tegra K1 GPU  gets 0.73 GOPs/W whereas DadianNao gets 12.8 GOPs/W, or ~17.5x (see https://media.nips.cc/Conferences/2015/tutorialslides/Dally-NIPS-Tutorial-2015.pdf page 109). The 1.17x we report is over DaDianNao.
We will release our tools. The modified Caffee for testing precision configurations is here: https://github.com/patrickjudd/caffe . The code was modified to enable measuring classification accuracy with fixed-point precisions at a bit granularity. We will cleanup and release our scripts for searching the precision configuration space and the performance models.
The paper is closely related to Judd et al 2016. How does it compare in terms of efficiency ? That comparison seems to be missing.
The main novelty over Judd et al seems to be handling fully connected layers but recent networks have basically just convolutional layers (googlenet, resnet, etc.). Any comments on that ?
Thank you for your questions. We provide the requested measurements which we didn't include as we wanted to keep the manuscript as close as possible to the suggested page count. We will include as appendix. We also provide some reasons why extending Stripes with Tartan is a good design choice given the additional flexibility, low cost, performance and energy efficiency improvements for FC layers and the importance of FC layers for other networks. In more detail:
**** Compared to STR (Judd et al 2016), TRT:
- Is as fast in convolutional layers (CLs)
-Is 61% faster on fully connected layers (FCLs)
-Has an area overhead of 14%
-Uses 3.6% more energy for CLs, a very low overhead and still remains much lower than DaDN.
-Uses 21.4% less energy for FCLs
**** Are FCLs important?
1.
While for image classification CNNs CLs dominate the execution time, accelerators like STR speedup them up, and according to Amdahl's law increase the relative importance of FCLs. CLs are executed 2x faster with SRT, and 4x on top with the Bit-Pragmatic accelerator https://arxiv.org/abs/1610.06920. Tartan is compatible with both.
For example, if FCLs account for 10% of the execution and CLs for 90%, if CLs are accelerated by 90%, the 10% of FCLs is now 31%.
2. Other networks, even on the image recognition domain, do make heavier use of fully connected layers (for example, long short-term memory caption-generating networks such as NeuralTalk http://arxiv.org/abs/1412.2306). Speech recognition tasks also use fully connected layers, for example Deep Speech model consists of five fully-connected layers of 2048 neurons each with one bidirectional recurrent layer (https://arxiv.org/pdf/1412.5567v2.pdf )
**** Why extending STR with TARTAN is important:
3. The area and power overhead over STR are a very small price to pay in exchange for:
a) Making STR much more compelling by enabling acceleration for more layer types, and  b) reducing the STR’s energy overhead  in these layers when compared to DaDN.
4. Tartan uses lower precision parameters and thus larger networks can fit on chip, which is especially important as FC layers dominate parameter size even on mostly convolutional networks.
5. Making fully connected layers less expensive also enables more network design choices.
This paper proposed a hardware accelerator for DNN. It utilized the fact that DNN are very tolerant to low precision inference and outperforms a state-of-the-art bit-parallel accelerator by 1.90x without any loss in accuracy while it is 1.17x more energy efficient. TRT requires no network retraining. It achieved super linear scales of performance with area.
The first concern is that this paper doesn't seem very well-suited to ICLR. The circuit diagrams makes it more interesting for the hardware or circuit design community.
The second concern is the "take-away for machine learning community", seeing from the response, the take-away is using low-precision to make inference cheaper. This is not novel enough. In last year's ICLR, there were at least 4 papers discussing using low precision to make DNN more efficient. These ideas have also been explored in the authors' previous papers.
Thank you for your review. The CFP lists"hardware" as one of the topics.
To the best of our knowledge there is no other work that exploits precision in the way Stripes and Tartan does. Performance here scales at the granularity of a single bits. The relation to prior work from our own group is explained in detail the paper. Stripes does not improve performance nor energy efficiency for FC layers. No other accelerator exploits the full range of precision nor does it a programmable way.
I do not feel very qualified to review this paper. I studied digital logic back in university, that was it. I think the work deserves a reviewer with far more sophisticated background in this area. It certainly seems useful. My advice is also to submit it another venue.
This seems like a reasonable study, though it's not my area of expertise. I found no fault with the work or presentation, but did not follow the details or know the comparable literature.
There seem to be real gains to be had through this technique, though they are only in terms of efficiency in hardware, not changing accuracy on a task. The tasks chosen (Alexnet / VGG) seem reasonable. The results are in simulation rather than in actual hardware.
The topic seems a little specialized for ICLR, since it does not describe any new advances in learning or representations, albeit that the CFP includes "hardware".  I think the appeal among attendees will be rather limited.
Please learn to use parenthetical references correctly. As is your references make reading harder.
Summary:
The paper describes how the DaDianNao (DaDN) DNN accelerator can be improved by employing bit serial arithmetic.  They replace the bit-parallel multipliers in DaDN with multipliers that accept the weights in parallel but the activations serially (serial x parallel multipliers).  They increase the number of units keeping the total number of adders constant.  This enables them to tailor the time and energy consumed to the number of bits used to represent activations.  They show how their configuration can be used to process both fully-connected and convolutional layers of DNNs.
Strengths:
Using variable precision for each layer of the network is useful - but was previously reported in Judd (2015)
Good evaluation including synthesis - but not place and route - of the units.
Also this evaluation is identical to that in Judd (2016b)
Weaknesses:
The idea of combining bit-serial arithmetic with the DaDN architecture is a small one.
The authors have already published almost everything that is in this paper at Micro 2016 in Judd (2016b).  The increment here is the analysis of the architecture on fully-connected layers.  Everything else is in the previous publication.
The energy gains are small - because the additional flip-flop energy of shifting the activations in almost offsets the energy saved on reducing the precision of the arithmetic.
The authors don’t compare to more conventional approaches to variable precision - using bit-parallel arithmetic units but data gating the LSBs so that only the relevant portion of the arithmetic units toggle.  This would not provide any speedup, but would likely provide better energy gains than the bit-serial x bit-parallel approach.
Overall:
The Tartan and Stripes architectures are interesting but the incremental contribution of this paper (adding support for fully-connected layers) over the three previous publications on this topic, and in particular Judd (2016b) is very small.  This idea is worth one good paper, not four.
Thank you for taking the time to read about our work, for your feedback, and opinion, however the review misrepresents the facts. My apologies this is long, I tried to edit for length however, you do raise several points:
1. The following statement is factually incorrect " The authors have already published almost everything that is in this paper at Micro 2016 in Judd (2016b). The INCREMENT HERE IS THE ANALYSIS  of the architecture on fully-connected layers.  Everything else is in the previous publication."  (emphasis ours). The facts are that Stripes does not get any performance improvement on FC layers and worse its energy efficiency suffers on FC layers compared to DaDN. TARTAN is a MODIFICATION of Stripes that solves both these problems: it improves performance and avoids the energy overhead for FC layers resulting in architecture that is faster than DaDN and at least as energy efficient for both FC and Conv layers when operating at the same frequency. There is nothing else in this paper other than the explanation of the modification and its analysis. This is all new material.
2. The statement “This idea is worth one good paper, not four” is also a misrepresentation of the previous publications and this one. There is really just one publication on Stripes in the MICRO conference, The IEEE CAL 4-pager is an introduction to the idea of using bit-serial computation for CNNs and is common practice in the architecture community. IEEE CAL’s mandate is the publication of innovative ideas that are not completely fleshed out hence the 4-pages. We will be happy to explain why IEEE CAL exists and what purpose it serves.
The 2015 Judd publication is an analysis of the precision variability along with a technique for finding these precisions. Tartan exploits this analysis and extends it to simultaneously adjust both weights and activations. Observing that there is precision variability and exploiting are related but clearly different. Others have observed that precision requirements vary but explored it in very different ways (e. g. , by hard-coding the precision in the design).
3. Power gating can be applied to TARTAN as well. In other work we have observed that there are many more activation bits that are zero than those that are one> Moreover, these zero bits  are in between bits that are one (See Bit-Pragmatic submission to ICLR), which your suggestion will not capture.
Further, this observation suggests that a bit-serial approach to processing activations may get more opportunities for power gating than a bit-parallel.
Moreover, there many other power and energy optimization techniques that could be explored for all designs.
In fact, the Stripes work showed that conventional subtype parallelism does not work well for performance since the precisions required are often just above a nice power of two. This approach *might* work for energy for DaDN but there is no reason to believe that it will not work for TARTAN or Stripes . In TARTAN synapses are added in a bit-parallel way once loaded to the SIPs.
Furthermore, power gating adds overhead in the logic and control and this is not free of trade-offs.  Other techniques energy optimizations include using higher VT transistors on non-critical paths. This is interesting follow up work that requires a non-trivial amount of design choices and optimizations that is in our opinion could it be covered in appropriate detail in one publication and that requires further non-trivial exploration. Having TARTAN published would motivate such a study.
4.
Further on the issue of energy or power efficiency, please consider that TARTAN is faster than DaDN.
So, a way to improve power and potentially energy might be to sacrifice some of the performance advantage of TARTAN and to operate at a lower frequency and potentially at a lower voltage. This cannot be done with DaDN.
Yet, we didn’t exploit this either. There is only so many things one can cover in one publication.
5. The opening statement “The idea of combining bit-serial arithmetic with the DaDN architecture is a small one” while perfectly appropriate since as reviewers we do have to make such judgement calls, hints to  a bias against this whole line of work. For whatever is worth, Stripes is getting a honorable mention as one of the most innovative ideas in Computer Architecture research this year in an upcoming IEEE MICRO Magazine issue (IEEE MICRO Top Picks – we do not get to write another report on it but the editors of the issue will comment on it --- IEEE MICRO Top Picks is the equivalent of best paper awards for the top-tier architecture conferences and is published annually – the selection is done by peer review --- 10 reviews this year --- over papers that have been accepted in the few architecture conferences).
I submit that many ideas in architecture are obvious in *retrospect* once someone sees the whole design being explained. For what is worth, it took us a while to come up with the idea of using bit-serial computation effectively after observing the variability in the precision requirements. It then took us a while to come up with TARTAN for FC layers which was not at all obvious to us – the Stripes MICRO paper has a section dedicated on how it can execute FC layers without a performance loss bit with no performance gain and a energy efficiency loss. Along the same lines, one could argue that DaDN presented a small idea -- it's just a wide vector machine one could say --, but that will be a major misrepresentation of how interesting that work has been.
We remain very excited about bit-serial computation for NNs in a modern context. Time will show whether this is a useful approach. We believe that alternate representation computing engine including serial-based ones are an interesting direction with a lot more to be explored than just one design.
TARTAN is a natural next step after Stripes that solves its inefficiencies for FC layers resulting a design that is better. It is fair to raise concerns about some of the design aspects but we think that it will be very valuable to allow this direction of research to grow further. The idea presented here is simple and implantable. It was not at obvious to us.
I edited the response for typos. Please read online the revised response. Apologies.
In Section 4.1 how exactly do you determine energy?  Did you capture toggle counts for each node during simulation?  On what workload?    Did you perform place and route?  If not, how were wire loads estimated?  and how was timing closed at 980MHz without accurate wire loads?  What process corner(s) were used for timing closure and energy estimation?
Thank you for the questions. The results are post layout and thus wire loads are taken into account.  For all designs being compared including the base, we used the default best-case libraries. The comparison is consistent. We will revise to include results also for the worst- and typical-case libraries as well. We will follow up as soon as these are in. The results will be worse for all and additional pipelining may be needed for all designs. For activities we used 50% as an approximation. We will revise to include activities for the networks.
Please do also take the following into consideration.
1. TARTAN being faster than DaDN could be clocked slower without sacrificing performance greatly improving power consumption.
2. There is lots of parallelism and the design can be pipelined even further if needed thus absorbing delays. The pipelining can be done horizontally across weight/synapse lanes and vertically across activation/neuron lanes or both as needed (think of the input data flowing in as a wave).
3. There are obvious circuit-level energy and power reduction techniques that will improve energy efficiency.
For example, we could gate those parts of the circuit involved in computing with zero activation bits or involving zero weights. We can also reduce the width and/or the frequency of reads from the eDRAM taking advantage of the precision variability.
4 While here we present TARTAN in the context of Stripes, the technique can be applied to Pragmatic as well which further reduces the number of computations.
In our opinion, the key contribution here is a practical way to process both fully-connected and convolutional layers bit-serially so the the precision requirement variability of CNNs can be exploited for performance and/or energy efficiency without etching on stone what precision is to be used. We think that this is an important additional degree of freedom that has not be explored before that the ML community could potentially exploit.
We will revise with the additional promised results and post a comment when this is done. we hope to have this done by the first week of January.
The authors present TARTAN, a derivative of the previously published DNN accelerator architecture: “DaDianNao”. The key difference is that TARTAN’s compute units are bit-serial and unroll MAC operation over several cycles. This enables the units to better exploit any reduction in precision of the input activations for improvement in performance and energy efficiency.
Comments:
1. I second the earlier review requesting the authors to be present more details on the methodology used for estimating energy numbers for TARTAN. It is claimed that TARTAN gives only a 17% improvement in energy efficiency.
However, I suspect that this small improvement is clearly within the margin of error ij energy estimation.
2. TARTAN is a derivative of DaDianNao, and it heavily relies the overall architecture of DaDianNao. The only novel aspect of this contribution is the introduction of the bit-serial compute unit, which (unfortunately) turns out to incur a severe area overhead (of nearly 3x over DaDianNao's compute units).
3.
Nonetheless, the idea of bit-serial computation is certainly quite interesting. I am of the opinion that it would be better appreciated (and perhaps be even more relevant) in a circuit design / architecture focused venue.
Thank you for your comments and time and wish you all the best for 2017. Please consider the following:
1.a. The existing comparison is consistent across all engines and is on par with  what other architecture design evaluation works have been using in top-tier publication venues for hardware such as ISCA, MICRO, or ASPLOS. The tools being used are standard commercial energy estimation tools which are provided by the foundry we have access to.
So, the estimation we are getting is as good as it gets at this point and is standard practice in hardware design practice and research. If we were to build the chip, we would have to use exactly the same tools.
1.b Regardless of the energy estimation results, TARTAN is *faster* and is more amenable to frequency and voltage scaling.
3+1.c The interesting part in TARTAN is that bit-serial computation can be used to exploit precision variability to improve performance for both convolutional *and* fully-connected layers. In our opinion, this is a very valuable result that is worthwhile exposing to the ML community. We believe that it is essential for the two communities to work together to achieve the best results possible. TARTAN opens up a new degree of freedom for NN designers as it offers performance that scales with the precision being used. We have tried only per layer precisions but we suspect that precision tuning at a finer granularity may be possible.
Moreover, in another work that is directly compatible with TARTAN we have been reducing computations to only those bits that  are 1.
Overall, we hope that by collaborating with ML experts will be able to co-design the NNs and the hardware.
1.d we will provide energy estimations for all three design corners (which we have not seen been reported by other architecture works) but given points 1.a and 1.b above we ask that you reconsider your evaluation.
2.a A distinct advantage of TARTAN is that it was designed TARTAN as a plug-in replacement of the execution code of DaDianNao and with the explicit goal of making no changes to its memory architecture. Other configurations of TARTAN are possible.
2.b While the execution engine is larger than  DaDianNao's the overall area occupied by the execution engine is very small compared to the large on-chip memories. A small overall area overhead is a reasonable price to pay and a reasonable design choice for improved performance and for the ability to tune performance and accuracy compared to a conventional bit-parallel hardware vector engine like DaDianNao. For a point of reference consider the effect multimegabyte caches have on performance for general purpose processors. Modern CPUs have L3 caches that occupy more than 50% of the chip area, yet do not offer necessarily a 50% performance advantage.
2.c There room to further improve the design to further reduce area costs.
FOr example, processing 2 bits are time instead of 1 would reduce the number SIPS by half while sacrificing very little performance (observe the precisions reported).
Our apologies for the long delay. Our co-author that has the expertise to do this work was overseas for the Christmas break and she returned this Tuesday.
She has synthesized the designs for three cases: bc, tc, and wc for best, typical and worst case respectively. The previous results were for the bc. The detailed results are below.
In summary, efficiency improves for tc and wc. We have layout being synthesized and we expect to have the results in a day or so. The results below are pre-layout and use 50% activity factors. The layout results that we will post as soon as possible will be a testbench for a typical layer. As with STRIPES we expect that energy efficiency will be better with real inputs as the inputs exhibit many more zero bits. Please keep in mind that the results in the STRIPES publication in MICRO are post-layout.
More importantly, we are also synthesizing a different configuration cuts down area costs and improves energy efficiency even further. We will post the post-layout results shortly.
We hope that we will be given the opportunity to post the updated results before a decision is made.
Thank you.
Here are the detailed pre-layout results for all three cases and for fully-connected layers only. Best case is the configuration used in the original submission. We will update the writing with post-layout and actual activity-based results shortly. Effieciency numbers above 1.0 mean that TRT is better than DaDN.
Area & Efficiency, Fully-connected layers, whole chip TRT vs DaDN:
Best case
Baseline area - 77.91 mm2
TRT area - 108.61 mm2 (+39.4%)
Efficiency:
AlexNet  0.935
VGG_S    0.932
VGG_M    0.935
VGG_19   0.932
geomean  0.933
Typical case
Baseline area - 79.28 mm2
TRT area - 111.29 mm2 (+40.4%)
Efficiency:
AlexNet  1.014
VGG_S    1.011
VGG_M    1.014
VGG_19   1.010
geomean  1.012
Worst case
Baseline area - 83.5 mm2
TRT area - 121.39 mm2 (+45.3%)
Efficiency:
AlexNet  1.048
VGG_S    1.046
VGG_M    1.049
VGG_19   1.045
geomean  1.047
We feel uncomfortable completing the reviewer ratings as provided except for one case where a review misrepresents the facts and which we have addressed. This is because the reviews primarily state that the paper does not fit into ICLR. This is a question for the organizers and an interpretation of the CFP which clearly states "hardware".
In summary, the take-away for the ML community is this:
1. Adjusting the precision used per layer or even at a finer granularity of groups of 256 or activations and weights can lead to faster processing and higher energy efficiency. This adjustment can be done at runtime and at a single-bit granularity. Performance can be had without sacrificing accuracy, but if one is willing to sacrifice accuracy more performance and more energy efficiency can be had. We envision follow up work on runtime adjustment of precisions (e.g,, incremental adjustment) to achieve better response times or higher throughput.
In more detail:
2. There is an energy efficient high-performance hardware design that can offer performance inversely proportional to the precision being used per layer or even at a finer granularity (we do not present any results on this but the design obviously support it as-is).
2. This design does not hardwire the precisions at manufacturing time but instead allows programmatic control at runtime.
3. This capability opens up an additional design know that network designers can use to tradeoff execution time and accuracy.
4.
Earlier we had proposed a method to choose per layer precisions  for convolutional layers here we extend this method to fully-connected layers. This method was published on arxiv and has never been accepted to any peer reviewed publication. The only related publication to the work here is STRIPES (MICRO) (12 pages) and a pre-print at iEEE Computer Architecture Letters (4 pages) that explained the basic idea behind Stripes. This work extends STRIPES for Fully-connected layers --- Stripes did not improve performance for FC layers and its energy efficiency was worse than DaDianNao for those layers.
Concern summary:
1. Some concerns were raised about the energy and area measurements. We have posted an update and we will deliver the final results post-layout in a day or so. We will also deliver results on an optimized configuration that drastically reduces area overhead and improves energy efficiency as well. The delay in response was due to having received the reviews during the Xmas break when the author that can perform these measurements was unreachable due to travel.
2. Incremental over DaDianNao: Tartan is a general concept which can be integrated to many different architectures. We chose DaDN as the baseline architecture as it widely known and often compared against and offers the additional challenge of being a very wide vector-like architecture (doing bit-serial computation for a single lane -- product -- independently is easy but doing it for 4K terms in parallel without extremely wide memories is hard).
So, we disagree that this is an incremental improvement of DaDianNao. To draw an analogy, from hardware the seminal work on pattern based branch prediction was not an incremental improvement over out-of-order execution even though previous techniques included branch predictors. Not that we feel that TARTAN is at the same level as pattern based branch prediction but we use this example to illustrate what such an argument can lead to.
Also, STRIPES is receiving a honorable mention in the upcoming IEEE MICRO Topic PIcs in Computer Architecture researcher, which is akin to a best paper award in the field of computer architecture (each accepted paper in the last year in a top-tier conference receives 10 more peer reviews that state whether they feel the paper has the potential for high impact).
So, at least some people that are credible enough to be invited to that panel in the comp arch community think it's not incremental.
3. FC layers are not important: this is not true. They are still in use and more so in different applications.
Moreover, last years best paper award in ICLR rightfully went to an excellent work that addressed both pruning the model and proposing an optimized hardware architecture solely for FC layers.
4. TARTAN takes too much area: The units are larger but this is not where most of the area cost is. The area cost is in the surrounding memory so in the big picture the overall area cost is much lower.
also, in modern technology area is not the concern.
5. Energy efficiency is not that much better than DaDN and within the error margin of the tools. We used industry standard tools and the results are positive.
moreover, TARTAN is faster and has we can use frequency and voltage scaling to improve energy efficiency (which depends on voltage square) while still performing better. We show results assuming same voltage and frequency.
6. You did not use power gating for DaDN. Did we not use it for TARTAN either. It is not straightforward to do for DaDN as it is a bit parlallel engine and there are not that many zero values to necessarily justify the logic needed to enable power gating. Which is to say that this ia a non-trivial task.
Moreover, the same technique can be applied to TATRAN. We have results reported in another submission that show a heavy bias toward the zero bit value which suggestes that power gating will most likely be a lot more effective for TARTAN than DaDN.
In summary, combined this work, with STRIPES and the earlier arxiv report present a comprehensive hardware/software approach to exploiting precision to improve performance and energy efficiency for CNNs. Stripes performed very well for convolutional layers but did poorly for fully-connected layers. TARTAN fixes this.
Here are our post-layout results with actual data-driven activity factors. The 2-bit configuration, detailed in the updated paper, is a design that processes two bits at once using half as many SIPs, increasing efficiency and reducing area overhead. It requires that precision is an even number.
We report results for the typical design case. Pre-layout results showed that the worst case design corner results in a larger advantage for TRT. The results show increased energy efficiency compared to the pre-layout results. This is expected as the pre-layout results used 50% activity factors whereas here we use actual activity factors measured on a typical layer. This is consistent with the behavior observed during the STRIPES work.
Fully-connected layers, whole chip (TSMC 65nm typical case):
Baseline (DaDN) area - 80.41 mm2
TRT area - 120.04 mm2
Efficiency (TRT vs baseline):
AlexNet  1.062
VGG_S    1.059
VGG_M    1.063
VGG_19   1.059
geomean  1.061
TRT 2-bit area - 100.43 mm2
Efficiency (TRT 2-bit vs baseline):
AlexNet  1.228
VGG_S    1.235
VGG_M    1.268
VGG_19   1.237
geomean  1.242
Numbers greater than 1 mean less energy used overall by TRT.

Meta-review:
role: AC
data: comment: Hardware is listed on the call-for-papers as relevant topic for ICLR 2017, and so the paper is on-topic.

We worked hard to improve on the initial reviewer assignment for this paper in ensure that we would get hardware-knowledgeable reviewers on board for this paper, although we only partly succeeded (due to conflicts, tight review deadlines, and more).

We are also still missing questions and comments from one reviewer.

ICLR papers should succeed in communicating with the ICLR audience, i.e., introducing concepts for this audience, and using a language that is accessible for this community.  But it need not connect with all of the ICLR community, which is dominated by algorithmic concerns; many may not have much background or interest in hardware and that is fine.  This situation is not unique, e.g., hardware-related papers in computer graphics and computer vision conferences are in the same situation.  It is also true, I think, that the audience and reviewers of hardware-related conferences may not be well-placed to fully understand and comment on all relevant algorithmic considerations that a hardware implementation targets. 

There remains disagreement as to the utility of achieving improvements related to fully connected layers, which would be good to resolve. And there also remains disagreement on the improvements (conceptual and performance-related) with respect to the state of the art, as recently rebutted by the authors. 



role: PC
data: comment: This metareview is written based on the reviews of the two expert reviewers (scores of 5(5), 5(5)),
 given the wide disparity of reviewer expertise, and the relevance of hardware expertise to this particular paper.
 
 Quality, Clarity: The paper is clearly written. It does require hardware expertise to read and appreciate the ideas, methodology, and the results. There was a request to post a clear "take home" message for ML research, and the authors did post a clear statement in this regard.
 
 Originality: The size of the contribution is the major point of contention for this paper. The expert reviewers believe it to be an incremental/small idea, particularly in relation to the prior work of the authors. The authors provide rebut that while the implementation builds on their previous work, it can be seen as a general concept that could be applied to other architectures, and that this work targets the important case of fully-connected architectures, vs the case of convolutional architectures that were the case where the prior work did well.
 
 Overall: I have asked the expert reviewers for further input. In the absence of further information, updated scores, or some reasonably strong indication of excitement about the ideas from the expert reviewers, there is little choice but to currently lean towards "reject" in terms of this being an inspirational paper for ICLR.

Decision: Reject