Title: An Analysis of Deep Neural Network Models for Practical Applications
Authors: Alfredo Canziani
Adam Paszke
Eugenio Culurciello
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Applications

Abstract: Since the emergence of Deep Neural Networks (DNNs) as a prominent technique in the field of computer vision, the ImageNet classification challenge has played a major role in advancing the state-of-the-art. While accuracy figures have steadily increased, the resource utilisation of winning models has not been properly taken into account. In this work, we present a comprehensive analysis of important metrics in practical applications: accuracy, memory footprint, parameters, operations count, inference time and power consumption. Key findings are: (1) power consumption is independent of batch size and architecture; (2) accuracy and inference time are in a hyperbolic relationship; (3) energy constraint are an upper bound on the maximum achievable accuracy and model complexity; (4) the number of operations is a reliable estimate of the inference time. We believe our analysis provides a compelling set of information that helps design and engineer efficient DNNs.

TL;DR: Analysis of ImageNet winning architectures in terms of accuracy, memory footprint, parameters, operations count, inference time and power consumption.

Review discussion:
Would any curve that is 1) continuous 2) monotonic 3) bounded up asymptotically (here by the 100% accuracy ceiling) have to naturally be 'hyperbolic' by your definition? This statement pretty much mechanically derives from the nature of the quantity you're measuring, not from the data, so doesn't add any information.
Read section 3.7.
You're looking at the envelope of all possible model architectures, which means that models that do worse, but with more parameters / compute are not part of your evaluation (they don't get published). The exception is the VGG models, which are still relevant in spite of their inefficiency because they are so simple architecturally, but you also exclude them from your linear fits. These choices mechanically imply that more compute means no-worse models.
So your curves will go up and to the right, and will be bounded above by 100%. If I ask you to draw a curve that goes up and hits a ceiling asymptotically (because you never hit 100% accuracy), it would be very, very hard to draw a curve that has anything but a hyperbolic-looking shape. It can't be linear or you'd extrapolate 110% accuracy. It's also baked in the nature of error curves: if at every step of improvement you cut the error rate by a fixed fraction, your absolute accuracy goes up logarithmically. Diminishing returns are the norm.
All I'm arguing is that the statement that this relationship is hyperbolic contains no more information than stating that two data points are in a linear relationship with each other. It's expected from the way you've defined what you measure, not from the data itself.
Alright, let's try to stay focussed on the point.
What the data is showing is that, *locally*, *well optimised models* (both in terms of training / parameter search and computationally / efficiently designed) are a good fit for a hyperbole.
I am certainly not overgeneralising this as a new global dogma of the field. Perhaps I did not express myself too well in English.
The robust linear fit deals well with outliers by its own.
Moreover, I believe that we are more interested in foreseeing what the current upper bound stands at, rather than analyse why a specific model hasn't been trained well enough.
But, given VGG's popularity, I wanted to show how it performs with respect to the others model.
In particular, it got a speed up of 3x moving from cuDNN v4 to v5, making it now at least somehow usable on a TX1.
You run you analysis for network architectures that are mainly used for training DNNs.
However it is getting more and more common to heavily change the network that is actually used in production, e.g. by the network compression techniques that you mention in 3.8.
I could imagine there is a trade off in the architecture between efficient training and efficient use in production (that might be overcome by compression techniques). Could you clarify whether you propose to start with training networks like E-Net from the beginning. If so, how do other important factores like training times, generalizability etc. compare to other state-of-the-art networks?
Or are you suggesting criteria for how to choose the architecture of networks that are used in production (possibly as a result of compression or retraining)?
Then I think it might be helpful to have some details on which kinds of networks are actually used in production (as far as it is possible to get this kind of information).
Hi, thank you for your comment. I will try my best to address all your points.
"What I propose" is, given the massive amount of network that have been trained for multi-class classification-task on the ImageNet data set, how different models does relate not only in terms of accuracy (only goal to win the challenge) but also in terms of usability.
If you read the papers of these architectures you'll notice information, essential to the usage of these networks, is often missing.
When I started trying to make the point of the situation (which was the primary target of our article), I soon realised this was not possible unless I would have re-run all the experiments by my own.
For example, by using cuDNN v4 on a TX1, VGG takes ~600 ms.
Hence, if you aim at a real time application, you already know your network choice should be different.
"What I suggest" is that well performing (accuracy-wise) networks can be also efficient in terms of operations count. More specifically, architectures designed with efficiency in mind (see GoogLeNet and our ENet, which architecture is discussed elsewhere and its ImageNet adaptation has been summarised in a blog post) achieve the highest utilisation of their parameters, and can perform up to 50 fps. All this before further compression.
Moreover, these design choices scale up well and let you reach state-of-the-art performance (see Inception-v4, best performing architecture so far) while being far less computationally expensive w.r.t. other models.
Furthermore, I believe that providing better models to the compression mechanism would generate better zipped networks.
Finally, yes, I think too that it would be very interesting seeing how networks used in production do compare. Unfortunately this information is not quite available.
Please, let me know if I missed anything! :)
You make an important point here.
However, I feel as well that compression is indeed a crucial ingredient for production systems. Why did you choose not to include it?
If the focus would be rather on state-of-the-art networks that are not meant for direct deployment in production, I am missing a more rigorous treatment of the differences between these networks and their properties.
For instance, vulnerability to adversarial examples might be something to look at.
It would be great, if you can give some insight why you chose to exclude both viewpoints.
Thank you.
This paper tries to fill the current gap in terms of understanding how SOTA architectures compare to each other on a pletora of different performance metrics.
As pointed out in the article, most of the times information is missing or misleading (multiple crops / models are used concurrently in order to improve final accuracy).
Therefore, I wanted to increase the awareness about how different architectural design choices affect usability.
I agree that considering compression would be definitely interesting, and could be covered in a follow up paper, where several techniques can be compared one against the other.
Nevertheless, we believe that starting with an highly optimised network, where redundancy is less to begin with, would lead to better final results.
Hence, this thorough analysis of models practical performance.
A few issues with this paper:
1- I find finding #2 trivial and unworthy of mention, but the author don't seem to agree with me that it is. See discussions.
2- Finding #1 relies on Fig #4, which appears very noisy and doesn't provide any error analysis. It makes me question how robust this finding is. One would have naively expected the power usage trend to mirror Fig #3, but given the level of noise, I can't convince myself whether the null hypothesis of there being no dependency between batch size and power consumption is more likely than the alternative.
3- Paper is unfriendly to colorblind readers (or those with B/W printers)
Overall, this paper is a reasonable review of where we are in terms of SOTA vision architectures, but doesn't provide much new insight. I found most interesting the clear illustration that VGG models stand out in terms of being a bad tradeoff in resource-constrained environments (too many researchers are tempted to benchmark their model compression algorithm on VGG-class models because that's always where one can show 10x improvements without doing much.)
Thank you for your review.
1- I have to say that the most unexpected results (and therefore the one I was not absolutely confident about) is actually finding #2, which comes from Fig #9. As reported in the paper, all these architectures lined up (when sufficiently trained or efficiently designed) on the grey-white boundary.
2- The power measurements have not been trivial to obtain, for which I've redesigned the acquisition protocol half a dozen times.
Nevertheless, when I initially used the cuDNN v4, you could see a strong inverse dependance with the inference time (https://arxiv.org/pdf/1605.07678v2.pdf). Now, such strong dependency is no longer there.
3- I willingly decided to compromise B&W compatibility for intelligibility.
Otherwise it would have been unfriendly for everyone.
Finally, after analysing all models, we propose a novel architecture that stands out among all the other SOTAs, in terms of information density (accuracy per tuneable parameter).
Yet, the main objective was to provide a fair comparison among SOTAs and understand what design choices should be observed in order to create usable models.
The authors did solid work in collecting all the reported data.
However, most findings don't seem to be too surprising to me:
- Finding #1 mainly shows that all architectures and batch sizes manage to utilize the GPU fully (or to the same percentage).
- Regarding Finding #2, I agree that from a linear relationship in Figure 9 you could conclude said hyperbolic relationship.
However, for this finding to be relevant, it has to hold especially for the latest generations of models. These cluster in the upper left corner of Figure 9 and on their own do not seem to show too much of a linear behaviour.
Therefore I think there is not enough evidence to conclude asymptotic hyperbolic behaviour: For this the linear behaviour would have to be the stronger, the more models approach the upper left corner.
- Finding #3 seems to be a simple conclusion from finding #1: As long as slower models are better and faster models do draw the same power, finding #3 holds.
- Finding #4 is again similar to finding #1: If all architectures manage to fully utilize the GPU, inference time should be proportional to the number of operations.
Maybe the most interesting finding would be that all tested models seem to use the same percentage of computational resources available on the GPU, while one might expect that more complex models don't manage to utilize as much computational resources due to inter-dependencies.
However actual GPU utilization was not evaluated and as the authors choose to use an older GPU, one would expect that all models manage to make use of all available computational power.
Additionally, I think these findings would have to be put in relation with compressing techniques or tested on actual production networks to be of more interest.
Thank you for your comments.
I'm slightly confused by the contradiction in your review: "solid work" and "rejected".
If the work is solid and therefore it's valuable, then it should get visibility and get accepted, even though the findings are not surprising.
After all, we did not know this before this work. Correct?
I mean, Newton described how an apple falls on Earth. Well, it was not that surprising an object without support will fall on the ground, still his findings have been widely used.
About hardware utilisation, there was a substantial difference for architectures using kernels 3x3 with the previous cuDNN library. 1/3 third of the power was used and 3x slower inference time was obtained. Updating the system gave us uniform performance for all architectures.
So, please, let me know what you think it should be improved in order to change your opinion on point 4. Or, if it's already flawless, please try to reconsider your verdict.
Thank you.
Maybe Newton is a good way of explaining what I am missing in the paper: The contribution of Newton was not describing that an apple falls to earth, but describing how exactly the apple does so (and his claim that the natural state of an object is not rest but moving at a fixed speed was actually a paradigm shift).
Finding #2 would make a nice contribution in describing "how" instead of "that", but as I pointed out I do not agree that this finding is supported by the data.
Regarding your confusion about me calling the paper "solid work" but still rejecting it: I acknowledge that a lot of work was necessary to collect all the presented data and I think this work was done very well.
Still the reviewers are also asked to judge the originality and significance of the work and right now I miss a clear takehome message that would influence other researchers in their work (keep in mind that my rating was "OK but not good enough"). Your aim was to evaluate networks with respect to their applicability in production systems. As I pointed out in my pre-review question, if this is what you are interested in, then I would expect much more discussion of other techiques to make networks production ready: What do I gain from using E-Net instead of any other network and applying SqueezeNet or compression techniques to them?
Additionally, appart from the ImageNet task, for most tasks networks are not even trained end-to-end from scratch but initialized with networks trained from other tasks and then retrained. Especially here VGG has proven to be a valuable starting point which is one reason why it is still used so often. What are your recommendations for this setting?
You might argue that the takehome message would be "don't use too many parameters".
But this takehome message does not even appear in the abstract and as just pointed out I am missing the discussion of several alternatives for this claim to hold.
I think the differences coming from different cuDNN libraries are interesting and not obvious. It sounds like the previous cuDNN library had substantial problems running 3x3 convolutions (which surprises me). This might be an interesting additional observation to have in the paper.
Finally, to make this more constructive: this is how I think the paper could be improved: Have a much clearer takehome message and give sufficient evidence to support it. Discuss in what situations people will profit from your work. One way to do this might be a recommendation on what you think is the right way given some task to come up with a production-quality network -- both for the case of end-to-end training and for the much more common case of retraining existing networks. One important piece of evidence would be to give reasons why your findings still hold if you plan to (or already have) applied techniques like SqueezeNet and at which point in the process from the task to the final network your findings apply.
Thank you for taking the time to argument your (re)view. I wish I had such an input last spring, when I started working on this.
I did have input from industry folks, but not much from peers, which I recon may have wider horizons and deeper knowledge than myself. It's also a pity this is anonymous, since it may hinder future collaborations with interested subjects, at least from my side.
As of right now, I'm not sure whether I can savage anything to still be able to make it to this conference or not.
The paper evaluates recent development in competitive ILSVRC CNN architectures from the perspective of resource utilization. It is clear that a lot of work has been put into the evaluations. The findings are well presented and the topic itself is important.
However, most of the results are not surprising to people working with CNNs on a regular basis. And even if they are, I am not convinced about their practical value. It is hard to tell what we actually learn from these findings when approaching new problems with computational constraints or when in production settings. In my opinion, this is mainly because the paper does not discuss realistic circumstances.
Main concerns:
1) The evaluation does not tell me much for realistic scenarios, that mostly involve fine-tuning networks, as ILSVRC is just a starting point in most cases. VGG for instance really shines for fine-tuning, but it is cumbersome to train from scratch. And VGG works well for compression, too.
So possibly it is a very good choice if these by now standard steps are taken into account. Such questions are of high practical relevance!
2) Compressed networks have a much higher acc/parameter density, so comparison how well models can be compressed is important, or at least comparing to some of the most well-known and publicly available compressed networks.
3) There is no analysis on the actual topology of the networks and where the bottlenecks lie. This would be very useful to have as well.
Minor concern:
1) Why did the authors choose to use batch normalization in NiN and AlexNet?
Thank you for your review and for pointing out well identified improvement points. This is how we keep advancing in Science!
I definitely share your opinion and I have to say that these topics have not been included in this publication due to the allotted amount of time.
Nevertheless, I could cover them in the next article, given also the current wide usage of this paper and its previous version, released on arXiv.
Batch-normalised NiN and AlexNet have been included due to their availability as re-trained networks by the community.
Now some briefs replies to your points.
1) The paper addresses the accuracy gain vs. performance tradeoff while training different architectures.
Indeed, fine-tuning *is* the norm, but private data sets, used in realistic scenarios, are not available to the public.
Therefore, it would have made little sense to publish the results.
Furthermore, VGG can be compressed, because it's huge to begin with, but I guess this was obvious.
2) Agreed.
Still, the focus was on architecture choice for training from scratch.
3) Agreed. We used this analysis to design ENet, best performing architecture. The paper and blog explain in detail the design choices.
Once more, I agree on everything but on the rating ;)
Thank you for your constructive criticism.

Meta-review:
role: PC
data: comment: The paper presents an evaluation of off-the-shelf image classification architectures. The findings are not too surprising and don't provide much new insight.

Decision: Reject