Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Would any curve that is 1) continuous 2) monotonic 3) bounded up asymptotically (here by the 100% accuracy ceiling) have to naturally be 'hyperbolic' by your definition? This statement pretty much mechanically derives from the nature of the quantity you're measuring, not from the data, so doesn't add any information.
review_title: Hyperbolic relationships

final_score: rating: null
review_content: You run you analysis for network architectures that are mainly used for training DNNs. However it is getting more and more common to heavily change the network that is actually used in production, e.g. by the network compression techniques that you mention in 3.8. 

I could imagine there is a trade off in the architecture between efficient training and efficient use in production (that might be overcome by compression techniques). Could you clarify whether you propose to start with training networks like E-Net from the beginning. If so, how do other important factores like training times, generalizability etc. compare to other state-of-the-art networks?

Or are you suggesting criteria for how to choose the architecture of networks that are used in production (possibly as a result of compression or retraining)? Then I think it might be helpful to have some details on which kinds of networks are actually used in production (as far as it is possible to get this kind of information).
review_title: Relationship to network compression etc

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors did solid work in collecting all the reported data. However, most findings don't seem to be too surprising to me:

- Finding #1 mainly shows that all architectures and batch sizes manage to utilize the GPU fully (or to the same percentage).

- Regarding Finding #2, I agree that from a linear relationship in Figure 9 you could conclude said hyperbolic relationship.
However, for this finding to be relevant, it has to hold especially for the latest generations of models. These cluster in the upper left corner of Figure 9 and on their own do not seem to show too much of a linear behaviour. Therefore I think there is not enough evidence to conclude asymptotic hyperbolic behaviour: For this the linear behaviour would have to be the stronger, the more models approach the upper left corner.

- Finding #3 seems to be a simple conclusion from finding #1: As long as slower models are better and faster models do draw the same power, finding #3 holds.

- Finding #4 is again similar to finding #1: If all architectures manage to fully utilize the GPU, inference time should be proportional to the number of operations.

Maybe the most interesting finding would be that all tested models seem to use the same percentage of computational resources available on the GPU, while one might expect that more complex models don't manage to utilize as much computational resources due to inter-dependencies. However actual GPU utilization was not evaluated and as the authors choose to use an older GPU, one would expect that all models manage to make use of all available computational power.

Additionally, I think these findings would have to be put in relation with compressing techniques or tested on actual production networks to be of more interest.

review_title: solid work but not surprising

final_score: rating: 5: Marginally below acceptance threshold
review_content: A few issues with this paper:
1- I find finding #2 trivial and unworthy of mention, but the author don't seem to agree with me that it is. See discussions.
2- Finding #1 relies on Fig #4, which appears very noisy and doesn't provide any error analysis. It makes me question how robust this finding is. One would have naively expected the power usage trend to mirror Fig #3, but given the level of noise, I can't convince myself whether the null hypothesis of there being no dependency between batch size and power consumption is more likely than the alternative.
3- Paper is unfriendly to colorblind readers (or those with B/W printers)

Overall, this paper is a reasonable review of where we are in terms of SOTA vision architectures, but doesn't provide much new insight. I found most interesting the clear illustration that VGG models stand out in terms of being a bad tradeoff in resource-constrained environments (too many researchers are tempted to benchmark their model compression algorithm on VGG-class models because that's always where one can show 10x improvements without doing much.)
review_title: Interesting paper. Some flaws.

final_score: rating: null
review_content: You make an important point here.
However, I feel as well that compression is indeed a crucial ingredient for production systems. Why did you choose not to include it?

If the focus would be rather on state-of-the-art networks that are not meant for direct deployment in production, I am missing a more rigorous treatment of the differences between these networks and their properties. For instance, vulnerability to adversarial examples might be something to look at. 

It would be great, if you can give some insight why you chose to exclude both viewpoints.
review_title: Question on structure of paper

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper evaluates recent development in competitive ILSVRC CNN architectures from the perspective of resource utilization. It is clear that a lot of work has been put into the evaluations. The findings are well presented and the topic itself is important.

However, most of the results are not surprising to people working with CNNs on a regular basis. And even if they are, I am not convinced about their practical value. It is hard to tell what we actually learn from these findings when approaching new problems with computational constraints or when in production settings. In my opinion, this is mainly because the paper does not discuss realistic circumstances.

Main concerns:
1) The evaluation does not tell me much for realistic scenarios, that mostly involve fine-tuning networks, as ILSVRC is just a starting point in most cases. VGG for instance really shines for fine-tuning, but it is cumbersome to train from scratch. And VGG works well for compression, too. So possibly it is a very good choice if these by now standard steps are taken into account. Such questions are of high practical relevance!

2) Compressed networks have a much higher acc/parameter density, so comparison how well models can be compressed is important, or at least comparing to some of the most well-known and publicly available compressed networks.

3) There is no analysis on the actual topology of the networks and where the bottlenecks lie. This would be very useful to have as well.

Minor concern:
1) Why did the authors choose to use batch normalization in NiN and AlexNet?
review_title: null

Meta-review: The paper presents an evaluation of off-the-shelf image classification architectures. The findings are not too surprising and don't provide much new insight.
Decision: Reject