OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Authors: Michael Mathieu, Yann LeCun, Rob Fergus, David Eigen, Pierre Sermanet, Xiang Zhang
ICLR 2014

Abstract: We present an integrated framework for using Convolutional Networks for classification, localization and detection. We show how a multiscale and sliding window approach can be efficiently implemented within a ConvNet. We also introduce a novel deep learning approach to localization by learning to predict object boundaries. Bounding boxes are then accumulated rather than suppressed in order to increase detection confidence. We show that different tasks can be learnt simultaneously using a single shared network. This integrated framework is the winner of the localization task of the ImageNet Large Scale Visual Recognition Challenge 2013 (ILSVRC2013), and produced near state of the art results for the detection and classifications tasks. Finally, we release a feature extractor from our best model called OverFeat.


Peer review:
[Review]:
The authors present a system that is capable of classification, localization and detection, to the advantage of all three. The starting point of Krizhevsky’s 2012 work is adapted to produce an architecture capable of simultaneously solving all tasks, with the feature extraction component being shared across tasks. They propose to scan the entire convolutional net across the entire image at several scales, reusing the relevant computations from overlapping regions already processed, making both classification and (per-class) bounding box predictions at each and a clever scheme for aggregating evidence for the bounding box across predictions, and lots of other tricks. They evaluate on ILSVRC 2012 and 2013 tasks with excellent results.

Novelty: low-medium
Quality: high

Pros
- This is an excellent piece of applied vision research. The results on ILSVRC speak for themselves, really.
- It brings to light some potentially non-obvious (to a convolutional networks neophyte) advantages of convolutional nets for tasks like this, namely that dense application
- Details are copiously documented: this is an excellent example of authors paying serious mind to the reproducibility of their work, a tendency that is sorely lacking in computer vision and machine learning in general. Please keep up the good work in future publications.

Cons
- There isn’t a lot of methodological novelty, although there is some in the tricks employed to deal with subsampling, etc. (to my knowledge, these are novel). That said, it is a tour-de-force application paper, so I don’t see this as a serious drawback.
- The only serious barrier to publication I see is clarity of exposition in certain parts.
- Some discussion of not just which hyperparameters were chosen but how and why, including some rationale for departures from Krizhevsky’s architecture, would be nice (e.g. the learning rate schedule, your choice to drop contrast normalization, non-overlapping pooling regions, etc.). Providing the details as statement of fact is good, but insights into how you made some of these decisions would make for more compelling reading, especially to those familiar with the Krizhevsky work.
- The organization of the paper could also use work: essential ideas should be distilled into an 8ish page manuscript and the details (which, as I said, are an extremely positive feature of this paper) relegated to an appendix.

Detailed comments:
- If you have some rough idea of the relative importance of horizontal flipping and other tricks described in 3.3, it would be useful to know. I don’t expect an exhaustive ablative analysis, but even an informal statement as to which elements seem to be the most critical would be interesting.
- The exposition on “dense application” and why this is a computational savings is less clear than it could be (section 3.5). Basically what you are trying to get across, I think, is that applying a convolutional net to every PxP window of an MxN image, where P << M and P << N, can be performed efficiently by convolving each layer of filters and doing the pooling and so on with the entire image at once, reusing computation for overlapping window regions, and thus it is much more efficient than if you had some arbitrary black box that you had to apply at every window location and reuse no computation whatsoever. However the text was very unclear on this point, and a reader with less background may not understand what you mean (which would be a shame, as this is a very important point, practically speaking). I’m sure I’ve heard this idea spoken of before -- is this the first time it’s appeared in print? If not, I’d make sure to include a citation.