Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
Authors: Julian Ibarz, Ian Goodfellow, Sacha Arnoud, Vinay Shet, Yaroslav Bulatov
ICLR 2014

Abstract: Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural-network that operates directly off of the image pixels. This model is configured with 11 hidden layers all with feedforward connections. We employ the DistBelief implementation of deep neural networks to scale our computations over this network. We have evaluated this approach on the publicly available SVHN dataset and achieve over 96% accuracy in recognizing street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art and achieve 97.84% accuracy. We also evaluated this approach on an even more challenging dataset generated from Street View imagery containing several 10s of millions of street number annotations and achieve over 90% accuracy. Our evaluations further indicate that at specific operating thresholds, the performance of the proposed system is comparable to that of human operators and has to date helped us extract close to 100 million street numbers from Street View imagery worldwide.


Peer review:
[Review]:
This submission describes an approach for digit and sequence recognition that results in improved performance on the StreetView house number dataset using a simple structured output to recognize the entire sequence as an ordered set of detections and a very deep convolutional network (11 layers).  The approach is end-to-end, without requiring multiple networks or a composite of techniques. 

The approach has very high accuracy and is attractive in its simplicity, although the authors make clear that it could not be extended to, for instance, general text recognition in images.  The relevance for ICLR is high, although the paper has some significant omissions that keep it from being a very strong candidate for acceptance.

First, the method is not clear, in particular the interface between the varia softmax digit classifiers and the output of the convnet. It is unclear whether there is any representation of locality in this interface beyond the assignment of different classifiers for each position.  In the introduction, it is stated that the subtasks of localization, segmentation, and recognition are solved in an integrated way, but section 3 introduces the dataset, which has localized inputs- numbers which fill at least ⅓ of the image. This needs to be clarified.

Second, to further the contribution of the paper, additional experiments or analysis could have been performed to understand the features, the architecture, the loss function, or other aspects of the approach. With these missing, the submission is somewhat thin and the contribution lessened.

Smaller issues: the variables used in the plate model in Fig. 1 need to be defined in the caption and supporting text as well as later when the method is explained, and DistBelief should not be used in the abstract and intro without citation or footnote.