Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
Authors: Julian Ibarz, Ian Goodfellow, Sacha Arnoud, Vinay Shet, Yaroslav Bulatov
ICLR 2014

Abstract: Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural-network that operates directly off of the image pixels. This model is configured with 11 hidden layers all with feedforward connections. We employ the DistBelief implementation of deep neural networks to scale our computations over this network. We have evaluated this approach on the publicly available SVHN dataset and achieve over 96% accuracy in recognizing street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art and achieve 97.84% accuracy. We also evaluated this approach on an even more challenging dataset generated from Street View imagery containing several 10s of millions of street number annotations and achieve over 90% accuracy. Our evaluations further indicate that at specific operating thresholds, the performance of the proposed system is comparable to that of human operators and has to date helped us extract close to 100 million street numbers from Street View imagery worldwide.


Peer review:
[Review]:
The paper presents an application of deep neural networks to the problem of reading multi-digit housenumbers from StreetView images.  The basic architecture is essentially standard (maxout units, ReLu units, convolution, and several dense layers), but is unusually deep (11 layers).  The output of the detector is a softmax predictor for the length of a sequence, as well as softmax predictors for each digit in the sequence.  This simple output encoding is sufficient to achieve a high recall at very high precision that is competitive with human labelers.  The authors conclude that this particular OCR application may regarded as solved at this stage.

There is relatively little new in terms of algorithms here, but the results are excellent.  

The paper is clearly written, though the prose could be tightened up a bit.  If room can be made, I think a deeper analysis of the method’s success would be useful.  For example, is it possible that the “deeper” networks are fitting the training set better as a result of more model parameters?  Or is the depth truly the deciding factor?

Most surprising to me is the fact that there is no explicit need to model the label structure beyond the obvious:  a detector for sequence length and softmax outputs for digit classes, with a small tweak to choose the most likely length of the sequence during test time.  This follows along with recent work on detection systems that suggests sophisticated regressors are able to do something similar for object classes, so I think this otherwise simple component is a useful datapoint for that conversation.

Pros:

Simple off-the-shelf application with excellent performance;  perhaps high enough to count this task as “solved”.

A useful reference point for work on predicting structured outputs like character sequences.

Cons:

Essentially boiler-plate neural network.  A bit more detailed analysis of the result would be useful.