Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
Authors: Julian Ibarz, Ian Goodfellow, Sacha Arnoud, Vinay Shet, Yaroslav Bulatov
ICLR 2014

Abstract: Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural-network that operates directly off of the image pixels. This model is configured with 11 hidden layers all with feedforward connections. We employ the DistBelief implementation of deep neural networks to scale our computations over this network. We have evaluated this approach on the publicly available SVHN dataset and achieve over 96% accuracy in recognizing street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art and achieve 97.84% accuracy. We also evaluated this approach on an even more challenging dataset generated from Street View imagery containing several 10s of millions of street number annotations and achieve over 90% accuracy. Our evaluations further indicate that at specific operating thresholds, the performance of the proposed system is comparable to that of human operators and has to date helped us extract close to 100 million street numbers from Street View imagery worldwide.


Peer review:
[Review]:
The authors propose an integrated approach to sequence recognition in the case of limited number of characters (house numbers with 5 characters at most). Avoiding separating localization, segmentation and recognition is novel in this context. This is the right approach and results are good but the model is not well explained (see below).

Pros:
- integrated sequence recognition rather than traditional localization/segmentation/recognition step.
- new record on single digit
- accuracy high enough for real world deployment (although the ability to pay human operators for remaining errors in the 98% regime means that real world deployment is possible even at very low accuracy, depending how much money the company is willing to spend).

Cons:
- 'and has special code for handling much of the mechanics such as proposing candidate object regions' and 'we take a similar approach, but with less post-processing': this is misleading and debatable that there is less post-processing because the authors methods relies on a pre-detection step that gives relatively tight bounding boxes around the number, as indicated here: 'Beyond cropping the image close to the street number'. One could argue that there is as much post-processing in the other cited work once the detection is performed.
- what happens at the top of the network is not clear at all to me. Are they using an HMM or not? Does the 64x64 input image yield a grid of probabilities? then what is the size of that grid? Or is the network directly predicting N outputs, and based on the value of L, uses only the first L values out of N?
- 'a softmax classifier that is attached to intermediate features': digit softmax are located on the intermediate hidden layers? which ones?
- 'On this task, due to the larger amount of training data, we did not need to train with dropout': it would have been nice to see the numbers with and without dropout instead of just relying on this claim.