Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
Authors: Julian Ibarz, Ian Goodfellow, Sacha Arnoud, Vinay Shet, Yaroslav Bulatov
ICLR 2014

Abstract: Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural-network that operates directly off of the image pixels. This model is configured with 11 hidden layers all with feedforward connections. We employ the DistBelief implementation of deep neural networks to scale our computations over this network. We have evaluated this approach on the publicly available SVHN dataset and achieve over 96% accuracy in recognizing street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art and achieve 97.84% accuracy. We also evaluated this approach on an even more challenging dataset generated from Street View imagery containing several 10s of millions of street number annotations and achieve over 90% accuracy. Our evaluations further indicate that at specific operating thresholds, the performance of the proposed system is comparable to that of human operators and has to date helped us extract close to 100 million street numbers from Street View imagery worldwide.


Peer review:
[Review]:
In order to avoid the ArXiv approval delay, we have posted a revised copy of our paper directly here:
https://drive.google.com/file/d/0B64011x02sIkd3RwSDRpTXlKSzQ/edit?usp=sharing

'Anonymous 88a9', 'Anonymous ac02' and 'Anonymous dee9' have raised concerns
about the clarity of the description of the architecture and inference process.
We have updated the paper to add Appendix A, a worked example showing exactly
how the inference process works step by step. We hope this resolves the ambiguities
in the main text.

'Anonymous ac02' suggests to use larger shallow networks to have another
datapoint on the importance of a deep network. This is a great idea and we
have started experiments with models of varying sizes using 3 hidden layers to see
their improvements based on the number of parameters alone. We will post another
update when these experiments are complete but didn't manage to finish them during
the recommended rebuttal period.

'Anonymous dee9' suggested that we were doing heavy post-processing of the image
because we were doing tight crops around the street number. We have five responses
to this criticism:
1) We mistakenly omitted some details about the limitations of our preprocessing
of the larger internal dataset. These details make it clear that the preprocessing
is not especially useful. Specifically, the centroid is not well-known, the scale
is not known at all, and the crop size we use for that dataset is 128x128 compared
to the 54x54 we used on the public SVHN. Our results on this dataset therefore
indicate that the network is able to localize the house number itself as well as localizing the digits
within the number. They also demonstrate the network is able to handle wide variations
in the scale of the house number.
2) It is not computationally practical to run a convolutional network on a
completely uncropped Street View panorama, so some degree of preprocessing is
inevitable.
3) On the public SVHN, all pre-existing methods make use of more ground truth
localization information than our method does. All previous authors who have
published on this dataset use the version that is tightly cropped per-digit.
We use the much looser crop that identifies only the region in which the multi-digit
number occurs, but we still improve upon the state of the art for single digit
recognition. This demonstrates that the system is able to localize individual
digits on this dataset.
4) Other systems for transcription represent the concept of the sequence external
to the neural net, i.e. the sequence parsing is handled by post-processing techniques
such as HMM inference, non-maxima suppression, etc. Only our approach trains a neural
net with an internal concept of a sequence.
5) Our system uses *no post-processing at all* but only pre-processing, while other
systems use both pre-processing and post-processing.



'Anonymous dee9' suggests that we should have given the accuracy number with and
without dropout for our internal dataset to prove that 'we did not need to train
with dropout'. We agree that this claim is overreaching, we changed the paper to
say that we were not seeing large overfitting and decided to not use dropout primarily
in order to speed up the training, which is time-consuming for the larger models.