Build a Multi Digit Detector with Keras and OpenCV
By: Priya Dwivedi
Tags: Deep Learning, Artificial Intelligence, Data Science, TensorFlow, Machine Learning

Build a Multi Digit Detector with Keras and OpenCV

Doing cool things with data!

Introduction

Numbers are everywhere around us. Be it an alarm clock or the fitness tracker or a barcode or even a well packed delivery package from Amazon, numbers are everywhere. With MNIST data set, machine learning was used to read single handwritten digits. Now we are able to extend that to reading multiple digits as shown below. The underlying neural network does both digit localisation and digit detection. This can be quite useful in a variety of ML applications like reading labels in the store, license plates, advertisements etc.

Reading multiple digits

But why not use just OCR? Yes, OCR can be a good starting point to automatically detecting numbers but OCR doesn’t always work and sometime we need to train a neural network for our specific task.

Digit detection pipeline

The digit detection problem can be divided into 2 parts

Digits localisation Digits identification

Digits Localization :

An image can contain digits in any position and for the digits to be detected we need to first find the regions which contain those digits. The digits can have different sizes and backgrounds.

There are multiple ways to detect location of digits. We can utilize simple image morphological operations like binarization , erosion , dilations to extract digit regions in the images. However these can become too specific to images due to the presence of tuning parameters like threshold , kernel sizes etc. We can also use complex unsupervised feature detectors, deep models etc.

Digits Identification :

The localized digit regions serve as inputs for the digit identification process. MNIST dataset is the canonical data set for handwritten digit identification. Most data scientists have experimented with this data set. It contains around 60,000 handwritten digits for training and 10,000 for testing. Some examples look like :

MNIST Images

However, the digits in real life scenarios are generally very different. They are of different colours and generally printed like the below cases.

Day to day digit images

A bit for research leads us to one more public dataset SVHN — Street View House Numbers dataset. The dataset consists of house-number images gathered from Google’s street view and annotated. Sample images from SVHN below :

SVHN Images

This data set has a variety of digit combinations against many backgrounds and will work better for a generalized model.

Modelling in Keras

We chose this repo for implementing a multiple digit detector. It is well written and easy to follow.

Digit Localization is done using Maximally Stable Extremal Regions (MSER) method which serves as a stable feature detector. MSER is mainly used for blob detection within images. The blobs are continuous sets of pixels whose outer boundary pixel intensities are higher (by a given threshold) than the inner boundary pixel intensities. Such regions are said to be maximally stable if they do not change much over a varying amount of intensities. MSER has a lighter run-time complexity of O(nlog(log(n))) where n is the total number of pixels on the image. The algorithm is also robust to blur and scale. This makes it a good candidate for extraction of text / digits. To learn more about MSER, please check out this link.

Digit recognition is done using a CNN with convolution, maxpool and FC layers that classify each detected region into 10 different digits. The classifier gets to 95% accuracy on the test set.

We tested the repo on a variety of examples and found that it works quite well. See examples shared above.

There were some gaps where either the localizer didn’t work perfectly (digit 1’s location not detected) or the detector failed ( $ detected as 5).

Conclusion

We hope this blog proves to be a good starting point to understand how multi-digit detection pipeline works. We have shared a good github link that can be used to build a model on the SVHN data set. If this model doesn’t work well. you can collect your own data and fine tune the trained model.

I have my own deep learning consultancy and love to work on interesting problems. I have helped many startups deploy innovative AI based solutions. Check us out at — http://deeplearninganalytics.org/. If you have a project that we can collaborate on, then please contact me through my website or at info@deeplearninganalytics.org

You can also see my other writings at: https://medium.com/@priya.dwivedi

References: