How to Make Real-Time Handwritten Text Recognition With Augmentation and Deep Learning
By: Sushant Gautam
Tags: Handwritten, Deep Learning, Machine Learning, TensorFlow, Ocr

Use Convolutional Recurrent Neural Network to recognize the Handwritten line text image without pre segmentation into words or characters. Use CTC loss Function to train.

What is covered here:

Offline Handwritten Recognition

Understand the detailed architecture of the Handwritten Recognition system.

How to use the Data Augmentation technique to increase the accuracy and ability to work in real-time.

Why Deep Learning?

Fig1: Deep Learning vs Machine Learning & Deep Learning performance with Other Algorithm.

Machine Learning needs early Feature Extraction as features and performed classification on it. But Deep Learning acts as a “black box” which do feature extraction and classification on its own.

In above figure shows that the main task is to classify the given image as face or non-face. In the case of machine learning, it needs a feature of images such as edges, color, shape, etc., and performs classification on its own.

But deep learning extracts feature and perform classification on its own. Example of this given image to the Convolutional Neural Networks that every layer of CNN takes the feature and finally Fully Connected Layer perform classification

The main conclusion is Deep Learning self extracts features with deep neural networks and classifies itself. Compare to traditional Algorithms its performance increase with the Amount of Data.

This article is all for building your own handwritten recognition system with TensorFlow. It covers detailed intuition about architecture and how I reach the solution and increase accuracy. In the end, I will provide the Github repo link where the pre-trained model is provided. Note that you can build a handwritten recognition system in any language where the architecture remains the same. In the end, you can build a handwritten recognition system in your own language provided the dataset to it. Let’s get started!

Basic Intuition on How it Works.

Fig2: Visualization Output from each layer

First Use Convolutional Recurrent Neural Network to extract the important features from the handwritten line text Image.

The output before the CNN FC layer (512x100) is passed to the BLSTM which is for sequence dependency and time-sequence operations. The output of BLSTM is 100x80 i.e 100 timesteps and 80 characters including blank.

Then CTC LOSS Alex Graves is used to train the RNN which eliminates the Alignment problem in Handwritten since handwritten have a different alignment of every writer. We just gave them what is written in the image (Ground Truth Text) and BLSTM output, then it calculates loss simply as log("gtText") ; aim to minimize the negative maximum likelihood path.

; aim to minimize the negative maximum likelihood path. Then, CTC finds out the possible paths from the given labels. Loss is given by for (X, Y) pair is:

Fig3: CTC loss formula

Finally, CTC Decode is used to decode the output during Prediction.

Detail Architecture

Fig4: Detail Architecture

Altogether there are 3 steps:

Multi-scale feature Extraction → Convolutional Neural Network 7 Layers Sequence Labeling (BLSTM-CTC) → Recurrent Neural Network (2 layers of LSTM) with CTC Transcription → Decoding the output of the RNN (CTC decode)

Fig5: 3 steps used in handwritten recognition

Now hope that you have understood the basic intuition on how it works. Let’s see the code and methods to increase accuracy.

Code

How to increase Accuracy?

Basically, on improving accuracy related to images you will perform data augmentation where the same image in a different form will be created which is given to the model for training and it can be robust to handle images of different types.

But for real-time handwritten recognition, we need to self creat a handwritten line image shown below to make the system capable of recognizing handwritten lines with a pen also. You may observe that IAM dataset images have thick width of the line. So we perform augmentation both on the IAM dataset and self-created line images to make the system more robust and able to work in real-time.

Fig 6: IAM dataset line Image

Fig 7: Self-created handwritten line Image

For this handwritten recognition task, data augmentation is reduced line thickness, random noise, blur filter, and random stretch. Other techniques will also work well, but I found these 4 techniques are sufficient for now. You notice that I haven’t introduced the random rotation to it, since rotation doesn’t preserve the original property of a handwritten image.

If you see the IAM dataset, the line images are thick in size, we can use the dilation technique to make the thinner in line width where the real-time handwritten found to be thin.

Fig 8: Reduce the line thickness of IAM dataset using dilation

Each augmentation in image output is shown below:

Random Noise Added

Fig 9: Random Noise Added on a given image

Random Stretch

Fig 10: Random Stretch on a given image

Blur Image

Fig 11: Blur on a given image

Complete code to Augment IAM dataset handwritten lines is given below:

IAM dataset Augment

Get Code and data

Implementation found at Handwritten Line Text Recognition with Deep Learning. Real-time handwritten and save it as IAM dataset format. Found at: https://www.kaggle.com/sushant097/english-handwritten-line-dataset

Output

Prediction Output on IAM test dataset

Fig 12: Output on IAM test dataset

Prediction Output on real handwritten image

Fig 13: Output on real handwritten image

Further Improvement

Line segmentation can be added for full paragraph text recognition. For line segmentation, you can use the A* path planning algorithm or CNN model to separate paragraphs into lines.

Better Image preprocessing such as: reduce background noise to handle real-time images more accurately.

Better Decoding approach to improve accuracy. Some of the CTC Decoder found here.

What about localize handwritten on the page and recognize it with an end-to-end approach. (Object localization concept to localize handwritten and segmentation)

Conclusion

We have discussed how CRNN (CNN + LSTM) is able to recognize text in images with the detailed architecture of it. The architecture consists of 7 CNN and 2 LSTM layers and outputs a character-probability matrix. This matrix used for CTC loss calculation and decoding. We have also discussed how can data augmentation techniques increases accuracy and make a real-time handwritten recognition system with detailed code. Finally, further improvement of this system was given.

References

[1] A. Hannun, Sequence ModelingWith CTC (2017),

[2] T. Bluche, J. Louradour, and R. Messina Scan Attend, and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention (2016)

[3] B. Shi and X. Bai and C. Yao An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition (2015)

[4] H. Scheidl Build a Handwritten Recognition System with Tensorflow (2018)