LSTM-based Handwriting Recognition by Google
By: Edward Ma
Tags: Machine Learning, Artificial Intelligence, Data Science, Handwriting, Naturallanguageprocessing

Handwriting is a one of the challenge in NLP task. It is because it can be various among different people. On the other hand, some characters (e.g. English) are quite similar. As a human beginning, we leverage contextualize information, lexicon matching.

Sometimes, “O” can be written as “0” while human begin has the capability to distinguish whether it is “O” or “0” from contextualize information. For example, “0” will be used in phone number while “O” will be used as part of English word. Another skill is lexicon searching. It helps to guess words even though we cannot recognize every single characters.

Handwriting in different language (Carbune et al., 2019)

How can we tackle in via deep learning era? This story will discuss about Fast Multi-language LSTM-based Online Handwriting Recognition (Carbune et al., 2019) and the following are will be covered:

Data

Architecture

Experiment

Data

Carbune et al. leverage both open and close dataset to validate the model. As usual, IAM-OnDB dataset is used to train a model. During the experiment, they use two representations which are Raw Touch Points and Bézier Curves .

Raw Touch Points

Data will be convert to 5-dimensional points which are x coordinate, y coordinate, timestamp of the touchpoint since the first touch, pen-up or pen-down and new stroke or not. Some preprocessing a necessary

Since the size of image can be various, normalization of x and y coordinates are necessary. Surrogate are 20% larger than observed touch points if writing area is unknown

Equidistant linear resampling along the strokes with value of 0.05. In other word, a line of length 1 will have 20 points.

Bézier Curves

Authors also evaluate Bézier Curves whether it is better than Raw Touch Points . From experiment, Bézier Curves demonstrates a better result. Bézier Curves is a natural way to present trajectories in space. Authors found that it is more promising that Raw Touch Points . Authors use cubic polynomials of Bézier curves to calculate features.

Show “go” in bézier curve method (Google AI Blog)

You may check out this story if you want to refresh the concept of Bézier Curves . After calculating the curve, 10-dimensional vectors will be generated and feeding into neural network. They are :

Vector between the endpoints (red dots). It is blue line in the figure.

Distances between the control points (green dots) and the endpoints. They are green dashed lines in the figure.

Angels between each control point and end points. That is green arcs in the figure.

Three time coefficients (Not shown in figure)

A boolean indicator whether this is pen-up or pen-down

Bézier curve used to feed the network (Carbune et al., 2019)

Architecture

Handwriting challenge is a well defined problem and there are multiple approaches to deal with it. Segment-and-decode classifiers is one of the example. It splits word to sub-words and classifying it one by one. Another line is using Hidden Markov Models (HMM). It use a chain concept to recognize handwriting and returning string.

Carbune et al. use Bézier Curves as a feature and feeding into Bidirectional LSTM to learn the feature and using softmax layer to get a probability distribution over all possible characters. This is not the end of model, softmax layer generates a sequence of classes and Carbune et al. use Connectionist Temporal Classification (CTC) to decode it and getting the finalized output.

Experiment

Authors measure the character error rates (CER) between different model on IAM-OnDb. For CER, lower is better. You may notice that deeper LSTM is better and curves to outperform than raw touch points.

Comparison on IAM-OnDB for different input representation and layers (Carbune et al., 2019)

Comparison among different models (Carbune et al., 2019)

CER of Bézier Curves model (Carbune et al., 2019)

Take Away

Different from previous study, this approach calculate input features (ie. Bézier Curves ) rather than using CNN or other neural network to learn the feature.

) rather than using CNN or other neural network to learn the feature. Bézier Curves seems quite make sense that it can represent the input without any loss of data.

About Me

I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related. Feel free to connect with me on LinkedIn or following me on Medium or Github.

Extension Reading

More about Bézier Curves

Reference

V. Carbune, P. Gonnet, T. Deselaers, H. A. Rowley, A. Daryin, M Calvo, L. L. Wang, D. Keysers, S. Feuz, P. Gervais. Fast Multi-language LSTM-based Online Handwriting Recognition. 2019.