Creating a Japanese Handwriting Recognizer | by Aiyu Kamate | Towards Data Science

This summer, I worked on a machine learning project to create an App that recognizes Japanese characters, using python’s tensorflow library.
Before we get started, here’s the final result!
We will need to install a couple libraries. Here are the versions I used:
Python 3.7 (any python3 version should work)Tensorflow 2.2.0Keras 2.4.3Numpy 1.16.4 (this isn’t the newest version but this prevents tensorflow from printing bunch of future warnings)matplotlib — newest versionPIL — newest versionskimage — newest versionsklearn — newest versioncorelmltools 3.2
Python 3.7 (any python3 version should work)
Tensorflow 2.2.0
Keras 2.4.3
Numpy 1.16.4 (this isn’t the newest version but this prevents tensorflow from printing bunch of future warnings)
matplotlib — newest version
PIL — newest version
skimage — newest version
sklearn — newest version
corelmltools 3.2
If you don’t know how to install the libraries, type this in your terminal:
pip install library_name (e.g. matplotlib)
To install a specific library:
pip install library_name==version_name (e.g. tensorflow==2.2.0)
Note that if tensorflow doesn’t work after installation, try running the following command in terminal:
/Applications/Python 3.7/Install Certificates.command
Change the 3.7 to the python version you’re using if you are using a different version.
There are two main machine learning problems: Classification problem, and Regression problem. The main difference between the two is that a Classification problem have categorical answers, while a Regression problem has continuous values as its answers. An example of a classification problem would be predicting whether an image is a cat or a dog, while an example of a regression problem would be predicting prices of houses.
Handwriting Recognition is a Classification problem.
Here’s what you should do to create a machine learning model that can recognize handwriting, or any other Classification Problems in general:
Get enough data. You need thousands of examples to train your model to learn something it had zero knowledge on. The more, the better.Give labels to each data, so your model knows the answers. (some database may already do the steps 1 and 2 for your convenience)Split your data into training data and testing data, so your model can learn from some examples, and test its newly learnt knowledge on some others. A good ratio is about 8:2 for training data:testing data.Build your model. Here’s the fun and time consuming part. Each dataset will have different models that suit them the most. You just have to try different models and tweak minor parts to find the best model.Run your code and train your model!
Get enough data. You need thousands of examples to train your model to learn something it had zero knowledge on. The more, the better.
Give labels to each data, so your model knows the answers. (some database may already do the steps 1 and 2 for your convenience)
Split your data into training data and testing data, so your model can learn from some examples, and test its newly learnt knowledge on some others. A good ratio is about 8:2 for training data:testing data.
Build your model. Here’s the fun and time consuming part. Each dataset will have different models that suit them the most. You just have to try different models and tweak minor parts to find the best model.
Run your code and train your model!
I used the ETL Character Database, which had over ten thousands of Japanese characters written by different people, and the KMNIST database, which had old Japanese characters. Here are the 5 types of machine learning models I created:
Hiragana (ETL 8)
Katakana (ETL 1)
Kanji (ETL 8)
Kuzushiji (KMNIST)
3036 Kanji (ETL 9)
To explain briefly, Kanji are Chinese characters that were adopted in Japanese, while Hiragana and Katakana are the syllabic scripts (essentially the Japanese alphabets); Hiragana is the main alphabet, and each Hiragana has a corresponding Katakana, but Katakana are used for foreign words. Kuzushiji is the old Japanese Hiragana, but much more complex.
After downloading the ETL 1 and 8 dataset, I had to extract the data to fit my tensorflow model. Here’s how exactly I manipulated the Katakana dataset:
Here’s a simple explanation:
The contents of the read_record function was provided on the ETL website; it basically reads the file’s data.the read_kana function extracts the data and formats it into numpy arrays; this is useful because tensorflow machine learning and numpy arrays work nicely together. We also group the same characters together so that we can make the labels easily.
The contents of the read_record function was provided on the ETL website; it basically reads the file’s data.
the read_kana function extracts the data and formats it into numpy arrays; this is useful because tensorflow machine learning and numpy arrays work nicely together. We also group the same characters together so that we can make the labels easily.
After this, I reshaped each image size to 48*48 pixels, created the labels for each characters, and split the images and labels to train images, train labels, test images, and test labels. Here’s my code:
I did the same thing for Hiragana and Kanji with some minor changes (e.g. number of characters and writers, image sizes, the way the read the original file, the duplicate characters are slightly different), so I won’t go into detail here. Check out my github page to see the exact code!
github.com
KMNIST already provided each of training images, training labels, testing images, and testing labels, so no extra steps are necessary here.
A very helpful tool to visualize your data is matplotlib.pyplot; this helps a lot with debugging, because you can see if all the data were extracted/formatted correctly. Here’s an example:
Finally, the fun part: training the model!
We will be using a Convolutional Neural Network (CNN). CNN consists of the process of convolution and pooling, and essentially keeps the important parts of an image so that the model can make a better prediction.
We will also be using ImageDataGenerator in tensorflow.keras, which is a technique of image data augmentation that creates modified versions of images, such as rotated, zoomed, and flipped versions, to train so that the model doesn’t overfit (recognizes training examples well but doesn’t understand different kind of outside data).
After adding/removing layers and tweaking the exact numbers in each layers, I was able to create a pretty good model for all characters. To be exact,
Katakana - 98.6% accuracyHiragana - 98.4% accuracyKanji (879 Kanji characters that are used regularly) - 98.6% accuracyKuzushiji - 95.6% accuracyKanji3036 (3036 Kanji characters, includes ones that aren’t used usually) - 97.9% accuracy
Katakana - 98.6% accuracy
Hiragana - 98.4% accuracy
Kanji (879 Kanji characters that are used regularly) - 98.6% accuracy
Kuzushiji - 95.6% accuracy
Kanji3036 (3036 Kanji characters, includes ones that aren’t used usually) - 97.9% accuracy
Here are the exact CNN model I used for each Japanese Characters:
To show an example, I’ll show the exact code for my Katakana model. The model I created for the other characters can be seen in my github page.
github.com
Some explanation of my models:
CNN and a neural network were used for all models. The characters with more classes have a larger neural network after the CNN. The activation used for most models were relu , one of the most popular activation due to its simplicity, but tanh was used for Kanji3036 due to the large number of classes.ImageDataGenerator was used since people may have smaller handwriting or write in a tilted way. However, ImageDataGenerator was not used for Kuzushiji, since there are too many different variations of writing for each character, and this actually hurts the model.The optimizer parameter was set to adam since it can achieve the best results very fast, loss was set to sparse_categorical_crossentropy , and metrics was set to accuracy .fit_generator was used to train the model, since ImageDataGenerator was used.ReduceLROnPlateau was used to reduce the learning rate if the accuracy wasn’t improving for a while, and EarlyStopping was used to stop the training if the accuracy still didn’t improve for a longer time. Reducing the learning rate helped improve the accuracy by 1–2%, while early stopping prevented the model from overfitting to the training data..h5 is basically the format to save your model so that you don’t need to re-run your code every time you want to re-train your model.
CNN and a neural network were used for all models. The characters with more classes have a larger neural network after the CNN. The activation used for most models were relu , one of the most popular activation due to its simplicity, but tanh was used for Kanji3036 due to the large number of classes.
ImageDataGenerator was used since people may have smaller handwriting or write in a tilted way. However, ImageDataGenerator was not used for Kuzushiji, since there are too many different variations of writing for each character, and this actually hurts the model.
The optimizer parameter was set to adam since it can achieve the best results very fast, loss was set to sparse_categorical_crossentropy , and metrics was set to accuracy .
fit_generator was used to train the model, since ImageDataGenerator was used.
ReduceLROnPlateau was used to reduce the learning rate if the accuracy wasn’t improving for a while, and EarlyStopping was used to stop the training if the accuracy still didn’t improve for a longer time. Reducing the learning rate helped improve the accuracy by 1–2%, while early stopping prevented the model from overfitting to the training data.
.h5 is basically the format to save your model so that you don’t need to re-run your code every time you want to re-train your model.
Once you have your h5 file, you can load it by keras.models.load_model("your_model_name.h5"), see your model by model.summary() , and test the accuracy on test data by model.evaluate(test_images,test_labels).
Before creating the app, we need use coremltools to convert the .h5 format to .mlmodel format to be able to use the models in the app. Here, we need to downgrade keras 2.4.3 to 2.2.4, because coremltools only supports keras 2.2.4 or older versions.
keras 2.2.4 also have some issues with tensorflow 2, so I had to run the following command to upgrade a keras file:
I also had to go into the optimizers.py file within keras and manually change this in line 75:
original: allowed_kwargs = {'clipnorm','clipvalue'}
changed version: allowed_kwargs = {'clipnorm','clipvalue','name','learning_rate'}
Finally, run the following code:
The katakanajapanese file can be found here.
input_names="image" changes the input from an array to an image. Very important step because on XCode, handling image is much easier.
As always, the code for the other Japanese characters can be seen here:
github.com
Finally, let’s get into actually making the app for iOS!
Here’s the general process.
Create a new XCode project and choose the single view app and swift language. Then drag and drop your mlmodel to the project.Create a space where you can draw things by hand touch, and create a button that clears the handwriting.Create a system that checks the handwritten character and detects the character through the imported mlmodel.
Create a new XCode project and choose the single view app and swift language. Then drag and drop your mlmodel to the project.
Create a space where you can draw things by hand touch, and create a button that clears the handwriting.
Create a system that checks the handwritten character and detects the character through the imported mlmodel.
The exact implementation can be seen here:
The functions touchesBegan and touchesMoved handle events when the screen is touched; in this code, it allows the user to handwrite.The classification and recognize functions together do the process of recognizing the user’s handwriting and predicting what the character is. convertImage makes the input size (48*48 pixels in this case) fit the model.
The functions touchesBegan and touchesMoved handle events when the screen is touched; in this code, it allows the user to handwrite.
The classification and recognize functions together do the process of recognizing the user’s handwriting and predicting what the character is. convertImage makes the input size (48*48 pixels in this case) fit the model.
The code here makes the most basic app for recognizing handwritten Japanese!
You can add features by providing second and third choice characters and enabling switching modes between Hiragana, Katakana, and Kanji, for example.
Though the codes of the models themselves are actually short, creating the best models was definitely the most time consuming part of the project.
The results were rewarding; about 98~99% accuracy for all characters, especially for Kanji, is good!
In future machine learning projects, I’d like to work on larger and more complicated dataset and achieve higher accuracy, and also try Regression problems.
Perhaps I could try to predict the number of gold medals Japan would win at Tokyo Olympics next year!
[1] Electrotechnical Laboratory, Japanese Technical Committee for Optical Character Recognition, ETL Character Database, 1973–1984.
[2] Tarin Clanuwat and Mikel Bober-Irizar and Asanobu Kitamoto and Alex Lamb and Kazuaki Yamamoto and David Ha, Deep Learning for Classical Japanese Literature, KMNIST, 2018/12/3