5 Open Source Tools You Can Use to Train and Deploy an OCR Project | by Lai Woen Yon | Towards Data Science

Part I  - 5 open-source tools you can use to train your own data and  deploy it for your next OCR project!Part II - From labelling to serving your OCR model! (Coming soon)
There are a number of cases we need to detect text in an image.
To digitalize their operations, banks need to store all paperwork into a cloud database. For this, the documents must be scanned and converted into a machine-readable format.In the property industry, home buyers and agents typically fill out their agreement forms in paper form. To save the forms in the cloud, you need OCR software, which converts the text into machine-readable files.Among Edtech startups, there are some who heavily rely on OCR. As one example, Photomath, a startup that breaks down problems into simple steps to help people understand mathematics. The app offers users the convenience of scanning questions they have on paper and having them translated into machine-readable format through scanning.
To digitalize their operations, banks need to store all paperwork into a cloud database. For this, the documents must be scanned and converted into a machine-readable format.
In the property industry, home buyers and agents typically fill out their agreement forms in paper form. To save the forms in the cloud, you need OCR software, which converts the text into machine-readable files.
Among Edtech startups, there are some who heavily rely on OCR. As one example, Photomath, a startup that breaks down problems into simple steps to help people understand mathematics. The app offers users the convenience of scanning questions they have on paper and having them translated into machine-readable format through scanning.
OCR can be done using either traditional computer vision techniques or more advanced deep learning techniques. The focus of this article will only be on tools that use deep learning models. As a bonus, I will also include scripts that will allow you to experience all the models at once.
github.com
EasyOCR is an open-source and ready-to-use OCR with almost 80 supported languages. You can choose to train the model with your own data (you can follow their example dataset to format your own dataset) or use the existing models to serve your own application.
The following script can be used to run the code:
# installationpip install easyocr# importimport easyocr# inferencereader = easyocr.Reader(['en'])result = reader.readtext(img_path)
Here’s a quick test to see how accurate this OCR software is. The image below is from pexels.com, and the OCR tool will detect the text in the picture. In a perfect model, it should be able to output “MAKE THIS DAY GREAT!”.
Here is the result:
You can see that the result is not perfect. It misinterpreted the character ‘G’ and misidentified ‘H’ as the small letter ‘h’.
github.com
PaddleOCR is an open-source product developed by the Baidu team in China. I have been using this software tool for quite a while and I am really amazed by how much the team has done to make this free product as powerful as any commercial OCR software in the market. The models used in the framework were trained using State-Of-The-Art (SOTA) techniques (such as CML knowledge distillation and CopyPaste data expansion strategy) and with tons of printed and handwritten images. This makes it one of the most powerful open-source OCR software. Here is a number of things that you can do with the open-source code:
You can use their existing models for your applications. They also provide an extremely lightweight and yet powerful model called PP-OCRv2 so that you don’t need to worry about a large memory problem.They support multiple languages such as Chinese, English, Korean, Japanese, German and etc.They have multiple tools to support you for data labeling. For example, they provide the PPOCRLabel for you to quickly label the text in the image. As data is important to train the OCR model, they also have a tool called Style-text for you to quickly synthesize your image so that you have more images to train your model, making it robust to use in the production environment.You can fine-tune the model on your dataset with the provided script.
You can use their existing models for your applications. They also provide an extremely lightweight and yet powerful model called PP-OCRv2 so that you don’t need to worry about a large memory problem.
They support multiple languages such as Chinese, English, Korean, Japanese, German and etc.
They have multiple tools to support you for data labeling. For example, they provide the PPOCRLabel for you to quickly label the text in the image. As data is important to train the OCR model, they also have a tool called Style-text for you to quickly synthesize your image so that you have more images to train your model, making it robust to use in the production environment.
You can fine-tune the model on your dataset with the provided script.
Here is how you can use it:
# installationpip install paddleocr paddlepaddle# importfrom paddleocr import PaddleOCR# inferenceocr = PaddleOCR(use_angle_cls=True, lang='en')result = ocr.ocr(img_path, cls=True)
Let’s use the same image above to examine the model performance:
It is amazing to see such an accurate output from a model that has yet been trained on a similar image before.
huggingface.co
github.com
TrOCR was initially proposed in TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models by Minghao Li, Tengchao Lv, Lei Cui and etc. It is developed based on the image Transformer encoder and an autoregressive text decoder (Similar to GPT-2). The code has been included in the famous Huggingface library so we can use the trained model directly from the library.
# installationpip install transformers# importfrom transformers import TrOCRProcessor, VisionEncoderDecoderModelfrom PIL import Image# inferencemodel_version = "microsoft/trocr-base-printed"processor = TrOCRProcessor.from_pretrained(model_version)model = VisionEncoderDecoderModel.from_pretrained(model_version)image = Image.open(img_path).convert("RGB")pixel_values = processor(image, return_tensors="pt").pixel_valuesgenerated_ids = model.generate(pixel_values)generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
Unlike the other two, the models only output the final text output without the text location. The model is meant for text recognition so you should not expect it to detect the text in the image. The output that you would receive after running the above script is “MASKAY”. As expected, the result is poor because there isn’t a text detection model in the pipeline.
github.com
MMOCR is another open-source OCR tool that was developed under the famous OpenMMLab project. The project is developed by the team from The Chinese University of Hong Kong and has been one of the leading projects in the area of Computer Vision.
It supports multiple SOTA models and also allows you for training and deployment. Among the highlights of this open-source tool is its ability to provide a key information extraction model in addition to other modules (Text detection and recognition) in the same pipeline. Key information extraction solves problems in which users must match a certain template in order to extract data from a document. The conventional approach would make it difficult to use documents with unknown templates.
Once we have installed the tool (yes, it is not as convenient as the first two tools), we can use it to detect our sample image and test its performance. We get a similar result to PaddleOCR, except that the exclamation mark does not appear at the end of “GREAT”. As a whole, this tool provides almost everything into a single pipeline, so it’s quite powerful to use.
github.com
pypi.org
For people who have done OCR projects before, you should be very familiar with this library. Here is a little bit of history about Tesseract-OCR:
Tesseract was originally developed at Hewlett-Packard Laboratories Bristol and at Hewlett-Packard Co, Greeley Colorado between 1985 and 1994, with some more changes made in 1996 to port to Windows, and some C++izing in 1998. In 2005 Tesseract was open sourced by HP. From 2006 until November 2018 it was developed by Google. — from https://github.com/tesseract-ocr/tesseract
Python-Pytesseract is a wrapper for the Tesseract-OCR engine. Using it is very straightforward. Since the model hasn’t been updated since December 26, 2019, and uses a very simple LSTM-based model, the result might not be as desirable as the rest of the approach. In spite of this, the model can be trained using new data, so that you can tailor it to your needs.
To test the tool, I used the same image above. Despite preprocessing the image with OpenCV, I got an empty result. It seems that the tool does not work well with text on scenes. The performance should be better for normal documents with relatively clean backgrounds.
Here is a comparison of the different open-source OCR tools:
This rating is solely based on my own experience with the open-source tools. Each of them has its own advantages. Our project requires that we select the right tool based on a set of specific requirements.
Here is the Colab notebook if you want to give them a try!
Open-source tools cannot be completely relied upon when deployed for real-world service. Especially for some applications, it requires more than 99% accuracy. In order to accomplish this, we can collect data from our business and label them before training the open-source tool. My next step will be to show you how Label Studio can streamline the entire process. From labeling to serving models, I’ll cover it all!
If you have not read my Label Studio article, feel free to read it here:
towardsdatascience.com
Do follow me and like my post to get the latest updates on the series.
Woen Yon is a Data Scientist based in Singapore. His experience includes developing advanced artificial intelligence products for several multinational enterprises.
Woen Yon has worked with a handful of smart people to offer web solutions including web crawling services and website development for local and international start-up business owners. They are well aware of the challenges of building quality software. Please do not hesitate to drop him an email at wushulai@live.com if you need assistance.
He loves making friends! Feel free to connect with him on LinkedIn and Medium