Pre-Processing in OCR!!!. A basic explanation of the most widely... | by Susmith Reddy | Towards Data Science

Welcome to part II, in the series about working of an OCR system. In the previous post, we briefly discussed the different phases of an OCR system.
Among all the phases of OCR, Preprocessing and Segmentation are the most important phases, as the accuracy of the OCR system highly depends upon how well Preprocessing and Segmentation are performed. So, here we are going to learn some of the most basic and commonly used preprocessing techniques on an image.
The main objective of the Preprocessing phase is To make as easy as possible for the OCR system to distinguish a character/word from the background.
Some of the most basic and important Preprocessing techniques are:-
1) Binarization2) Skew Correction3) Noise Removal4) Thinning and Skeletonization
Before discussing these techniques, let’s understand how an OCR system comprehends an image. For an OCR system, an Image is a multidimensional array (2D array if the image is grayscale (or) binary, 3D array if the image is coloured). Each cell in the matrix is called a pixel and it can store 8-bit integer which means the pixel range is 0–255.
Let’s go through each preprocessing technique mentioned above one-by-one
Binarization: In layman’s terms Binarization means converting a coloured image into an image which consists of only black and white pixels (Black pixel value=0 and White pixel value=255). As a basic rule, this can be done by fixing a threshold (normally threshold=127, as it is exactly half of the pixel range 0–255). If the pixel value is greater than the threshold, it is considered as a white pixel, else considered as a black pixel.
Binarization: In layman’s terms Binarization means converting a coloured image into an image which consists of only black and white pixels (Black pixel value=0 and White pixel value=255). As a basic rule, this can be done by fixing a threshold (normally threshold=127, as it is exactly half of the pixel range 0–255). If the pixel value is greater than the threshold, it is considered as a white pixel, else considered as a black pixel.
But this strategy may not always give us desired results. In the cases where lighting conditions are not uniform in the image, this method fails.
So, the crucial part of binarization is determining the threshold. This can be done by using various techniques.
→ Local Maxima and Minima Method :
C(i,j) is the threshold for a defined size of locality in the image (like a 10x10 size part). Using this strategy we’ll have different threshold values for different parts of the image, depending on the surrounding lighting conditions but the transition is not that smooth.
→ Otsu’s Binarization: This method gives a threshold for the whole image considering the various characteristics of the whole image (like lighting conditions, contrast, sharpness etc) and that threshold is used for Binarizing image.This can be accomplished using OpenCV python in the following way:
ret, imgf = cv2.threshold(img, 0, 255,cv2.THRESH_BINARY,cv2.THRESH_OTSU) #imgf contains Binary image
-> Adaptive Thresholding: This method gives a threshold for a small part of the image depending on the characteristics of its locality and neighbours i.e there is no single fixed threshold for the whole image but every small part of the image has a different threshold depending upon the locality and also gives smooth transition.
imgf = cv2.adaptiveThreshold(img,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY,11,2) #imgf contains Binary image
2. Skew Correction: While scanning a document, it might be slightly skewed (image aligned at a certain angle with horizontal) sometimes. While extracting the information from the scanned image, detecting & correcting the skew is crucial.Several techniques are used for skew correction.
→ Projection profile method→ Hough transformation method→ Topline method→ Scanline method
However, the projection profile method is the simplest, easiest and most widely used way to determine skew in documents.
In this method, First, we’ll take the binary image, then
project it horizontally (taking the sum of pixels along rows of the image matrix) to get a histogram of pixels along the height of the image i.e count of foreground pixels for every row.
Now the image is rotated at various angles (at a small interval of angles called Delta) and the difference between the peaks will be calculated (Variance can also be used as one of the metrics). The angle at which the maximum difference between peaks (or Variance) is found, that corresponding angle will be the Skew angle for the image.
After finding the Skew angle, we can correct the skewness by rotating the image through an angle equal to the skew angle in the opposite direction of skew.
import sysimport matplotlib.pyplot as pltimport numpy as npfrom PIL import Image as imfrom scipy.ndimage import interpolation as interinput_file = sys.argv[1]img = im.open(input_file)# convert to binarywd, ht = img.sizepix = np.array(img.convert('1').getdata(), np.uint8)bin_img = 1 - (pix.reshape((ht, wd)) / 255.0)plt.imshow(bin_img, cmap='gray')plt.savefig('binary.png')def find_score(arr, angle):    data = inter.rotate(arr, angle, reshape=False, order=0)    hist = np.sum(data, axis=1)    score = np.sum((hist[1:] - hist[:-1]) ** 2)    return hist, scoredelta = 1limit = 5angles = np.arange(-limit, limit+delta, delta)scores = []for angle in angles:    hist, score = find_score(bin_img, angle)    scores.append(score)best_score = max(scores)best_angle = angles[scores.index(best_score)]print('Best angle: {}'.formate(best_angle))# correct skewdata = inter.rotate(bin_img, best_angle, reshape=False, order=0)img = im.fromarray((255 * data).astype("uint8")).convert("RGB")img.save('skew_corrected.png')
3. Noise Removal: The main objective of the Noise removal stage is to smoothen the image by removing small dots/patches which have high intensity than the rest of the image. Noise removal can be performed for both Coloured and Binary images.One way of performing Noise removal by using OpenCV fastNlMeansDenoisingColored function.
import numpy as np import cv2 from matplotlib import pyplot as plt # Reading image from folder where it is stored img = cv2.imread('bear.png') # denoising of image saving it into dst image dst = cv2.fastNlMeansDenoisingColored(img, None, 10, 10, 7, 15) # Plotting of source and destination image plt.subplot(121), plt.imshow(img) plt.subplot(122), plt.imshow(dst) plt.show()
More about Noise removal & Image smoothening techniques can be found in this wonderful article
4. Thinning and Skeletonization: This is an optional preprocessing task which depends on the context in which the OCR is being used.  → If we are using the OCR system for the printed text, No need of performing this task because the printed text always has a uniform stroke width.  → If we are using the OCR system for handwritten text, this task has to be performed since different writers have a different style of writing and hence different stroke width. So to make the width of strokes uniform, we have to perform Thinning and Skeletonization.
This can be performed using OpenCV in the following way
import cv2import numpy as npimg = cv2.imread('j.png',0)kernel = np.ones((5,5),np.uint8)erosion = cv2.erode(img,kernel,iterations = 1)
In the above code, Thinning of the image depends upon kernel size and no.of iterations.
In this article, we have seen some of the basic and most widely used Preprocessing techniques which gives us a basic idea of what’s happening inside the OCR system. An example of preprocessing workflow can be seen in the below image.
I hope you got an essence of how Preprocessing is performed in the OCR.
In part-III, we’ll see the Segmentation techniques used by the OCR system.
Happy Learning !!!!
Any doubts, Suggestions & Corrections are Welcome. 😃
[1] Shafii, M., Sid-Ahmed, M. Skew detection and correction based on an axes-parallel bounding box. IJDAR 18, 59–71 (2015). https://doi.org/10.1007/s10032-014-0230-y
[2] Jyotsna, S. Chauhan, E. Sharma and A. Doegar, “Binarization techniques for degraded document images — A review,” 2016 5th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, 2016, pp. 163–166, doi: 10.1109/ICRITO.2016.7784945.
[3] A. Papandreou and B. Gatos, “A Novel Skew Detection Technique Based on Vertical Projections,” 2011 International Conference on Document Analysis and Recognition, Beijing, 2011, pp. 384–388, doi: 10.1109/ICDAR.2011.85.
[4] K. Lin, T. H. Li, S. Liu and G. Li, “Real Photographs Denoising With Noise Domain Adaptation and Attentive Generative Adversarial Network,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 2019, pp. 1717–1721, doi: 10.1109/CVPRW.2019.00221.
[5] Choudhary, Amit & Rishi, Rahul & Savita, Ahlawat. (2013). A New Character Segmentation Approach for Off-Line Cursive Handwritten Words. Procedia Computer Science. 17. 88–95. 10.1016/j.procs.2013.05.013.