Checkbox/Table cell detection using OpenCV-Python
By: Sreekiran A R
Tags: Python, Data Science, Artificial Intelligence, Opencv, Programming

Source: Image by Author

Introduction

In this age of Digital Transformation, Information Extraction is one of the key areas of Business interest, where we need to extract relevant information from unstructured data sources like scanned invoices, bills, etc into structured data, using Computer Vision and Natural Language Processing. Here, the primary steps we are dependent on are Optical Character Recognition and Document Layout Analysis.

Optical Character Recognition (OCR) is for detecting the text from the image, and Document Layout Analysis(DLA) is where we try to get additional metadata from the documents like identifying headers, paragraphs, lines, words, tables, key-value pairs, etc.

In this article, we are focusing on solving one problem in DLA, detecting boxes in an image, which can be a checkbox or a Table cell. I am taking you through the journey of solving this problem with three examples

Create a Version0 Solution for Digital documents with checkboxes. Test the solution on a low-quality scanned document with checkboxes, find the corner cases, and finetune the algorithm to create a Version1. Test the Version1 solution on a low-quality scanned document with a table and see how it works out.

1. Version 0 — Box detection for Digital Documents

The term Born-Digital refers to materials that originated in digital form. The advantage of a Born-digital document is that it is the purest form of data. We are trying to solve this ideal scenario first. In reality, most of the documents we will be dealing with scanned documents, where issues like poor quality, noise, skewness & orientation issues, etc are to be taken into consideration to arrive at a solution.

But let’s focus on our first solution, as Homer Simpson would say

That’s a problem for future Me! Man I don’t envy that guy :P

This is the sample image we are using for solving our problem.

1.1 Import Necessary libraries and reading the image

import os

import cv2

import numpy as np image_path='digital.jpg'

image=cv2.imread(image_path)

1.2 Binarisation

For a lot of Image Processing problems that you will encounter in the future, Binarisation will be the primary step.

Why?

The original image we get will have 3 channels and each channel will represent the Blue, Green, and Red values of each pixel of the image. But, for a lot of processes, we do not need that much information.

So we convert the image to grayscale, Where the original 3 channel image is reduced to a single-channel image and each pixel value is between 0 and 255, 0 being black and 255 being white.

Still, to reduce the complexity further, we do the Thresholding. It is nothing but making all the pixels either ON or OFF based on a threshold on the pixel values. Hence, the term Binarisation.

Now, we have a simple image with all the pixels either 0 or 255. As Steve Jobs said, sometimes

Simplicity is the Ultimate Sophistication!

gray_scale=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)

th1,img_bin = cv2.threshold(gray_scale,150,225,cv2.THRESH_BINARY)

img_bin=~img_bin

After binarisation, the image looks like this.

binary image: [Source: Image by Author]

1.3 Applying Morphological Operations

When solving Image processing problems, Morphological Transformation is a term that you will come across a lot of times.

We need to know these four terms; Dilation, Erosion, Opening, Closing.

In simple terms,

Dilation: Makes the image thicker → Used to join broken parts

Erosion: Makes the image thinner → Used to remove noise, separate nearby components

Opening: Erosion then Dilation → Used in Noise removal

Closing: Dilation then Erosion → Filling small holes

You can read more about it from here.

Now, we create two kernels; for detecting horizontal and vertical lines.

Assumption 1: we are assuming a checkbox will be at least 15x15 size.

Why?

To avoid boxes in text getting detected as False positives.

Why did I choose 15 pixels? Just a value to get started. You might need to change it later depending on our data, or we can set the line_min_width as a function of the image size.

### selecting min size as 15 pixels

line_min_width = 15

kernal_h = np.ones((1,line_min_width), np.uint8)

kernal_v = np.ones((line_min_width,1), np.uint8)

Applying Horizontal Kernel on the image

img_bin_h = cv2.morphologyEx(img_bin, cv2.MORPH_OPEN, kernal_h)

Source: Image by Author

Applying Vertical Kernel on the image

img_bin_v = cv2.morphologyEx(img_bin, cv2.MORPH_OPEN, kernal_v)

Source: Image by Author

Merging Horizontal and Vertical Images

img_bin_final=img_bin_h|img_bin_v

Source: Image by Author

Now, as you can see from the above image, we can filter most of the text as noise, we do a Connected Component Analysis on the image to get the Bounding Boxes of the checkboxes.

Basically, what it does is simply what the Bucket icon in our childhood hero, Microsoft Paint does!

_, labels, stats,_ = cv2.connectedComponentsWithStats(~img_bin_final, connectivity=8, ltype=cv2.CV_32S)

So, after calling the function, we are making use of two outputs; labels and stats .

labels is a GrayScale image with each connected Component getting assigned to a different value from its neighbor.

stats is a tuple of lists with each tuple having (x,y,w,h,area) of the detected bounding boxes.

Visualizing Connected Component Image

Source: Image by Author

As you can see from the above image, all the checkboxes are identified. Now Let’s use the stats to draw the detected boxes on the input image.

for x,y,w,h,area in stats[2:]:

cv2.rectangle(image,(x,y),(x+w,y+h),(0,255,0),2

If you noticed, I have skipped the first 2 values of stats. Why?

They are the background and residue pixel bounding boxes, which we do not require in this case. You can try to plot the stats values to get a better idea.

Source: Sampleforms [https://www.sampleforms.com/] [modified by Author]

Voila!! we have our Version0 Solution!

2. Testing on Low-Quality documents

Now that we have Version 0, Let’s test it on a low-quality document and see how it performs..!

This is our input image:

Current Algorithm output:

Source: 2Captha [https://2captcha.com/blog/_scanned] [Modified by Author]

Whoaa..! Almost, we missed two boxes…Let’s see our CC image to see why this happened.

Zoomed image to understand the issue [Source: Image by Author]

As you can see, there is a slight gap in the boxes that we could not detect, resulting in the overflow.

So, what technique do we apply for joining broken parts? Yes, you guessed it Right! We are going to apply a layer of Dilation to the image to sort it. Before, calling the Connected Components function, we add

final_kernel = np.ones((3,3), np.uint8)

img_bin_final=cv2.dilate(img_bin_final,final_kernel,iterations=1)

Now, let's see the output!

CC image [Source: Image by Author]

Final output [Source: 2Captha [https://2captcha.com/blog/_scanned] [Modified by Author]]

As you can see, we have detected all the bounding boxes correctly.

So, we have successfully solved a corner case and created version 1.

Let’s see the complete code now. I created it as a function.

def detect_box(image,line_min_width=15):

gray_scale=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)

th1,img_bin=cv2.threshold(gray_scale,150,225,cv2.THRESH_BINARY)

kernal_h=np.ones((1,line_min_width), np.uint8)

kernal_v=np.ones((line_min_width,1), np.uint8)

img_bin_h=cv2.morphologyEx(~img_bin, cv2.MORPH_OPEN, kernal_h)

img_bin_v=cv2.morphologyEx(~img_bin, cv2.MORPH_OPEN, kernal_v)

img_bin_final=img_bin_h|img_bin_v

final_kernel=np.ones((3,3), np.uint8)

img_bin_final=cv2.dilate(img_bin_final,final_kernel,iterations=1)

ret, labels, stats,centroids = cv2.connectedComponentsWithStats(~img_bin_final, connectivity=8, ltype=cv2.CV_32S)

return stats,labels

It took less than 15 lines of code to solve this problem. Isn’t OpenCV amazing!?

3. Testing for detecting Table cells

Now, let’s test our algorithm on a low-quality document for table detection.

Input Image:

Source: Image by Author

Output Images:

Source: Image by Author

Looks rather beautiful ain’t it?

Output [Source: Image by Author]

There it is!! We were able to extract all the boxes from an image. Notice that here I’m not skipping any stats we detected. So, I’m getting the entire image as a bounding box, and also the full table itself as a bounding box.

4. What are the next steps?

Now that we can detect checkboxes and table cells from an image, Let’s check how can we make use of the information.

4.2 Table Extraction

Here, we were able to identify all the table cells. But, if it’s a table, a client would like to extract it into a structured format like CSV or Excel Sheet.

How can we achieve that?

As I said above, the stats the variable will give the Bounding Box and area of all the table cells and the table itself.

Now, it's just a matter of Coordinate Geometry.

As most Math teachers would say,

Most of those were covered in your previous year, I won’t be covering them :P

4.1 Checkboxes

Usually, the use case associated with a checkbox will be mostly,

Whether it’s checked or not → Check if a bounding box is empty or not What is the text content inside the box (cross or tick usually)?

As most Math teachers would say,

Most of those will be covered in next year, So I am skipping them :P

Conclusion

Thanks for spending your valuable time, hope you had fun learning!

Please find the full code explained in this GitHub repo. You can also check out my StackOverflow answer for the same.

Please connect with me at LinkedIn for feedbacks and suggestions.

As always,