Checkbox/Table cell detection using OpenCV-Python | by Sreekiran A R | Towards Data Science

In this age of Digital Transformation, Information Extraction is one of the key areas of Business interest, where we need to extract relevant information from unstructured data sources like scanned invoices, bills, etc into structured data, using Computer Vision and Natural Language Processing. Here, the primary steps we are dependent on are Optical Character Recognition and Document Layout Analysis.
Optical Character Recognition (OCR) is for detecting the text from the image, and Document Layout Analysis(DLA) is where we try to get additional metadata from the documents like identifying headers, paragraphs, lines, words, tables, key-value pairs, etc.
In this article, we are focusing on solving one problem in DLA, detecting boxes in an image, which can be a checkbox or a Table cell. I am taking you through the journey of solving this problem with three examples
Create a Version0 Solution for Digital documents with checkboxes.Test the solution on a low-quality scanned document with checkboxes, find the corner cases, and finetune the algorithm to create a Version1.Test the Version1 solution on a low-quality scanned document with a table and see how it works out.
Create a Version0 Solution for Digital documents with checkboxes.
Test the solution on a low-quality scanned document with checkboxes, find the corner cases, and finetune the algorithm to create a Version1.
Test the Version1 solution on a low-quality scanned document with a table and see how it works out.
The term Born-Digital refers to materials that originated in digital form. The advantage of a Born-digital document is that it is the purest form of data. We are trying to solve this ideal scenario first. In reality, most of the documents we will be dealing with are scanned documents, where issues like poor quality, noise, skewness & orientation issues, etc are to be taken into consideration to arrive at a solution.
But let’s focus on our first solution, as Homer Simpson would say
That’s a problem for future Me! Man I don’t envy that guy :P
This is the sample image we are using for solving our problem.
import osimport cv2import numpy as npimage_path='digital.jpg'image=cv2.imread(image_path)
For a lot of Image Processing problems that you will encounter in the future, Binarisation will be the primary step.
Why?
The original image we get will have 3 channels and each channel will represent the Blue, Green, and Red values of each pixel of the image. But, for a lot of processes, we do not need that much information.
So we convert the image to grayscale, Where the original 3 channel image is reduced to a single-channel image and each pixel value is between 0 and 255, 0 being black and 255 being white.
Still, to reduce the complexity further, we do the Thresholding. It is nothing but making all the pixels either ON or OFF based on a threshold on the pixel values. Hence, the term Binarisation.
Now, we have a simple image with all the pixels either 0 or 255. As Steve Jobs said, sometimes
Simplicity is the Ultimate Sophistication!
gray_scale=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)th1,img_bin = cv2.threshold(gray_scale,150,225,cv2.THRESH_BINARY)img_bin=~img_bin
After binarisation, the image looks like this.
When solving Image processing problems, Morphological Transformation is a term that you will come across a lot of times.
We need to know these four terms; Dilation, Erosion, Opening, Closing.
In simple terms,Dilation: Makes the image thicker → Used to join broken partsErosion: Makes the image thinner → Used to remove noise, separate nearby componentsOpening: Erosion then Dilation → Used in Noise removalClosing: Dilation then Erosion → Filling small holes
You can read more about it from here.
Now, we create two kernels; for detecting horizontal and vertical lines.
Assumption 1: we are assuming a checkbox will be at least 15x15 size.
Why?
To avoid boxes in text getting detected as False positives. Why did I choose 15 pixels? Just a value to get started. You might need to change it later depending on our data, or we can set the line_min_width as a function of the image size.
### selecting min size as 15 pixelsline_min_width = 15kernal_h = np.ones((1,line_min_width), np.uint8)kernal_v = np.ones((line_min_width,1), np.uint8)
Applying Horizontal Kernel on the image
img_bin_h = cv2.morphologyEx(img_bin, cv2.MORPH_OPEN, kernal_h)
Applying Vertical Kernel on the image
img_bin_v = cv2.morphologyEx(img_bin, cv2.MORPH_OPEN, kernal_v)
Merging Horizontal and Vertical Images
img_bin_final=img_bin_h|img_bin_v
Now, as you can see from the above image, we can filter most of the text as noise, we do a Connected Component Analysis on the image to get the Bounding Boxes of the checkboxes.
Basically, what it does is simply what the Bucket icon in our childhood hero, Microsoft Paint does!
_, labels, stats,_ = cv2.connectedComponentsWithStats(~img_bin_final, connectivity=8, ltype=cv2.CV_32S)
So, after calling the function, we are making use of two outputs; labels and stats.
labels is a GrayScale image with each connected Component getting assigned to a different value from its neighbour.
stats is a tuple of lists with each tuple having (x,y,w,h,area) of the detected bounding boxes.
Visualizing Connected Component Image
As you can see from the above image, all the checkboxes are identified. Now Let’s use the stats to draw the detected boxes on the input image.
for x,y,w,h,area in stats[2:]:    cv2.rectangle(image,(x,y),(x+w,y+h),(0,255,0),2)
If you noticed, I have skipped the first 2 values of stats. Why?
They are the background and residue pixel bounding boxes, which we do not require in this case. You can try to plot the stats values to get a better idea.
Voila!! we have our Version0 Solution!
Now that we have Version 0, Let’s test it on a low-quality document and see how it performs..!
This is our input image:
Current Algorithm output:
Whoaa..! Almost, we missed two boxes...Let’s see our CC image to see why this happened.
As you can see, there is a slight gap in the boxes that we could not detect, resulting in the overflow.
So, what technique do we apply for joining broken parts? Yes, you guessed it Right! We are going to apply a layer of Dilation to the image to sort it. Before, calling the Connected Components function, we add
final_kernel = np.ones((3,3), np.uint8)img_bin_final=cv2.dilate(img_bin_final,final_kernel,iterations=1)
Now, let's see the output!
As you can see, we have detected all the bounding boxes correctly.
So, we have successfully solved a corner case and created version 1.
Let’s see the complete code now. I created it as a function.
def detect_box(image,line_min_width=15):    gray_scale=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)    th1,img_bin=cv2.threshold(gray_scale,150,225,cv2.THRESH_BINARY)    kernal_h=np.ones((1,line_min_width), np.uint8)    kernal_v=np.ones((line_min_width,1), np.uint8)    img_bin_h=cv2.morphologyEx(~img_bin, cv2.MORPH_OPEN, kernal_h)    img_bin_v=cv2.morphologyEx(~img_bin, cv2.MORPH_OPEN, kernal_v)    img_bin_final=img_bin_h|img_bin_v    final_kernel=np.ones((3,3), np.uint8)   img_bin_final=cv2.dilate(img_bin_final,final_kernel,iterations=1)    ret, labels, stats,centroids = cv2.connectedComponentsWithStats(~img_bin_final, connectivity=8, ltype=cv2.CV_32S)    return stats,labels
It took less than 15 lines of code to solve this problem. Isn’t OpenCV amazing!?
Now, let’s test our algorithm on a low-quality document for table detection.
Input Image:
Output Images:
Looks rather beautiful ain’t it?
There it is!! We were able to extract all the boxes from an image. Notice that here I’m not skipping any stats we detected. So, I’m getting the entire image as a bounding box, and also the full table itself as a bounding box.
Now that we can detect checkboxes and table cells from an image, Let’s check how can we make use of the information.
Here, we were able to identify all the table cells. But, if it’s a table, a client would like to extract it into a structured format like CSV or Excel Sheet.
How can we achieve that?
As I said above, the stats variable will give the Bounding Box and area of all the table cells and the table itself.
Now, it's just a matter of Coordinate Geometry.
As most Math teachers would say,
Most of those were covered in your previous year, I won’t be covering them :P
Usually, the use case associated with a checkbox will be mostly,
Whether it’s checked or not → Check if a bounding box is empty or notWhat is the text content inside the box (cross or tick usually)?
Whether it’s checked or not → Check if a bounding box is empty or not
What is the text content inside the box (cross or tick usually)?
As most Math teachers would say,
Most of those will be covered in next year, So I am skipping them :P
Thanks for spending your valuable time, hope you had fun learning!
Please find the full code explained in this GitHub repo. You can also check out my StackOverflow answer for the same.
Please connect with me on LinkedIn for feedback and suggestions.
As always,
Happy Coding :D