A table detection, cell recognition and text extraction algorithm to convert tables in images to excel files | by Hucker Marius | Towards Data Science

Let’s say you have a table in an article, pdf or image and want to transfer it into an excel sheet or dataframe to have the possibility to edit it. Especially in the field of preprocessing for Machine Learning this algorithm will be exceptionally helpful to convert many images and tables to editable data. In the case that your data exists of text-based PDFs there is already a handful of free solutions. The most popular ones are tabular, camelot/excalibur, which you can find under https://tabula.technology/, https://camelot-py.readthedocs.io/en/master/, https://excalibur-py.readthedocs.io/en/master/.
However, what if your PDF is image-based or if you find an article with a table online? Why not just take a screenshot and convert it into an excel sheet? Since there seems to be no free or open source software for image-based data (jpg, png, image-based pdf etc.) the idea came up to develop a generic solution to convert tables into editable excel-files.
But that’s enough for now, let’s see how it works.
The algorithm consists of three parts: the first is the table detection and cell recognition with Open CV, the second the thorough allocation of the cells to the proper row and column and the third part is the extraction of each allocated cell through Optical Character Recognition (OCR) with pytesseract.
As most table recognition algorithms, this one is based on the line structure of the table. Clear and detectable lines are necessary for the proper identification of cells. Tables with broken lines, gaps and holes lead to a worse identification and the cells only partially surrounded by lines are not detected. In case some of your documents have broken lines make sure to read this article and repair the lines: Click here.
First, we need the input data, which is in my case a screenshot in png-format. The goal is to have a dataframe and excel-file with the identical tabular structure, where each cell can be edited and used for further analysis.
Let’s import the necessary libraries.
For more information on the libraries:cv2 — https://opencv.org/pytesseract — https://pypi.org/project/pytesseract/
import cv2import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport csvtry:    from PIL import Imageexcept ImportError:    import Imageimport pytesseract
The first step is to read in your file from the proper path, using thresholding to convert the input image to a binary image and inverting it to get a black background and white lines and fonts.
#read your filefile=r'/Users/YOURPATH/testcv.png'img = cv2.imread(file,0)img.shape#thresholding the image to a binary imagethresh,img_bin = cv2.threshold(img,128,255,cv2.THRESH_BINARY |cv2.THRESH_OTSU)#inverting the image img_bin = 255-img_bincv2.imwrite('/Users/YOURPATH/cv_inverted.png',img_bin)#Plotting the image to see the outputplotting = plt.imshow(img_bin,cmap='gray')plt.show()
medium.com
The next step is to define a kernel to detect rectangular boxes, and followingly the tabular structure. First, we define the length of the kernel and following the vertical and horizontal kernels to detect later on all vertical lines and all horizontal lines.
# Length(width) of kernel as 100th of total widthkernel_len = np.array(img).shape[1]//100# Defining a vertical kernel to detect all vertical lines of image ver_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1, kernel_len))# Defining a horizontal kernel to detect all horizontal lines of imagehor_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (kernel_len, 1))# A kernel of 2x2kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 2))
The next step is the detection of the vertical lines.
#Use vertical kernel to detect and save the vertical lines in a jpgimage_1 = cv2.erode(img_bin, ver_kernel, iterations=3)vertical_lines = cv2.dilate(image_1, ver_kernel, iterations=3)cv2.imwrite("/Users/YOURPATH/vertical.jpg",vertical_lines)#Plot the generated imageplotting = plt.imshow(image_1,cmap='gray')plt.show()
And now the same for all horizontal lines.
#Use horizontal kernel to detect and save the horizontal lines in a jpgimage_2 = cv2.erode(img_bin, hor_kernel, iterations=3)horizontal_lines = cv2.dilate(image_2, hor_kernel, iterations=3)cv2.imwrite("/Users/YOURPATH/horizontal.jpg",horizontal_lines)#Plot the generated imageplotting = plt.imshow(image_2,cmap='gray')plt.show()
We combine the horizontal and vertical lines to a third image, by weighting both with 0.5. The aim is to get a clear tabular structure to detect each cell.
# Combine horizontal and vertical lines in a new third image, with both having same weight.img_vh = cv2.addWeighted(vertical_lines, 0.5, horizontal_lines, 0.5, 0.0)#Eroding and thesholding the imageimg_vh = cv2.erode(~img_vh, kernel, iterations=2)thresh, img_vh = cv2.threshold(img_vh,128,255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)cv2.imwrite("/Users/YOURPATH/img_vh.jpg", img_vh)bitxor = cv2.bitwise_xor(img,img_vh)bitnot = cv2.bitwise_not(bitxor)#Plotting the generated imageplotting = plt.imshow(bitnot,cmap='gray')plt.show()
After having the tabular structure we use the findContours function to detect the contours. This helps us to retrieve the exact coordinates of each box.
# Detect contours for following box detectioncontours, hierarchy = cv2.findContours(img_vh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
The following function is necessary to get a sequence of the contours and to sort them from top-to-bottom (https://www.pyimagesearch.com/2015/04/20/sorting-contours-using-python-and-opencv/).
def sort_contours(cnts, method="left-to-right"):    # initialize the reverse flag and sort index    reverse = False    i = 0    # handle if we need to sort in reverse    if method == "right-to-left" or method == "bottom-to-top":    reverse = True    # handle if we are sorting against the y-coordinate rather than    # the x-coordinate of the bounding box    if method == "top-to-bottom" or method == "bottom-to-top":    i = 1    # construct the list of bounding boxes and sort them from top to    # bottom    boundingBoxes = [cv2.boundingRect(c) for c in cnts]    (cnts, boundingBoxes) = zip(*sorted(zip(cnts, boundingBoxes),    key=lambda b:b[1][i], reverse=reverse))    # return the list of sorted contours and bounding boxes    return (cnts, boundingBoxes)# Sort all the contours by top to bottom.contours, boundingBoxes = sort_contours(contours, method=”top-to-bottom”)
The further steps are necessary to define the right location, which means proper column and row, of each cell. First, we need to retrieve the height for each cell and store it in the list heights. Then we take the mean from the heights.
#Creating a list of heights for all detected boxesheights = [boundingBoxes[i][3] for i in range(len(boundingBoxes))]#Get mean of heightsmean = np.mean(heights)
Next we retrieve the position, width and height of each contour and store it in the box list. Then we draw rectangles around all our boxes and plot the image. In my case I only did it for boxes smaller then a width of 1000 px and a height of 500 px to neglect rectangles which might be no cells, e.g. the table as a whole. These two values depend on your image size, so in case your image is a lot smaller or bigger you need to adjust both.
#Create list box to store all boxes in  box = []# Get position (x,y), width and height for every contour and show the contour on imagefor c in contours:    x, y, w, h = cv2.boundingRect(c)    if (w<1000 and h<500):        image = cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,0),2)        box.append([x,y,w,h])plotting = plt.imshow(image,cmap=’gray’)plt.show()
Now as we have every cell, its location, height and width we need to get the right location within the table. Therefore, we need to know in which row and which column it is located. As long as a box does not differ more than its own (height + mean/2) the box is in the same row. As soon as the height difference is higher than the current (height + mean/2) , we know that a new row starts. Columns are logically arranged from left to right.
#Creating two lists to define row and column in which cell is locatedrow=[]column=[]j=0#Sorting the boxes to their respective row and columnfor i in range(len(box)):    if(i==0):        column.append(box[i])        previous=box[i]    else:        if(box[i][1]<=previous[1]+mean/2):            column.append(box[i])            previous=box[i]            if(i==len(box)-1):                row.append(column)        else:            row.append(column)            column=[]            previous = box[i]            column.append(box[i])print(column)print(row)
Next we calculate the maximum number of columns (meaning cells) to understand how many columns our final dataframe/table will have.
#calculating maximum number of cellscountcol = 0for i in range(len(row)):    countcol = len(row[i])    if countcol > countcol:        countcol = countcol
After having the maximum number of cells we store the midpoint of each column in a list, create an array and sort the values.
#Retrieving the center of each columncenter = [int(row[i][j][0]+row[i][j][2]/2) for j in range(len(row[i])) if row[0]]center=np.array(center)center.sort()
At this point, we have all boxes and their values, but as you might see in the output of your row list the values are not always sorted in the right order. That’s what we do next regarding the distance to the columns center. The proper sequence we store in the list finalboxes.
#Regarding the distance to the columns center, the boxes are arranged in respective orderfinalboxes = []for i in range(len(row)):    lis=[]    for k in range(countcol):        lis.append([])    for j in range(len(row[i])):        diff = abs(center-(row[i][j][0]+row[i][j][2]/4))        minimum = min(diff)        indexing = list(diff).index(minimum)        lis[indexing].append(row[i][j])    finalboxes.append(lis)
In the next step we make use of our list finalboxes. We take every image-based box, prepare it for Optical Character Recognition by dilating and eroding it and let pytesseract recognize the containing strings. The loop runs over every cell and stores the value in the outer list.
#from every single image-based cell/box the strings are extracted via pytesseract and stored in a listouter=[]for i in range(len(finalboxes)):    for j in range(len(finalboxes[i])):        inner=’’        if(len(finalboxes[i][j])==0):            outer.append(' ')        else:            for k in range(len(finalboxes[i][j])):                y,x,w,h = finalboxes[i][j][k][0],finalboxes[i][j][k][1], finalboxes[i][j][k][2],finalboxes[i][j][k][3]                finalimg = bitnot[x:x+h, y:y+w]                kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 1))                border = cv2.copyMakeBorder(finalimg,2,2,2,2,   cv2.BORDER_CONSTANT,value=[255,255])                resizing = cv2.resize(border, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC)                dilation = cv2.dilate(resizing, kernel,iterations=1)                erosion = cv2.erode(dilation, kernel,iterations=1)                                out = pytesseract.image_to_string(erosion)                if(len(out)==0):                    out = pytesseract.image_to_string(erosion, config='--psm 3')                inner = inner +" "+ out            outer.append(inner)
The last step is the conversion of the list to a dataframe and storing it into an excel-file.
#Creating a dataframe of the generated OCR listarr = np.array(outer)dataframe = pd.DataFrame(arr.reshape(len(row),countcol))print(dataframe)data = dataframe.style.set_properties(align="left")#Converting it in a excel-filedata.to_excel(“/Users/YOURPATH/output.xlsx”)
That’s it! Your table should now be stored in a dataframe and in an excel-file and can be used for Nature Language Processing, for further analysis via statistics or just for editing it. This works for tables with a clear and simple structure. In case your table has an extraordinary structure, in the sense that many cells are combined, that the cells size varies strongly or that many colours are used, the algorithm may has to be adopted. Furthermore OCR (pytesseract) is nearly perfect in recognizing computer fonts. However, if you have tables containing handwritten input, the results may vary.
If you use it for your own table(s), let me know how it worked.
medium.com
Also Read:
How to Fix Broken Lines in Table Recognition