How to recognize and fix broken lines in table recognition | by Hucker Marius | Towards Data Science

When documents are digitalized via scanning or via photo, the image quality can suffer from wrong settings or bad conditions. In the case of table recognition, this can lead to a broken table structure. Consequently, some lines might have light flaws or even holes and the table as a whole is not recognized as a coherent system. Sometimes the tables are also created without lines on some sides of the cells. In this case, the system closes the lines with the lines of the cells above. There is a great variety of tables and cell types and as so often the proposed code might not work perfectly for all. Nonetheless, with small adaptations its useful for many cases.
Most cell recognition algorithms are based on lines and cell structure. Not having lines leads to a bad recognition rate due to “forgotten” cells. The same is the case for this approach. The lines are necessary. In case your table does not have clearly recognizable lines it won’t work. But now, let’s take a look at the code!
First, we need to do the imports. In this case, it is restricted only to two imports: OpenCV and NumPy.
import cv2import numpy as np
Then we need to load the image/document containing the table. In case it is a whole document with text surrounding the table you need to recognize the table first and cut the image to the table size. (To read more about table recognition and cutting to the table size, click here.)
# Load the imageimage = cv2.imread(‘/Users/marius/Desktop/holes.png’, -1)
As you can see in the input image the lines of the cells in the second row are not fully connected. In table recognition, the second row would not be recognized and considered by the algorithm because the cells are not a closed box. The solution proposed in this article works not only for this case of gaps. It works also for other broken lines or holes in tables.
Now, we need to obtain the size of the image (height and width) and store it in the variables hei and wid.
(hei,wid,_) = image.shape 
The next step is the grayscaling and blurring via a Gaussian Filter. This helps in the recognition of the lines. For more on grayscaling, click here.
#Grayscale and blur the imagegray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)blur = cv2.GaussianBlur(gray, (3,3), 0)
Then we need to threshold our image. If you want to learn a bit more about thresholding, you can read about it in this article: Click Here (It’s all the way down to the section Binarizing an image).
#Threshold the imagethresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]
The findContours algorithm of OpenCV is then used to get the position of all contours. For all contours, a bounding rectangle is drawn to create the boxes/cells of the table. The boxes are then stored in the list box with the four values x, y, width, height.
#Retrieve contours contours, hierarchy = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)#Create box-listbox = []# Get position (x,y), width and height for every contour for c in contours:    x, y, w, h = cv2.boundingRect(c)    box.append([x,y,w,h])
All heights, widths, x’s and y’s are then separately stored in lists and the minimum height, width and x and y are computed. Furthermore, the maximum y and x are necessary.
#Create separate lists for all valuesheights=[]widths=[]xs=[]ys=[]#Store values in listsfor b in box:    heights.append(b[3])    widths.append(b[2])    xs.append(b[0])    ys.append(b[1])#Retrieve minimum and maximum of listsmin_height = np.min(heights)min_width = np.min(widths)min_x = np.min(xs)min_y = np.min(ys)max_y = np.max(ys)max_x = np.max(xs)
The stored values are now used to understand where the table is located. The minimum y value can be used to get the uppermost row of the table, which can be regarded as the starting point of the table. The minimum x value is the left edge of the table. To get the approximate size we need to retrieve the maximum y value, which is the cell or row at the bottom of the table. The y-value of the last row represents the upper edge of the cell and not the bottom of the cell. To regard the whole size of the cell and the table it is necessary to add the last rows cell height to the maximum y to retrieve the full height of the table. The maximum x will be the last column and consecutively, the right-most cell/row of the table. The x-value is the left edge of each cell and consecutively, we need to add the width of the last column to the maximum x-value to retrieve the full width of the table.
#Retrieve height where y is maximum (edge at bottom, last row of table)for b in box:    if b[1] == max_y:        max_y_height = b[3]#Retrieve width where x is maximum (rightmost edge, last column of table)for b in box:    if b[0] == max_x:        max_x_width = b[2]
In the next step, all horizontal and vertical lines are extracted and stored separately. This is done by creating a kernel which thresholds and applies morphological operations. The horizontal kernel has a size of (50,1). You can play around with the size depending on the size of your image. The vertical kernel has a size of (1,50).
Morphological operations perform transformations of the detected structures based on their geometry (Soille, p.50, 1998). Dilation is one of the most applied and most basic morphological operations. If at least one pixel under the kernel is white, the pixel being viewed of the original image will be considered white. Consequently, the white areas are enlarged. Please be aware that due to inversion the background is black and foreground white, meaning that the table lines are currently white. The dilation can be seen as the most important step. The holes and broken lines are now repaired and for further table recognition, all cells will be regarded.
# Obtain horizontal lines maskhorizontal_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (50,1))horizontal_mask = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, horizontal_kernel, iterations=1)horizontal_mask = cv2.dilate(horizontal_mask, horizontal_kernel, iterations=9)# Obtain vertical lines maskvertical_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (1,50))vertical_mask = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, vertical_kernel, iterations=1)vertical_mask= cv2.dilate(vertical_mask, vertical_kernel, iterations=9)
Both masks, the horizontal and the vertical, are then merged to one table using the bitwise_or operation of OpenCV. To retrieve the original back- and foreground the image is inverted by subtracting the cv2.bitwise_or from 255.
# Bitwise-and masks togetherresult = 255 — cv2.bitwise_or(vertical_mask, horizontal_mask)
In case the table is surrounded by text and not standing alone (in my example it’s not surrounded), we cut it out and set it on a white background. Now we need the size of the table retrieved earlier. We cut the final image to the table size by using the minimum y (which is the edge at top), the maximum y + the height of the maximum y cells (which is the edge at the bottom), the minimum x (which is the left edge) and the maximum x + the width of the maximum x cells (which is the right edge). The image is then cropped to the size of the table. A new background of the original size of the document is created and completely filled with white pixels. The center of the image is retrieved and the repaired table is merged with the white background and set right into the center of the image.
#Cropping the image to the table sizecrop_img = result[(min_y+5):(max_y+max_y_height), (min_x):(max_x+max_x_width+5)]#Creating a new image and filling it with white backgroundimg_white = np.zeros((hei, wid), np.uint8)img_white[:, 0:wid] = (255)#Retrieve the coordinates of the center of the imagex_offset = int((wid — crop_img.shape[1])/2)y_offset = int((hei — crop_img.shape[0])/2)#Placing the cropped and repaired table into the white backgroundimg_white[ y_offset:y_offset+crop_img.shape[0], x_offset:x_offset+crop_img.shape[1]] = crop_img#Viewing the resultcv2.imshow(‘Result’, img_white)cv2.waitKey()
This is the result. The method can be used for multiple typed of broken lines, gaps, and holes in tables. The result is the basis for further table recognition as explained in my other article. The explained method was applied on an empty table. You can also apply it on a table containing text or surrounded by text. For a table containing text, it is still necessary to merge the original image containing a table with data with the final image with repaired holes. This can be done with a OpenCV bitwise operation and should not be to complex.
I hope you enjoyed my tutorial and can use this for your own projects.