Keeping track of what we eat, part 1: Pre-processing the Receipts
By: Konstantin Golubev
Tags: Image Processing, Scanning, Preprocess

Keeping track of what we eat, part 1: Pre-processing the Receipts

So I decided to find out what we, a family of four, consume. Apart from pure curiosity on how many bananas we eat each year, I wanted to find out where our cumulative nutrition lies relative to the official recommendation in the US and Europe. Since more or less everything we buy comes with a detailed receipt, I thought it would be the easiest to extract data from them.

Two assumptions here are: there is a detailed receipt for everything we eat; and we do not throw away food. Neither is entirely correct but you know.

Pipeline Sketch

The pipeline sketch looked like this:

Take photos of the receipts, and transfer them to the PC. Pre-process the receipts. Extract the information. ???? PROFIT!!!

Step 1 currently looks like this. My wife and I just send pics to a Telegram chat. Then I manually download them to the computer. It works fine in the meantime, and I might substitute this with a Telegram bot that would feed the pics automatically to the pipeline in the future, but first: receipt pre-processing.

Receipt Scanning Tool

Constructing a Scanning App has become a standard exercise to get oneself familiar with image processing libraries. There are several articles and posts about how to do that: [1], [2], [3], [4]. But the general plan usually is the same and goes as follows.

Step 1: Thresholding and Smoothing

Import the pic in grayscale -> convert to black and white (literally) via thresholding -> dilate to remove small artifacts

# path to image

path = 'img/IMG_0108.JPG' # Load an color image and transform to grayscale

img = cv2.imread(path)

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) # thresholding

ret, thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_OTSU) # dilate thresholded image - merges top/bottom

kernel = np.ones((3,3), np.uint8)

dilated = cv2.dilate(thresh, kernel, iterations=3)

Step 2: Finding Corners

Find the contour bounding the largest area -> reduce the number of points in the contour -> approximate the contour with four points. These four points serve as the corner points of the quadrilateral that bounds the receipt.

# find contours and choose the largest one

contours = cv2.findContours(dilated,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)

#contours = imutils.grab_contours(contours)

contours.sort(key = cv2.contourArea, reverse = True) contour = contours[0]

peri = cv2.arcLength(contour, True)

approx = cv2.approxPolyDP(contour, 0.02 * peri, True)

Finally, warp and crop the pic so that the quadrilateral becomes a rectangle.

The code worked beautifully on the first two examples I had. Then I ran it on the receipts that accumulated during the coding time. And it did work in most cases, but sometimes things went wrong in different ways:

For example, here the final contour approximation step outputs less than four points. This was resolved by switching to Otsu’s threshold instead of a constant in the thresholding step.

Here the program just totally missed the receipt on the pic. This happened because out of all contours on the pic I was choosing the one with the most points, while the more reasonable choice is the one bounding the largest area.

Conclusion: I don’t need it

After I pre-processed the pic, I fed to Tesseract. Turns out that the result didn’t differ much from the one I got without preprocessing the receipt. So I decided to just skip the preprocessing part. ¯\_(ツ)_/¯

Links