Automatic Skew Correction Using Corner Detectors and Homography
Tags: Opencv, Homography, Machine Learning, Image Processing, Corner Detection

Automatic Skew Correction Using Corner Detectors and Homography

In this post, I will talk about one of the main applications of homography: Skew Correction and how we can achieve it. I will be using, cv2.findHomography() to compute the Homography matrix and cv2.warpPerspective() to transform the images. I will use two example images (figure 3 and figure 8) for this purpose.

This post is divided into the following parts:

1. Introduction to homography

2. Example 1

— a) Corner detection (Shi Tomasi)

— b) Homography calculation and perspective transform

3. Example 2

— a) Masking Thresholding and Contour detection

— b) Corner detection (using Douglas-Peucker algorithm)

— c) Homography calculation and perspective transform

4. Conclusion

5. References

1. Introduction to Homography

A digital image of an object can be understood as an instance of perception of the object projected on the camera plane. An object can be perceived in different ways with respect to the orientation of the camera. For example, the projection (image) we obtain when the camera plane and the object are placed parallel to each other is different than the projection we obtain when they are placed obliquely. Essentially, the different projections (on the same plane) are interchangeable. Homography is a concept that can help us achieve this.

Figure 1: Different views of the same object. Image source: here

Formally, Homography (or sometimes collineation) is a mapping from a plane to itself such that the collinearity of any set of points is preserved. It transforms collinear points into collinear points within a plane.

[1]A 2D homography is an invertible mapping h from P2 to itself such that three points x1, x2, x3 lie on the same line if and only if h(x1), h(x2), h(x3)do. A mapping h : P2→P2 is a homography if and only if there exist a non‐singular 3x3 matrix H such that for any point in P2 represented by a vector x it is true that h(x)=H

Homography mapping H between two points x and x` can be represented as:

x` = H . x

Figure 2: Formula for homography

2. Example 1

Now let’s begin with the first example image. Here we have a quadrangle, and our goal is to un-warp it so that we can get a regular rectangle.

Figure 3: Example image 1

The first step is to read the image and convert it from BGR to RGB format for display.

image = cv2.imread('images/example_1.png')

image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

plt.imshow(image)

plt.title('Original Image')

plt.show()

We start off by detecting corner points of the quadrangle.

Why detect corners though? Why can’t we use other points?

We can but, corners make things easier. We approach this problem by detecting corner coordinates of the quadrangle and mapping them to the corners of its corresponding rectangle.

Figure 4: Mapping corners of the example quadrangle with its corresponding rectangle.

2. a) Corner Detection (Shi-Tomasi)

For this example, we will be using [2] Shi-Tomasi corner detector which is implemented in cv2.goodFeaturesToTrack() . Here is the code:

cv2.goodFeaturesToTrack() takes four arguments: Grayscale image, numbers to corners to detect, quality of corners (a value between 0–1, below which all possible corners are rejected) and the minimum euclidean distance between two corners.

This gives us the corners of the quadrangle, the order of which is shown in the figure.

Figure 5(a): Corners detected using Shi-Tomashi algorithm

Figure 5(b): Coordinates of corner points

Important: Note the order of the corners. Top-left — A, Top-right — B, Bottom-left — C, Bottom-right— D. Computation in the upcoming steps is dependent on this order. See Figure 4.

Now, to calculate the homograpy matrix we will also need the destination points, i.e. the corners of the corresponding rectangle (A’, B’, C’ and D’)

For this, we need to find the height and width of the target rectangle. We do this by calculating the maximum of the two widths and the maximum of the two heights as described in the code snippet below:

Figure 6: Output — Coordinates of destination points and approximated size of the original image

Now that we have our source and destination points, we can calculate the homography matrix.

2. b) Homography calculation and perspective transform

To calculate the homography matrix, we use cv2.findHomography() . It takes the source and destination points that we calculated in the preceding steps. Minimum of four source and corresponding destination points are required to calculate the homography matrix. But, we can also send more than 4 pairs. In that case, cv2.findHomography() uses algorithms such as RANSAC to calculate the best homography matrix with minimum reprojection error.

After obtaining the homography matrix, we pass it to cv2.warpPerspective() warp our image. Below is the code:

Figure 7(a): Homography matrix

Figure 7(b): Output — correcting skew in the quadrangle

As you can see in figure every pixel of the image (not just the quadrangle) undergoes some transformation. Since the background is totally irrelevant for us, we can just crop the transformed rectangle to get the useful part.

cropped = un_warped[0:h, 0:w]

plt.imshow(cropped)

plt.show()

Figure 7(c): Output — skew corrected and cropped image

Hence we un-warped a warped rectangle! Now let’s move to the next example.

3. Example 2

In this example, we will try to correct the skew in a more realistic image. Figure 8 shows a picture of my notebook. While taking this picture, I had tilted my phone so that the image plane and camera plane were oblique to each other. Our goal here is to transform this image into a bird’s eye view.

Figure 8: Example 2

Unlike the first example, we won’t be using Shi-Tomasi corner detector in this image. It is because, this image has numerous candidate corners (along the spirals, within the text etc.) and Shi-Tomasi is not well suited for such textured images. In fact, the Shi-Tomasi algorithm is not wrong to identify corners inside the handwritten text…some of them do look like corners. But, to achieve our goal, we need to direct our code to look at the bigger picture and find corners at the border of the notebook and not within. For this, we mask the area of the notebook and use contour detection.

The first step is the same as that of example 1. We read the image and convert it into RGB format.

image = cv2.imread('images/example_1.png')

image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

plt.imshow(image)

plt.title('Original Image')

plt.show()

3. a) Masking, Thresholding and Contour detection

To mask the notebook, we create a 5x5 kernel and apply it over the grayscale image as shown in the code snippet below:

Figure 9(a): Output — applied filter to the original image

As you can see in the figure, by applying the filter, we get a nicely blurred image of the notebook and almost all the hand-written characters in it are gone. Now, we apply [3] [4] Otsu Threshold for binary classification of the image; i.e. classify each pixel as either black or white. This makes things easier because now, we can detect the notebook contour with without any problems.

Figure 9(b): Output — Thresholding using OTSU

For contour detection, we use [5] cv2.findContours() . We are only interested in the largest contour. So, that’s what we take.

cv2.drawContours() is used to draw the contour points. We draw the contour in a blank canvas as shown in the code snippet.

Figure 9(c): Output — Largest contour detected

3. b) Corner detection (using Douglas-Peucker algorithm)

Now that we got the contour of the notebook, we need to find the corner points. We will use cv2.approxPolyDP() . This function uses [6] Douglas-Peucker algorithm to approximate a curve or a polygon with fewer vertices so that the distance between them is less than epsilon as shown in the figure below:

[6] Source: By Mysid — Own work; self-made in Inkscape and Gimp. Based on File:Douglas Peucker.png by de:User:Leupold., CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=20461106

Hence, by using cv2.approxPolyDP() with the appropriate epsilon value, we can get a pretty good approximation of our contour cnt . We can exploit this feature to get our corner points as shown in the following code snippet:

By using, epsilon = 2% of the contour arc length we got good corner approximation.

Figure 10: Corners detected

As you can see in figure, the order of the corners do not match with the order specified here. So, we will arrange them in the format discussed in the previous section.

3. c) Homography calculation and perspective transform

Now, we will use get_destination_points() to get the corresponding corner vertices of our un-warped notebook and unwarp() to perform perspective transform using homography (refer to example 1). Finally, we will crop the notebook area to remove the unwanted background.

Here is our result!

Figure 11(a): Output — Corner points, destination points, approximated size and homography matrix for example 2 image.

Figure 11(a): Output — correcting skew in notebook image

Figure 11(b): Output — skew corrected and cropped image

Looks pretty good!

4. Conclusion

Thus, from the two examples, we learnt how to use OpenCV's functions to calculate and utilize homography matrix for skew correction. We also learnt two important ways of detecting corners. First, the Shi-Tomasi method, which works well with smooth images with a consistent background and second, the contour detection method, which when used carefully works for all kinds of textured images. Please note that these are not the only methods available for corner detection in OpenCV. But, these are the ones I use most frequently in my projects. For alternative options, you can look for Harris corner detector, SUSAN, FAST corner detector etc.

For the full code, refer to my GitHub here.

5. References

These are the references I used in this post:

[1] Homography: https://ags.cs.uni-kl.de/fileadmin/inf_ags/3dcv-ws11-12/3DCV_WS11-12_lec04.pdf

[2] Shi-Tomashi corner detector Opencv: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_shi_tomasi/py_shi_tomasi.html

[3] Image thresholding Opencv: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_thresholding/py_thresholding.html

[4] OTSU thresholding algorithm: https://en.wikipedia.org/wiki/Otsu%27s_method

[5] Contour Detection Opencv: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_contours/py_contour_features/py_contour_features.html

[6] Douglas-Peucker algorithm: https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm