Tutorial: Stereo 3D reconstruction with openCV using an iPhone camera. Part II.
By: Omar Padierna
Tags: Python, Computer Vision, Tutorial, Opencv

Welcome to the second part of this 3 part series about stereo reconstruction. In this section we will talk about calibrating your camera.

As mentioned before, the lens in a camera distorts the pictures you take. This is troublesome when it comes to 3D reconstruction and therefore we need to correct this. Before correcting though we need to know the internal parameters of the camera we’re using.

Sometimes these parameters are unknown, luckily for us , OpenCV has an algorithm specifically dedicated to this so let’s get crackin’

We will be using Python 3.7.1, OpenCV 3.4.4. as well as Numpy, Glob, tqdm and Pillow so be sure to have all those things installed first.

The process of calibrating a camera in OpenCV consists in making the computer scan a picture with a chessboard pattern and identify the internal corners several times with different pictures.

An example of pattern calibration courtesy of OpenCV

Most tutorials out there concerning camera calibration deal with the webcam and/or other video cameras and as such they’re tailored towards analyzing every individual frame (or a number of them).

In our case, since we want to calibrate for our phone cameras we can’t do this. In order for the computer to properly calibrate the camera, it needs several examples of the same pattern.

When dealing with a video feed we can analyze every frame as the user moves the pattern in front of the camera until the pattern is detected a number of times (this is normally an arbitrary measurement but the general rule is to have around 10 detections at least).

Since we’re not dealing with a video feed we have to take several pictures of the same pattern instead.

Step 1. Get a chessboard pattern.

Go to this link and print this chessboard pattern on a piece of paper. As a best practice, make sure that every square is 30 mm long (although this isn’t too important).

Step 2. Put chessboard pattern on a white wall.

When I was running the algorithm I noticed that the more stuff in the background the longer it took to calculate the camera matrix. Therefore, I used a white wall and pasted the chessboard pattern on it. Make sure it is perfectly flat.

Redecorating the house with chessboard patterns

Step 3. Take several pictures of the pattern.

This step is quite important, make sure that the pictures you take have good illumination, and that the pattern is captured from different angles. Also make sure that the pattern is located in different parts of the screen.

Bad calibration can happen if you only take pictures that are centered. Make sure there’s a lot of variation in your pictures.

A very good example on how to take pictures of the pattern. Courtesy of Utkarsh Sinah from AI shack.

A great resource on camera calibration and how to capture the pattern can be found in Utkarsh Sinah’s blog (AI shack). If you’re interested in the C++ way of calibrating your camera, you should go there.

It is important to point out that not all pictures will be suitable for detecting the pattern. It is hard to know beforehand which pictures will work and therefore it is a good idea to take as many pictures as possible. I took 64.

Step 4. Let’s get coding.

Once you’ve taken enough pictures it is time to write some code (remember that the entire code is here). The first step is to pick the chessboard size. While the size is completely arbitrary, it is recommended that you pick an asymmetrical size (ie. a rectangle, not a square). I chose 7x5

import cv2

import numpy as np

import glob

from tqdm import tqdm

import PIL.ExifTags

import PIL.Image #============================================

# Camera calibration

#============================================ #Define size of chessboard target. chessboard_size = (7,5)

The second step is to define a grid where all the points will be stored. This is to store the points in an ordered manner, like so: (0,0,0), (1,0,0), (2,0,0) ….,(6,5,0)

#Define arrays to save detected points

obj_points = [] #3D points in real world space

img_points = [] #3D points in image plane #Prepare grid and points to display objp = np.zeros((np.prod(chessboard_size),3),dtype=np.float32) objp[:,:2] = np.mgrid[0:chessboard_size[0], 0:chessboard_size[1]].T.reshape(-1,2)

Since we’re dealing with several images, we can open them iteratively using Glob. Additionally, since the corner detection algorithm in OpenCV takes some time to process we can wrap our loop with tqdm to know how long it has been since the last image and how many images are left.

#read images calibration_paths = glob.glob('./calibration_images/*') #Iterate over images to find intrinsic matrix

for image_path in tqdm(calibration_paths): #Load image

image = cv2.imread(image_path)

gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

print("Image loaded, Analizying...")

#find chessboard corners

ret,corners = cv2.findChessboardCorners(gray_image, chessboard_size, None) if ret == True:

print("Chessboard detected!")

print(image_path)

#define criteria for subpixel accuracy

criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

#refine corner location (to subpixel accuracy) based on criteria.

cv2.cornerSubPix(gray_image, corners, (5,5), (-1,-1), criteria)

obj_points.append(objp)

img_points.append(corners)

It is inside this loop where all the magic is happening. After loading the image we have to convert it to grayscale and then we use the findChessboardCorners algorithm.

This algorithm will return the corners detected and a flag called ret that will be true if the algorithm was able to detect the pattern.

In order to improve the accuracy in calibration algorithm, refine the location of the corners to subpixel accuracy. In this case we have to define the criteria required to locate.

Criteria is defined as follows criteria = (type, number of iterations, accuracy). In this case we are telling the algorithm that we care both about number of iterations and accuracy (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER) and we selected 30 iterations and an accuracy of 0.001

cv2.cornerSubPix is the algorithm focused on relocating the points. it receives the image, the corners a window size, zeroZone and the actual criteria. The window size is the search area.

It isn’t very crucial to obsess over this algorithm, I simply decided to comment on the parameters because most tutorials just gloss over this algorithm. For more information on how it works you can check here.

After analyzing all the pictures, we then run the cv2.calibrateCamera algorithm. This is the algorithm that outputs the camera parameters. This algorithm returns the camera matrix (K) distortion coefficients (dist) and the rotation and translation vectors (rvecs and tvecs).

#Calibrate camera

ret, K, dist, rvecs, tvecs = cv2.calibrateCamera(obj_points, img_points,gray_image.shape[::-1], None, None) #Save parameters into numpy file

np.save("./camera_params/ret", ret)

np.save("./camera_params/K", K)

np.save("./camera_params/dist", dist)

np.save("./camera_params/rvecs", rvecs)

np.save("./camera_params/tvecs", tvecs)

Notice that we saved these values into different numpy files. The reason I chose to do so was because of practicality. Since this script takes a while to run, it is inconvenient (and unnecessary) to execute it every time we want to reconstruct something.

As such, it is easier to simply save everything into a numpy file and load it later. Why numpy and not XML or JSON? because with numpy files there’s no need to parse the data.

To do 3D reconstruction there are 3 parameters we really care about, the camera matrix, the distortion coefficients and the focal length. The focal length can be derived from the camera matrix.

Nonetheless, for the sake of learning I decided to obtain the focal length from the Exif data contained in the picture. Most camera phones save information about the focal length (and other parameters) in the exif data.

#Get exif data in order to get focal length.

exif_img = PIL.Image.open(calibration_paths[0]) exif_data = {

PIL.ExifTags.TAGS[k]:v

for k, v in exif_img._getexif().items()

if k in PIL.ExifTags.TAGS} #Get focal length in tuple form

focal_length_exif = exif_data['FocalLength'] #Get focal length in decimal form

focal_length = focal_length_exif[0]/focal_length_exif[1] np.save("./camera_params/FocalLength", focal_length)

Exif data can be parsed into a dictionary using Pillow, nonetheless, the keys will be given in numerical terms which is the reason we have to use the ExifTags module in order to convert them into a readable form.

Finally, we need to have a way to measure how accurate our calibration was. We have two ways to do this. The visual way and the numerical way.

The numerical way consists in calculating the total error when it comes to projecting the points. If you notice, at the beginning of the script we declare two arrays, the object points (3D points) and the image points(2D points).

These were obtained iteratively during the calibration process. The goal here is to project the 3D points into a 2D plane using the rotation and translation vectors calculated in the calibration loop. These new projected points (called img_points2 in the script) are then compared to the image points obtained in the calculation loop.

We then calculate the error for every point and get the average. This error should be as close to 0 as possible. In my case, the error was 0.44 which slightly better than chance.

The visual way to do this is to use the undistort algorithm on an image taken with this camera (preferably one of the patterns that shows some curve distortion). The goal is for the algorithm to get rid of lens distortions, if it does so correctly then you have a good calibration.

Undistort results obtained by Utkarsh Sinah. Courtesy of Utkarsh Sinah from AI Shack

If your error is too high make sure that you detected the chessboard at least 10 times and make sure that the pictures are varied.

I must also warn you that you have to be patient, especially when it comes to large images. In my case, the calibration algorithm took 1:30 hours to complete.

Once the calibration is done, you’re ready to calculate a disparity map, which is the topic we will cover in the next part.

See you!