12 / 17 / 2010

Object Depth Estimation (ODE) is a common problem faced in Computer Vision, and it's often solved using either Monocular or Stereo ODE algorithms. Since both use different tactics and have estimation problems at different distance ranges, it may be beneficial to use them in tandem and use Machine Learning techniques to combine the results in order to decrease errors. 1. 1.1. Motivation Introduction The need to identify object locations relative to a fixed point, based on single or multiple 2-dimensional images is a common problem in Computer Vision, and comes up often in Robotics. This has led to an entire area of study, referred to as Object Depth Estimation (ODE), to form directly around solving this issue. Thus, there has been a considerable amount of research into both single (monocular) and multiple (stereo) image ODE algorithms. Monocular algorithms typically parse an image into objects and determine depth based on the location of the object in the image. A big issue to this approach is the susceptibility to scaling errors. These can be corrected for in some cases, based on various features of the object, but it remains as a significant source of error. Stereo algorithms tend to focus on the differences between the views of an object based on angle. However, this tends to give inaccurate results on objects at a distance. Since both algorithms attempt to solve the same problem using different methods, but have individual drawbacks, it seems intuitive that a combination has the possibility of decreasing errors. 1.2. Approach Introduction The new solution suggested by this paper is to use supervised machine learning techniques, such as Linear Regression, on the results of using both algorithms independently. For the purposes of training and evaluation, a large data set of images from two separate angles paired with a reading from a Hokuyo Laser Range Finder is used. Multiple Machine Learning techniques are compared on the monocular and stereo ODE algorithm outputs, but this paper focuses on a Markov Random Field model. 2. There are several important components to this project that must be incrementally completed to ensure successful implementation of object depth estimation. These sections are enumerated as follows: Approach 2.1. Data set collection and Hardware setup Before implementing any algorithms, we had to successfully complete taking a large set of data so that we have a database of stereo images and ground truth from laser range data. However, taking an accurate set of stereo images proved more of a challenge than anticipated. We had to get 2 identical cameras with similar intrinsic properties (such as focal length, skew, lens curvature etc.). These had to be mounted together with a predefined physical vertical separation along with the Hokuyo Laser Range finder and had Rohan Sharma, David Skiff 12/17/2010 to be axis aligned to minimize skew between the cameras and to align image pixels with laser range (this constitutes as manual alignment). The resulting data set contains over a hundred images of various rectangular and circular objects in different orientations, and locations in the field of view. Most of the data was collected in one session to reduce the amount of error introduced due to removal and repositioning of the camera between sessions. 2.2. Image Rectification, Laser Alignment and Edge Extraction After taking the data sets, the 2 stereo images contained in each data set had to be rectified such that there is 1-1 correspondence between the columns. Since the cameras were mounted with a vertical separation, each column of the images was aligned with each other such that an accurate disparity calculation could be performed based on the base edges of objects observed in each image. Therefore, the extreme left and right ends of the 2 images were modified such that each vertical column in the 2 images matches each other. Since there was only a horizontal variation in the image (extra columns at the end), the end columns of the 2 images had to be compared against one another to find a close match and any additional columns were removed from the image. This rectification was constant throughout the entire image database (that was taken in one session). As a second step, the laser range data had to be formatted and aligned with the rectified stereo images to get accurate ground truth information about each column of the images. This was done by manually aligning different parts of the laser range (starting from the center) to see if the laser data matched with the obstacles viewed (was performed once). Once the laser angles were aligned with the image edges, linear interpolation had to be performed to get range estimations for each column. S Every time we take data sets with a new vertical separation, we would have to change the image alignment code as we would never be able to get the same angle orientation of the 2 cameras and laser. The results section contains 2 images that represent 1 set of images aligned with the laser data. 2.3. Stereo Object Depth Estimation Algorithm Stereo vision is a technique for inferring depth of an object using a combination of 2 or more cameras that are a certain distance apart from each other. For our implementation, we have a combination of 2 cameras that are a certain distance d apart from each other (physical separation). Thus, if we assume that a certain section of an image has a disparity x between the 2 images, the depth of the object can be calculated as follows: Z = d.f Xb − Xt = d.f x Where d = Physical separation between the camera f = Focal length of the camera x = Disparity of the image patch in the stereo images (ground edges in our case) Z = Actual depth estimation Rohan Sharma, David Skiff 12/17/2010 Figure1: Stereo vision setup illustration The focal length of the camera was converted to pixel representation using the following formula: Focal Length(pixels) = Imagewidth(pixels) ∗ � focal length(mm) lenswidth(mm) � To run this algorithm on the images, several pre-processing image steps had to be completed. We used a combination of filters to extract the edges from the picture. First, we converted the image into a grayscale image as color information is not required for this algorithm. We then ran the grayscale images through a 7x7 Gaussian blur filter to smooth out any edges created due to the carpet texture. This was implemented so that the first edge that we observe from the bottom of the images is the ground edge of the object and not random edges due to the texture of the carpet. This image was then run through a Canny edge detector with a threshold value of [low high] = [0.05 0.25] to give fine edges for the bottom for the object. These images (rectified and edge extracted) are then compared for the bottom most edge in each image column/segment and the disparity in the pixel positions in between the 2 images is found. These values are then run through the formula to get an estimate of the depth of the object’s edge. Comparison of this range data with the laser range information is provided in the results section. 2.4. Monocular Ground Ranging Object Depth Estimation Algorithm The monocular ground ranging algorithm tries to find the depth using just the pixel information of the object edge. This algorithm is based on the theory that objects that are far away in the image are present at a higher pixel level (from the bottom) in the image and appear closer to the horizon. Thus, this algorithm tries to estimate the distance to an object based on the number of pixels from the bottom of the image to the object edge. Using a similar concept from the previous algorithm, the colored image is converted into a grayscale image and a Gaussian filter will be convolved with it to blur out the edges caused by any carpet texturing. Following this, a canny edge detector is run to find the bottom most edge of the object and the pixel distance to the edge is stored. Using the ground truth information from the laser range data, a linear regression algorithm (using the least mean squares algorithm) is run on the top and bottom image sets to find the weight parameters for the 2 sets. These learned parameters are used along with the pixel information to estimate depth of the object’s edge. The linear regression algorithm and feature set used is as follows: θ = (A′A)−1 ∗ (A′y) Where Rohan Sharma, David Skiff 12/17/2010 Θ = Weights for linear regression (of size 3) A = feature matrix y = laser data vector The feature vector is created using the 3 features as given below: A = [x log(x) x2] Here, each x is simply a very large column vector that consists of the pixel position for each column of the image (in the training set). This vector consists of all the columns of all images (#sets * 626, where 626 is number of columns after rectification). 2.5. Machine Learning Algorithm Finally, using the stereo vision and monocular ranging algorithms on the 2 images, we have 3 range values that estimate the depth of the object and this makes up our feature set for learning. The following figure illustrates our model and represents different modules used to get single range estimation by combining the algorithms explained above, Figure2: Learning and Error analysis Model The range data will be split into different segments based on the distance they are trying to estimate (that is 1 group for closer ranges less than 1 meter, one for ranges between 1-3 meters and one group for ranges above 3 meters). We also have laser range data that represents the ground truth distance for the objects. Thus, the feature set and laser range data together represent the training set and test based on how we split up the data for testing. Following this partition of data, a number of different Machine Learning algorithms were implemented in Matlab [3] in order to ascertain an optimal method. The following were tried: Rohan Sharma, David Skiff 12/17/2010 1. Linear Regression – This is basic linear