1 DENSE STEREO MATCHING USING MACHINE LEARNING

Introduction Many researches in computer vision have been focused on developing algorithms to accurately determine depth maps. In stereo vision, a pair of cameras at two different locations capture a left and a right image which are slightly different images of the same scene. Due to different locations of the cameras, a 3D point will show up at different pixels in each image. To perform stereo matching is to map a pixel in the left image to the pixel in the right image that is corresponding to the same 3D point. The difference of locations of matched left and right pixel is called disparity. Disparity is inversely proportional to depth. An image can have only a finite number of objects and hence a finite number of disparity values. Many algorithms such as SIFT have been developed to perform sparse matching, which produce matches for only a few keypoint pixels. However in some applications such as robots’ navigation, it is better to have disparity value and depth for every single pixel in an image. In this project we explore an application of machine learning in performing dense matching which will produce matches for all pixels in the images. Background Sparse Matching Technique Sparse matching algorithms produce disparity values for keypoints. There are many sparse matching techniques such as matching by using Scale Invariant Feature Transformation (SIFT). In this project, we use the algorithm developed by Geiger et al. to produce disparity values for keypoints, which will be used as our training data[1]. Geiger’s algorithm first preprocesses images by filtering input images with 5x5 blobs and corner mask and then applying non-maximumand nonminimum-suppression[2]. Speed Up Robust Features (SURF) descriptors of each pixels are computed and the descriptors are passed to a Sobel filter. For a given keypoint pixel in the left image, sum of absolute differences (SAD) between its filter response and the filter response of every pixel in the right image are computed and the pixel of the right image that produces the smallest SAD is picked as the match. DAISY Daisy is a computationally efficient dense descriptor. In this technique initially the gradient at each pixel location in different directions is calculated. This can be represented by Go(u;v) = max((δI/δo),0), where o is the direction of the gradient, and I is the intensity. Next we convolve this with Gaussian kernels of varying standard deviation ie Σ1 in one layer (a layer is a group of circles with center equidistant from the pixel in consideration) around each pixel and Σ2 in another layer and so on. We use a 3 layered version of Daisy as shown in the figure (same as used by the developers of Daisy: E.Tola, V.Leptit and P.Fua in their paper[3][4]). In the figure, the Gaussian smoothing is proportional to the circle radius. Also this entire figure computes the daisy parameters for the central pixel. The overlapping circles result in smooth results. Overview of Our Approach We explore applications of learning algorithms including multi-class support vector machine(SVM) and K-mean clustering algorithm. We use Geiger’s matching algorithm to generate disparity values for keypoint pixels. These pixels and their corresponding feature vectors are used to train our algorithm. The disparity values for rest of the pixels in the image are calculated using our trained model. Our approach is justified since for a given image, the number of disparity values is finite, which would most likely be covered in one of the training pixels. We initially use daisy parameters as our feature vector. Later on we add the normalized pixel location(x, y coordinate) to the daisy features. In addition, to generate some training examples in smooth regions, we performed pixel padding according to a heuristic that nearby pixels are likely to have the same disparity value. Finally, we try increasing the training size by finding the disparity values of edge pixels using correlation based matching. Multi-class Support Vector Machine Our approach is to use Geiger’s matching algorithm to generate disparity values for keypoint pixels. These disparity values are used as class labels (target variable). Each keypoint pixel in the left image of a given pair of images is a training example. For a training example i in the training set of size  m , we have a feature vector  x and a class label  c , the pixel’s disparity value calculated from Geiger’s algorithm. We use these keypoints pixels and their disparity values to train our multi-class SVM model. Our algorithm extracts features from each of the remaining pixels in the left image, and uses the trained SVM model to predict the disparity value. In this report, non-keypoint pixels in the left image are referred to as pixels in a test set or test pixels. In this project, two sets of features are used and compared. While the first set of feature consists of 200 Daisy descriptors of a pixel, the second set uses both Daisy descriptors and the normalized x and y coordinates denoting the location of the pixel. Let n be the length of each feature vector then  x is a n x 1 vector. Therefore, a training set consists of a  m x 1 label vector  C and a  m x n feature matrix  X .