Dense 3 D Mapping with Monocular Vision

Currently, the vast majority of autonomous mapping in robotics relies on direct measurements from costly devices such as sonar, infrared, and laser range finders. Adapting established methodologies from the structure-from-motion (SFM) subfield of computer vision to data commonly available in robotics, we have created a unique toolkit, able to render visually dense 3d maps from odometrically annotated monocular vision. We then leverage the richness of our representation to augment the widely-used Monte Carlo Localization algorithm (MCL). The new version, coined Monte Carlo Correction, provides for pose updates and hypothesis merging on top of MCL, and demonstrates the potential of visually dense 3d maps as a basis for amplifying recent advances in robotics. We thus show that although computationally intensive, monocular vision provides a low-cost, yet highly capable alternative to currently popular mapping sensors. Problem and Motivation As humans, most of us take the computation involved in our visual perception of the world for granted. Despite efforts in the robotics and computer vision communities, a method of representing visual data to guide an agent around an environment, that is as efficient and effective as human vision, has not yet been developed. Currently, the vast majority of robotic navigation relies on direct range-to-obstacle measurements from relatively costly devices such as sonar or infrared rings ($200-400) and laser range finders ($4000). For this work, the primary sensor is a $25 webcamera. In fact, most systems doing autonomous navigation typically would benefit from the additional information that a camera would provide. The research we are conducting contributes to finding a cost effective solution to the representation problem and investigates how popular spatial reasoning algorithms can be adapted to take advantage of available visual information. Our newly developed toolkit allows for construction of visually dense 3d maps from data gathered by a platform equipped solely with monocular vision and odometry. In the present phase of our work, we are investigating the potential of this representation for improvement of popular spatial reasoning algorithms. We believe that dense 3d maps show strong potential for amplifying many existing navigational algorithms, and thus a tool for constructing such maps will be beneficial to the robotics community. Background and Related Work To put our research into context with other work, we have created a taxonomy of visionbased mapping, shown in Figure 1. The horizontal axis of the taxonomy corresponds to the density of image information represented in the map, while the vertical axis indicates how the relation of map primitives to each other is incorporated into the representation. The most common type of data representation chosen for robotic systems using vision is one with sparse image information, depicted on the right side of our taxonomy. For example, Kosecka and Li, [1], used scale-invariant (SIFT) features and histograms of 2d images to localize in indoor environments. This sparse collection of 2d features characterizes the upper right-hand corner of Figure 1’s taxonomy. Another such example is a monocular map building system developed by Neira et al., [2]. Their maps were also sparse, based on vertical edges in the environment, but because these edges were mapped in a global set of coordinates their approach inhabits the lower right-hand area in Figure 1. Figure 1: Taxonomy of vision based mapping We are particularly interested in creating and navigating based on maps that classify in the lower left corner of our taxonomy, since this type of map representation incorporates the most information from acquired visual data. Previous vision based mapping work in this region of our taxonomy has been done by Iocchi et al., [3], who used a stereo vision system to create 3d maps of planar indoor environments. A number of other mapping techniques have been developed using different permutations of stereo cameras, laser range finders, and panoramic vision. Our focus, however, is on the more modest sensing capability of monocular vision. To accomplish this we draw upon the structurefrom-motion (SFM) subfield of the computer vision community, which has experienced great success creating dense 3d reconstructions from single camera data, as demonstrated by Pollefeys et al.’s method of modeling structure using a hand-held video camera, [4]. Our approach bridges the gap between computer vision and robotics by adapting established SFM techniques, [5], for use in traditional robotics domains. We take advantage of robot odometry and the planarity of indoor environments to create visually dense 3d environmental reconstructions using a single camera. Just as our robotic focus differentiates our work from mainstream computer vision, the limited nature of our sensor suite also distinguishes us from sensor-rich world acquisition methods more commonly found on high-end robotics platforms. Approach and Uniqueness The map building process consists of five principal stages: feature identification and tracking, feature triangulation, surface fitting, estimation of homographies between images and the 3d surface, and mosaicking. This section will give an overview of each of these stages, highlighting places where our approach departs from other work. The feature identification and tracking (Figure 2a) on the input image sequence relies on Stan Birchfield’s KLT tracker [6]. This first processing stage outputs the coordinates of every tracked feature, for each image that it appears in. Each feature is assigned a unique ID upon its first tracking between images, and maintains that ID as long as it continues to be tracked through the sequence of images. These tracked features’ coordinates are then passed to our Triangulator. An earlier, offline calibration provides the internal camera parameters, such as focal length. With the odometrically annotated data, the Triangulator computes the camera rotations and translations for each image. A least-squares estimation follows, providing the 3d coordinates of each of the features (Figure 2b). A critical component of the Triangulator is that it allows for feature IDs to be propagated to the 3d points, thus maintaining a direct correspondence with the original images. This correspondence enables the later visual mosaicking of the estimated surfaces. To estimate planar surfaces from the 3d point coordinates, the RANSAC [7] algorithm is used to find the optimal set of planes to fit the points (Figure 2c). For each plane the algorithm generates 2000 hypotheses and the optimal fit is decided based on a set of heuristics. This subsystem outputs a set of plane normals, a list of feature points to which each plane is fit, and the 2d coordinates of each feature when projected onto its plane. It is during this stage that most noisy points resulting from mistracking of features or Map