Detection and Tracking of Point Features

The factorization method described in this series of reports requires an algorithm to track the motion of features in an image stream. Given the small inter-frame displacement made possible by the factorization approach, the best tracking method turns out to be the one proposed by Lucas and Kanade in 1981. The method de nes the measure of match between xed-size feature windows in the past and current frame as the sum of squared intensity di erences over the windows. The displacement is then de ned as the one that minimizes this sum. For small motions, a linearization of the image intensities leads to a Newton-Raphson style minimization. In this report, after rederiving the method in a physically intuitive way, we answer the crucial question of how to choose the feature windows that are best suited for tracking. Our selection criterion is based directly on the de nition of the tracking algorithm, and expresses how well a feature can be tracked. As a result, the criterion is optimal by construction. We show by experiment that the performance of both the selection and the tracking algorithm are adequate for our factorization method, and we address the issue of how to detect occlusions. In the conclusion, we point out speci c open questions for future research. Preface In principle, the stream of images produced by a moving camera allows the recovery of both the shape of the objects in the eld of view, and the motion of the camera. Traditional algorithms recover depth by triangulation, and compute shape by taking di erences between depth values. This process, however, is sensitive to noise for distant scenes. To overcome this problem, we have developed a factorization method to decompose an image stream directly into object shape and camera motion, without computing depth as an intermediate step. To explore this new method, we designed a series of eleven technical reports, as shown in gure 1, going from basic theory to implementation. The rst two report, already published as CMU-CS-90-166 and CMU-CS91-105, explore the idea in the case of planar motion, in which images are single scanlines, and then for unrestricted, 3D motion and complete images. The method used to select and track points from frame to frame is described in detail in the present report, number 3. As to the future reports, number 4 and 5 address shape and motion reconstruction and tracking for line features, rather than points. Reports number 6 and 7 deal with perspective e ects and degenerate motion. Occlusion can be handled by our method, as shown in report number 8, while number 9 examines how to detect cases of spurious, that is, non-rigid, motion. The factorization algorithm in report number 2 deals with the whole stream of images at once; report number 10, in contrast, proposes an implementation that can work with an inde nitely long stream of images. Report number 11 considers a more radical departure from the assumption of a static scene than spurious motion, that of several bodies moving independently in the eld of view. 6. perspective 8. occlusion 9. spurious motion 11. multiple motion 2. point features in 3D motion motion 7. degenerate 1. planar motion 4. line features in 3D motion 3. detection and tracking of line features 10. implementation issues 3. detection and tracking of point features Figure 0.1: The reports in the series. Chapter