Exploring the Practical Limits of Optical Flow

This report documents our experience with different optical flow estimation methods and our attempt to use optical flow both qualitatively and quantitatively. Special attention is devoted to improving the Lucas-Kanade method to obtain dense flow. We use a simple clustering technique to find looming objects. This method has the potential of supporting obstacle avoidance using optical flow. Experiments using real images demonstrate that this simple clustering is effective for certain scenes. We also point out when this technique will fail. We try to use optical flow quantitatively to recover the structure of a piecewise planar environment. First, we use the widely-known 8-parameter planar flow equations to locate individual planes in the scene. Second, in lieu of full flow, we try to use normal flow to compute both the ego-motion and the structure. Both trials fail ungracefully, mostly due to noisy flow data. We describe the mathematics of both methods and our experimental results. This work is supported by the Apple Aid through NAUEOD contract N00174-03-C-0017 and the NSF RI grant No. EIA-0080124. 1 Motivation and Goals Our goals for the summer of 2003 were two. First, to change or improve the flow algorithm to allow the computation of dense flow (previous work relied on sparse flow computed at only a few image locations). Second, to use dense flow information in a more quantitative way to reconstruct descriptions of the environment or the robot’s motion (in the form of plane equations describing surfaces, or robot ego-motion descriptions, say). Thus we wanted to use the flow in order to calculate and map the three dimensional structure of the environment. We hoped to be able to use the flow data quantitatively to judge the depth and orientation of objects and planes in the environment. In this report, we first survey different flow estimation methods we tried and our own conclusion as which ones produce dense and accurate optical flow. We then describe several unsuccessful attempts to use optical flow and normal flow quantitatively. These attempts include using dense flow and simple clustering to find looming objects, using full flow to find planar structures in the scene and using normal flow to reconstruct a 3D planar scene. One reason of our failure is the violation, in real scenes, of the usual assumption that the motion field corresponds to the flow field. This assumption is especially brittle in indoor scenes with lots of reflective floors and walls. Yet, the more fundamental problem is to obtain accurate optical flow, be it full flow or normal flow. Our experience indicates that for common scenes, current flow estimation methods don’t provide usable raw data. 2 Most Recent Prior Work Under This Contract. In the 2002-2003 academic year, we worked on a navigation control system that could direct a mobile robot around obstacles [28]. The method was to estimate optic flow divergence in eight regions of interest (ROI) in the flow field. In each ROI a standard optic flow algorithm (an Open Source version of Lucas and Kanade’s algorithm) was applied to a small number of points. The algorithm was slow enough that the resulting “flow” was sparse (at a low resolution and only over a few areas). The obstacle avoidance system initializes itself by moving in a flat, obstacle free area, where it gathers statistics on flow in the ROIs. Subsequently it compares flow statistics with those from the initial “pure ground plane” situation. Deviations signal moving objects or surfaces not lying in the ground plane: potential obstacles. The main statistic used was the (vector-field) divergence of flow: generally a change in the density or direction of flow vectors. This technique was successfully used with dense flow by Nelson [26]. If the flow in any of the ROIs diverged in such a way as to indicate an obstacle, the robot would be told to steer the other way. This work was reasonably successful but not backed up by theory and thus was rather domaindependent. It also needed much tweaking of thresholds. 3 Flow definitions and properties To facilitate our discussions, we briefly define relevant flow terms here. For details, readers are referred to standard computer vision textbooks,e.g.[19; 5]. Motion flow or motion field is the 2D projection of 3D motion. For a static scene, let (tx, ty, tz) be the camera’s translational velocities