Active Hidden Models for Tracking with Kernel Projections

We introduce Active Hidden Models (AHM) that utilize kernel methods traditionally associated with classification. We use AHMs to track deformable objects in video sequences by leveraging kernel projections. We introduce the “subset projection” method which improves the efficiency of our tracking approach by a factor of ten. We successfully tested our method on facial tracking with extreme head movements (including full 180-degree head rotation), facial expressions, and deformable objects. Given a kernel and a set of training observations, we derive unbiased estimates of the accuracy of the AHM tracker. Kernels are generally used in classification methods to make training data linearly separable. We prove that the optimal (minimum variance) tracking kernels are those that make the training observations linearly dependent. The design of visual tracking methods is a fundamental goal of computer vision. Tracking methods are needed to solve important video understanding problems such as interpreting human activities for video-conferencing, surveillance, human-computer interaction, sign-language recognition, and driver assistance. In this paper, we introduce a new approach to visual tracking that differs significantly from prior work described in the computer vision literature. Our approach is based on techniques from classification theory, in particular, kernel projections. We show how this fundamental technique can become extremely valuable in the new context of tracking. Our exemplar based method tracks rigid or deformable objects by uncovering their hidden states. We introduce Active Hidden Models (AHMs) to describe the feasible object states in the hidden state space. We train a Active Hidden Model in advance of the tracking process by applying a kernel function to a large set of training images. Using a kernel function, we can map the training images, which are considered points in an observation space, to their corresponding points in the feature/state space. We select training images that show the objects of interest in representative rigid or nonrigid transformations. We then define tracking as the problem of identifying the image in the observation space that best describes the rigid or non-rigid object transformation seen in the current video frame. To find the best matching image, we map the current object image into the hidden space via the same kernel function and project the resulting point onto the subspace that defines the Active Hidden Model of the object. We then reconstruct a fundamental appearance of this hidden point, resulting in the best-effort reconstruction of the object. To design a practical tracking system that runs in real time, we introduce a method to approximate the kernel projection. Our experiments show that only a small subset of the training set (» 1=10) is needed to approximate a projection accurately. In some of our experiments, a subset of size of 2 is sufficient for an accurate projection. We provide unbiased estimates of the accuracy of our AHM tracking approach given a kernel function and a training set. We show that the optimal (minimum variance) kernels are those that make the training data linearly dependent in the state space. We successfully applied our AHM-based tracking method to three problem domains ‐ recovering face pose and location, understanding facial expressions, and tracking the deformations of candy wrapper. For each domain, we show that our tracking method can recognize the object transformations accurately and in real time. The computer vision literature includes various tracking approaches that also leverage results of classification theory; for example, tracking can be interpreted as a modelbased binary classification problem which decides whether a region in the current image matches the object model or