Multimodal probabilistic person tracking and identification in smart spaces

Smart spaces and environments that perceive their occupants’ actions and offer intelligent human-centered services are an active topic of research. In this context, the tracking and identification of persons (referred to as “identity tracking”) plays an important role, as it provides fundamental contextual knowledge upon which further analysis of activities or interactions can be performed. The overall goal is to simultaneously keep track of multiple identities evolving in the space using unobtrusive sensors, such as distantly placed cameras and microphones. Further, this is to be accomplished in everyday scenarios imposing little or no constraint on the natural behavior of users, such as in meeting rooms, office areas, living rooms, etc. One of the main problems facing identity tracking is that in such realistic scenarios, reliable cues for person-specific identification are hard to obtain with the sensors described above. Generally observable features based on a person’s overall appearance, such as the color of clothing, body height, etc., can be ambiguous (e.g. when all persons wear black) and may well vary considerably with time or environmental conditions (e.g. taking off one’s jacket, sitting down). On the other hand, more invariant and person-specific features such as those gained by face or voice identification may only seldom be observable (such as when a good view of the face is available or when the person takes his or her turn speaking in a conversation). The main idea followed in this thesis to overcome this problem is to actively and opportunistically capture reliable identification cues for each occupant whenever they become available and to keep track of identified persons until further observations can be made. This involves using focusable sensors such as steerable cameras to obtain high resolution facial close-ups and microphone arrays to determine the origin of speech. The difficulties to be dealt with are twofold: Firstly, single observations gained through face or voice identification are inherently noisy, being influenced by lighting conditions, low resolution, imperfect facial alignment, environmental noise, crosstalk, etc. This implies that identification cues need to be accumulated in time and multiple modalities should be used to increase the accuracy of identification. Secondly, in realistic scenarios, the tasks of automatically detecting and tracking persons in the first place cannot be assumed solved with perfect accuracy. Persons may be missed, tracks may be confused or lost. This means that person identities need to be correctly recovered when observations again become available. While some amount of work has been done on the field of tracking and identification using sensor networks with overlapping or even non-overlapping views, none of the approaches so far tackle all the related problems efficiently. Most integrated approaches rely on general appearance features, such as color, and build on the assumption that features for identification are jointly available with features for tracking with every observation made. While some approaches use person-specific features such as provided by face identification, they still rely on the continuous availability of high resolution face images in very restrictive setups. Approaches that use acoustic features for identification typically assume that the number of persons is known a priori, that speakers take frequent turns, and do not keep track of their locations except in very restrictive setups. More importantly: Almost all approaches found in the literature that target multiple users are limited to applications where the detection (and spatio-temporally local tracking) of persons can be realized flawlessly and build on the results of this step for identification. A better approach would be to integrate identification and tracking into one framework, such that errors in tracking are less detrimental to overall accuracy. In this thesis, a new methodology is introduced for the multimodal tracking and identification of multiple persons by seeking and integrating reliable ID cues whenever they become observable. The method opportunistically integrates person-specific identification cues that can only sparsely be observed for each person over time and keeps track of the location of identified persons while ID cues are not available. It also fuses the acoustic and visual modalities to increase its robustness and flexibility and probabilistically integrates tracking and identification at the same level, such that errors made by one are compensated by the other. Finally, it represents a general framework for ID-Tracking in that it supports different types and configurations of sensors, different modalities with varying reliability or availability, and is scalable to different rooms, sensor setups, etc. The developed method is a generative approach based on Bayesian filtering of high level tracking and ID cues. It is implemented as a particle filter which approximates the probability density of the persons’ presence, locations and identities by a set of samples or particles. The belief of the particle filter is propagated in time and updated which each new observation concerning person locations or identities. In this thesis, a new formulation is developed to represent the belief of the particle filter about the state of the world (the person locations and identities), and to keep the state space tractable while avoiding to rely on flawless detection and tracking results. Moreover, the formulation allows the integration of sparsely available observations concerning identities together with tracking-related observations, which arrive at a constant and high framerate. The performance of the developed system is evaluated in the context of small meetings with several users taking place in “smart” rooms, and the effect of using single or multiple modalities on identification accuracy is investigated. Experiments have been made using a large multimodal database of recordings captured in 5 different instrumented rooms. This database, which was used in two international evaluations, the CLEAR evaluation workshops, allowed a thorough, quantitative evaluation of the approach. Note that the accuracy metrics used in the CLEAR workshops themselves were developed in the course of this thesis in an effort to provide a clear and generally applicable methodology for the quantitative evaluation of multiple target tracking performance. Experiments involving real-time identity tracking were also made with a distributed implementation of the approach using several cameras and microphones in a smart room. The results show that the integrated approach is robust to tracking failures and degrades gracefully with decreasing tracking accuracy. They also show that the fusion of audio and visual modalities can help achieve noticeable identity tracking accuracies, even in relatively uncontrolled situations with multiple persons, occlusions, cross-talk, etc., and using available state-of the art tracking, face and voice identification components.