Active speaker localisation and tracking using audio and video

This thesis is concerned with the problem of tracking active speakers using audio and video data. Particular focus is placed on the task of tracking the current active speaker in a lecture room environment using multiple cameras and multiple microphones. A database of lecture recordings corresponding to this scenario from the European Integrated Project, Computers in the Human Interaction Loop (CHIL) is used to support work presented throughout the thesis. Within the lecture room environment, the problem of extracting reliable audio and video based features for detecting people is explored. In the audio domain, the use of time-delay estimates from multiple pairs of microphones is examined for localising active speakers. Fundamental limitations on localisation accuracy using this approach are also discussed. In the video domain, background modelling, face detection and skin colour detection are considered as candidate features for the visual detection of people. A new skin colour model is introduced which models for the non-linear dependence of skin-tone on luminance and is shown be effective in modelling skin colour under low illumination. Following the evaluation of audio and video features for locating people, a review of existing techniques which aim to fuse audio and video information for tracking is presented. This thesis makes two critical analyses in relation to the joint audio-video based localisation problem. The first analysis examines the expected accuracy of audio-based localisation using multiple microphones and video-based localisation using multiple cameras. Within this analysis, the theory of estimating localisation uncertainty for both localisation techniques is unified under a common framework. Using this, single modality localisation is compared to localisation through different audio-video based fusion strategies. The different fusion strategies evaluated are the fusion of audio and video location estimates in the positional domain, the audio domain and the video domain. For each of these fusion methods, it is found that little is to be gained in terms of localisation accuracy through a fusion based approach, in comparison to the accuracy of single modality video-based localisation. The second critical analysis made in this thesis evaluates the localisation performance of a configuration of microphone arrays in a lecture room. Theoretical bounds on the accuracy of time-delay estimates obtained using the microphone arrays are employed in this analysis. A theoretical bound on the accuracy of localising an active speaker is developed to include important aspects influencing localisation performance such as, the signal-to-reverberant ratio and the speaker and microphone directionality characteristics. It is argued that the sub-optimal placement of microphone arrays results in poor localisation performance. The configuration of microphone arrays in the CHIL lecture room is taken as an example case to demonstrate this point. A novel algorithm is proposed which uses simulated annealing to automatically determine the optimal placement of microphone arrays to minimise localisation uncertainty. The effectiveness and practicality of the proposed algorithm is shown for a number of example