Particle Flow PHD Filtering for Audio-Visual Multi-Speaker Tracking
Creators: name: Liu, Yang
Publisher: name: University of Surrey
Year: 2020

description: Tracking an unknown and time-varying number of targets (e.g., speakers) in indoor environments using audio-visual (AV) modalities has received increasing interest in numerous ﬁelds including video conferencing, individual speaker discrimination, and human-computer interaction. The audio-visual sequential Monte Carlo probability hypothesis density (AV-SMCPHD) ﬁlter is a popular baseline for multi-target tracking, offering an elegant framework for fusing audio-visual information and dealing with a varying number of speakers. However, the performance of this ﬁlter can be adversely affected by the weight degeneracy problem, where the weights of most of the particles may become very small, while only few remain signiﬁcant, during the iteration of the algorithm. To address this issue, this thesis proposes the AV-SMC-PHD ﬁlter by incorporating particle ﬂows deﬁned in terms of the ordinary differential equation and the Fokker-Planck equation. This thesis considers both zero and non-zero diffusion particle ﬂows (ZPF/NPF), and developed two new algorithms, AV-ZPF-SMC-PHD and AV-NPFSMC-PHD, where the speaker states from the previous frames are also considered for particle relocation. The particle ﬂow migrates particles from the prior distribution to the posterior distribution, using a homotopy function which deﬁnes the ﬂow in synthetic time. The proposed methods can mitigate the particle degeneracy of the AV-SMC-PHD ﬁlter and improve tracking accuracy. Another issue is that the performance of the multi-speaker tracking algorithms is often degraded by mis-detection and clutter in the measurements. To address this issue, this thesis proposes an intensity particle ﬂow (IPF) SMC-PHD ﬁlter based on the intensity function derived from the measurements, informed by the clutter density and the detection probability. The IPF-SMC-PHD ﬁlter improves tracking accuracy, but induces a high computational overhead, due to the requirement for computing the sum of the likelihood intensity functions and the third-order differentiation of the likelihood density. As a result, the computational complexity of IPF is proportional to the cube of the number of measurements. To address this problem, this thesis proposes a labelled particle ﬂow (LPF) algorithm where particle labels are estimated from the measurements from multiple sensors and then used to update particles and estimate speaker states. Since the LPF only uses the ﬁrst differentiation of the likelihood density and replaces the clustering step by the sum of particle states, LPF offers a higher computational eﬃciency as compared with other particle ﬂow methods where a clustering method is often used to estimate the target states. All the proposed methods are extensively evaluated using different datasets, such as AV16.3, AVDIAR and CLEAR. The results show that the weight degeneracy problem has been mitigated by our proposed methods which offer higher tracking accuracy than the baseline methods in a variety of scenarios such as occlusion and rapid movements of the speakers.