Entropy rate estimators and their application to blind source separation
Creators: name: Fu, Geng-Shen
Publisher: name: Maryland Shared Open Access Repository
Year: 2014
Subjects: Blind source separation
Entropy rate
Independent component analysis
Independent vector analysis
Maximum entropy distributions
Order selection

description: The characterization of a signal is one of the key issues in statistical signal processing and is vital to the success of many methods. Thus, the goal of characterization techniques is to achieve a desirable balance between flexibility and maintaining a simple form, so as to allow for generalization and efficient implementation. In this dissertation, we propose a new density estimation technique, which uses the principle of maximum entropy with Gaussian kernels. By jointly using global and local measuring functions, the new density estimation method enjoys a high level of flexibility while providing a simple parametric form for the estimated density function. When considering a signal with sample dependence, the signal cannot be fully characterized by its density function. To address this issue, we propose the use of entropy rate to provide a powerful tool for data analysis, since it accounts for all types of diversity: higher-order statistics, sample dependence, noncircularity for a complex-valued process, and dependence across processes for a vector-valued process. We propose entropy rate estimators for real-valued processes based on both Markovian and invertible source models. We extend the entropy rate estimator to complex-valued and vector-valued processes by using a widely linear whitening filter and vector autoregressive model, respectively. The new entropy rate estimators are shown to exploit all these statistical properties effectively. These entropy rate estimators can be used in a wide array of applications in statistical signal processing. In this work, we mainly focus on: model order selection, real-valued independent component analysis, complex-valued independent component analysis, and independent vector analysis. Detecting the number of latent signals in a given number of observations, or order detection, is a key issue in many signal processing problems. Many of the commonly used methods to solve this problem are based on information theoretic criteria. In order to facilitate the use of information theoretic criteria, these approaches address the issue of dependent samples by downsampling the dataset. However, downsampling the data decreases the sample size, which can cause degradation in the accuracy of the order estimation. Using an entropy rate estimator, we write the likelihood based on the entire data set and use it in an information theoretic framework. We propose two entropy-rate-based order estimation methods and show the performance improvement they offer with examples. In addition, the consistency of the entropy-rate-based methods is shown when the sample size goes to infinity. By assuming latent sources are statistically independent, independent component analysis separates underlying sources from a given linear mixture. Since in many applications, latent sources are both non-Gaussian and have sample dependence, it is desirable to exploit both properties jointly. In this work, we propose to use the mutual information rate to construct a general framework for analysis and derivation of algorithms that account for both types of diversity. We introduce four algorithms for the problem of independent component analysis, and show that their performance can approach the Cramer-Rao lower bound. Complex-valued signals frequently arise in applications such as communications, radar, and biomedicine, as well as processing when real-valued signals are in a transform domain. Independent component analysis, when implemented in the complex domain, enjoys the addition of yet another type of diversity, noncircularity of the sources---underlying components. Using the new entropy rate estimator, we develop a new complex-valued independent component analysis algorithm, which exploits all three types of diversity, non-Gaussianity, sample dependence, and noncircularity. In addition, we establish the general form of the Fisher information matrix and Cramer-Rao lower bound and show the identification conditions for the complex-valued independent component analysis problem. Experimental results show that the new method accounts for all three types of diversity effectively and provides desirable performance for a wide variety of sources. Additionally, the entropy rate estimator for vector-valued processes is used for the independent vector analysis problem, which is a generalization of independent component analysis to multiple datasets and adds the use of one more diversity, the dependence across multiple datasets. Using mutual information rate, we propose a new method that takes all these types of diversity into account.