In this paper, both non-mixing and mixing local minima of the entropy are analyzed from the viewpoint of blind source separation (BSS); they correspond respectively to acceptable and spurious solutions of the BSS problem. The contribution of this work is twofold. First, a Taylor development is used to show that the exact output entropy cost function has a non-mixing minimum when this output is proportional to any of the non-Gaussian sources, and not only when the output is proportional to the lowest entropic source. Second, in order to prove that mixing entropy minima exist when the source densities are strongly multimodal, an entropy approximator is proposed. The latter has the major advantage that an error bound can be provided. Even if this approximator (and the associated bound) is used here in the BSS context, it can be applied for estimating the entropy of any random variable with multimodal density
As for the mutual information, simulations results in @cite_11 suggest that mixing local entropy minima exist in specific cases (i.e. when the source pdfs are strongly multimodal, which sometimes occur in practice, for sinusoid waveforms among other). These results, based on density estimation using the Parzen kernel method, are confirmed by other simulations using directly entropy estimation, such as Vasicek's one in @cite_8 or based on the approximator analyzed in this paper in @cite_18 . Rigorously speaking, the above results do not constitute an absolute proof since error bounds are not available for the approximation procedure. By contrast, a theoretical proof is given in @cite_14 , but for a specific example only (two bimodal sources sharing the same symmetric pdf). The existence of mixing local entropy minima has also been shown in @cite_9 (without detailed proof) in the case of two non symmetric sources with strongly multimodal pdfs.
@cite_18 @cite_14 @cite_8 @cite_9 @cite_11
2135806772 2146830462 2110841704 2132084190 2130788103
Recent simulation results have indicated that spurious minima in information-theoretic criteria with an orthogonality constraint for blind source separation may exist. Nevertheless, those results involve approximations (e.g., density estimation), so that they do not constitute an absolute proof. In this letter, the problem is tackled from a theoretical point of view. An example is provided for which it is rigorously proved that spurious minima can exist in both mutual information and negentropy optima. The proof is based on a Taylor expansion of the entropy. The marginal entropy h(Z) of a weighted sum of two variables Z = αX + βY, expressed as a function of its weights, is a usual cost function for blind source separation (BSS), and more precisely for independent component analysis (ICA). Even if some theoretical investigations were done about the relevance from the BSS point of view of the global minimum of h(Z), very little is known about possible local spurious minima.In order to analyze the global shape of this entropy as a function of the weights, its analytical expression is derived in the ideal case of independent variables. Because of the ICA assumption that distributions are unknown, simulation results are used to show how and when local spurious minima may appear. Firstly, the entropy of a whitened mixture, as a function of the weights and under the constraint of independence between the source variables, is shown to have only relevant minima for ICA if at most one of the source distributions is multimodal. Secondly, it is shown that if independent multimodal sources are involved in the mixture, spurious local minima may appear. Arguments are given to explain the existence of spurious minima of h(Z) in the case of multimodal sources. The presented justification can also explain the location of these minima knowing the source distributions. Finally, it results from numerical examples that the maximum-entropy mixture is not necessarily reached for the 'most mixed' one (i.e. equal mixture weights), but depends of the entropy of the mixed variables. Marginal entropy can be used as cost function for blind source separation (BSS). Recently, some authors have experimentally shown that such information-theoretic cost function may have spurious minima in specific situations. Hence, one could face spurious solutions of the BSS problem even if the mixture model is known, exactly as when using the maximum-likelihood criterion. Intuitive justifications of the spurious minima have been proposed, when the sources have multimodal densities. This paper aims to give mathematical arguments, complementary to existing simulation results, to explain the existence of such minima. This is done by first deriving a specific entropy estimator. Then, this estimator, although reliable only for multimodal sources with small-overlapping Gaussian modes, allows one to show that spurious minima may exist when dealing with such sources. This paper presents two approaches for showing that spurious minima of the entropy may exist in the blind source separation context. The first one is based on the calculation of first and second derivative of the output entropy and The second one is based on entropy approximator for multimodal variable having small overlap between the modes. It is shown that spurious entropy minima arise when the source distribution becomes more and more multimodal.