# Independent Component Analysis Based on Information Bottleneck

Authors: Qiao Ke; Jiangshe Zhang; H. M. Srivastava; Wei Wei; Guang-Sheng Chen
Journal: Abstract and Applied Analysis
 (2015)

---

## Abstract

The paper is mainly used to provide the equivalence of two algorithms of independent component analysis (ICA) based on the information bottleneck (IB). In the viewpoint of information theory, we attempt to explain the two classical algorithms of ICA by information bottleneck. Furthermore, via the numerical experiments with the synthetic data, sonic data, and image, ICA is proved to be an edificatory way to solve BSS successfully relying on the information theory. Finally, two realistic numerical experiments are conducted via FastICA in order to illustrate the efficiency and practicality of the algorithm as well as the drawbacks in the process of the recovery images the mixing images.


---

## Body

## 1. Introduction

Information theory is found by Claude Elwood Shannon (1948) in one of his famous academic papers, “A Mathematical Theory of Communication,” where he gave the definition of information and information entropy based on the probability theory which build a bridge between the information theory and the numerical mathematics. Some basic conceptions (entropy, negentropy, mutual information, and so on) in the information theory have been successfully used to elaborate the independent components (ICs) and to deal with the problems on the application of the blind source separation (BSS). In the past decades, the information theory has been applied successfully into many fields such as clustering [1], medical examination [2], independent component analysis [3], feature learning [4], and telecommunication [5–8]. The purpose of this paper is to use the information bottleneck to derive the maximum of the mutual information (MI) between the mixing data and the recovery data which is no more than the MI of the recovery data and the original sources.The rest of the paper is organized as follows. In Section2, we first explain the information theory and introduce some important formulas. In Section 3, based on the entropy, the mutual information (MI), and negentropy, information bottleneck is used to illustrate the equivalence of the two classical algorithms, informax [3] and FastICA [9]. At last, by a series of experiments of synthetic data, sonic data, and image in Section 4, it is easy to compare the accuracy and complexity of the two algorithms. However, the ambiguity of the direction and scale of the recovery matrix lead the results of the image to the opposite.


## 2. Information Theory

According to the explanation of communication theory by Warren Weaver, “information” is not related to what you do say but to what you could say. That is, information is a measure of one’s freedom of choice when one selects a message [6, 7, 10].At first, people focused attention on the “meaningful” or “relevant” information, which is crucial in solving the problem of transmitting information. Then, some scholars argue that lossy source compression provides a natural quantitative approach to “relevant information” [11, 12].So, information bottleneck, which is going to seek for a tradeoff between the compression and the representation and preserving meaningful information, could be decomposed into the following aspects:(1)
how to define the “meaningful” or “relevant” information;(2)
how to extract the efficient representation of relevant information in order to transmit it speedily;(3)
how to recover the information as exactly and comprehensively as possible only based on the efficient representation of relevant information.People regard the possible results of the uncertainty or fuzzy as the surprise or information [13], and the smaller probability of the results occurring, the bigger surprise or the more information people obtained. So, entropy H(X), a measure of the chaotic degree, is defined to measure the uncertainty of information. Assume that X is a discrete random variable, and probability density function (pdf) p(x)=Pr(X=x),x∈χ; then, entropy is defined as(1)HX=-∑x∈χ‍pxlogpx.Moreover, it is easy to generalize it to more than two random variables, the joint entropy. On the other hand, mutual information (MI), a measure of dependency of two different random variable sets, is regarded as the reduction of uncertainty of the random variable, given the other random variable. Consider two random variables X and Y, with the joint pdf p(x,y) and marginal pdfs p(x) and p(y), respectively. MI can be written as follows:(2)IX;Y=∑x∈χ‍∑y∈Y‍p(x,y)logp(x,y)p(x)p(y)=Ep(x,y)logp(x,y)p(x)p(y).It is easy to prove the following equations about MI based on information entropy:(3)IX;Y=H(X)+H(Y)-H(X,Y)=H(Y)-H(Y∣X)=HX-H(Y∣X).According to the last two terms, we can find the relationship between MI and entropy. if and only ifX and Y are irrelevant.


## 3. The Equivalence of the Two ICA Algorithms Based on the IBN

Information bottleneck (IBN) [14] is used to make sure to recover the compressed information X, which is presumed to be good representation or compression of the original information S, to the recipient in terms of Y in the following type: (4)S⟹X⟹Y.Now, in the terminology of information theory and optimizing theory, there are two inconsistent optimal problems that, on the one hand, we would make sure to minimize MI between the original informationS and the compressed information X and, on the other hand, we want to capture the maximum of mutual information between Y and X. Obviously, the amount of information about Y in X~ is given by (5)IX;Y=ΣyΣx~py,xlogpy,xpypx≦IS;Y,while the mutual information between the independent sources and the mixing signals is determinate but unknown with the precondition of ICA.ICA is studied to find the independent sources asyi,Y=(y1,y2,…,yn), which is equal to the original independent sources, S=(si1,si2,…,sin) ignoring the ambiguity of the direction and scale. Furthermore, the independent sources are the most concise, while any linear transformation of the independent sources obtains the redundance information(6)IS;Y=H(S)-H(S∣Y)≦HS.If and only if Y are the independent sources the equation is true. That is to say, we need to find the recovery matrix W, Y=WX, in order to obtain the independent sources. Because of the precondition of the unknown independent sources and mixing matrix, the optimal problem of ICA is written by IBN [14] as follows:(7)maxYI(X;Y),where H(S) is an theoretic maximum and H(Y) is an approximate maximum.


### 3.1. Infomax Method

Infomax method [3] is used to tackle the problem of separating the mixture signals X, attempting to look for the weight matrix W without both the mixture matrix A and the original signal S. We attempt to illustrate BBS in the following: (8)S→X=ASmixtureX→Y=WXrecoveryY.According to the optimization problem of ICA (16), we could rewrite it as follows: (9)maxWIX;Y=HY-HYX.That is also regarded as(10)maxWHY.The equation can be differentiated with respect to a parameter W, involved in the mapping from X to Y: (11)∂∂WI(X;Y)=∂∂WH(Y).Therefore, MI between the mixtures X and recoveries Y could be maximized by maximizing the entropy of the recoveries alone. And then the gradient method is used to obtain the learning method [3].Considering the slow convergence and nonprecise and low accuracy,Hyva¨rinen gave a FastICA based on the maximum of the negentropy, which is regarded as the measure for the independency of the signal.


### 3.2.  FastICA Method

According to the informax method and IBN, the BBS is equal to the optimal problem as follows:(12)maxYH(Y).Then, the optimal problem can be adapted as(13)minYI(Y)=∑i=1n‍H(yi)-H(Y),where Y=(y1,y2,…,yn) and yi(i=1,2,…,n) are the ICs. How can we identify and measure the independence of the recovery data? The equivalence of the non-Gaussian random variables and negentropy is illustrated based on the Central-Limit Theorem [9].Theorem 1 (Central-Limit Theorem [15]).
Given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined expected value and well-defined variance, will be approximately normally distributed.According to the Central-Limit Theorem, if and only if the recovery dataY is a permutation of the original independent sources X, the non-Gaussian random variables reaches the maximum. Consider(14)IY=∑i=1n‍H(yi)-H(X)-logdet(W)=(-H(X)-logdet(W))+∑i=1n‍(H(yigauss)-J(yi))via the normalization of mixing data: (15)EyyT=WExxTWT=I,where I is the n_order identical matrix. So, (10) is rewritten as(16)maxYJ(Y)s.t.jjW2=1which is equivalent to (7) and (10). So, we can obtain the equivalence of the two classical algorithms in the point of IBN.The approximation of negentropy and fixed-point algorithm are applied to derive the learning rule [9].


## 3.1. Infomax Method

Infomax method [3] is used to tackle the problem of separating the mixture signals X, attempting to look for the weight matrix W without both the mixture matrix A and the original signal S. We attempt to illustrate BBS in the following: (8)S→X=ASmixtureX→Y=WXrecoveryY.According to the optimization problem of ICA (16), we could rewrite it as follows: (9)maxWIX;Y=HY-HYX.That is also regarded as(10)maxWHY.The equation can be differentiated with respect to a parameter W, involved in the mapping from X to Y: (11)∂∂WI(X;Y)=∂∂WH(Y).Therefore, MI between the mixtures X and recoveries Y could be maximized by maximizing the entropy of the recoveries alone. And then the gradient method is used to obtain the learning method [3].Considering the slow convergence and nonprecise and low accuracy,Hyva¨rinen gave a FastICA based on the maximum of the negentropy, which is regarded as the measure for the independency of the signal.


## 3.2.  FastICA Method

According to the informax method and IBN, the BBS is equal to the optimal problem as follows:(12)maxYH(Y).Then, the optimal problem can be adapted as(13)minYI(Y)=∑i=1n‍H(yi)-H(Y),where Y=(y1,y2,…,yn) and yi(i=1,2,…,n) are the ICs. How can we identify and measure the independence of the recovery data? The equivalence of the non-Gaussian random variables and negentropy is illustrated based on the Central-Limit Theorem [9].Theorem 1 (Central-Limit Theorem [15]).
Given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined expected value and well-defined variance, will be approximately normally distributed.According to the Central-Limit Theorem, if and only if the recovery dataY is a permutation of the original independent sources X, the non-Gaussian random variables reaches the maximum. Consider(14)IY=∑i=1n‍H(yi)-H(X)-logdet(W)=(-H(X)-logdet(W))+∑i=1n‍(H(yigauss)-J(yi))via the normalization of mixing data: (15)EyyT=WExxTWT=I,where I is the n_order identical matrix. So, (10) is rewritten as(16)maxYJ(Y)s.t.jjW2=1which is equivalent to (7) and (10). So, we can obtain the equivalence of the two classical algorithms in the point of IBN.The approximation of negentropy and fixed-point algorithm are applied to derive the learning rule [9].


## 4. Experiments

Based on the infomax learning rule, the experiments presented here were obtained using the synthetic data as the original data plotted in Figure1(a). The result by infomax is listed in Figure 1(b) corresponding to the recovery matrices (17). Obviously, wa=W∗A is the product of the recovery matrix and the mixture matrix so that it would be the permutation of the approximate diagonal matrix. Then, we can easily find that only one substantial entry (boxed) exists in each row and column(17)wa=-0.191-0.015-0.8000.680-0.020-0.2230.0230.500-0.063.Figure 1
ICA. The synthetic independent data are plotted in (a), and the recovery data are shown in (b) corresponding to the matrixwa in (17). In terms of every column of the matrix wa, the substantial entry, waij, is almost a reflection of the transformation between the original data (i) and the recovery data (j) by multiplying the substantial entry waij, accompanied by the nonzero entries, wit,t≠j. For example, in the first result of the ICA experiments, wa21=-0.6804 is just a proof that the original data (1) is recovered into the recovery data (2) with a multiplicator wa21 and some noises based on the minor numbers of wa22 and wa23.
(a)
Synthetic independent sources
(b)
Recovery dataIn order to illustrate the efficiency of FastICA algorithm and the limitation on no more than one Gaussian variable, we list some numerical results on the blind mixing signals shown in Figures2, 3, and 4, using the nonquadratic function G1 to approximate the negentropy. Consider(18)wa=-0.707-0.0080.0010.009-0.5000.0000.001-0.0180.833.Figure 2
Based on the FastICA algorithm, we separate the randomly mixing data of the sinusoid, the rectangular curve, and the sawtooth curve successfully. At the same time, the product matrixwa of the separation matrix W and the random mixing matrix A is presented in (18).
(a)
Independent sources
(b)
Recovery data
(c)
Mixing dataFigure 3
In this numerical experiment, we add the Gaussian into the above experiment and succeed in blindly separating the mixing data. The product matrixwa is given in (19).
(a)
Synthetic independent source
(b)
Recovery data
(c)
Mixing dataFigure 4
The algorithm is not very efficient in separating the sinusoid from the two Gaussian signals in the mixing data. The more Gaussian variables there are, the more difficult it is to recover the original data.
(a)
Independent sources
(b)
Recovery data
(c)
Mixing dataFigure2 is an obvious proof to declare the efficiency of the algorithm separating the randomly mixing data of the sinusoid, the rectangular curve, and the sawtooth curve successfully. And (18) revealed that the matrix W is an elementary transformation of the approximative inverse of the mixing matrix A(19)wa=-0.000.03-0.83-0.140.710.010.000.20-0.060.01-0.051.50-0.010.500.01-0.08.Then, it is necessary and meaningful to add the Gaussian variable into the original data to prove the efficiency of the algorithm so that the result is shown in Figure3 and the product matrix wa is in (19). At last, based on the two Gaussian signals in the mixing data, the algorithm is not efficient to separate the two Gaussian signals apart shown in Figure 4.Furthermore, the average iterative steps on the first three numerical experiments based on FastICA algorithm are shown in Table1.Table 1
The comparison of the different data on the iterative steps.
Original data set
The iterative steps
Figure2
Less than 15 steps
Figure3
Less than 20 steps
Figure4
Not stationaryAfter the experiments on the synthetic data, the algorithm is also efficient on the real sonic data in Figure5 and image data in Figure 6. In the process of separating the image data, the picture in Figure 7 can be obtained, because the matrix W, which alters the picture in the opposite color, is not the exact inverse of A(20)wa=-0.3769.31814.1930.175,(21)wa=0.0690.7980.8980.015.Figure 5
Real sonic data (FastICA). Using the real sonic data from the website, we also can get the recovery data and the product matrix (20).
(a)
Mixing data
(b)
Recovery data
(c)
Independent sourcesFigure 6
Using the picture from the website and Gaussian noise mixing with the matrixA=[2,3;2,1], the Gaussian noise and the original picture are shown in the first line, the two mixing pictures in the second line, and the recovery pictures in the third line.
(a)
Independent images
(b)
Mixing images
(c)
Recovery imagesFigure 7
The recovery matrixW is not the exact inverse of the mixing matrix A, while the recovery data y has the different orders with s and is very accurately estimated, up to multiplicative signs (FastICA).


## 5. Conclusion

The algorithm of independent component analysis is enlightened from BSS, which is a very successful application of the information theory in speech recognition, image separation without knowing the linear transformation. But, there are also some disadvantages. For example, there exist the strong preconditions that the original data should be independent and the transformation should be linear.



---

*Source: 386201-2015-06-07.xml*