# Iterative Receiver Design for the Estimation of Gaussian Samples in Impulsive Noise +
## 3 Factor Graph and Sum-Product Algorithm
### Kullback-Leibler Divergence  
The KL divergence is an information theoretic tool (see, e.g., [20]) that can be used to measure the similarity between two probability distributions. We can search a pre-selected approximating family \(\mathcal{F}\) of (possibly \"simple\") distributions, to find the distribution \(q(x)\) closest to a given distribution \(p(x)\), which is possibly much more \"complicated\" than \(q(x)\) (such as, e.g., a mixture). Such an operation is called projection and is accomplished by minimizing the KL divergence, as:  
\[\text{proj}[p(x)]=\arg\min_{q\in\mathcal{F}}\quad\text{KL}(p(x)||q(x))\quad. \tag{23}\]  
A common choice is to let \(q(x)\) belong to an exponential family, \(q(x)=\exp\left(\sum_{j}G_{j}(x)v_{j}\right)\), where \(G_{j}(x)\) are the so-called features of the family and \(v_{j}\) are the so-called natural parameters of the distribution. Gaussian pdfs of Tikhonov pdfs, just to name a couple, are common distributions that naturally belong to exponential families, each with its own set of features (\(G_{j}(x)=x^{j}\) (\(j=0,1,2\)) for the Gaussian; \(G_{1}(x)=\cos(x)\) and \(G_{2}(x)=\sin(x)\) (\(j=1,2\)) for the Tikhonov) and each with its own set of constraints on the natural parameters (e.g., \(v_{2}<0\), for the Gaussians), which force the family to include proper distributions only. Exponential families have the pleasant property of being closed under the multiplication operation. Furthermore, it can be shown [20] that the projection operation in (23) simply reduces to equating the expectation of each feature with respect to the true distribution \(p(x)\), \(E_{p}[G_{i}(x)]\), with that computed with respect to the approximating distribution \(q(x)\), \(E_{q}[G_{i}(x)]\). Such a procedure of matching the expectations (of the features) is at the heart of (and gives the name to) the celebrated EP (expectation propagation) algorithm [21], further applied in Section 4 to our problem.  
To get a better understanding of this procedure, let us apply it to project a Gaussian mixture onto a single Gaussian pdf. Suppose \(p(x)\) is the following Gaussian mixture:  
\[p(x)=\sum_{i}\alpha_{i}g(x;\mu_{i},\sigma_{i}^{2})\quad, \tag{24}\]  
where \(\sum_{i}\alpha_{i}=1\) for normalization. We shall approximate it by the nearest Gaussian \(q(x)\) in the sense of the KL divergence, i.e., by applying (23). For that purpose, given that the expectations of the features of a Gaussian distributions are simply the mass, the mean, and the mean-squared value of that Gaussian (i.e., the expectations of one, \(x\) and \(x^{2}\)), the matching of these moments computed under \(p(x)\) with those computed under \(q(x)\) simply amounts to equating the mean and variance of the two distributions (assuming that both \(p(x)\) and \(q(x)\) are normalized).  
\[E_{q}[x]=E_{p}[x] = \sum_{i}\alpha_{i}\mu_{i} \tag{25}\] \[E_{q}[x^{2}]=E_{p}[x^{2}] = \sum_{i}\alpha_{i}\Big{(}\sigma_{i}^{2}+\mu_{i}^{2}\Big{)} \tag{26}\]  
are the resulting algebraic equations from which the parameters (both the natural parameters and the moments) of the best approximating Gaussian \(q(x)\) are derived.