# Mutual Information, the Linear Prediction Model, and CELP Voice Codes
## 6 Log Ratio of Entropy Powers  
We can use the definition of the entropy power in Equation (6) to express the logarithm of the ratio of two entropy powers in terms of their respective differential entropies as [8]  
\[\log\frac{Q_{X}}{Q_{Y}}=2[h(X)-h(Y)]. \tag{11}\]  
We can write a conditional version of Equation (6) as  
\[Q_{X|Y_{n}}=\frac{1}{(2\pi e)}\text{exp}\,2h(X|Y_{n})\leq Var(X|Y_{n}), \tag{12}\]  
and from which we can express Equation (11) in terms of the entropy powers at successive stages in the signal processing chain (Figure 5), as  
\[\frac{1}{2}\log\frac{Q_{X|Y_{n}}}{Q_{X|Y_{n-1}}}=h(X|Y_{n})-h(X|Y_{n-1}). \tag{13}\]  
If we add and subtract \(h(X)\) to the right-hand side of Equation (13), we then obtain an expression in terms of the difference in mutual information between the two stages as  
\[\frac{1}{2}\log\frac{Q_{X|Y_{n}}}{Q_{X|Y_{n-1}}}=I(X;Y_{n-1})-I(X;Y_{n}). \tag{14}\]  
From the series of inequalities on the entropy power in Equation (9), we know that both expressions in Equations (13) and (17) are greater than or equal to zero. Figure 5: N-link system block diagram (adapted from [19]). These results are from [8] and extend the data processing inequality by providing a new characterization of the information loss between stages in terms of the entropy powers of the two stages. Since differential entropies are difficult to calculate, it would be particularly useful if we could obtain expressions for the entropy power at two stages and then use Equations (13) and (17) to find the difference in differential entropy and mutual information between these stages. We are interested in studying the change in the differential entropy and mutual information brought on by different signal processing operations by investigating the log ratio of entropy powers. In the following, we highlight several cases where Equation (11) holds with equality when the entropy powers are replaced by the corresponding variances. The Gaussian and Laplacian distributions often appear in studies of speech processing and other signal processing applications [10; 15; 20], so we show that substituting the variances for entropy powers in the log ratio of entropy powers for these distributions satisfies Equation (11) exactly. For two i.i.d. Gaussian distributions with zero mean and variances \(\sigma_{X}^{2}\) and \(\sigma_{Y}^{2}\), we have directly that \(Q_{X}=\sigma_{X}^{2}\) and \(Q_{Y}=\sigma_{Y}^{2}\), so  
\[\frac{1}{2}\log\frac{Q_{X}}{Q_{Y}}=\frac{1}{2}\log\frac{\sigma_{X}^{2}}{\sigma _{Y}^{2}}=[h(X)-h(Y)], \tag{15}\]  
which satisfies Equation (11) exactly. Of course, since the Gaussian distribution is the basis for the definition of entropy power, this result is not surprising. For two i.i.d. Laplacian distributions with variances \(\lambda_{X}^{2}\) and \(\lambda_{Y}^{2}\)[21], their corresponding entropy powers \(Q_{X}=2e\lambda_{X}^{2}/\pi\) and \(Q_{Y}=2e\lambda_{Y}^{2}/\pi\), respectively, so we form  
\[ \begin{split}\frac{1}{2}\log\frac{Q_{X}}{Q_{Y}}=& \frac{1}{2}\log\frac{\lambda_{X}^{2}}{\lambda_{Y}^{2}}\\ =&[h(X)-h(Y)].\end{split}  \tag{16}\]  
Since \(h(X)=\ln(2e\lambda_{X})\), the Laplacian distribution also satisfies Equation (11) exactly [5]. We thus conclude that we can substitute the variance, or for zero mean Laplacian distributions, the mean squared value for the entropy power, in Equation (11), and the result is the difference in differential entropies. Interestingly, using mean squared errors or variances in Equation (11) is accurate for many other distributions as well. It is straightforward to show that Equation (11) holds with equality when the entropy powers are replaced by mean squared error for the logistic, uniform, and triangular distributions as well. Furthermore, the entropy powers can be replaced by the ratio of the squared parameters for the Cauchy distribution. Therefore, the satisfaction of Equation (11) with equality occurs in more than one or two special cases.