# Mutual Information, the Linear Prediction Model, and CELP Voice Codes
## 7 Mutual Information in the Short-Term Prediction Component  
[PERSON] [5] used Equation (17) to investigate the change in mutual information as the predictor order, denoted in the following by \(N\), is increased for different speech frames. Based on several analyses of the MMSPE and the fact that the log ratio of entropy powers can be replaced with the log ratio of MMSPEs for several different distributions, as outlined in Section 6 and in [9], we can use the expression \[\frac{1}{2}\log\frac{MMSPE(X,X_{N-1})}{MMSPE(X,X_{N})}=I(X;X_{N})-I(X;X_{N-1}), \tag{17}\]  
as in [5], to associate a change in mutual information with a change in the predictor order. Figure 6 (bottom) shows 160 time domain samples from a narrowband (200 to 3400 Hz) speech sentence sampled at 8000 samples/s, and the top plot is the magnitude of the spectral envelope calculated from the linear prediction model using Equation (1). We show the \(MMSPE\) and the corresponding change in mutual information for predictor orders \(N=1,2,...,10\) in Table 1. Notice that the chain rule assumption indicated by Figure 5 is retained by the analysis chain in Figure 3 even as we increment \(N\) as in Equation (17) and in Table 1 if we use a lattice implementation of the prediction [15]. However, for most of our analyses in what follows, we consider fixed order predictors. From Table 1, we see that the mutual information between the input speech frame and a 10 th order predictor is 1.52 bits/sample. We can examine the mutual information between the input speech and a 10 th order linear predictor for other frames in the same speech utterance.  
To categorize the differences in the speech frames for easy reference, we use a common indicator of predictor performance, the signal-to-prediction error (SPER) in dB [22], also called the prediction gain [15], defined as  
\[SPER(dB)=10\log_{10}\frac{MSE(X)}{MMSPE(X,X_{10})}, \tag{18}\]  
where \(MSE(X)\) is the average energy in the utterance, and \(MMSPE(X,X_{10})\) is the minimum mean squared prediction error achieved by a 10 th order predictor. The \(SPER\) can be calculated for any predictor order, but we choose \(N=10\), a common choice in narrowband speech codecs and the predictor order that, on average, captures most of the possible reduction in mean squared prediction error without including long-term pitch prediction. For a normalized \(MSE(X)=1.0\), we see that for the speech frame in Figure 6, the \(SPER=9.15\) dB.  
Several other speech frames by the same speaker are analyzed in [5], and the results for these frames are tabulated in Table 2. From this table, it is evident that the mutual information between the input speech and a 10 th order linear predictor can change quite dramatically across frames, even with the same speaker. We observe that the change in mutual information in some sense mirrors a change in \(SPER\), with a larger \(SPER\) implying a larger mutual information. However, the explicit  
\begin{table}
\begin{tabular}{c c c} \hline \hline \(N\) & \(MMSPE(X,X_{N})\) & \(I(X;X_{N})-I(X;X_{N-1})\) \\ \hline
0 & 1.0 & 0 bits/letter \\ \hline
0–1 & 0.402 & 0.656 bits/letter \\ \hline
1–2 & 0.328 & 0.147 bits/letter \\ \hline
2–3 & 0.294 & 0.0795 bits/letter \\ \hline
3–4 & 0.2465 & 0.125 bits/letter \\ \hline
4–5 & 0.239 & 0.0234 bits/letter \\ \hline
5–6 & 0.2117 & 0.0869 bits/letter \\ \hline
6–7 & 0.212 & 0.0 bits/letter \\ \hline
7–8 & 0.125 & 0.381 bits/letter \\ \hline
8–9 & 0.1216 & 0.0206 bits/letter \\ \hline
9–10 & 0.1216 & 0.0 bits/letter \\ \hline
0–10 Total & 0.1216 & 1.52 bits/letter \\ \hline \hline \end{tabular}
\end{table}
Table 1: Change in mutual information from Equation (17) as the predictor order is increased: Frame \(3237\), \(SPER=9.15\) dB; MMSPE, minimum mean squared prediction error.  
characterization in terms of mutual information is new and allows new analyses of the performance of CELP codecs in terms of bits/sample reduction in rate that can be associated with the short-term predictor. We provide this analysis in Section 9.