# A Reinforcement Learning Approach to Speech Coding
## 6 Error Shaping  
For speech coding, the time domain mean squared error is a poor indicator of how well the overall system is performing with or without lookahead; however, perceptually shaping the error in the frequency domain before the sum of the squared error calculation can yield good performance and is computationally tractable. The error shaping employed in this work has the form  
\[W(z)=\frac{1-\sum_{k=1}^{8}a_{k}k^{k}z^{-k}}{1-\sum_{k=1}^{8}a_{k}\gamma^{k}z^{-k}} \tag{20}\]  
where the \(a_{k},k=1,2,\ldots,8\) are the AR predictor coefficients calculated based upon a block of the input speech, that is, the observed state of the Environment, and \(\mu\) and \(\gamma\) are constants to be selected. Although various combinations of \(\mu\) and \(\gamma\) are possible, their values should fall in the interval between 0 and 1, and based on experiments, we set the parameters \(\mu=1\) and \(\gamma=0.8\). Other values of \(\mu\) and \(\gamma\) can be and are used in block-based CELP codecs, and the current codec is not particularly sensitive to variations in the values chosen here. More specifically, the desired spectral error shaping is not achieved by fixed values of \(\mu\) and \(\gamma\) across all speech segments, but adaptive methods for these parameters have not been determined in prior experiments for tree coders or for the standardized CELP based codecs. The calculation of the perceptual error filter coefficients in Equation (20) uses a block-based method on the input speech samples since error shaping is only performed at the Encoder, and there is no need to transmit the perceptual weighting filter coefficients to the Decoder. We calculate the model coefficients from the input speech frames by using nonoverlapping rectangular windows of the 200 future input samples with an update rate of every 200 samples. This corresponds to a latency of 25 ms at 8000 samples/s and constitutes the maximum encoding delay associated with the current codec. We have experimented with using 100 past samples and 100 future samples in this process and the results are very similar. To reduce latency, this parameter estimation process for error shaping deserves further attention. In the time domain, we denote the spectrally shaped weighted error sequence by \(e_{w}(k)=[s(k)-x(k)]_{w}\), where \(x(k)\) and \(s(k)\) are as in Figure 1, so that the cost-to-go function is \(\sum J_{k}=\sum e_{w}^{2}(k)=\sum J_{k}=\sum[s(k)-x(k)]_{w}^{2}\) where the sum is over \(L\) time instants starting with the current instant \(k\). The perceptual error shaping filter provides only an imperfect and approximate indicator of the actual quality and intelligibility of the reconstructed speech, but it is critically important to obtaining acceptable system performance. The goal of the perceptual weighting filter is to keep the reconstruction error spectral envelope below that of the input speech envelope across the frequency band of interest, since this allows the speech spectrum to mask the error spectrum. An example is given in Figure 3.  
Figure 3: Perceptual spectral error shaping example. In Figure 3 we see that the unweighted error spectral envelope falls above the input speech spectral envelope at some frequencies and the perceptual error weighting is able to push the error envelope down below that of the input speech; however, the error spectral envelope becomes very close to the input spectral envelope at 2000 Hz and just above 3000 Hz. The most desirable result would be for the error spectral envelope to be a constant amount below the input spectral envelope across the entire band. This is difficult to achieve, and it is well known that even with the error spectral shaping, the envelope of the error spectrum may not fall below that of the input spectrum. However, this is the best known approach available and the weighted squared error is important, even though imperfect, in reproducing a pleasing reconstruction at the Decoder.