# A Reinforcement Learning Approach to Speech Coding
## 4 Speech Coding as Reinforcement Learning
### The System Model  
From decades of accumulated domain knowledge of the speech modeling process, we know that a reliable model for speech sequences is a discrete autoregressive (AR) process  
\[s(k)=\sum_{i=1}^{M}a_{i}s(k-i)+w(k) \tag{1}\]  
where the autoregressive parameters \(a_{i},i=1,2,...M\), are called the linear prediction coefficients in speech processing applications, and \(w(k)\) is the excitation sequence [18]. The excitation can be a periodic sequence or a noise sequence or a mix of the two. The excitation sequence can thus be modeled as  
\[w(k)=\sum_{i=-1}^{1}p_{i}w(k-d+i)+\epsilon(k) \tag{2}\]  
where \(d\) is the long term pitch redundancy and \(\epsilon(k)\) is some random driving sequence. The AR coefficients \(a_{i},i=1,2,\ldots M\) are time-varying as are the excitation signal characteristics  
Figure 1: Reinforcement learning for speech coding. \(d\), \(p_{i}\), \(i=-1,0,1\), and \(\epsilon(k)\). All of this knowledge of the Environment can be used to construct the System Model to be controlled and the various adaptive identification algorithms. The System Model to be controlled cannot include actual speech samples since they are not available at the remote Decoder, and are, in fact, what we are trying to reproduce using our control sequence. Furthermore, even if the System Model were exact, we do not know the various parameters, which must be adaptively identified, but based on Equations (1) and (2), we can write that the System Model at least should include the structure  
\[x(k)=\sum_{i=1}^{M}\widehat{a}_{i}x(k-i)+\sum_{i=-1}^{1}\widehat{\beta}_{i}u( d+i)+u(k) \tag{3}\]  
where the hats \(\widehat{a}_{i}\) and \(\widehat{\beta}_{i}\) indicate estimated parameters and \(u(k)\) is the control sequence. This model needs to be modified further in order to capture the fact that the AR and long term redundancy parameters are being adaptively estimated based on noisy measurements, that is, not the input speech itself, since the input speech is what we are attempting to reproduce at the Decoder. Research has shown that when estimating AR coefficients based on noisy measurements, the recursion should include a moving average (MA) component in addition to the AR component [20]. In addition, it is a less well-known result that if the AR coefficients are being estimated within the prediction recursion loop, the coefficient adaptation can start to track itself and not the observed input speech, but the inclusion of a MA component can help prevent this from happening [21]. Incorporating the MA part, Equation (3) becomes  
\[x(k)=\sum_{i=1}^{N_{k}}\widehat{a}_{i}x(k-i)+\sum_{i=-1}^{1}\widehat{\beta}_{ i}u(d+i)+\sum_{i=1}^{N_{k}}\widehat{b}_{i}u(k-i)+u(k) \tag{4}\]  
Equation (4) constitutes the System Model to be controlled. This System Model also requires the estimation/identification of the \(\widehat{a}_{i}\), the \(\widehat{\beta}_{i}\), the pitch lag \(d\), and the \(\widehat{b}_{i}\). Since the adaptive algorithms to identify these parameters are also driven by the control sequence \(u(k)\), we have a dual control or reinforcement learning problem. The adaptive identification algorithms for estimating these parameters are described in Section 5. The critical importance of Equation (4) and the recognition that the control \(u(k)\) not only drives the reconstruction but also simultaneously drives the parameter estimation/identification algorithms is the essence of this contribution. Both adaptive DPCM (ADPCM) and tree coders can have adaptive parameter estimation algorithms but the excitation in these systems is wholly chosen to achieve good reconstruction. More specifically, in ADPCM the prediction error is simply quantized to yield a quantized prediction error that most closely approximates the quantizer input. The ADPCM system does not choose the quantized value with the knowledge that this value will be driving the several parameter estimation/system identification algorithms.