# A Reinforcement Learning Approach to Speech Coding
## 10 Exploitation and Exploration  
Using the control sequences defined in Section 8 and their corresponding control gain policies, we can study the effects of exploitation and exploration in the choice of the control policy. Before we focus on the control sequences, however, we need to specify all of the other system details. The System Model to be driven by the dual control sequences is given by Equation (4). The AR coefficient adaptation is accomplished by the least squares lattice algorithm shown in Figure 2 and delineated in Section 5.1, wherein the MA coefficient adaptation is specified by Equation (16), and the quasi-periodic excitation is adapted as in Section 5.2. The perceptual weighting is performed as described in Section 6 and the approximate Cost-to-Go, \(\sum\
olimits_{k}=\sum\
olimits[s(k)-x(k)]_{\omega}^{2}\), is calculated as in the section on the \(L\) Step lookahead, Section 9, using the \(M\) control tree search combined with the single control symbol release rule. All of the speech samples used in this paper are taken from the ITU-T P.501 standard [31]. We see from Figure 1 and the details of the adaptation algorithms that the control sequence performs the dual roles of exploration, driving the adaptation of the system parameters, and exploitation, cautiously driving the System Model to track the input speech measured from the Environment. With the System Model and adaptation algorithms specified, we now consider the performance of the reinforcement learning method using the excitation of the System Model consisting of the 40 sample randomly interleaved four-level and two-level multitree segments as shown in Figure 5 combined with the expansion of the scaling by 1.6 for outer levels and contraction by 0.8 for inner levels for the four-level tree and no scale adjustment for the two-level tree. The performance is measured by the Value Function described in Section 7. The results are shown in the first row of Table 1 for the speech utterances Female 1 (F1), Female 2 (F2), Male 1 (M1), and Male 2 (M2). In this row, the average PESQ-MOS is 3.438, which although it is less than the desired target of 3.5, is the best performance ever achieved by a tree structured speech codec at the low average bit rates shown. To further explore the reinforcement learning approach in Figure 1, we compare this result to inserting a 140 sample five-level control sequence at the beginning of each Voiced speech segment before employing the 40 sample random sequence with the previous adaptation for the four level and no adaptation for the two-level tree. The reason for inserting this more impulse-like excitation at the beginning of each Voiced segment is \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Control** & **Value Function** & **F1** & **F2** & **M1** & **M2** & **Avg** & **Std Dev** \\ \hline \multirow{2}{*}{Random 4-2} & PESQ-MOS & 3.377 & 3.376 & 3.443 & 3.556 & 3.438 & \multirow{2}{*}{0.073} \\  & Rate (kbits/s) & 5.92 & 5.57 & 4.09 & 5.41 & 5.25 & \\ \hline Random 4-2 & PESQ-MOS & 3.425 & 3.515 & 3.544 & 3.569 & 3.513 & \multirow{2}{*}{0.054} \\ plus 5-level & Rate (kbits/s) & 6.03 & 5.66 & 4.15 & 5.49 & 5.33 & \\ \hline Random 4-2, & PESQ-MOS & 3.471 & 3.562 & 3.584 & 3.599 & 3.554 & \\
5-level, 5 pol & Rate (kbits/s) & 6.03 & 5.66 & 4.15 & 5.49 & 5.33 & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: PESQ-MOS value function results for three control policies/full utterances. to force the control to explore other sequences and larger excitations that will force the parameter adaptation algorithms out of any conservative behaviors.