# A Reinforcement Learning Approach to Speech Coding
## 10 Exploitation and Exploration
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Control** & **Value Function** & **F1** & **F2** & **M1** & **M2** & **Avg** & **Std Dev** \\ \hline \multirow{2}{*}{Random 4-2} & PESQ-MOS & 3.646 & 3.581 & 3.784 & 3.699 & 3.678 & \multirow{2}{*}{0.074} \\  & Rate (kbits/s) & 12 & 12 & 12 & 12 & 12 & 12 \\ \hline \multirow{2}{*}{Random 4-2} & PESQ-MOS & 3.747 & 3.798 & 4.022 & 3.793 & 3.840 & \multirow{2}{*}{0.107} \\  & Rate (kbits/s) & 12.24 & 12.24 & 12.24 & 12.24 & 12.24 & 12.24 & \multirow{2}{*}{0.104} \\ \hline \multirow{2}{*}{Random 4-2, 5-level, 5 pol} & PESQ-MOS & 3.766 & 3.827 & 4.039 & 3.819 & 3.863 & \multirow{2}{*}{0.104} \\  & Rate (kbits/s) & 12.24 & 12.24 & 12.24 & 12.24 & 12.24 & 12.24 & \multirow{2}{*}{0.032} \\ \hline \multirow{2}{*}{
\begin{tabular}{c} AMR, \\ Narrowband \\ \end{tabular} } & PESQ-MOS & 4.04 & 4.001 & 4.089 & 4.063 & 4.048 & \multirow{2}{*}{0.032} \\  & Rate (kbits/s) & 12.2 & 12.2 & 12.2 & 12.2 & 12.2 & \multirow{2}{*}{0.032} \\ \hline \hline \end{tabular}
\end{table} 
Table 2: PESQ-MOS value function results for three control policies and AMR/voiced only. values in the 3.7 and above range implies near transparent voice quality and intelligibility with little distortion. We see from the last two rows of Table 2 that the AMR-NB 12.2 kbits/s coding mode consistently achieves a PESQ-MOS of 4.0, or Very Good, which is better than the performance of our reinforcement learning based approach on average. For Male 1, almost identical performance is achieved. Since we have used the two Female, F1 and F2, and two Male sentences, M1 and M2, in Tables 1 and 2 in our research optimizations, we now consider additional female and male sentences outside of the optimization set to evaluate the performance of reinforcement learning. To this end, Table 3 shows the performance of our three control policies and the comparison to AMR-NB for female speakers outside of the optimization set and voiced segments only, which are the regions where reinforcement learning is used. We see that there is a slight upward trend in the PESQ-MOS, but just less than one standard deviation gain, when we insert the 140 samples of the five-level sequence at the beginning of each voiced segment and then including the 5 polarity memory control gain; however, the additional increase in PESQ-MOS achieved by using the five-polarity memory for the control gain is certainly negligible in terms of audible perception. Further, the performance change when we insert the 140 samples of the five-level sequence at the beginning of each voiced segment is not uniform across all sentences. In comparison to AMR, AMR outperforms our reinforcement learning approach on average, but for F9 and F11 our codec performs better. is true for analysis-by-synthesis speech codecs that operate in the bit rate range of 8 to 16 kbit/s (roughly). This difference is usually attributed to the fact that female pitch periods are shorter and thus, for fixed-length-block-based coders such as AMR, there are more pitch periods in a frame that are averaged together. For our proposed codec structure, this long term redundancy due to pitch is difficult to track by our current backward adaptive algorithm, so additional pitch pulses in a shorter period of time create tracking difficulties. Note that a difference in performance between female and male speakers may not hold for all speech codec structures. For example, low bit rate codecs that operate at 4 kbits/s and below [34], have very different forms than the analysis-by-synthesis CELP codecs [35]. Further, newer neural-based codecs do not show differences between female and male speakers, but the structure of these speech codecs and their design procedures are quite different than CELP and other low bit rate codecs and their performance cannot be determined from PESQ-MOS [36]. An additional note is that the results for the sentences in Tables 3 and 4 are not as good as in Table 2, as should be expected since the four sentences in Table 2 were used in our codec optimizations.