# Voice Transformation Using Two-Level Dynamic Warping and Neural Networks
## 4 Spectral Warping Using NN  
The proof-of-concept experiments above demonstrated that spectral mapping followed by phase reconstruction produces effective VT. For a complete, viable VT system, it is necessary to establish conversion rules for transforming speech when the target speaker saying the desired phrase is not available. These conversion rules are represented using a NN which is trained to accept spectral inputs from the source speech and to produce the spectral warping function paths, \(\mathbf{a}_{F}\) and \(\mathbf{b}_{F}\). The process is outlined in Figure 8. The input data for training the NN is the magnitude of the spectrum of a segment of speech that has been time-aligned with the target speech. The spectral information for segments of speech from source and target are used in DW to find the \((\mathbf{a}_{F},\mathbf{b}_{F})\) spectral warping information, as suggested by Figure 8a. The NN is trained to produce this \((\mathbf{a}_{F}(i),\mathbf{b}_{F}(i))\) information when the magnitude spectrum for the segment \(\mathbf{s}_{1_{TA}}\) is the input. This is suggested by the NN diagram in Figure 8b.  
It was found that better VT was produced when the NN uses the spectral magnitude from the preceding segment \(\mathbf{s}_{1_{TA}}(i)\) and the following segment \(\mathbf{s}_{1_{TA}}(i+1)\) as inputs,  
\begin{table}
\begin{tabular}{c c} \hline \hline
**No.** & **Two-Level DW Method** & **MCD [dB]** \\ \hline
1 & DW, with phase reconstruction & 2.4 \\
2 & DW, without phase reconstruction & 6.3 \\ \hline \hline \end{tabular}
\end{table}
Table 1: MCDs obtained for 2-Level DW with and without GLA.  
Figure 8: Spectrogram for male, female and warped male to female. (**a**) Making DFW training data. (**b**) Training a neural network—one input spectral vector. (**c**) Training a neural network—three input spectral vectors.  
to provide context for the current segment \(\mathbf{s}_{1_{TA}}(i+2)\). This configuration is suggested in Figure 8c.  
_Interpolating the Data to Achieve Constant Length_  
As noted above, the length of the warping path \((\mathbf{a}_{F},\mathbf{b}_{F})\) may vary among the different segments of speech. This varying length poses a problem for a NN which has a fixed number of outputs. Rather than zero-pad the outputs of the NN to produce a desired length (which would create artifacts) the \(\mathbf{a}_{F}\) and \(\mathbf{b}_{F}\) are interpolated to produce a modified \(\mathbf{a}_{F}\) and \(\mathbf{b}_{F}\) which is the same (maximum) length for all segments of the speech vector. In our experiments the maximum length of the warping paths was 482 (as determined from training data). The interpolation is described using Matlab-like notation and functions. Let \(\max_{\text{index}}\) denote the maximum length of the interpolated vectors. The interpolation procedes in two steps. First, interpolated indices are computed according to  
\[\tilde{n}=[1:\max_{\text{index}}]\times\frac{\text{length}(\mathbf{a})}{\max_ {\text{index}}};\]  
Then interpolation among the \(\mathbf{a}\) values using these interpolated indices is computed by  
\[\tilde{\mathbf{a}}=\text{interpolate}(\mathbf{a},[1:\text{length}(\mathbf{a}) ],\tilde{n})\]  
The sequence \(\mathbf{b}\) is similarly interpolated. Figure 9 illustrates un-interpolated function paths for \(\mathbf{a}\) and \(\mathbf{b}\) with interpolated paths (\(\tilde{\mathbf{a}}\) and \(\tilde{\mathbf{b}}\) respectively). The interpolated \(\tilde{\mathbf{a}}\) and \(\tilde{\mathbf{b}}\) have the same general shape as the original \(\mathbf{a}\) and \(\mathbf{b}\), but have length 482, as desired. The NN is trained to produce the interpolated warping sequences at its output.