# Voice Transformation Using Two-Level Dynamic Warping and Neural Networks
## 3 Voice Transformation Process
### Experiment Description and a Proof of Concept Result  
Experiments were carried out on CMU ARCTIC database [42]. The database consisted of seven primary sets of recordings, recorded by two US males, one Canadian male with English accent, one Scottish male with English accent, one Indian male, and two US females. Each speaker recorded a set of 1132 English utterances, most being between one and four seconds long and between 5 to 15 words long. The prompt list was designed to be phonetically balanced American English and have diphone coverage representative of the source material. The sound files were recorded in a soundproof booth at 16,000 samples/sec with simultaneous EGG measurements (the speech needed to be clean, or else background disturbances affected the transformed speech). The lexical and phonetic descriptions derived from the US English front-end module. In experiments presented here, one US male and one US female were chosen to do VT from male to female. While this was a limited experiment, since the transformation process was general (being based on DW of spectral features), this was considered a valid proof of concept for this initial experiment. After identifying the starting point and ending point of the first 720 sentences of CMU ARCTIC database, each sentence was temporally segmented into 32-ms segments, using a Hamming window with 16-ms overlapping, zero-padded, then transformed using a 512-point FFT then normalized as described above. DW was applied on these features and the features of the first 600 sentences were used to train the NN and a separate set of 120 sentences were used for testing the trained NN. Figure 3(a) shows a typical spectrogram for a male speaker, using the phrase \"Author of the danger trail, [PERSON], etc.\" from the CMU ARCTIC database. Figure 3(b) shows the spectrogram for a female speaker for the same phrase. Figure 3(c) shows the VT from male speaker to female speaker. After transformation, the overall energy distribution looked similar to the spectrogram of the female speaker. The fine-scale variations (the ripples in frequency) evident in Figure 3(a,b) are due to pitch. These lines are also in Figure 3(c), but the pitch details are fuzzier. The resulting speech had very severe processing artifacts. (It did not sound very good.) Since the transformation only involved shifting magnitude spectral information, we were led to suspect that the problem might result from a mismatch between the magnitude and phase information of the warped signal. Figure 4(a) shows the spectral feature information for one segment of speech for the female speaker. Figure 4(b) shows the spectral feature information for one time-aligned segment of speech for the male speaker. By applying the DFW, the peaks and valleys of the male spectrum were generally aligned to the peaks and valleys of the female spectrum, as portrayed in Figure 4(c). In warping the spectral information, artifacts were introduced into the signal which were not typical of speech. For example, Figure 4(c) shows that the warping smeared some spectral values across several frequency bins (indicated with green circles) in the plot, resulting in regions of the spectrum that had constant spectral magnitude. We found that the quality of the speech was significantly improved by using a phase reconstruction algorithm on the warped speech. In this work, the Griffin-Lim algorithm [43] was applied to do phase reconstruction after the spectral transformation has been accomplished [44]. Figure 6 shows the spectrogram for the warped speech after phase reconstruction. The pitch lines were stronger (less fuzzy) after the GLA phase reconstruction. Figure 7 shows the spectral feature information for one segment of speech for the female speaker with the time aligned spectral warped segment for male from the phrase mentioned above. Figure 4: Spectrogram for male, female and warped male to female.