# Voice Transformation Using Two-Level Dynamic Warping and Neural Networks
## 3 Voice Transformation Process  
### Two-Level Dynamic Warping  
The transformation in our VT is accomplished using a two-level DW transformation which aligns signals both temporally (DTW) and spectrally (DFW). Let \(i=1,2\) denote the index of a speaker, with \(i=1\) representing the source speaker and \(i=2\) representing the target speaker. Let \(\ell\) be the index of a segment of speech, where successive segments may be overlapping. Let \(\mathbf{r}_{i}(\ell,:)\) denote the time-domain signal associated with the \(\ell\)th segment of speech. This is a vector of length \(2K=512\). This was transformed using a 512-point FFT. This was normalized as follows. Let \(D_{i}(\ell,j)\), \(i=1,2\), \(\ell=1,2,\ldots,L_{i}\), \(j=1,2,\ldots,K\) be the magnitude of the positive frequency coefficients in the DFT of \(\mathbf{r}_{i}(\ell,:)\). The feature vector is produced by \[\mathbf{s}_{i}(\ell,j)=\frac{D_{i}(\ell,j)}{\frac{1}{KL_{i}}\sqrt{\sum_{\ell=1}^{L_ {i}}\sum_{j=1}^{K}D_{i}(\ell,j)^{2}}},\qquad\ell=1,2,\ldots,L_{i},\quad j=1,2, \ldots,K.\]  
A sequence of these feature vectors is denoted as  
\[\mathcal{S}_{i}=\{\mathbf{s}_{i}(1,\cdot),\mathbf{s}_{i}(2,\cdot),\ldots, \mathbf{s}_{i}(L_{i},\cdot)\},\qquad i=1,2. \tag{1}\]  
Here, \(L_{1}\) and \(L_{2}\) denote the number of source and target feature vectors. Let \(d_{T}(\ell_{1},\ell_{2})\) denote the minimum cost between the sequences \(\mathcal{S}_{1}\) (source) and \(\mathcal{S}_{2}\) (target), up to segment indices \(\ell_{1}\) and \(\ell_{2}\), computed using DTW to align the segments in time. (The subscript \(T\) refers to time warping.) As usual for DTW [41], \(d_{T}(\ell_{1},\ell_{2})\) is recursively computed as  
\[d_{T}(\ell_{1},\ell_{2})=d(\mathbf{s}_{1}(\ell_{1},\cdot),\mathbf{s}_{2}(\ell _{2},\cdot))+\min\{d_{T}(\ell_{1}-1,\ell_{2}),d_{T}(\ell_{1},\ell_{2}-1),d_{T} (\ell_{1}-1,\ell_{2}-1)\}. \tag{2}\]  
Here, \(d(\mathbf{s}_{1}(\ell_{1},\cdot),\mathbf{s}_{2}(\ell_{2},\cdot))\) is the distance between individual feature vectors at segments \(\ell_{1}\) and \(\ell_{2}\). This is computed using DFW, as described below. Conceptually, the dynamic warping aligns the peaks and valleys between the source and target data, as suggested by Figure 2. When \(\ell_{1}=L_{1}\) and \(\ell_{2}=L_{2}\), the temporal warping process determines an overall warped metric distance \(d_{T}(L_{1},L_{2})\) between the sequences \(\mathcal{S}_{1}\) and \(\mathcal{S}_{2}\). DTW computes a sequence of indices \(\mathbf{a}_{T}=(a_{T}(1),a_{T}(2),\ldots,a_{T}(N_{T}))\) and \(\mathbf{b}_{T}=(b_{T}(1),b_{T}(2),\ldots,b_{T}(N_{T}))\), which are called the temporal warping function paths. The temporal warping function paths define a time-aligned (TA) function \(\mathbf{s}_{1,TA}\) such that the sequence  
\[\mathbf{s}_{1,TA}(a_{T}(j),\cdot)=\mathbf{s}_{1}(b_{T}(j),\cdot),\quad j=1,2, \ldots,N_{T}\]  
is as similar to \(\mathcal{S}_{2}\) as possible. The distance between two feature vectors in (2), \(d(\mathbf{s}_{1}(\ell_{1},\cdot),\mathbf{s}_{2}(\ell_{2},\cdot))\), can itself be computed using warping between the components of these individual vectors. Because these vectors represent frequency information, this is DFW. DFW is computed in a way similar to the DTW. Let \(\mathbf{s}_{i}=\mathbf{s}_{i}(\ell_{i},\cdot)\), \(i=1,2\) denote the spectral information vectors at segment \(\ell_{i}\) that is passed to the DFW function from the (outer) DTW function. The elements of \(\mathbf{s}_{i}\) are denoted by \(s_{i}(k)\). DFW is applied to calculate the distance between \(\mathbf{s}_{1}\) and \(\mathbf{s}_{2}\) as  
\[d_{F}(k_{1},k_{2})=\text{dist}(s_{1}(k_{1}),s_{2}(k_{2}))+\min\{d_{F}(k_{1}-1, k_{2}),d_{F}(k_{1},k_{2}-1),d_{F}(k_{1}-1,k_{2}-1)\}, \tag{3}\]  
where the subscript \({}_{F}\) refers to a frequency (or spectral) warping process, and \(\text{dist}(s_{1}(k_{1}),\)\(s_{2}(k_{2}))\) represents the metric distance between elements of the feature spectral vectors. Figure 2: Distance measure for outer dynamic warping for VT.