## 2 The Structure of the Proposed VC Method  
The overall procedure proposed for the VC method appears in Figure 1 wherein a typical conventional VC scheme is also presented for comparison. The first step of VC is analysis that extracts a set of speech feature parameters of both the reference and target speakers. The spectral envelop parameter and spectral fine structure were used as feature parameters, which were associated with the vocal tract transfer function and prosody information, respectively. The linear prediction coefficient cepstrum (LPCC) was chosen to represent the spectral envelope and the spectral fine structure was represented with the pitch period. In practice, even if the reference/target speakers utter the same words, it is unlikely that a synchronized set of feature sequences would be produced. Dynamic Time Warping (DTW) [39] was first applied in a preprocessing step in order to time-align these sequences. Time-alignment using DTW produced frame-level synchronized sequences, but waveform-level synchronization between the neighboring frames, is not guaranteed. This could potentially result in occurrence of undesired pitch-pulse misalignment. To cope with this problem, a synchronized overlap and add (SOLA) method [40] was applied to the frame-level time-alignedtarget speech. A pairing of reference speech and time-aligned (both in frame-level and waveform-level) target speech was used to construct the conversion rules and for evaluation. An example of the final time-aligned target speech that is subsequently used for construction of the conversion rules and evaluation is shown in Figure 2. This shows that the onset/termination times of the time-aligned target speech are relatively consistent with those of reference speech. When \(\mathcal{F}_{H}\) and \(\mathcal{F}_{E}\) denote the conversion functions for the LPCC and the spectral fine structure, respectively, then the optimal conversion rules for conventional VC methods, \(\mathcal{F}_{H}^{*}\), \(\mathcal{F}_{E}^{*}\) are given by  
\[\mathcal{F}_{H}^{*} = \arg\min_{\mathcal{F}_{H}}D_{H}(\mathbf{H}_{t},\mathbf{\hat{H}}_ {t}) \tag{1}\] \[\mathcal{F}_{E}^{*} = \arg\min_{\mathcal{F}_{E}}D_{E}(\mathbf{E}_{t},\mathbf{\hat{E}}_ {t}) \tag{2}\]  
where \(\mathbf{\hat{H}}_{t}=\mathcal{F}_{H}(\mathbf{H}_{r})\), \(\mathbf{\hat{E}}_{t}=\mathcal{F}_{E}(\mathbf{E}_{r})\). \(\mathbf{H}_{r}=\{\mathbf{h}_{r,n}\}_{n=1}^{N}\) and \(\mathbf{H}_{t}=\{\mathbf{h}_{t,n}\}_{n=1}^{N}\) are the sets of the reference and time-aligned target LPCCs, respectively, and \(N\) is the total number of the parameters for constructing the conversion rules. In a similar manner, \(\mathbf{E}_{r}=\{\mathbf{e}_{r,n}\}_{n=1}^{N}\) and \(\mathbf{E}_{t}=\{\mathbf{e}_{t,n}\}_{n=1}^{N}\) are the sets of the spectral fine structures for reference and target speakers, respectively. \(D_{H}\) and \(D_{E}\) are the objective functions for the LPCC and the spectral fine structures, respectively. The mean squared error (MSE) was mostly adopted as the objective function in previous VC methods. Equation (1) indicates that the conversion rules for two parameters are independently obtained by minimizing each objective function, as shown in the top of Figure 1.  
In the proposed method, construction of the two conversion rules was achieved by minimizing the unique distance measurement,  
\[\mathcal{F}_{H}^{*},\ \mathcal{F}_{E}^{*}=\arg\min_{\mathcal{F}_{H},\mathcal{F}_{E} }D_{\text{PD}}(\mathbf{H}_{t},\mathbf{\hat{H}}_{t},\mathbf{E}_{t},\mathbf{ \hat{E}}_{t}) \tag{3}\]  
where \(D_{\text{PD}}\) is the perceptual distance measure that is explained in the next section. As illustrated in Figure 1, the distance measurement is computed using the synthesized speech signals and the target speech signals. That is a major difference from conventional VC schemes, in which the distance measurements are independently computed using the corresponding feature parameters. Independent minimization of each feature parameter leads to producing the converted speech which is close to the target speech.