###### Abstract  
In voice conversion (VC), it is highly desirable to obtain transformed speech signals that are perceptually close to a target speaker's voice. To this end, a perceptually meaningful criterion where the human auditory system was taken into consideration in measuring the distances between the converted and the target voices was adopted in the proposed VC scheme. The conversion rules for the features associated with the spectral envelope and the pitch modification factor were jointly constructed so that perceptual distance measurement was minimized. This minimization problem was solved using a deep neural network (DNN) framework where input features and target features were derived from source speech signals and time-aligned version of target speech signals, respectively. The validation tests were carried out for the CMU ARCTIC database to evaluate the effectiveness of the proposed method, especially in terms of perceptual quality. The experimental results showed that the proposed method yielded perceptually preferred results compared with independent conversion using conventional mean-square error (MSE) criterion. The maximum improvement in perceptual evaluation of speech quality (PESQ) was 0.312, compared with the conventional VC method. voice conversion; joint conversion; perceptual distance measure +
Footnote †: journal: Medical  
minimize the mean squared errors between the converted and target MFCCs [20; 21] or they minimize the squared errors between the warped version of the reference spectra and the target spectra [22; 23]. The statistical properties (e.g., the mean and standard deviation) of the converted pitch periods are the major concerns in prosody conversion [2; 24]. This indicates that the perceptual aspects have been given limited consideration in current VC techniques. Since converted speech will be listened to by a human, it is highly desirable to adopt human auditory-based distance as an objective function to be minimized. Such a distance measure has already been used for various forms of speech processing procedures, such as speech enhancement [25], speech recognition [26], speech coding [27], speech synthesis [28; 29] and speech quality evaluation [30]. This distance measure has not yet been adapted as an objective metric in VC. In the proposed VC method, the conversion rules for both the spectral envelope and the pitch were designed so that the perceptual differences between the converted spectra and that of the target speech is minimized. The perceptual evaluation of speech quality (PESQ) [30], which has been widely used in speech quality evaluation was adopted to compute the perceptual differences. The PESQ is one of the best known objective metrics for speech coding [30] and speech enhancement [25]. We extended the utility of the PESQ to construct VC mapping rules and evaluate the quality of the converted speech. The two conversion functions, one for the spectral envelope and the other for the spectral fine structure, were cascade connected and incrementally trained to reduce the unique objective function. This differs from the conventional VC approaches wherein the conversion rules for each feature parameter are independently constructed by minimizing the separate distance measurements. One of the advantages gained by the construction of cascaded conversion rules is that one step can compensate for the conversion errors in another, thereby further reducing the overall conversion error. The conversion function was implemented by using the deep neural networks (DNN) [31], which were widely employed in the VC methods [32; 33; 34; 35; 36; 37]. The objective function of the DNN is different from the previous DNN-based methods where the log-likelihood [32] or the joint probability function [33] was adopted. The WaveNet vocoder [38], which was originally developed for text-to-speech (TTS) was fine-tuned to improve the quality of the converted speech signals [34; 36]. The perceptual aspects, however, were not addressed in construction of the conversion rules.