###### Abstract
This resulted in reproduced speech signals that were less similar to original speech from a perceptual perspective. Since the reproduced speech from SSI is received by a human ear, it is highly desirable to adopt a human auditory-based distance measure as an objective function. The usefulness of the human auditory-based distance metric has already been verified in various speech processing fields that include speech coding [33], speech enhancement [34], speech recognition [35], speech synthesis [36], and speech quality evaluation [37]. In the proposed SSI method, the speech estimation rules were constructed by minimizing the perceptual differences between the estimated spectra and that of target speech. The perceptual distance was computed in a manner similar to the procedure used to obtain a perceptual evaluation of speech quality (PESQ) [37], which is widely used in speech quality evaluation. And hence, it can be expected that the perceptual qualities of the reproduced speech signals are improved by employing the perceptual distance. In our previous study [19], perceptual aspects were not considered in construction of the speech estimation rules, only the MSE criterion was employed. Another drawback of the estimation rules adopted in current SSI schemes was that the spectral fine structure was not well estimated, and only an approximated spectrum was obtained. Since the degree of voicing in the vowel- and voiced-consonant regions is closely related to the spectral fine structure, a deficiency in the harmonic spectral structure seriously degrades the intelligibility and naturalness of the reproduced speech, particularly in the voiced regions. To alleviate this problem, both the fundamental frequency (\(f_{0}\)) and the voicing state (voiced/unvoiced) have been estimated from a non-acoustic modality to produce a quasi-periodic source signal [22; 27; 38]. The quality of the voiced regions was partially improved by combining the estimated source signals with a spectral envelope represented by the MCC. However, clear improvements in reproduced speech have not been guaranteed [22; 27; 30; 38]. This was due to severe blurring artifacts of the estimated speech spectrum even though periodic sources were applied [27]. The inaccuracies of f0 estimation and of voiced/unvoiced (V/UV) decisions when using a non-acoustic modality has also been a major reasons for degrading the naturalness of synthesized speech. In the present study, harmonic enhancement (HE) techniques were adopted to improve the intelligibility and naturalness of synthesized speech. Our previous studies associated with the UDS-based speech enhancement scheme showed that a moderate degree of pitch estimation error was obtained when the pitch periods were predicted using an artificial neural network (ANN) with the features derived from UDS [18]. The perceptual quality of the reproduced speech was improved by HE when a moderate degree of pitch estimation error was maintained [18]. In the subsequent section, we quantitatively analyze the errors of pitch estimation and V/UV decisions for each modality in order to verify the usefulness of non-acoustic pitch estimation. A fixed enhancement factor was adopted in the previous study [18]. In the present study we found, however, that adjusting the enhancement of the harmonic components according to the voicing strength improves the HE. Therefore, the filter coefficients of the HE were determined by UDS in the proposed method. In the training step, two cascade-connected estimators, one for the speech spectrum and one for the HE filter coefficients, were alternatively established to reduce the unique objective function. Such a process can compensate for the estimation errors from one step to another, which thereby reduces the overall error. The remainder of the paper is organized as follows. First, Section 2 explains the procedure of the baseline SSI system and the limitations. Possible solutions for the problems are explained and the accuracies of the pitch estimation and V/UV decisions are also presented in Section 2.