###### Abstract  
Moderate performance in terms of intelligibility and naturalness can be obtained using previously established silent speech interface (SSI) methods. Nevertheless, a common problem associated with SSI has involved deficiencies in estimating the spectrum details, which results in synthesized speech signals that are rough, harsh, and unclear. In this study, harmonic enhancement (HE), was used during postprocessing to alleviate this problem by emphasizing the spectral fine structure of speech signals. To improve the subjective quality of synthesized speech, the difference between synthesized and actual speech was established by calculating the distance in the perceptual domains instead of using the conventional mean square error (MSE). Two deep neural networks (DNNs) were employed to separately estimate the speech spectra and the filter coefficients of HE, connected in a cascading manner. The DNNs were trained to incrementally and iteratively minimize both the MSE and the perceptual distance (PD). A feasibility test showed that the perceptual evaluation of speech quality (PESQ) and the short-time objective intelligibility measure (STOI) were improved by 17.8 and 2.9%, respectively, compared with previous methods. Subjective listening tests revealed that the proposed method yielded perceptually preferred results compared with that of the conventional MSE-based method. s 2  
highly correlated with the corresponding audio speech signals and not affected by high levels of ambient audio interference [3]. Various types of modalities satisfying these conditions have been adopted in previously developed SSI methods, such as the Doppler frequency shifts caused by GHz microwaves [4; 5; 6; 7; 8], the ultrasound images (UI) of a vocal tract [9; 10; 11], the visual shape of the mouth [12; 13; 14; 15; 16], acoustic Doppler sonic signals [17; 18; 19; 20; 21], and signals recorded by a non-audible microphone (NAM) [22; 23], or an electromyogram (EMG) [24; 25; 26; 27; 28; 29; 30; 31]. Among them, the Doppler-based methods have several advantages over other methods. The problems associated with contact sensing in the EMG- and NAM-based methods can be avoided by non-contact sensing. The Doppler-based techniques are free from image processing-related problems such as inaccuracy in the tracking of regions of the mouth. The underlying assumption of the Doppler-based SSI methods is that the primary sources of Doppler frequency shifts are vocal vibrations of the body surfaces that cause sounds. A sinusoidal signal with a fixed frequency is incident to a speaker's mouth and neck regions, and the Doppler shifts of the returned signals are used to synthesize the voice. Our previous study [19] showed that the Doppler-based SSI method using an ultrasonic wave (40 kHz) could be implemented using relatively small and simple hardware by comparison with the use of GHz microwaves and the overall quality of the synthesized speech was superior to other SSI methods and modalities. The main objective of the present study was to improve the quality of the speech synthesized using the ultrasonic Doppler signal (UDS)-based SSI method. In particular, the perceptual aspects of the reproduced speech were the major concerns of designing the speech estimation rules. In most previous SSI studies [13; 21; 22; 24; 25; 27; 30], the speech signals were represented as Mel-cepstral coefficients (MCCs), which approximated the speech spectrum in the perceptual domain [32]. The objective functions to be minimized in the training stage of those studies was given by the Mel-cepstral distance (MCD). Since the MCC is computed based on human auditory systems, perceptual aspects were considered to some extent in the SSI techniques that were designed to minimize the MCD. In most SSI schemes, however, the differences perceived by the human ear were not directly taken into consideration in construction of the speech estimation rules. Instead, the speech estimation rules were constructed by minimizing the mean squared errors (MSE) between the original and estimated MCCs.