## 2 Baseline UDS-Based SSI
### Performance of Pitch Estimation and V/Uv Decisions  
In most of the current pitch estimation methods, the pitch period is estimated from the audible speech that is not available in the SSI systems [32; 41]. The pitch estimation methods using non-audible signals were investigated in the previous SSI studies in which EMG was used [22; 27; 38]. V/UV decisions using EMG were also proposed in at least one other study [38]. In these studies, the quality of the reproduced speech signals using impulse trains with the estimated pitch periods was investigated. However, neither the quantitative analysis of the pitch estimation accuracy nor the V/UV decision errors were sufficiently discussed in the previous studies. Figure 2: Examples of original (bold line) and the predicted (dotted line) speech spectra. In the present study, experiments were performed to verify the possibility of pitch estimation and V/UV decisions using lip images, EMG, and ultrasonic signal. A pitch estimation method using non-acoustic signals was proposed in our pervious study [18], where a DNN was trained using a set of the features derived from the non-acoustic signals and the ground truth pitch periods that are detected by audible speech. This method was similar to the previous DNN-based pitch estimation method [41] in which a DNN provided the likelihood of each of the candidate pitch periods, and a sequence of the optimal pitch periods was obtained via a Viterbi-trellis search. In the original work [41], spectral feature vectors derived from noisy speech were inputted to a DNN. In the present study, however, feature vectors derived from the non-acoustic signals were inputted to a DNN. All necessary parameters for pitch estimation such as the number of candidate pitch periods, the weights for the posterior probability, and the transition probability were independently determined for each modality so that the overall accuracy was maximized. A DNN was also employed to implement V/UV decisions given by two output nodes, and each output node corresponded to either the voiced frame or the unvoiced frame. The performance of pitch estimation and V/UV decision-making was evaluated for each modality by using a validation dataset that was constructed from the three subjects (two males and one female). Each subject pronounced 60 isolated Korean words 10 times. This resulted in a total of 25,232 stimuli that split into 17,904 stimuli for training and 7328 stimuli for the actual test. Although the samples for each modality were not recorded simultaneously (because of changes in the shapes of the mouth region by attaching the EMG electrodes), the differences in speech signals among the modalities were minimized by using the common utterance set and asking the subjects to pronounce each word in a consistent manner. The performance of pitch estimation was evaluated using the two standard metrics: the gross pitch error (GPE) rate and the fine pitch error (FPE) [42]. The GPE frames are defined as voiced frames where the error between the estimated pitch period and the ground truth is greater than 0.625 ms. The GPE rate is then given by  
\[\text{GPE}_{\text{rate}}=\frac{N_{\text{GPE}}}{N_{\text{p}}} \tag{5}\]  
where \(N_{\text{GPE}}\) and \(N_{\text{p}}\) denote the number of GPE frames and voiced frames, respectively. The FPE is represented by using the mean (\(\mu_{\text{FPE}}\)) and the standard deviation (\(\sigma_{\text{FPE}}\)) that reflect the bias in the \(f_{0}\) estimation and the accuracy, respectively. \[\mu_{\text{FPE}} = \frac{1}{N_{\text{p}}}\sum_{i=1}^{N_{\text{p}}}\epsilon_{i} \tag{6}\] \[\sigma_{\text{FPE}} = \sqrt{\frac{1}{N_{\text{p}}}\sum_{i=1}^{N_{\text{p}}}(\epsilon_{ i}-\mu_{\text{FPE}})^{2}}\] (7) \[\epsilon_{i} = \Big{|}\hat{f}_{0}^{i}-f_{0}^{i}\Big{|}, \tag{8}\]  
where \(\hat{f}_{0}^{i}\) and \(f_{0}^{i}\) denote the estimate and the ground truth of \(f_{0}\), respectively, at the \(i\)-th frame in the voiced frames.