## 2 Background and Related Studies
### Past Work on Isarn Speech Synthesis  
HMM-based speech synthesis for Isarn was developed in [5]. The waveform of the speech units was not used directly, instead linguistic and acoustic features were extracted from a speech corpus. In the training stage, the waveform of speech is converted into acoustic features, including \(F_{0}\) and spectral features. The components of the acoustic parameters depend on the speech vocoder. Transcription is converted into linguistic features used as input by a text analysis module. The acoustic models are then trained using the extracted acoustic features and linguistic features. A duration model is also trained to determine phone duration. In the synthesis stage, linguistic contextual features are extracted from the input text and fed to the duration model. A sequence of speech parameters is then generated using the trained acoustic model and the information obtained concerning the phone duration. Finally, the waveform of speech is synthesized through the speech vocoder.  
In HMM-based speech synthesis for Isarn, the tonal syllable is modeled by using two or three contextual phone models, including the initial phone model, vowel phone model, and final phone model (optional). The six tones in Isarn are represented in terms of tone-context features.  
However, the HMM-based speech synthesis for Isarn still generates unnatural speech related to the generation of \(F_{0}\). Therefore, we examined past studies to identify a method to improve the performance of Isarn speech synthesis. We found that RNN-based speech synthesis had achieved the best performance for other languages [11; 12; 37; 38]. We also preliminarily implemented RNN-based speech synthesis for Isarn, and observed the synthetic speech. RNN-based speech synthesis often generated unnatural speech due to the generation of inappropriate \(F_{0}\) contours. Therefore, we attempted to improve the \(F_{0}\) model for Isarn speech synthesis.  
At a linguistic level, the proposed model does not consider temporal dependencies across frames, but across syllables. In tonal languages, the tone is indicated by the \(F_{0}\) contour at the syllable level [1]. The deviation in the \(F_{0}\) contour also occurs across syllables. From the perspective of  
Figure 2: Comparison of \(F_{0}\) contours of the Isarn utterance (“Please call him to meet the teacher.” in English translation) pronounced by the same speaker in (**a**) isolation and (**b**) continuous speech.  
feature representation, we propose a sampling-based approach that represents the \(F_{0}\) contour of the syllable by using sampled \(F_{0}\) values and their dynamic features. We expect that the sampling-based method can provide rich information for modeling suprasegmental features of the \(F_{0}\) contour.  
This model consists of two steps. First, the \(F_{0}\) contour of the syllables is represented by the sampling-based method. Second, linguistic features are mapped into the extracted parameters by using the RNN. The development of the proposed model can be divided into two parts: \(F_{0}\) contour modeling and synthesis, as shown in Figure 3. Details of each part are described below.