###### Abstract  
The modeling of fundamental frequency (\(F_{0}\)) in speech synthesis is a critical factor affecting the intelligibility and naturalness of synthesized speech. In this paper, we focus on improving the modeling of \(F_{0}\) for Isarn speech synthesis. We propose the \(F_{0}\) model for this based on a recurrent neural network (RNN). Sampled values of \(F_{0}\) are used at the syllable level of continuous Isarn speech combined with their dynamic features to represent supra-segmental properties of the \(F_{0}\) contour. Different architectures of the deep RNNs and different combinations of linguistic features are analyzed to obtain conditions for the best performance. To assess the proposed method, we compared it with several RNN-based baselines. The results of objective and subjective tests indicate that the proposed model significantly outperformed the baseline RNN model that predicts values of \(F_{0}\) at the frame level, and the baseline RNN model that represents the \(F_{0}\) contours of syllables by using discrete cosine transform. t Date of print  
In the last few years, several types of neural network-based methods of speech synthesis have been proposed to overcome the limitations of HMM-based speech synthesis. Deep neural networks (DNNs) [10] have been used in place of decision tree-based clustering. However, DNN ignores the sequential nature of speech because it assumes that each frame is sampled independently. To capture long-term dependencies, the recurrent neural network (RNN) with long-short term memory (LSTM) has also been applied as an acoustic model [11; 12; 13]. End-to-end speech synthesis using advanced neural network architectures has also been proposed, such as in Tacotron [14] and Deep Voice [15]. These techniques directly convert raw text into speech waveforms and generate speech that sounds more natural than both parametric-based and concatenative approaches. However, they require a large set of speech and text pairs for training, which is time consuming and expensive to collect [16]. These models are thus challenging to apply to low-resource languages. Thus, this study focuses on parametric-based speech synthesis approach. As described in [17; 18], the shape of the \(F_{0}\) contour of a syllable can deviate due to several factors, such as tone co-articulation, stress, and declination effects. Thus, the contour should be modeled at the syllable level than the frame level. Several models are available for representing the \(F_{0}\) contour, such as the tilt model [19; 20], Fujizaki's model [21], pitch target approximation models [22], and the discrete cosine transform (DCT) [23; 24]. These models substantially improve performance in terms of the generation of \(F_{0}\). Nevertheless, some \(F_{0}\) representation models require manual annotation to correct (e.g., the Fujizaki and pitch target approximation models) [25; 26]. Although the RNN can capture suprasegmental characters of the \(F_{0}\) contour, its internal structure of connections considers \(F_{0}\) at the frame or segment level only. In particular for tonal languages, tone contour and its deviation are also characterized at higher levels (e.g., syllable and word) [27; 28; 29]. In this paper, we propose a tone modeling technique based on the RNN for speech synthesis in Isarn. We first propose a sampling-based method of representation of \(F_{0}\) that can capture rich information concerning the tone contours within a syllable for Isarn. Syllable-level features are then modeled by the RNN to learn the suprasegmental characters of \(F_{0}\) at higher prosodic levels. We also explore several architectures of RNN models and training strategies to generate tone contours. In terms of the linguistic level used for modeling and the representation of \(F_{0}\), we compare the proposed model with a frame-based model and other established \(F_{0}\) transforms, such as the DCT, using objective and perceptual evaluations.