## 3 Proposed Method
### \(F_{0}\) Contour Modeling
#### 3.1.3 RNN Training  
We use the RNN to map input features to output features. The RNN has been proposed to overcome the limitations of a feedforward neural network (FFNN) that ignores temporal information in sequential data. In the RNN, information from previous time steps is considered as input to the next time step. Given a sequence of input feature vectors, \([x_{1},...,x_{T}]\), and a sequence of output feature vectors, \([y_{1},...,y_{T}]\), the RNN computes the hidden state vectors and output vectors for a given input sequence as follows:  
\[h_{t}=\sigma_{h}(\mathbf{W}_{xh}x_{t}+\mathbf{W}_{hh}h_{t-1}+b_{h}) \tag{4}\]  
\[y_{t}=\sigma_{y}(\mathbf{W}_{hy}h_{t}+b_{y}) \tag{5}\]  
where, \(x_{t}\), \(y_{t}\), and \(h_{t}\) are the input vector, output vector, and hidden state vector, respectively, at time \(t\). \(\sigma_{h}\) is the activation function of the hidden layer, \(\sigma_{y}\) is the activation function of the output layer, \(W_{xh}\) denotes the weight matrix between the input and the output layers, \(W_{hh}\) is the weight matrix between consecutive hidden states, \(W_{hy}\) is the weight matrix between the hidden and the output layers, and \(b_{h}\) and \(b_{y}\) are the bias vectors of the hidden layer and the output layer, respectively. Typically, the conventional RNN can use only information from the past. The bi-directional RNN has been developed to use information from past and future inputs. It outperforms the unidirectional RNN in many tasks, such as the front-end of text-to-speech systems [43], speech recognition [44],speech synthesis [11], and machine translation [45, 46]. The bi-directional RNN processes the input sequence forward and backward to capture past and future information, respectively, in each layer. Then, the two hidden states are concatenated to produce the output. The iterative process is as follows:  
\[\overrightarrow{h}_{t}=\sigma_{h}(W_{x\overrightarrow{h}}x_{t}+W_{ \overrightarrow{h}\overrightarrow{h}}\overrightarrow{h}_{t-1}+b_{ \overrightarrow{h}}) \tag{6}\]  
\[\overleftarrow{h}_{t}=\sigma_{h}(W_{x\over h}\underbrace{\star}_{t}+W_{ \overleftarrow{h}\over h}\overleftarrow{h}_{t-1}+b_{\overleftarrow{h}}) \tag{7}\]  
\[y_{t}=\sigma_{y}(W_{\overrightarrow{h}y}\overrightarrow{h}_{t}+W_{ \overleftarrow{h}y}\overleftarrow{h}_{t}+b_{y}) \tag{8}\]  
where \(\overrightarrow{h}\) and \(\overleftarrow{h}\) denote the forward and backward hidden state vector sequences, respectively. In practice, the performance of the RNN is limited when modeling long-term dependencies in sequential features, called the vanishing gradient problem. We use the recently proposed recurrent units, such as the gated recurrent unit (GRU) [45] and recurrent long short-term memory (LSTM) unit [47], to solve this problem. Based on past work [48], we employed the recurrent bidirectional LSTM (BLSTM) unit. To train the model, the input features are a sequence of linguistic features. Each input feature vector contains 153-dimensional linguistic features extracting from linguistic features in Table 1. The output feature vector includes 3\(K\)-dimensional of sampled \(F_{0}\) values with dynamic features, where \(K\) is the number of sampling points per syllable. Theoretically, contextual information is modeled by the internal connection of the recurrent model structure. We thus seek to eliminate contextual information and include only tone for context. The output feature is a sequence of \(F_{0}\) vectors and its dynamic future sampled from the original \(F_{0}\) contour. Both the input and output features are normalized to have zero mean unit variance. The hyper-parameters (i.e., the number of hidden layers, number of hidden units, and learning rate) of all models were tuned to achieve close to optimal results on the development set. The model weights were optimized using the Adam-based back-propagation algorithm [49]. To avoid over-fitting, we applied early stopping criteria to stop training when the validation loss had stopped decreasing in 10 consecutive epochs.