# Using Transfer Learning to Realize Low Resource Dungan Language Speech Synthesis
## 2 Models and Methods
### Pre-Trained Tacotron2-Based Mandarin Acoustic Model  
The Mandarin acoustic model is initially trained using a large-scale Mandarin corpus. Our proprietary Chinese text analyzer extracts these sentences' initials, finals, and associated prosodic labels. The extracted acoustic features encompass the Mel spectrogram from the large-scale Mandarin corpus within the Tacotron2 framework.  
Given the similar pronunciation between the Dungan language and Mandarin, we employ the mapping-transfer learning method [44] to obtain a Dungan (target language) acoustic model by transferring knowledge from Mandarin (source language), which can be formulated as follows:  
\[f_{\theta,W}:\mathcal{X}_{\mathcal{L}}\rightarrow\mathcal{Y} \tag{4}\]  
where \(\theta\) is the parameters of the acoustic model, \(W\) denotes learnable symbol embeddings, and \(\mathcal{Y}\) represents the space of Mandarin. \(\mathcal{X}_{\mathcal{L}}\) is the text space for the Dungan language.  
\[\mathcal{X}_{\mathcal{L}}=\Big{\{}\{s_{t}\}_{t=1}^{T}\mid\forall ts_{t}\in \mathcal{L},T\in\mathbb{N}\Big{\}} \tag{5}\]  
where \(\mathcal{L}\) is the unit set for the Dungan language, \(S_{t}\) is the \(t\)-th unit of Dungan unit sequence, and \(T\) is the length of the unit sequence.  
In the encoder, we input a Dungan unit sequence represented by character embeddings. This is passed through a stack of three convolutional layers, followed by batch normalization and ReLU activations. Subsequently, the output from the final convolutional layer is fed into a bidirectional LSTM layer to generate the Dungan unit features.  
Mapping-based transfer learning involves mapping instances from \(\theta_{src}\) and \(\theta_{tgt}\) into a new acoustic parameter space. In this process, we can directly use \(W_{src}\) and \(\theta_{src}\) decoded from the Mandarin acoustic model by the decoder. \(\theta_{src}\) and \(\theta_{tgt}\) can take embeddings as input and generate speech. However, because \(s_{src}\) and \(s_{tgt}\) come from different symbol sets, i.e., \(\mathcal{L}_{src}\
eq\mathcal{L}_{tgt}\), the same concept cannot be directly applied to \(W_{src}\) and \(W_{tgt}\). To address this issue, Dungan units are embedded in \(W_{tgt}\) to facilitate relearning during the transmission process.  
We adopt the forward attention mechanism, which uses cumulative attention weights to calculate the context vector.  
Figure 6: Procedure of training the Dungan language acoustic model with transfer learning.  
The decoder is an autoregressive recurrent neural network that predicts a \(\theta_{tgt}\) from the encoder input Dungan unit sequence one frame at a time. We can use \(\theta_{src}\) learned from the Mandarin acoustic model to initialize \(\theta_{tgt}\) in the new acoustic parameter space. The output from the initial timestep is first processed through a Pre-net consisting of two fully connected layers. This output is combined with the forward attention context vector and passed through a pair of LSTM layers. The combination of the LSTM outputs and attention context vectors undergo three distinct linear transformations to predict the target spectrogram frame, stop token, and estimated residual. Subsequently, the predicted acoustic features are subjected to five convolutional layers, generating a residual to enhance the reconstruction of the Dungan acoustic model.