# Using Transfer Learning to Realize Low Resource Dungan Language Speech Synthesis
###### Abstract  
This article presents a transfer-learning-based method to improve the synthesized speech quality of the low-resource Dungan language. This improvement is accomplished by fine-tuning a pre-trained Mandarin acoustic model to a Dungan language acoustic model using a limited Dungan corpus within the Tacotron2+WaveRNN framework. Our method begins with developing a transformer-based Dungan text analyzer capable of generating unit sequences with embedded prosodic information from Dungan sentences. These unit sequences, along with the speech features, provide -unit sequence with prosodic labels, Mel spectrograms> pairs as the input of Tacotron2 to train the acoustic model. Concurrently, we pre-trained a Tacotron2-based Mandarin acoustic model using a large-scale Mandarin corpus. The model is then fine-tuned with a small-scale Dungan speech corpus to derive a Dungan acoustic model that autonomously learns the alignment and mapping of the units to the spectrograms. The resulting spectrograms are converted into waveforms via the WaveRNN vocoder, facilitating the synthesis of high-quality Mandarin or Dungan speech. Both subjective and objective experiments suggest that the proposed transfer learning-based Dungan speech synthesis achieves superior scores compared to models trained only with the Dungan corpus and other methods. Consequently, our method offers a strategy to achieve speech synthesis for low-resource languages by adding prosodic information and leveraging a similar, high-resource language corpus through transfer learning. D 2024 17 June 2024 17 July 2024 18 July 2024 18 July  
DNN-based Dungan speech synthesis [9; 10], the synthesized speech quality was not high due to the limited training corpus. Speech synthesis technologies encompass unit selection-based concatenative speech synthesis [11], hidden Markov model (HMM)-based statistical parametric speech synthesis (SPSS) [12], and deep learning-based speech synthesis [13; 14]. While deep learning has significantly advanced speech synthesis technology, methods such as long short-term memory (LSTM) and bidirectional LSTM [15; 16] have addressed temporal information limitations. Moreover, end-to-end speech synthesis models [17] like Tacotron [18] and Tacotron2 [19] have demonstrated the ability to map text directly to speech. When trained with large-scale text-to-speech pairs, these models produce synthesized speech using high-quality vocoders such as the Griffin-Lim algorithm [20], WaveNet [21], and WaveRNN [22]. However, such systems require substantial training corpora. For low-resource languages, the lack of training corpus makes it difficult for end-to-end models to learn the prosodic structure of sentences, resulting in a lack of prosodic changes in synthesized speech, which affects its naturalness, posing challenges for speech synthesis of low-resource languages. Cross-language transfer learning [23; 24; 25] has been employed to mitigate the issue of insufficient training corpora for speech synthesis in low-resource languages. This technique entails training a language model using a combination of a large corpus from a high-resource language and a smaller corpus from a low-resource language, followed by adapting this model to the low-resource language. Transfer learning in speech synthesis has proven to be an effective strategy for producing speech in low-resource languages by harnessing the capabilities of a high-resource language acoustic model [26; 27]. In our prior research on Tibetan speech synthesis [28; 29; 30; 31; 32], we determined that integrating prosodic information through transfer learning-based techniques enhances the quality of synthesized speech for low-resource languages such as Tibetan. Building on this insight, the present study implements a sequence-to-sequence (seq2 seq) approach for Dungan language speech synthesis, leveraging transfer learning and prosodic information within the Tacotron2+WaveRNN framework.