# Semi-Supervised Learning for Robust Emotional Speech Synthesis with Limited Data
###### Abstract  
Emotional speech synthesis is an important branch of human-computer interaction technology that aims to generate emotionally expressive and comprehensible speech based on the input text. With the rapid development of speech synthesis technology based on deep learning, the research of affective speech synthesis has gradually attracted the attention of scholars. However, due to the lack of quality emotional speech synthesis corpus, emotional speech synthesis research under low-resource conditions is prone to overfitting, exposure error, catastrophic forgetting and other problems leading to unsatisfactory generated speech results. In this paper, we proposed an emotional speech synthesis method that integrates migration learning, semi-supervised training and robust attention mechanism to achieve better adaptation to the emotional style of the speech data during fine-tuning. By adopting an appropriate fine-tuning strategy, trade-off parameter configuration and pseudo-labels in the form of loss functions, we efficiently guided the learning of the regularized synthesis of emotional speech. The proposed SMAL-ET2 method outperforms the baseline methods in both subjective and objective evaluations. It is demonstrated that our training strategy with stepwise monotonic attention and semi-supervised loss method can alleviate the overfitting phenomenon and improve the generalization ability of the text-to-speech model. Our method can also enable the model to successfully synthesize different categories of emotional speech with better naturalness and emotion similarity. speech synthesis; low resource; emotional speech corpus; transfer learning; pseudo label +
Footnote †: journal: _Applied_  
start to finish, and many studies have shown that transfer learning can reduce the amount of data required for the final target task. This idea was also validated in the emotional speech synthesis study by [PERSON] et al. [9], where the synthesized speech was objectively perceived by an emotional speech recognition model. However, due to the sparse amount of emotional speech data used for fine-tuning, the model, after parameter updating, suffers from overfitting and unstable sound quality. The model is affected by the non-linguistic rhythmic features of emotional speech, and the synthesis of test texts outside the training set (out-of-domain) is not satisfactory. Because the attention module used does not adapt well to the new speaking style of emotional speech. The mean opinion score (MOS) of the synthesized emotional speech is below 3 points, which is obviously not good enough for listening; thus, the sound quality still needs to be improved. [PERSON] et al. [10] also did some research under limited emotional speech data conditions; they used the GST [11] model as a style encoder, first pre-trained to maturity on a multi-speaker dataset, and then fine-tuned to act as an emotional encoder with small amount of emotional speech, and achieved emotional voice conversion under low data resource conditions through these two stages of training. Their method allows the TTS model to learn the actual emotional information. To address the above issues, we use a pre-trained and mature neutral Tacotron2 model fine-tuned with an emotional speech dataset that does not require manual screening. In order to enhance the alignment learning of the sequence-to-sequence (Seq2 Seq) model, the original location-sensitive attention (LSA) mechanism is replaced by a stepwise monotonic attention mechanism during training, and the pre-trained backbone network is used to provide the pseudo-labels. The pre-trained HiFi-GAN [12] vocoder is finally used to synthesize the target's emotional speech. The main contributions of this paper include three aspects. 1.