# Semi-Supervised Learning for Robust Emotional Speech Synthesis with Limited Data
## 3 Proposed Methods
### Training Process  
The model was pre-trained using the LJ-Speech dataset ([[https://keithito.com/LJ-Speech-Dataset/](https://keithito.com/LJ-Speech-Dataset/)]([https://keithito.com/LJ-Speech-Dataset/](https://keithito.com/LJ-Speech-Dataset/)), accessed on 7 January 2022), containing nearly 24 h of speech from a female speaker and the corresponding text. Another expressive dataset without emotional annotates is the audio dataset of the Blizzard Challenge 2013 ([[https://www.synsig.org/](https://www.synsig.org/)]([https://www.synsig.org/](https://www.synsig.org/))  
Figure 2: Network structure of SMAL-ET2 model. index.php/Blizzard_Challenge_2013, accessed on 15 February 2022), named BC2013, has a total speech duration of 16.8 h and contains 8248 utterances, with which the reference model is pre-trained to adapt the model to expressive speech in order to provide pseudo-label loss during fine-tuning. The model is fine-tuned with the neutral dataset of the speaker before being fine-tuned with the sentiment subset of the ESD (Emotional Speech Dataset) ([[https://github.com/HLTSingapore/Emotional-Speech-Dataset/](https://github.com/HLTSingapore/Emotional-Speech-Dataset/)]([https://github.com/HLTSingapore/Emotional-Speech-Dataset/](https://github.com/HLTSingapore/Emotional-Speech-Dataset/)), accessed on 16 February 2022) in order to adapt it to the style of the new speaker and then fine-tuned with the happy, sad, surprise, and angry subsets, respectively. When fine-tuning the model with a single emotional speech dataset, the limited training of the target data leads to an overfitting phenomenon, where only three thousand iterations fail to produce intelligible speech, and the audio is completely noisy, while even after nearly 40,000 iterations, it still fails to produce the correct sound for utterances outside the training set. To alleviate the overfitting phenomenon, the approach in this paper makes the pre-trained model the initial reference model and further guides the training of the fine-tuned model after using its generated pseudo-label. The main purpose is to introduce a reference loss function to recalculate the mean squared error (MSE) of the model prediction output to achieve the effect of MSE regularization. The main loss is calculated by Equation (7), and the additional loss is calculated by Equation (8). The total loss of the model training is calculated by Equation (9):  
\[\mathcal{L}_{main}=\frac{1}{N}\sum_{k=1}^{K}\Bigl{\|}\mathcal{M}_{f}^{(k)}- \mathcal{M}_{g}^{(k)}\Bigr{\|}^{2}, \tag{7}\]  
\[\mathcal{L}_{add}=\frac{1}{N}\sum_{k=1}^{K}\Bigl{\|}\mathcal{M}_{f}^{(k)}- \mathcal{M}_{r}^{(k)}\Bigr{\|}^{2}, \tag{8}\]  
\[\mathcal{L}_{final}=\mathcal{L}_{main}+\theta\mathcal{L}_{add}. \tag{9}\]  
The main loss \(\mathcal{L}_{main}\) is the MSE loss between the true speech (ground truth, GT) Mel-spectrum \(\mathcal{M}_{g}^{(k)}\) and the Mel-spectrum \(\mathcal{M}_{f}^{(k)}\) predicted by the fine-tuning model.\(N\) indicates the number of samples. The additional loss \(\mathcal{L}_{add}\) is the MSE loss between the Melspectrums \(\mathcal{M}_{r}^{(k)}\) predicted by the initial reference model and \(\mathcal{M}_{f}^{(k)}\) predicted by the fine-tuning model, respectively. The final total loss is obtained by configuring the trade-off parameter \(\theta\) of these two losses. If the value of \(\theta\) is too high, the fine-tuning model will focus too much on the output of the initial reference model. If it is too small, the performance learned from the source data will be lost. The purpose of the additional reference loss is to maintain a certain degree of generalization of the fine-tuning model, while the main loss is to maintain the emotional style of the target speech. For the value of the trade-off parameter, experiments were conducted to compare the final value of 0.1, and it was determined that the best results of speech synthesis were achieved. A comparison of the results of the experiments with different values is shown in Section 4.