# Semi-Supervised Learning for Robust Emotional Speech Synthesis with Limited Data
## 2 Related Work
Semi-supervised training methods can truly improve the data efficiency in end-to-end speech synthesis [32]. While the task for TTS is generally in supervised mode, a study  
Figure 1: Illustration of low-resource transfer learning. by the Ping An Technology team [33] showed that a semi-supervised learning method based on generating pseudo tags could free up exponentially more data and greatly reduce training costs. This semi-supervised learning scheme using pseudo labels to further guide the acoustic model learning significantly improves the speech quality of the test data and achieves naturalness and robustness of speech synthesis under limited emotional speech data conditions. This approach was inspired by the research in image processing, where [PERSON] et al. [34] proposed an efficient method to pre-train an EfficientNet model on labeled ImageNet images and use it as a teacher model to generate pseudo-labels on 300 million unlabeled images and then combine the labeled and pseudo-labeled image data to complete the self-training of the student model. Seq2 Seq models with attention mechanisms are more effective in modeling long-term dependencies at different temporal levels, such as words, phrases, and discourse [10]. By learning attention alignment, the sequence-to-sequence model can capture the dynamics of rhythm in discourse and also predict speech duration, which is a key rhythmic factor affecting speech emotions, such as longer duration in sad, relaxed, and tired states and faster and shorter duration in surprised and joyful emotions [35]. The formula for the attentional mechanism is shown in Equations (1)-(3). \[e_{i,j}=Attention\big{(}h_{i-1},x_{j}\big{)}, \tag{1}\]  
\[\alpha_{i,j}=\frac{exp\big{(}e_{i,j}\big{)}}{\sum_{k=1}^{n}exp(e_{i,k})}=softmax (e_{i,v})_{j}, \tag{2}\]  
\[c_{i}=\sum_{j=1}^{n}\alpha_{i,j}x_{j}. \tag{3}\]  
In which the encoder output \(x=[x_{1},x_{2},\dots,x_{n}]\) is used as a memory strip, and at each time step \(i\), the trainable attention mechanism is able to evaluate the energy value for each \(x_{j}\) by the hidden state \(h_{i-1}\) given by the previous decoder. The energy values are then normalized to obtain the alignment vector \(\alpha_{i}\) to generate the content vector \(c_{i}\). In order to obtain better performance, many variants of attention mechanisms have emerged for application to speech synthesis research. The guided attention (GA) mechanism proposed by the DCTTS [36] model accelerates convergence by penalizing non-diagonal alignments, while the location-sensitive attention (LSA) mechanism is used in Tacotron2. LSA uses the alignment information from previous steps to the current inference alignment information to reduce repetition and omission problems, implicitly encouraging monotonicity and completeness. Due to the nature of monotonic alignment between phoneme sequences and acoustic feature sequences, the forward attention mechanism [37] (FA), monotonic attention mechanism [6] (MA), and dynamic convolutional attention mechanism [38] (DCA) have been successively applied to TTS models with Seq2 Seq properties those have achieved good results. To ensure completeness, the stepwise monotonic attention mechanism [39] (SMA) adds a restriction to MA; the hard-aligned position can only be moved at most one step in each decoding step, the alignment vector is recomputed by Equations (4)-(6):  
\[p_{i,j}=sigmoid\big{(}e_{i,j}\big{)}, \tag{4}\]  
\[z_{i,j}:Bernoulli\big{(}p_{i,j}\big{)}, \tag{5}\]  
\[\alpha_{i,j}=\alpha_{i-1,j-1}\big{(}1-p_{i,j-1}\big{)}+\alpha_{i-1,j}p_{i,j} \tag{6}\]  
The set \"selection probability\" \(p_{i,j}\), which indicates the zero-patch operation. The experimental results of this study show that the quality of the synthesized speech using the TTS model with SMA is better than that of the baseline methods using LSA, MA, FA, GA, etc.