# Sequence-to-Sequence Acoustic Modeling with Semi-Stepwise Monotonic Attention for Speech Synthesis
## 3 Proposed Methods
The frame length was 64 ms and the frame shift was 15 ms. Phone sequences were adopted as model input and the initials and finals of Mandarin Chinese were treated as phones for simplification. A phone embedding vector, a tone embedding vector and a prosodic position embedding vector were concatenated to represent each phone. The Adam optimizer [18] was used, the training epochs were 200 and the training batch size was 80. The initial learning rate was \(1\times 10^{-3}\), and then the learning rate exponential decayed by 0.9 times every 10 epochs. A WaveNet vocoder was built to reconstruct waveforms in our experiments. Finally, three Tacotron2-based acoustic models were built for comparison. (Audio samples are available at [[https://xiaozhah.github.io/SSMA_demos](https://xiaozhah.github.io/SSMA_demos)]([https://xiaozhah.github.io/SSMA_demos](https://xiaozhah.github.io/SSMA_demos)), accessed on 17 April 2021). * The attention mechanism was stepwise monotonic attention (SMA) [9]. In both training and synthesis stages, the phrase boundaries of texts were not available. The initial bias \(b\) was 3.5 and the noise scale \(\gamma\) was 2.0. The soft mode was used at the training stage, while the hard mode was used at the synthesis stage. * This model adopted SSMA instead of SMA and other model structure and hyperparameters were the same as SMA-PB. In both training or synthesis stage, this model used the same data as SMA-PB. The initial bias \(b\) and \(b^{l}\) was 3.5, and the noise scale \(\gamma\) was 2.0. The soft mode was used at the training stage, while the hard mode was used at the synthesis stage. * This model had the same structure and hyperparameters as SMA. In both training and synthesis stages, this model used the texts with manually labeled phrase boundaries. ### Objective Evaluation

The objective performance of SMA-PB, SSMA-PB, and SMA+PB on predicting the acoustic features of test sentences was evaluated. The metrics included mel-cepstral distortion (MCD), F0 root mean square error (RMSE), F0 correlation (CORR), and un-voiced/voiced (UV) error percentage. The frame-level MCD was calculated as:

\[\text{MCD}=\frac{10}{\log 10}\sqrt{2\sum_{m=1}^{M}(c_{r}(m)-c_{s}(m))^{2}}, \tag{1}\]

where \(c_{r}\) and \(c_{s}\) are mel-cepstral coefficients (MCCs) from natural and synthetic speech, respectively, and \(M\) is their order. The frame-level F0 RMSE was calculated as:

\[\text{RMSE}=1200\sqrt{\left(\log_{2}(F_{r})-\log_{2}(F_{s})\right)^{2}}, \tag{2}\]

where \(F_{r}\) and \(F_{s}\) represents F0 values extracted from natural and synthetic speech, respectively. The F0 CORR was defined as the F0 values correlation coefficient between synthetic speech and natural speech in the voiced segment. The UV error percentage was the ratio of the number of unmatched U/V frames between natural and synthetic speech to the total number of frames. For calculating the four metrics, twelve-dimensional MCCs, and F0 values were extracted from synthetic speech at 5 ms frame shift by STRAIGHT [19] analysis. The FastDTW algorithm [20] based on MCCs was adopted to align predicted acoustic features toward reference ones for calculating the four metrics. The results are shown in Table 1.

From this figure, we can see that SMA+PB achieved the best accuracy of acoustic feature prediction. This is reasonable since it utilized manually labeled phrase boundaries in both training and synthesis stages. Comparing SMA-PB with SMA+PB, it can be found that the objective performance of SMA degraded significantly when phrase boundaries were not available. Comparing SMA-PB, SSMA-PB achieved smaller MCD and comparable F0 distortion, which may be that the MCD was more related to pause segments in utterances. ### Subjective Evaluation

Twenty sentences with at least one phrase boundary were randomly selected from the test set. These phrase boundaries were not used when synthesizing them with SMA-PB and SSMA-PB. The utterances synthesized using the SMA-PB, SSMA-PB, and SMA+PB systems were compared by two groups of AB preference tests on their naturalness.