# Evaluation of Tacotron Based Synthesizers for Spanish and Basque
## 4 Evaluation
### Naturalness and Quality  
In this section, the results of two independently developed MOS tests are presented. The first one was carried out for the voices of the two corpora recorded in the premises of our laboratory and was organized by our research group; the second one is part of the external evaluation campaign provided by the the Blizzard Challenge Organisation 2021 [34], where we participated with the synthesis system described in Section 3.3. In this latter case, we just present the results for reference.  
In our test, listeners were asked to evaluate the naturalness and the quality for each utterance using a 1-5 point scale, being 1 the lowest score assigned to the signal and 5 the highest. For this evaluation, we used the testing dataset described in Section 2.2, Corpus (a). The DNN-based systems used to synthesize the sentences were composed of the Taco-PAG acoustic model along with the neural vocoder model WG-MA described in Section 3.4.  
In addition to the assessment of the signals obtained with the DNN based systems, we also included signals from other two sources in the evaluation:  
* Signals from the natural reference speech.
* Synthetic signals obtained using an HMM-based speech synthesis sytem (HTS) based TTS system for Spanish and Basque developed in our research group ([[https://sourceforge.net/projects/ahotts/](https://sourceforge.net/projects/ahotts/)]([https://sourceforge.net/projects/ahotts/](https://sourceforge.net/projects/ahotts/)) (accessed on 20 December 2021)) [21].  
In total, 33 listeners took part in the evaluation. As the test includes sentences in both Basque and Spanish, all listeners were bilingual in both languages. From the 450 available utterances, each participant rated 6 random signals per language (2), speaker (2) and method (3), i.e., a total of 72 utterances.  
Figure 5 shows the results obtained in this evaluation. The box-plot graph representing the assessment of the quality evaluation is shown in Figure 5. The scores assigned to the signals obtained with the DNN-based approach are in all case higher than those assigned to the signals obtained with HTS-based synthesis. Nevertheless, the score remains lower than that of natural speech. A preference for the female voice is shown for the natural speech in both languages. This preference is also shown in the evaluation of the DNN based signals, where female speech obtained higher rates than male speech.  
Figure 5 presents the box-plot graph representing the naturalness ratings. As in the quality evaluation, DNN-based systems were ranked above the HTS systems, but they received a lower score comparing with the natural speech signal. In the naturalness evaluation, participants also preferred female voices over male voices in natural speech signals, which is reflected in the scoring of the DNN based systems as well.  
Different aspects of the speech signals were rated in the evaluation test performed by the Blizzard Challenge Organisation (i.e., intelligibility, similarity to to original speaker and naturalness). In this work, we only include the naturalness scores as a reference, but the complete evaluation can be accessed in [34]. In total, 12 systems participated in the challenge. The synthetic signals obtained with these systems were rated by 313 participants. Figure 6 shows the results of this evaluation. Our proposed system was labelled as letter 'E' and the reference natural speech was labelled as letter 'R'.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Baseline** & **Taco-PAG** & **Improvement** \\ \hline Female Spanish & 1791 & 1103 & 38.41\% \\ Female Basque & 2596 & 1941 & 25.23\% \\ Male Spanish & 1077 & 95 & 91.18\% \\ Male Basque & 1206 & 274 & 71.31\% \\ Blizzard Spanish & 361 & 90 & 75.07\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Number of utterances with errors and relative improvement in percentage.