# Evaluation of Tacotron Based Synthesizers for Spanish and Basque
## 5 Conclusions  
Neural networks allow the generation of synthetic voices that show high resemblance to natural ones, but require large amounts of data to train them. In languages with difficult access to large amounts of data, it is important to get the most out of the available data. One of the limitations of neural networks trained with few data is the appearance of synthesis errors. We propose a novel algorithm to automatically assess these errors based on the alignment of the last decoding steps of the acoustic model. This algorithm allows evaluating the results of the baseline Tacotron 2 model and showing the improvement achieved when the guided attention is introduced. The results show that the implementation of the guided attention decreases the number of errors for all the developed voices.  
In the MOS evaluation of quality and naturalness, female voices obtained better results than male voices for both languages and dimensions. This is the case even for the natural recordings, which implies that using recordings with high MOS values is important to get good subjective results in the synthetic voices. Therefore, it would be interesting to obtain the MOS values of several speakers to select the most suitable one prior to recording the database.  
We also applied an objective evaluation measure that assesses the naturalness of synthetic signals, NISQA. The best results were obtained for all voices by the guided attention Tacotron 2 system combined with a WaveGlow vocoder adapted to the gender of the speaker.  
Further work includes the evaluation of the effect that data augmentation techniques may have on the quality and naturalness of the generated synthetic voices. Another way of training with more data would be combining data from both languages and/or several speakers in the same model. Taking into account the phonetic resemblance between Spanish and Basque, mixing sentences from both languages may contribute to a more robust training of the common phonemes.  
Conceptualization, V.G., I.H. and E.N.; Data curation, V.G.; Formal analysis, V.G., I.H. and E.N.; Funding acquisition, I.H. and E.N.; Investigation, V.G., I.H. and E.N.; Methodology, V.G., I.H. and E.N.; Project administration, I.H. and E.N.; Software, V.G.; Supervision, I.H. and E.N.; Validation, I.H.; Visualization, V.G. and E.N.; Writing--original draft, V.G., I.H. and E.N.; Writing--review and editing, V.G., I.H. and E.N. All authors have read and agreed to the published version of the manuscript.  
This work was funded by the Basque Government (Project refs. PIBA 2018-035, IT-1355-19). This work is part of the project Grant PID 2019-108040 RB-C21 funded by MCIN/AEI/10.13039/501100011033.  
**Informed Consent Statement:** Not applicable.  
**Data Availability Statement:** Basque and Spanish databases are accessible via the following link [[http://metashare.ilsp.gr:8080/repository/search/?q=ahosyn](http://metashare.ilsp.gr:8080/repository/search/?q=ahosyn)]([http://metashare.ilsp.gr:8080/repository/search/?q=ahosyn](http://metashare.ilsp.gr:8080/repository/search/?q=ahosyn)) (accessed on 20 December 2021). For the Spanish Blizzard database, please contact the Blizzard Challenge Organisation.  
**Acknowledgments:** We would like to thank the organizers of the Blizzard Challenge 2021 for providing the data and carrying out the evaluations. We would also like to express our gratitude to all the participants of the evaluation we performed.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **HTS Based** & **Taco-PAG+WG-MA** & **Taco-PAG+WG-BA** \\ \hline Female Spanish & 2.97 \(\pm\) 0.05 & 3.34 \(\pm\) 0.05 & **4.20 \(\pm\) 0.04** \\ Female Basque & 3.07 \(\pm\) 0.04 & 3.46 \(\pm\) 0.07 & **3.76 \(\pm\) 0.01** \\ Male Spanish & 3.62 \(\pm\) 0.04 & **3.64 \(\pm\) 0.08** & 3.58 \(\pm\) 0.01 \\ Male Basque & 2.95 \(\pm\) 0.04 & **3.56 \(\pm\) 0.08** & 3.35 \(\pm\) 0.01 \\ Blizzard Spanish & - & 3.63 \(\pm\) 0.01 & **4.38 \(\pm\) 0.03** \\ \hline \hline \end{tabular}
\end{table}
Table 4: NISQA assessment scores with 95% confidence interval.