# Evaluation of Tacotron Based Synthesizers for Spanish and Basque
## 2 Materials and Methods
### Testing Datasets  
The deployed TTS systems were evaluated using two different written text datasets. The naturalness and quality evaluation of the synthetic voices was performed using in-domain sentences taken from novels and tales, similar to the sentences used in the training corpora. To assess the robustness of the systems, we used a dataset that contains sentences extracted from the Basque Parliament.  
(a) Texts from tales and novels: This dataset contains utterances extracted from different books. It is a phonetically balanced dataset with 450 sentences both in Basque and Spanish. This is the corpus used to assess the naturalness and quality of the synthetic speech. The right column of Table 2 presents a summary of the text length related information in this corpus. As shown in the table, the text length in this dataset follows a similar distribution to the one used in the training corpus. This choice was intentionally made to prevent failures due to alignment issues in long utterances.  
(b) Parliamentary texts: This dataset was supplied by MintzAI project [23] ([[http://www.mintzai.eus/indice.html](http://www.mintzai.eus/indice.html)]([http://www.mintzai.eus/indice.html](http://www.mintzai.eus/indice.html)) (accessed on 20 December 2021)). The sentences in the dataset are transcriptions from the Basque Parliament speeches, in both Basque and Spanish. This dataset is used to assess the robustness of the deployed models. From the entire dataset, 20,000 utterances were randomly selected. The left column of Table 2 shows the most relevant information of the size and lengths of this dataset. It can be seen that the lengths of the utterances contained in this corpus, with an average of \(21.27\pm 16.55\) and \(19.46\pm 16.81\) words per sentence in Basque and Spanish, greatly differ from those of the training dataset. In particular, the presence of very long utterances is an especially difficult scenario for Tacotron 2 model [24].  
Figure 2 reveals the distribution of the number of words per utterance in each testing dataset. Compared to the distributions shown in Figure 1, a high resemblance can be found between the sentence length distribution of the training dataset and the corpora extracted from novels and tales. On the contrary, the length distribution of the sentences extracted from the parliamentary texts greatly differ from that of the training dataset. The motivation behind this choice is to assess the robustness of the systems in a more challenging environment.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**Parliamentary Texts**} & \multicolumn{2}{c}{**Texts from Novels and Tales**} \\ \cline{2-5}  & **Spanish** & **Basque** & **Spanish** & **Basque** \\ \hline Number of utterances & 20,000 & 20,000 & 450 & 450 \\ Number of words & 425,467 & 389,252 & 3748 & 3064 \\ Unique words & 10,704 & 18,563 & 1378 & 1703 \\ Words in shortest & 1 & 1 & 3 & 2 \\ utterance & 268 & 133 & 15 & 14 \\ Avg. words per utterance & \(19.46\pm 16.81\) & \(21.27\pm 16.55\) & \(8.33\pm 2.44\) & \(6.81\pm 8.75\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Description of the testing datasets.  
These datasets were processed according to the same processing applied to the transcriptions of the training dataset.