## 5 Results
### Speech-to-Text Evaluation
#### 5.1.2 Closed-Set Condition Results  
Regarding the closed-set condition results, Table 8 presents the overall results of the three primary systems. The best results, with a WER of 19.57%, were obtained by the G14-SIGMA system. The G14-SIGMA closed-set condition system was similar to the one used in the open-set condition; the difference was in the data used for training the acoustic and language models. In the open-set condition, acoustic and language models were trained with the data used in the closed condition augmented with an additional 265 h of audio and text coming from news, interviews, and subtitles. This data augmentation allowed a reduction of the WER from 19.57% to 17.26%. It is interesting to compare results among the three systems in terms of the data used for training. The main difference was the way they obtained correctly transcribed audio from the training and development partitions. G14-SIGMA used manual transcription of 350 h of the train partition for both acoustic and language models. However, G6-VICOMTECH-PRHLT and G7-MLLP-RHTW used automatic aligned and filtered audio from the train and development partitions for training acoustic models, a total of 136 h and 218 h, respectively, and all text files given in the RTVE2018 database for training language models. Although the three systems were using different ASR toolkits, the results showed a high correlation between the amount of training data obtained from the training set and the performance: the more training data, the better the results. It is valuable to highlight that the filtering techniques used by G6-VICOMTECH-PRHLT and G7-MLLP-RHTW degraded the performance by less than 3% compared to the human-transcription used by G14-SIGMA.  
In terms of the TNWER (Table 9), it is interesting to note that G14-SIGMA almost got the same TNWER (17.90%) as in the open-set condition (17.26%), which means that most of the errors in the closed-set conditions were related to stop-words, verbal conjugations, and word number or gender. A comparison between open-set and closed-set conditions for the G6-VICOMTECH-PRHLT and G7-MLLP-RHTW systems was not possible as they used different ASR architectures.  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Punc.Marks** & **Periods** & & **Periods** & & **Periods and Commas** \\ \hline
**Team** & G5 & & G6 & & & G6 & \\
**System** & P & P & C1 & C2 & P & C1 & C2 \\
**Overall WER** & 20.92 & 24.52 & 33.00 & 26.66 & 24.52 & 33.00 & 26.66 \\
**Overall PWER** & 26.04 & 29.75 & 29.19 & 28.12 & 33.00 & 33.59 & 31.59 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Speech to text open-set condition. WER (%) and punctuation marks evaluation (PWER) (%) for the primary systems per team. Team and system descriptions are in Section 4.1.1.  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Team** & & **G6** & & **G7** & **G14** \\ \hline
**System** & P & C1 & C2 & P & C1 & P \\
**TV Show** & & & & & & \\
**AFI** & 24.22 & 25.01 & 25.99 & 25.39 & 25.29 & **19.75** \\
**AV** & 33.94 & 37.75 & 42.17 & 36.17 & 37.19 & **27.75** \\
**DH** & 45.62 & 48.36 & 51.30 & 50.82 & 50.35 & **43.50** \\
**EC** & 16.70 & 17.27 & 20.81 & 16.68 & 16.56 & **15.63** \\
**LA24H** & **10.47** & 10.73 & 12.74 & 12.04 & 11.96 & 11.25 \\
**LM** & 28.28 & 29.58 & 34.40 & 26.68 & 26.60 & **23.96** \\
**LT24 HTer** & 20.80 & 21.38 & 24.82 & 19.15 & 19.29 & **18.01** \\
**SG** & 17.7 & 18.92 & 22.65 & 18.79 & 18.75 & **15.43** \\
**Overall WER** & 22.22 & 23.16 & 26.66 & 21.98 & 21.96 & **19.57** \\ \hline \hline \end{tabular}
\end{table}
Table 8: Speech-to-text closed-set condition. WER(%) per TV show and team. Team and system descriptions are in Section 4.1.2. See Table 4 for show descriptions. Best results by TV show are in bold.