## 3 IberSpeech-RTVE 2018 Evaluation Tasks
### Speech-to-Text Challenge
#### 3.1.2 Performance Measurement  
The STT system output was evaluated with different metrics, but they were ranked by the word error rate. All the participants had to provide as output for evaluation a free-form text with no page, paragraphs, sentence, or speaker breaks using the UTF-8 charset [[http://www.utf-8.com/](http://www.utf-8.com/)]([http://www.utf-8.com/](http://www.utf-8.com/)) per test file. The text might include punctuation marks to be evaluated with an alternative metric.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Show** & **Acronym** & **\# of Shows** & **Duration** & **Audio Features** \\ \hline \hline Al filo de lmposible & AFI & 9 & 4:10:03 & Poor quality audio in some outdoor shots. \\  & & & & Few speakers. Exterior shots. \\ \hline Arranca en Verde & AV & 2 & 1:00:30 & Good audio quality in general. \\  & & & & Most of the time, 2 speakers in a car. \\ \hline Dicho y Hecho & DH & 1 & 1:48:00 & Much speech overlap and speech inflections. \\  & & & & About 8 speakers, most of them comedians. \\  & & & & Studio and exterior shots. \\ \hline España en Comunidad & EC & 22 & 8:09:32 & Good audio quality in general. \\  & & & & Diversity of speakers. \\  & & & & Studio and exterior shots. \\ \hline La Mañana & LM & 4 & 8:05:00 & Much speech overlap, \\  & & & & speech inflections, and live audio. \\  & & & & Studio and exterior shots. \\ \hline La Tarde en 24H (Tertulia) & LT24 HTer & 9 & 8:52:20 & Good audio quality, \\  & & & & overlapped speech on rare occasions, \\  & & & & up to 5 speakers. Television studio. \\ \hline LatinoAmerica en 24H & LA24H & 8 & 4:06:57 & Good audio quality. \\  & & & & Many speakers with a Spanish \\  & & & & Latin American accent. \\  & & & & Studio and exterior shots. \\ \hline Saber y Ganar & SG & 4 & 2:54:53 & Good audio quality. \\  & & & & Up to 6 speakers per show. \\  & & & & Television studio. \\ \hline \hline \end{tabular}
\end{table}
Table 4: S2T test dataset characteristics.  
The word error rate (WER) was the primary metric for the STT task. The text was normalized removing all the punctuation marks; numbers were written with letters; and text was lower-cased. The WER is defined as:  
\[WER=\frac{S+D+I}{N_{r}} \tag{1}\]  
where N\({}_{r}\) is the total number of words in the reference transcription, S is the number of substituted words in the automatic transcription, D is the number of words from the reference deleted in the automatic transcription, and I is the number of words inserted in the automatic transcription not appearing in the reference.  
In addition to the primary metric, other alternative metrics were computed, but not taken into account for the challenge ranking.  
Punctuation marks evaluation (PWER): The WER was computed with the punctuation marks given by the STT system. Periods and commas were processed as words.  
Text normalized word error rate (TNWER): Text normalization techniques such as stop-word removal and lemmatization were applied to the STT output. In this sense, common errors such as verbal conjugations, gender or number substitutions, articles, determiners, and quantifiers deletion/insertions had no impact on the performance evaluation metric. The same text normalization was applied to both the reference and automatic transcriptions before proceeding to calculate WER. The Freeling [[http://nlp.lsi.upc.edu/freeling/](http://nlp.lsi.upc.edu/freeling/)]([http://nlp.lsi.upc.edu/freeling/](http://nlp.lsi.upc.edu/freeling/)) lemmatizer was used.