# MLLP-VRAIN Spanish ASR Systems for the Albayzin-RTVE 2020 Speech-to-Text Challenge: Extension
## 4 Closed-Condition Systems
### Experiments and Results  
An empirical study similar to that described in Section 3 was carried out using the development and test sets in Table 3. The first step was to compare n-gram and Transformer LMs, as well as their combination, in terms of perplexity on the _dev1-dev_ set. Table 6 shows the results obtained, including the optimal interpolation weights we found for their combination. As a second step, we studied the effect of the theoretical acoustic latency on system performance. Figure 1 shows the WER as a function of the AM context window size, for window sizes under one second, and using the simpler n-gram LM in isolation. For comparison, the performance of the open-condition system is also shown. From the results in Figure 1, and in agreement with the open-condition experimentation of Section 3, an acoustic window size of 0.6 s was selected for further experiments under the streaming setup aimed at measuring real latencies. \begin{table}
\begin{tabular}{c c c} \hline \hline
**LM Comb.** & **PPL** & **Weights (\%)** \\ \hline ng & 164 & - \\ tf & 103 & - \\ ng + tf & 84 & 70 + 30 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Perplexity (PPL) and interpolation weights, computed over the _dev1-dev_ set, of the n-gram (ng) and Transformer (tf) LMs and their combination when trained with restricted data. For the final streaming and latency tests, the WER was plotted against empirical latencies with different pruning parameters and LM combinations. All measurements were made on the same hardware we used to assess the open condition systems (Section 3). Figure 3 shows the WER as a function of the mean empirical latency computed over _dev1-dev_. For comparison, the corresponding results for the open-condition systems are also included. In contrast to the open-condition results, a significant improvement is achieved by combining the n-gram and Transformer LMs, instead of just using the Transformer LM alone. Therefore, for comparative purposes, the systems using interpolated n-gram and Transformer LMs are considered our final closed-condition systems. Table 7 contains a summary of the most relevant results obtained with our systems, both under open and closed data conditions. For the closed-condition systems, results for data-unrestricted LMs are also included to check how far we can go without expanding the acoustic data, better reflecting a usual scenario where text data is relatively much easier to obtain than audio with transcriptions. For comparison, Table 7 includes the figures for the 2018 contest winners and results achieved by other contestants in the 2020 challenge. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**System** & **Duration (h)** & _dev2_ & _test-2018_ & _test-2020_ \\ \hline _open-p-streaming\_1500 ms\_nlt_ & 3924 & 11.2 & 11.6 & **16.0** \\ _open-c2-streaming\_600 ms\_t_ & 3924 & 12.0 & 12.3 & **16.9** \\ \hline _closed-streaming\_600 ms\_nt2018_ & 205 & 15.0 & 15.3 & 23.5 \\ + _open ng_ & & 15.6 & 15.9 & 23.2 \\ + _open ng+tf_ & & 13.3 & 13.7 & **20.4** \\ \hline _closed-streaming\_1500 ms\_nt2018_ & 205 & 14.7 & 15.3 & 23.1 \\ + _open ng_ & & 15.3 & 15.8 & 22.9 \\ + _open ng+tf_ & & 13.0 & 13.7 & **19.9** \\ \hline
2018 open-cond. winner [28] & 3800 & 15.6 & 16.5 & - \\
2018 closed-cond. winner [28] & 205 & 20.0 & 22.0 & - \\ \hline
2020 _Vicomtch_[23] & 743 & n/p & n/p & 19.3 \\
2020 _BRNO_[24] & 780 & 12.8 & 13.3 & 23.2 \\
2020 _Sigma-UPM_[25] & 615 & n/p & n/p & 27.7 \\
2020 _Biometric Vox System_[26] & 1000 & 17.8 & 22.0 & 30.3 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: WER of our main (open) submissions and our closed-condition systems described in this section, including for comparison the winning open- and closed-condition systems of the 2018 challenge, computed over the _dev2_, _test-2018_ and _test-2020_ sets. Highlited in bold, the most relevant results to compare open and closed conditions in the main test set _test-2020_.