# MLLP-VRAIN Spanish ASR Systems for the Albayzin-RTVE 2020 Speech-to-Text Challenge: Extension
## 3 MLLP-VRAIN Systems
### Experiments and Results
The motivation behind this change is that the computation of LSTM LM scores is quite computationally expensive, and its contribution to PPL is negligible with respect to the n-gram LM + TLM combination (3% relative improvement). We thus increased system latency stability while experiencing nearly no degradation in terms of WER: 11.6 and 16.1 points on _test-2018_ and _test-2020_, respectively. Both streaming ASR systems, _p-streaming_1500 ms_nt_ and _c1-streaming_1500 ms_nt_, share the same theoretical latency of 1.5 s due to the context window size. As stated in Section 3.3, this parameter can be adjusted at decoding time. This allows us to configure the decoder for lower latency responses or better transcription quality. Hence, our final goal for the challenge was to find a proper system configuration providing state-of-the-art, stable latencies with minimal WER degradation. Figure 1 illustrates the evolution of WER on _dev1-dev_ as a function of the context window size, limited to one second at maximum. As we focused on gauging AM performance, we used the \(n\)-gram LM in isolation for efficiency reasons. In light of the results, we chose a window size of 0.6 s, as it brings a good balance between transcription quality and theoretical latency. The last step to set up our latency-focused streaming system was to measure WER and empirical latencies as a function of different pruning parameters and LM combinations. In our experiments, latency is measured as the time elapsed between the instant at which an acoustic frame is generated, and the instant at which it is fully processed by the decoder. Latency figures are provided at the dataset level, computed as the average of the latencies observed at the frame level on the whole dataset. Figure 2 shows WER vs mean empirical latency figures, computed over _dev1-dev_, with different pruning parameter values, and comparing the LM combinations including the Transformer LM. These measurements were made on an Intel i7-3820 CPU @ 3.60 GHz, with 64 GB of RAM and a GeForce RTX 2080 Ti GPU. On the one hand, we can see how combinations involving LSTM LMs are systematically shifted rightwards with respect to other combinations. This means that the LSTM LM has a clear negative impact on system latency, with little to no effect on system quality. This evidence corroborates our decision to remove the LSTM LM to define our contrastive system _c1-streaming_1500 ms_nt_. On the other hand, the TLM alone generally provides a good baseline that is slightly improved in terms of WER if we include the other LMs. However, this comes at the cost of increasing latency. Hence, we selected the Transformer LM in isolation for our final latency-focused streaming system. This system was our contrastive system no. 2, identified as _c2-streaming_600 ms_t_. Its empirical latency on _dev1-dev_  
Figure 1: WER as a function of context window size (in seconds) for the streaming setup, computed over the _dev1-dev_ set. This figure includes both the setup in this section (dashed line) and the setup for the closed-condition system described in Section 4 (solid line). was 0.81 \(\pm\) 0.09 s (mean \(\pm\) stdev), and its performance was 12.3 and 16.9 WER points on _test-2018_ and _test-2020_, respectively. This is, with just a very small relative WER degradation of 6% with respect to the primary system, we got state-of-the-art (mean = 0.81 s) and very stable (stdev = 0.09 s) empirical latencies. This system has a baseline consumption (when idle) of 9 GB RAM and 3.5 GB GPU memory (on a single GPU), adding 256 MB RAM and one CPU thread per decoding (audio stream). For instance, decoding four simultaneous audio streams in a single machine would use four CPU threads, 10 GB RAM and 3.5 GB GPU memory. Table 5 summarises the results obtained for all four participant ASR systems on the _dev2_, _test-2018_ and _test-2020_ sets, and also includes the results obtained with our 2018 open-condition system for comparison.