# MLLP-VRAIN Spanish ASR Systems for the Albayzin-RTVE 2020 Speech-to-Text Challenge: Extension
## 3 MLLP-VRAIN Systems
### Language Modelling  
Regarding language modelling, we trained count-based (n-gram) and neural-based (LSTM, Transformer) language models (LMs) to perform one-pass decoding with different linear combinations of them [16], using the text data sources and corpora described in Table 2.  
On the one hand, we trained 4-gram LMs using SRILM [36] with all text resources plus the Google-counts v2 corpus [37], accounting for 102G running words. The vocabulary size was limited to 254 K words, with an OOV ratio of 0.6% computed over our internal development set.  
On the other hand, regarding neural LMs, we considered the LSTM and Transformer architectures. In both cases, LMs were trained using a 1-gigaword subset randomly extracted from all available text resources, except Google-counts. Their vocabulary was defined as the intersection between the n-gram vocabulary (254 K words) and that derived from the aforementioned training subset. We did this to avoid having zero probabilities for words that are present in the system vocabulary but not in the training subset. This is taken into account when computing perplexities by renormalizing the unknown-word score accordingly.  
Specific training details for each neural LM architecture are as follows. Firstly, LSTM LMs were trained using the CUED-RNNLM toolkit [38]. The Noise Contrastive Estimation (NCE) criterion [39] was used to speed up model training, and the normalization constant learned from training was used during decoding [40]. Based on the lowest perplexity observed on our internal development set, a model with a 256-unit embedding layer and two hidden LSTM layers of 2048 units was selected. Secondly, Transformer LMs (TLMs) were trained using a customized version of the FairSeq toolkit [41], with the following configuration minimizing perplexity in our internal development set: 24-layer network with 768 units per layer, 4096-unit feed-forward neural network, 12 attention heads, and an embedding of 768 dimensions. These models were trained until convergence with batches limited to 512 tokens. Parameters were updated every 32 batches. During inference, Variance Regularization was applied to speed up the computation of TLM scores [21].  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Resource** & **Duration (h)** \\ \hline Internal: entertainment & 2932 \\ Internal: educational & 406 \\ Internal: user-generated content & 202 \\ Internal: parliamentary data & 158 \\ Voxforge [35] & 21 \\ \hline RTVE2018: _train_ & 187 \\ RTVE2018: _dev1-train_ & 18 \\ \hline TOTAL & 3924 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Transcribed Spanish speech resources for AM training.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Corpus** & **S (K)** & **RW (K)** & **V (K)** \\ \hline OpenSubtitles [42] & 212,635 & 1,146,861 & 1576 \\ UFAL [43] & 92,873 & 910,728 & 2179 \\ Wikipedia [44] & 32,686 & 586,068 & 3373 \\ UN [45] & 11,196 & 343,594 & 381 \\ News Crawvl [46] & 7532 & 198,545 & 648 \\ Internal: & 4799 & 59,235 & 307 \\ entertainment & 1665 & 47,542 & 247 \\ el Geriatrico [48] & 2677 & 46,637 & 291 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of Spanish text resources for LM training. S = Sentences, RW = Running words, V = Vocabulary. Units are in thousands (K).