## 5 Validation of the Basque Parliament Database through ASR
Figure 5: CER and WER performance, disaggregated per language, on the dev and eval sets when using the Basque Parliament language model. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{**CER**} & \multicolumn{4}{c}{**WER**} \\ \hline
**Set** & **LM** & **eu** & **es** & **bi** & **all** & **eu** & **es** & **bi** & **all** \\ \hline  & no & 1.47 & 1.26 & 1.52 & 1.33 & 4.90 & 3.19 & 3.99 & 3.66 \\ dev & wiki & 1.52 & 1.25 & 1.54 & 1.34 & 4.92 & 2.90 & 3.99 & 3.46 \\  & bp & 1.50 & 1.22 & 1.52 & 1.31 & 4.80 & 2.74 & 3.56 & 3.30 \\ \hline  & no & 1.46 & 1.14 & 1.50 & 1.24 & 4.60 & 2.84 & 3.76 & 3.26 \\ eval & wiki & 1.55 & 1.14 & 1.53 & 1.26 & 4.80 & 2.60 & 3.42 & 3.11 \\  & bp & 1.46 & 1.10 & 1.42 & 1.20 & 4.35 & 2.47 & 3.08 & 2.90 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: CER and WER performance, disaggregated per language, obtained on the dev and eval sets using acoustic models trained on the train-clean dataset and three configurations: not using any language model (no-LM); using a bilingual 3-gram language model estimated on out-of-domain texts: Wikipedia pages in Spanish and Basque (wiki-LM) and using a bilingual 4-gram language model estimated on the Basque Parliament train transcriptions (bp-LM). ASR performance is good even when no language model is used (3.26 WER on the eval set), which means that the acoustic models match the background and channel conditions, speakers, etc., of the test set. This is an important result because the main motivation for creating the Basque Parliament database was training in-domain acoustic models and building a competitive ASR system, able to produce highly reliable transcriptions. The integration of lexical and language models in the search for the optimal sequence of graphemes/words does not always improve performance. For instance, the use of generic lexical and language models extracted from Wikipedia (wiki-LM) leads to worse results in terms of CER, although it does generally lead to better performance in terms of WER. On the other hand, the Basque Parliament lexical and language models (bp-LM) do always help, both in terms of CER and WER, which underlines the importance of in-domain lexical and language models. The best overall WER on the eval set (2.90) represents a 28% relative WER reduction with regard to our previous best result (4.02). A language-wise analysis shows that the best performance is found for Spanish (2.47 WER on the eval set) and the worst performance for Basque (4.35 WER on the eval set), meaning 32% and 15% relative WER reductions with regard to our previous best results for Spanish (3.66 WER) and Basque (5.13 WER), respectively. It seems that the imbalance between Spanish and Basque segments in the training set could explain the difference in performance between the two languages. Finally, the good performance attained on bilingual segments (3.08 WER on the eval set) is also remarkable, with some of them featuring code-switching events. In fact, exploring the potential of integrating acoustic and language models into a single bilingual ASR system for other pairs of code-switching languages is an interesting line for future research.