How Important is a Language Model for Low-resource ASR? (2024)

Abstract: N-gram language models (LMs) are the innovation that first made large-vocabulary continuous automatic speech recognition (ASR) viable. With neural end-to-end ASR architectures, however, LMs have become an afterthought. While the effect on accuracy may be negligible for English and Mandarin, jettisoning the LM might not make sense for the world’s remaining 6000+ languages. In this paper, we investigate the role of the LM in low-resource ASR. First we ask: does using an n-gram LM in decoding in neural architectures help ASR performance? While it may seem obvious that it should, its absence in most implementations suggests otherwise. Second, we ask: when an n-gram LM is used in ASR, is there a relationship between the size of the LM and ASR accuracy? We have discovered that gut feelings on this question vary considerably, but there is little empirical work to support any particular claim. We explore these questions “in the wild” using a deliberately diverse set of 9 very small ASR corpora. The results show that: (1) decoding with an n-gram LM, regardless of its size, leads to lower word error rates; and (2) increasing the size of the LM appears to yield improvements only when the audio corpus itself is already relatively large. This suggests that collecting additional LM training text may benefit widely-spoken languages which typically have larger audio corpora. In contrast, for endangered languages where data of any kind will always be limited, efforts may be better spent collecting additional transcribed audio.

heading: 1 Introduction
text: Including an n-gram language model (LM) during decoding within an automatic speech recognition (ASR) system is a long-established critical practice (Lucassen and Mercer, 1984; Jelinek, 1976; Bahl et al., 1983, 1989). This has motivated appreciable research exploring different ways to build or augment LMs in order to improve ASR accu-
racy (Celebi et al., 2012; Bellegarda, 2004; Sagae et al., 2012; Sheikh et al., 2024).
With recent advances in end-to-end neural ASR, the role of the LM has diminished somewhat dramatically (Hannun et al., 2014; Watanabe et al., 2018), with the LM becoming largely optional. Using an LM in decoding is often challenging within many state-of-the-art deep learning ASR toolkits, where it is typically a poorly documented command-line option or a feature that users themselves must implement (Conneau et al., 2020b).
A number of studies have incidentally shown that decoding with an LM within s.o.t.a. neural architectures yields competitive or superior performance (Wang et al., 2022; Liu et al., 2019; Baevski and Mohamed, 2020; Meng et al., 2021; Javed et al., 2022), but the impact of the LM in these papers is discussed briefly, if at all. Additionally, the languages in these studies – typically English – are not at all constrained by data availability. Recent work by Javed et al. (2022) on 40 Indian languages with more limited resources shows a consistent improvement in accuracy when using an LM, but the corpora are still enviably large, with over 17K hours of audio and billions of words.
Two recent studies probe the effect of LM size on ASR performance in more genuinely underresourced settings. Sikasote and Anastasopoulos (2022a) report that a larger LM actually yields mildly worse performance for a 19-hour Bemba corpus. Liu et al. (2023b) explore the effect of LM size in six small corpora, finding a lack of word error rate (WER) improvement when using larger LMs, even in simulated low-resource settings.
The contradictory results in this prior work raise two questions. First, is an n-gram LM still a valuable component in ASR when resources are limited? Second, if using an LM does make a difference, how much additional text data beyond the transcripts of the audio training data should be used to train the LM – or in other words, how large
206
should the LM be? Training on additional texts could decrease the out-of-vocabulary (OOV) rate and reduce the overall sparsity of the LM, but the impact of the LM and its size on ASR accuracy remains strangely unexplored with few studies addressing the issue directly or thoroughly (Sikasote and Anastasopoulos, 2022a; Liu et al., 2023b).
This study aims to provide empirical evidence to answer these research questions. We explore the impact of decoding with LMs of different sizes within two ASR architectures on a diverse set of small corpora. We very deliberately choose to work with many corpora of varying sizes, recording quality, and linguistic properties rather than simulating a range of sizes using a single corpus. We conduct further regression analyses to explore for each language whether ASR performance is impacted by LM size and other factors of interest.
Addressing these questions has important implications for creating ASR datasets for underresourced languages. For many indigenous and endangered languages, additional texts for training larger LMs often come from a domain (e.g., the Bible) that is drastically different from that of the audio (e.g., linguistic fieldwork). This sort of domain mismatch might yield richer LMs that poorly model the target data. An improved understanding of the impact of n-gram LMs on ASR accuracy in such settings can help language communities choose how to allocate their limited resources for building more robust ASR systems.

heading: 2 Related work
text: There has been continuous progress over the past decade in advancing ASR technologies for lowresource languages (Thomas et al., 2013; Cui et al., 2014; Shi et al., 2021). Some of these languages, such as Wolof (Gauthier et al., 2016) and Swahili (Gelas et al., 2012), have large speaker populations; with reasonable time and financial support, more data is obtainable. For others, especially indigenous and endangered languages, it is difficult to collect additional data, but ASR can facilitate the creation of this data (Shi et al., 2021; Prud’hommeaux et al., 2021; Bartelds et al., 2023; Le Ferrand et al., 2023).
Our work goes beyond prior work that examines the impact of n-gram LMs on ASR performance (Sikasote and Anastasopoulos, 2022a; Javed et al., 2022; Liu et al., 2023b). First, we study nine languages from six language families, covering a
diverse set of typological properties. Second, we compare two ASR architectures (Section 3.3) for each language. This experimental design allows us to see whether the impact of LM size is consistent across both languages and models.

heading: 3 Experiments
text: It is not our goal to find the best performing ASR architecture or parameterizations. We are also not comparing WER across languages, a meaningless exercise given the range of languages and corpus size. Instead, our goal is to examine, for each language, the effect of (1) decoding with an LM in an end-to-end system, and (2) increasing the size of the LM, in order to see whether similar qualitative trends hold across these nine diverse languages.

heading: 3.1 Data sources
text: We use ASR datasets from nine typologically diverse languages, spanning six language families (see Table 1). Among these languages, Bemba (Sikasote and Anastasopoulos, 2022b), Wolof (Gauthier et al., 2016), Swahili (Gelas et al., 2012), Fongbe (Laleye et al., 2016) (Niger-Congo), Iban (Juan et al., 2014, 2015) (Austronesian) are widely-spoken under-resourced languages (Liu et al., 2022). The datasets for these five languages are publicly available and include additional texts. The dataset for Quechua is taken from the 2022 AmericasNLP Workshop Shared Task1. Additional texts are sourced from Agić and Vulić (2019); Ortiz Suárez et al. (2019); Conneau et al. (2020b); Wenzek et al. (2020), and Zevallos and Bel (2023). Hupa (Dene/Athabaskan) is a critically endangered language of North America with audio data derived from fieldwork and additional texts from a published grammar (Goddard, 1904). The corpus for Bininj Kunwo, an Arnhem language spoken in the Northern Territory of Australia, and that for Kréyol Gwadloupéyen, a French-based creole spoken on Guadeloupe (Glaude, 2013), consist of fieldwork recordings with additional texts from the Bible. The last three corpora were shared with us by the linguists who collected the data and are not currently publicly available.

heading: 3.2 LM settings
text: We explore three LM training settings, which we refer to as NO_LM, LM_BASE, and LM_LARGE.
1turing.iimas.unam.mx/americasnlp/2022_st.html
NO_LM means that a LM is not included in decoding. Both LM_BASE and LM_LARGE are trigram LMs; the former is trained on the transcripts of the acoustic training data, whereas the latter includes those transcripts plus all the additional text data.

heading: 3.3 ASR frameworks
text: We experiment with two ASR architectures (see also Appendix B). Kaldi DNN is a hybrid fully connected DNN implemented with the Kaldi toolkit (Panayotov et al., 2015), following the default sequence training parameters from Karel’s recipe2. Prior work shows reasonable performance from this DNN architecture in low-resource settings (Morris et al., 2022). Training and decoding within Kaldi requires an LM. For Wav2Vec2, we fine-tune from the pre-trained Wav2Vec XLSR-53 multilingual model (Conneau et al., 2020a), built upon the Wav2Vec 2.0 framework (Baevski et al., 2020) (see Table 4, Appendix A for training parameters). The test data is decoded without an LM and, using CTC decoding, with the two trigram LMs.3
For evaluation, we use random splits (Liu et al., 2023a), which yield more reliable estimates of performance that the “held-out speaker” approach. We randomly divide the dataset into 80:20 training and test sets, three times. We report the WER score averaged over the three splits.

heading: 3.4 Regression analysis
text: To further validate the effect of LM size on WER, we apply regression modeling. For each language, we first calculate the WER of every LM-decoded test utterance, which serves as the outcome variable in the regression. The variable of interest is the LM size, the number of tokens in the LM
2https://kaldi-asr.org/doc/dnn1.html 3Grid search for the α and β LM parameters yields no
appreciable WER improvement (Table 5 in Appendix C).
training corpus. We include perplexity (PPL) and out-of-vocabulary (OOV) rate as control variables. All fixed-effects have interactions with each other. Given that different LM training settings appear to influence the performance of Kaldi and Wav2Vec2 differently (Section 4; Table 2), we fit separate regression models for each. Recall that with the corpora we have, it is not our goal, nor is it appropriate, to compare WER across languages. Rather, we investigate whether the impact of LM size is consistently observed for every language. To that end, we employ a separate regression model for each language-architecture combination. (Every resulting dataset has a reasonable size, ranging from 1,290 for Kréyol to 15,510 for Bemba given each acoustic model.) This setup also means that the coefficient values for the same factors are not comparable across languages. In cases where speaker (or session, for Swahili) information is provided, we include the speaker/session of the utterance as a random effect, leading to a mixed-effects regression structure; otherwise a linear regression model is applied (Hupa, Quechua, Kréyol, and Kunwok).

heading: 4 Results
text: The WER results are presented in Table 2 (see also Table 5, Appendix C). First, it seems that having an LM, regardless of its size, leads to better performance. All WER results with LM_BASE and LM_LARGE are lower than those derived from NO_LM. Secondly, and counterintuitively for some readers, a larger LM does not increase accuracy for all languages. For Bemba and Swahili, a larger LM yields moderate improvement, but in other cases, regardless of the architecture, the WER for LM_LARGE and LM_BASE are comparable, or the WER for LM_LARGE is slightly worse.
Results from the regression analysis lend support
to these observations. When using Kaldi (Table 6 in Appendix D), there is a significant positive coefficient for LM size for all languages, indicating that a larger LM leads to a higher (worse) WER. These findings largely align with Table 2. Why, then, are the average WERs lower for LM_LARGE for Bemba and Swahili under Kaldi? The regression results suggest a relationship between LM size and WER when utterance perplexity and OOV rate are controlled for. Regression results for Wav2Vec2 resonate with patterns from Table 2 as well: while there is a significant negative coefficient for LM size for Bemba and Wolof, we fail to see the same pattern for other languages.
Why then does having a larger LM not improve model performance? We offer a few conjectures. First, we note in Table 1 that the LM size ratio for some languages is quite small (e.g., Hupa: 6.63; Kréyol: 2.75), while for Bemba, Swahili, Iban and Kunwok, where a larger LM does reduce WER, LM_LARGE is proportionally much higher than LM_BASE. The only exception is Quechua, whose LM size ratio ranks third but the WERs between the two settings are roughly the same.
This brings to our second conjecture: the domain mismatch between the transcribed audio sources and the additional texts may decrease ASR accuracy with LM_LARGE. For Quechua, Hupa, Kréyol, and Kunwok, the test data comes from everyday speech, whereas the additional texts are technical or religious texts. To explore this idea, we approximate domain similarity by measuring the average perplexity of the transcripts of the acoustic test data under the two LMs (the vocabularies for the two LMs were kept the same). As shown in Table 3, the perplexity scores for LM_LARGE are higher across languages (with exceptions for Wolof and Quechua) to different extents, suggesting that domain mismatch may play a role in the lack of WER improvement with LM_LARGE.
Finally, we consider the OOV rates of the acoustic test data. OOV rate will logically decrease with a larger LM, which could in turn yield a lower WER score. This seems true for Bemba, Swahili, Iban and Kunwok. For Wolof and Fongbe, LM_LARGE does not reduce the OOV rate much. On the other hand, for Quechua, Hupa, and Kréyol, the larger LM does not contribute to improved accuracy despite the lower OOV rate. One noticeable feature of these three languages is that they have relatively little audio training data available, which possibly subsumes the potential effect of LM size.

heading: 5 Conclusions and Future Work
text: Our results over nine languages, corroborated with regression analysis, show that: (1) decoding with an LM yields consistently lower WER; (2) a larger LM infrequently improves ASR accuracy. We propose that in order for a larger LM to be helpful, it needs to: (1) be much larger relative to the size of the transcripts of the audio; (2) result in lower OOV rates for the audio test data; (3) be coupled with sufficient audio training data.
We hope these results can be taken as guidance for ASR dataset creation strategies for low-resource languages. For widely-spoken languages, where additional text data is often readily available, resources might be best directed toward gathering that data. For indigenous and endangered languages, limited resources might be better spent ethically increasing the amount of audio data.
In addition to experimenting with more languages, particularly endangered languages, we see designing informative data selection and data augmentation methods for LM building as a fruitful
avenue for future work. This in turn can perhaps mitigate the issues of (audio) data limitations particularly for endangered languages.

heading: 6 Limitations
text: We would like to acknowledge two main limitations of our work: the number of languages and the number of model architectures studied. Given that our experiments target low-resource languages, this is naturally constrained by the lack of ASR dataset availability for languages as such. One possible solution is to create artificial low-resource scenarios from large datasets for languages with abundant training data in order to expand the language diversity covered in this work.
In terms of model architectures, here we explore Kaldi and Wav2Vec2 XLSR-53, two frameworks that have been shown empirically to work well in settings with limited training data. That said, it would be worthwhile in the future to employ other end-to-end systems (Shi et al., 2021) in order to test their limits in building language technology for low-resource languages.

heading: 7 Ethics Statement
text: We include a total of 10 datasets covering nine languages in our study. Of these datasets, six are publicly available. The data for Hupa, Kréyol, and Kunwok are developed in-house through academic relations with the respective speech communities of these languages; applications of these datasets are carefully considered via personal connections with elders and researchers from the communities.

heading: B Details of Computing Infrastructures
text: All models are trained on research computing clusters. For Kaldi, each model is trained on a single Nvidia A100 GPU with 8GB of memory, and Wav2Vec with 16GB of memory.

heading: C Full WER Results
text: The WER results from Wav2Vec2 with grid search are presented in Table 5.

heading: D Regression Results
text: Regression results given different model architectures for our variable of interest, LM size (the number of tokens used to train the language model), and the two control variables, the perplexity (PPL) and OOV are presented in Table 6 (Kaldi) and Table 7 (Wav2Vec2).

heading: Future Work
text: That said, it would be worthwhile in the future to employ other end-to-end systems (Shi et al. , 2021) in order to test their limits in building language technology for low-resource languages