An (unhelpful) guide to selecting the right ASR architecture for your under-resourced language
Year: 2023

Abstract:
Advances in deep neural models for automatic speech recognition (ASR) have yielded dramatic improvements in ASR quality for resource-rich languages, with English ASR now achieving word error rates comparable to that of human transcribers. The vast majority of the world’s languages, however, lack the quantity of data necessary to approach this level of accuracy. In this paper we use four of the most popular ASR toolkits to train ASR models for eleven languages with limited ASR training resources: eleven widely spoken languages of Africa, Asia, and South America, one endangered language of Central America, and three critically endangered languages of North America. We find that no single architecture consistently outperforms any other. These differences in performance so far do not appear to be related to any particular feature of the datasets or characteristics of the languages. These findings have important implications for future research in ASR for under-resourced languages. ASR systems for languages with abundant existing media and available speakers may derive the most benefit simply by collecting large amounts of additional acoustic and textual training data. Communities using ASR to support endangered language documentation efforts, who cannot easily collect more data, might instead focus on exploring multiple architectures and hyperparameterizations to optimize performance within the constraints of their available data and resources.

heading: 1 Introduction
text: The majority of significant academic and industry research on automatic speech recognition (ASR) (Povey et al., 2011; Hinton et al., 2012; Amodei et al., 2016; Watanabe et al., 2018; Baevski et al., 2020) has been evaluated on a small set of English language datasets (Panayotov et al., 2015; Godfrey et al., 1992). Word error rates (WER) for English ASR now approach those of human transcriptionists (Baevski et al., 2020; Radford et al., 2022), and
speakers of English can now reliably use ASR for text entry when using mobile devices. This level of accuracy, however, is attainable only for the handful of the world’s 7000 languages that, like English, have abundant training resources.
Most of the world’s languages, even ones spoken by tens of millions of speakers, currently lack datasets prepared specifically for training ASR models. The datasets that do exist are typically much smaller than English ASR datasets that have been available for decades, with no more than a few dozen hours of acoustic training data. As the Common Voice project (Ardila et al., 2020) has shown, collecting large amounts of data for widely spoken languages is possible, but using this kind of platform is likely to be impractical for the roughly 40% of the world’s languages that are endangered (Eberhard et al., 2022). A similar percentage of languages – again, even many that are widely spoken – lack an established writing system, which presents other obstacles to building large ASR corpora.
Fortunately, existing methods for training accurate ASR models for English and other highresource languages can be adapted to low-resource settings. Some toolkits include recipes for smaller datasets that require the training of fewer parameters. Other approaches rely on fine-tuning acoustic models pre-trained on massive multilingual speech datasets. Most recent work using these approaches, however, does not compare the performance of multiple competitive architectures across multiple diverse small ASR datasets. Thus, while we have access to transformative technology that can be harnessed to build reasonable models for languages with limited resources, we do not know which of the popular architectures is “better” or whether features of a particular dataset or language might make one architecture more suitable than another.
In this paper we explore four different popular ASR architectures, three of which are currently considered state of the art, that can be used even in
1008
low-resource settings: a hybrid DNN (Veselỳ et al., 2013); two approaches for fine-tuning from a multilingual pre-trained acoustic model (Conneau et al., 2020; Radford et al., 2022); and an end-to-end approach designed specifically for small datasets (Shi et al., 2021). We train models for eleven datasets for under-resourced languages, which are diverse in their linguistic properties, mechanisms for collection, relative sizes, and recording quality.
We find that no single approach to training ASR models in low-resource settings consistently outperforms any other, with the most outdated method turning out to be the most accurate surprisingly often. While unsatisfying in some ways, these results can help guide ASR researchers and language community members to select the architecture that is most compatible with their objectives and that can be feasibly supported with their available financial and personnel resources. For widely spoken
languages, where the goal of developing an ASR system is likely to be to support a voice-based app or a personal digital assistant, the best use of financial resources might be to collect large amounts of additional data in order to take advantage of stateof-the-art high-resource architectures. Linguists and members of endangered language communities hoping to use ASR to document and preserve their language cannot easily gather more data, and thus might see more benefit from carefully experimenting with multiple architectures to identify the approach that provides the best results for their particular language or existing dataset.

heading: 2 Related Work
text: Although most of the notable advances in ASR have focused on English and a few other languages with abundant data, there has been substantial inter-
est in ASR for languages with minimal training resources for quite some time (Besacier et al., 2014). Much of the work from the 2010s focused on the languages of the IARPA Babel project (Thomas et al., 2013; Miao et al., 2013; Cui et al., 2014; Grézl et al., 2014). Research initiated with the Babel datasets on methods of transfer learning and data augmentation in low-resource settings has continued apace (Khare et al., 2021; Vanderreydt et al., 2022; Guillaume et al., 2022b). With the success of the Kaldi toolkit, researchers began to collect and freely distribute their own Kaldi-ready datasets for under-resourced and endangered languages, several of which are explored in this paper (Gauthier et al., 2016; Laleye et al., 2016; Gelas et al., 2012; Juan et al., 2015; Pulugundla et al., 2018). More recent work has explored training monolingual endto-end models with substantially larger datasets than those used here (Shi et al., 2021), as well as transfer learning and fine-tuning from pretrained multilingual (Guillaume et al., 2022a; Sikasote and Anastasopoulos, 2022) or English models (Thai et al., 2020).

heading: 3 Datasets
text: Five of the datasets explored here are freely available datasets built by researchers, sometimes in collaboration with speech communities, specifically
for training ASR models for widely spoken but under-resourced languages of the global South: Bemba (Sikasote and Anastasopoulos, 2022), Fongbe (Laleye et al., 2016), Wolof (Gauthier et al., 2016), Swahili (Gelas et al., 2012), and Iban (Juan et al., 2014, 2015). Three datasets (Quechua, Bribri, Guarani) were created from existing recordings for the 2022 AmericasNLP Workshop Shared Task 1. The remaining datasets for three endangered languages of North America (Hupa, Oneida, and Seneca) were created using existing linguistic and community fieldwork recordings available to the authors through the affiliation of one of the authors with one of these communities and the generosity of the community elders.
While nearly any recorded speech can be transcribed and used to train an ASR system, a common approach for building a new ASR dataset is to ask speakers of the language to read aloud provided texts, which obviates the laborious task of transcription. With this strategy, speakers are often recorded in a studio or similarly controlled environment, resulting in more consistent recording quality. Alternatively, datasets can be created from existing audio data such as radio broadcasts or linguistic fieldwork recordings. Such recordings are often already transcribed but need to be segmented and
1http://turing.iimas.unam.mx/americasnlp/2022_st.html
time-aligned with the transcripts, which must often be done by hand. Table 1 provides details about these sorts of characteristics of the datasets, as well as information about the quantity of the training data for the acoustic and language models.
Information about the linguistic characteristics of the eleven languages is provided in Table 2. Seven of these languages are widely spoken by millions of people, and some have institutional or government recognition; one is endangered with around 7,000 speakers; and three are critically endangered with very few (perhaps only one, in the case of Hupa) first-language speakers and no more than a hundred second language learners. A diverse set of morphological, phonological, and phonetic features and properties are represented among these languages, and we note that they are all quite different typologically from most high-resource languages, including not only English and Chinese but also the major European languages.

heading: 4 ASR Architectures
text: The goal of this work is to explore whether any one of several popular and state-of-the-art ASR architectures is especially well suited for building models with small amounts of training data. We train models on the the eleven datasets described in Section 3 using four different architectures:
• A hybrid DNN (Veselỳ et al., 2013) implemented within the Kaldi toolkit (Povey et al., 2011), following Karel’s DNN recipe2 which uses a variety of feature optimizations including RMB pretraining, frame cross-entropy training, and MBR sequence-discriminative training. Decoding was performed with a trigram language model.
• A transducer-based end-to-end model for small datasets within ESPnet2 (Watanabe et al., 2018), following the recipe for Yoloxochitl Mixtec (Shi et al., 2021).
• Fine-tuning from a multilingual acoustic model using Wav2Vec2 XLSR-53 (Conneau et al., 2020), decoding both with and without a trigram language model and using the parameterizations specified in the Hugging Face Wav2Vec XLSR-53 tutorial.3
2https://kaldi-asr.org/doc/dnn1.html 3https://huggingface.co/blog/fine-tune-xlsr-wav2vec2
• Fine-tuning from the medium multilingual acoustic model with Whisper (Radford et al., 2022), using the parameterizations specified in the Hugging Face Whisper tutorial.4
Training and testing were carried out on a university high-performance computing cluster. Training times ranged between 2 and 24 hours depending on the architecture and dataset.

heading: 5 Results
text: Figure 1 shows the word error rates (WER) for four of the five approaches (Kaldi DNN, Wav2Vec XLSR with and without a language model (LM), and Whisper) when trained and tested on each of the eleven datasets. Note that prior baselines reported in the papers associated with the datasets for Wolof, Swahili, Fongbe, Hupa, and Iban, using non-s.o.t.a. architectures, and Bemba, using a slightly different configuration of Wav2Vec XLSR, are lower than the best reported architecture here. No prior WER results have been reported for the Oneida, Quechua, Bribri, and Guarani datasets.
We observe a large variation in WER across languages, which should not be surprising given the great variability in the quantity of training data, the type and audio quality of data collected, and the linguistic features of these languages. Datasets of less than 3 hours had consistently high WERs, but across the other datasets, there does not appear to be a clear relationship between amount of audio training data and WER. Though not shown in Figure 1, ESPnet yielded the worst performance by far for all languages, with only Wolof, the second largest dataset, achieving a WER below 65%. Again, this is not surprising given that this ESPnet recipe (Shi et al., 2021) was proposed for a much larger 60-hour indigenous language dataset.
More interestingly, we see no consistent ranking of the remaining four approaches across the eleven datasets. Using an LM during decoding with Wav2Vec XLSR always yields some improvement in WER over not using an LM, but the differences are often quite small. Notably, Swahili, which has the largest LM, sees only a tiny reduction in WER when that LM is used during decoding. The Kaldi hybrid DNN, despite being outdated, outperforms more than one of its state of the art rivals for Seneca, Fongbe, Iban, and Quechua. Whisper is dramatically better than other models for Wolof and Hupa,
4https://huggingface.co/blog/fine-tune-whisper
but substantially worse for Fongbe and Quechua. Though closely related and typologically similar, Seneca and Oneida show very different patterns, as do Fongbe and Wolof, two related languages with datasets recorded under similar conditions. The WER for Swahili is relatively stable across architectures, while WER is quite variable for Wolof, Hupa, Fongbe, and Oneida.
The rankings do not appear to be related to the method of speech collection (read vs. spontaneous) or the consistency of audio quality. In addition, whether or not a language is tonal, like Bemba, Fongbe, and Bribri, does not appear to predict the relative rankings of the four architectures.
We do note, however, two potential patterns, which merit further investigation with a larger set of languages. First, Fongbe, the only language of the eleven with isolating morphology (i.e., limited affixation) is one of only two languages where Whisper yielded the highest WER of the four systems. Second, the three languages with the largest phonesets, Wolof, Swahili, and Hupa, yielded the same relative ranking, with Whisper performing the best and Kaldi the worst. Although there is certainly not enough information here to draw conclusions, it is plausible that the design of a particular training architecture or the content of the pretrained models could render a system more appropriate for a language with a particular linguistic property.

heading: 6 Conclusions
text: Under-resourced language communities, whether large or small, need to know how to invest their lim-
ited resources when developing an ASR system for their language. Our findings suggest, unfortunately, that there are no obvious or simple guidelines to follow. Our future work will expand the set of languages explored here in order to establish connections between expected model performance and linguistic features and dataset characteristics. We also plan to explore the impact of language model size and domain on ASR accuracy and the relationship between language model and morphology.
Limitations
One limitation of this work is that we have included results for only eleven languages. Training ASR models, even on small datasets, requires significant computing and financial resources. Second, there are not that many freely available and well prepared ASR datasets that are readily compatible with all four ASR architectures. We sought to select a diverse set of languages and datasets with varying features in order to provide, we hope, a reasonable snapshot of how the state of the art performs in low-resource settings.
Ethics Statement
The Hupa, Oneida, and Seneca datasets were recorded with the approval of participating universities’ IRBs and with the enthusiastic cooperation of the elders and other linguistic consultants. The datasets for the remaining languages were downloaded from public Web pages. The Bribri dataset, like those of other endangered languages,
was created using linguistic fieldwork recordings. Of the others, some were collected by recruiting participants to read text (Wolof, Fongbe, Bemba, Guarani); others consist of transcribed radio and television broadcasts (Iban, Quechua); and the Swahili dataset includes both types of data. While the participants who provided recordings by reading text presumably gave consent for their voices to be used for ASR research, it is unlikely that speakers recorded in the course of a radio or television broadcast provided consent explicitly for their voices to be used in an ASR dataset. We expect, however, given that members of the speech community participated in these data collection projects, that ethical concerns were carefully considered.

heading: Acknowledgements
text: We are grateful for the continued support from the Hupa and Seneca indigenous communities. We would like to especially thank Mrs. Verdena Parker, of the Hoopa Valley Tribe, and Mrs. Sandy Dowdy, of the Seneca Nation of Indians, for their generous and valuable input and support. This material is based upon work supported by the National Science Foundation under Grant #2127309 to the Computing Research Association and Grant #1761562. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the Computing Research Association.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Second to last section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics section, last section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?

heading: If by "artifacts" you mean "datasets", then yes, they are all cited when they are first mentioned.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: In the Ethics section we mention that we downloaded some datasets that are publicly available. We also discuss the artifacts that we used that are not publicly available but were shared by indigenous communities with the authors.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics section.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Our own data from indigenous communities was collected under our IRBs. The other data was downloaded from OpenSLR. We explain in the Ethics section that we assume that data was collected ethically but we cannot confirm it ourselves.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Extensively in the data section and ethics sections of our paper.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4, I think.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Vaguely in section 4.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4, 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? We use speech data that was collected and transcribed as part of earlier projects, some by us and some by other groups.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Ethics section

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Ethics

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Not applicable. Left blank.

heading: Future Work
text: Our future work will expand the set of languages explored here in order to establish connections between expected model performance and linguistic features and dataset characteristics. We also plan to explore the impact of language model size and domain on ASR accuracy and the relationship between language model and morphology. Limitations
One limitation of this work is that we have included results for only eleven languages. Training ASR models, even on small datasets, requires significant computing and financial resources. Second, there are not that many freely available and well prepared ASR datasets that are readily compatible with all four ASR architectures. We sought to select a diverse set of languages and datasets with varying features in order to provide, we hope, a reasonable snapshot of how the state of the art performs in low-resource settings