Resisting the Lure of the Skyline: Grounding Practices in Active Learning for Morphological Inflection (2024)

Abstract: Active learning (AL) aims to reduce the burden of annotation by selecting informative unannotated samples for model building. In this paper, we explore the importance of conscious experimental design in the language documentation and description setting, particularly the distribution of the unannotated sample pool. We focus on the task of morphological inflection using a Transformer model. We propose context motivated benchmarks: a baseline and skyline. The baseline describes the frequency weighted distribution encountered in natural speech. We simulate this using Wikipedia texts. The skyline defines the more common approach, uniform sampling from a large, balanced corpus (UniMorph, in our case), which often yields mixed results. We note the unrealistic nature of this unannotated pool. When these factors are considered, our results show a clear benefit to targeted sampling.

heading: 1 Introduction
text: Active learning (AL) (Cohn et al., 1996) is a data annotation approach, where the aim is to direct annotation effort at examples that are maximally helpful for model performance. Most active learning work in NLP involves pool-based active learning (McCallum et al., 1998) where a small seed training set is used to create an initial model, and additional examples are selected and annotated from a large pool of unannotated data. Several selection strategies exist, including confidencebased (Lewis, 1995; Cohn et al., 1996; Muradoglu and Hulden, 2022), diversity-based (Brinker, 2003; Sener and Savarese, 2018; Yuan et al., 2020) and committee-based approaches (Liere and Tadepalli, 1997; Farouk Abdel Hady and Schwenker, 2010); these approaches aim to outperform a uniform random selection baseline.
AL is often advocated as a method to rapidly improve model performance in low-resource settings (Baldridge and Palmer, 2009; Ambati, 2012;
considered, for the seed, LMC(LEMMA,MSD), LMC(WORDFORM, MSD) and TMC experiments. The maroon lines mark the Frequency Stratified (Baseline) accuracy, and the blue lines mark the Uniform sampling (Skyline) accuracy.
Grießhaber et al., 2020), where limited annotation capacity needs to be directed intelligently. Nevertheless, AL performance is inconsistent in practice and both success-stories and failures are reported in the literature (Settles et al., 2008; Baldridge and Palmer, 2009; Althammer et al., 2023), demonstrating that it is non-trivial to beat a uniform random selection baseline.
Language documentation is a natural application for active learning. Approximately half the world’s languages face the grim forecast of extinction, with
47
around 35–42% of these still substantially undocumented (Krauss, 1992; Wurm, 2001; Bianco, 2002; Crystal, 2002; Austin and Sallabank, 2011; Seifart et al., 2018). However, data for training automated systems is often limited, and additional annotation bears a high opportunity cost, limited not only by resources but also native speaker availability.
Simulated active learning The gold standard of active learning experiments for language documentation is the use of human annotators in a genuine low-resource setting, as in studies such as Baldridge and Palmer (2009). However, for practical reasons, most AL research uses simulated active learning, where a small seed training set is sampled from a large existing annotated dataset, and the remaining annotated examples represent the pool from which new examples are selected. While this approach allows for experimentation without costly manual annotations, it introduces a number of confounding factors which can complicate interpretation of results.
Baldridge and Palmer (2009) note that unit annotation cost is generally assumed in simulated active learning experiments, but this approach can be unrealistic when selection strategies tend to choose ambiguous examples that are harder, and therefore slower, to annotate. In a similar vein, Margatina and Aletras (2023) argue that in simulations, the unannotated pool tends to be carefully curated and preprocessed (as it is formed from an existing annotated training set). These pools often display unrealistic distributions of classes and lexical and structural diversity, which can be a highly inaccurate reflection of data in the wild, where noise, irrelevant examples and repetitions abound. To ensure validity of the results of simulated active learning experiments (particularly for low-resource settings), it is important to mimic a setting with limited lexical diversity and characteristic class imbalance, as is present in natural language datasets.
Active learning for morphology In this paper, we analyze pool-based active learning for language documentation, focusing on models for morphological inflection. We first argue that existing typelevel morphological resources (such as Unimorph, Batsuren et al. 2022) are a poor representation of a realistic unannotated pool in language documentation settings, unless some notion of lexical frequency is injected into the data. We then present experiments on morphological inflection, which demonstrate that the composition of the unanno-
tated pool is highly influential for performance in simulated active learning experiments.
We employ two selection criteria: transformer model confidence as previously investigated by Muradoglu and Hulden (2022) and a novel language model-based selection criterion. Given a carefully designed, frequency stratified, pool of unannotated examples mimicking naturalistic text, these methods can beat a uniform random baseline by a sizable margin. However, given a naïvely constructed, unannotated pool (based on the Unimorph database), neither of the methods confers an advantage over the baseline.

heading: 2 Data
text: We conduct experiments on the UniMorph database of inflection tables (Batsuren et al., 2022)1 on a typologically diverse set of eight languages: Arapaho (arp), Finnish (fin), Georgian (kat), Quechua (que), Sakha (sah), Turkish (tur), Tuvan (tyv) and Zulu (zul). Our choice of languages is motivated by a balance between morphological complexity, data availability (both UniMorph and Wikipedia) and endangerment classification according to UNESCO Atlas of the World’s Languages in Danger (Moseley, 2010). Where possible, we have attempted to maximise the diversity of our subject languages. Across 8 languages, 6 language families2. Further, three of the languages considered (sah, tyv and arp) are considered endangered. We exclusively include adjectives and nouns in our experiments.3 This simplifies analysis while still representing substantial morphological diversity as nouns make up a sizable portion of text cross-linguistically (Hudson, 1994; Liang and Liu, 2013).
To model word frequencies, we extract the Wikipedias for each language and form the intersection of word types present in UniMorph (U) and Wikipedia (W): U ∩ W. We also retain the much larger part of the UniMorph database U \ W, representing types not found in the Wikipedia. Data sampling is visualized in Figure 2.4 Our development and initial seed training set are formed by sampling (without replacement) 500 and 1,000
1Released under the CC BY-SA 3.0 license 2Uralic, Kartvelian, Turkic (South Siberian, North Siberia, Western Oghuz), Quechuan, Algonquian, Bantu. 3If these inflect identically, we combine them into a category of nominals. See Table 4 for details. 4All data and code will be made available at https://github.com/michaelpginn/ active-learning-for-morphology/. Code released under the MIT license.
forms, respectively, from U ∩ W (with their UniMorph lemmata and MSDs). For each language, we additionally supplement the seed training set with four complete inflection paradigms extracted from U \ W to ensure that all inflections are covered by the seed training data5. From the remaining types in U \ W, we then sample 1,000 for testing. Thus, we ensure that there is no overlap between the data splits.
Using each of the different active learning selection strategies presented below in Section 3, we sample an additional 500 training examples. For our baseline method, those are sampled from U ∩ W, while for all the other methods, additional data comes from U \ W.

heading: 3 Experimental Setup
text: We perform experiments on the word inflection task (Cotterell et al., 2016; Goldman et al., 2023) with datasets consisting of triplets ⟨lexeme, MSD, inflected form⟩, e.g. ⟨smile, V;PST, smiled⟩. Models are trained to predict the correct inflected form based on the lemma and MSD. We train transformer (Vaswani et al., 2017) inflection models
5In a language documentation setting, this information could be supplied by the linguist.
using fairseq (Ott et al., 2019).6 In all experiments, we apply data augmentation using the lemma-copy mechanism (Liu and Hulden, 2022). We initially train models on the seed training set and use various sampling strategies to select 500 additional examples from the unused pool, evaluating the change in inflection performance when training on the augmented set. The test and development sets, disjoint with all training data, remain unchanged through this process.
We experiment with the following strategies: Frequency Stratified (Baseline) We use word frequencies from Wikipedia to perform weighted random sampling from the pool U ∩ W. This method serves as a linguistically motivated, realistic baseline, accounting for the Zipfian nature of language, and approximating realistic lexical diversity and the naturalistic distribution of inflected forms.
Wiki Uniform We additionally report results on a baseline which samples from U ∩ W without frequency weighting.
Uniform sampling (Skyline) Our second baseline (which we call Skyline, as it is near-unbeatable) uses uniform sampling without word frequency information from U \ W. This setting is unrealistic in a language documentation setting—due to the lexical diversity and balanced class distribution of the samples, rare paradigm slots are over-represented.
Oracle Inspired by Muradoglu and Hulden (2022), we sample forms which the model fails to inflect correctly. Since this requires knowledge of gold standard forms, the method can only be used for comparison. This strategy mimics feedback from a linguist or language expert. In many cases, there are more than 500 incorrectly inflected forms to choose from. When this happens, we select maximally erroneous examples, that is, the examples with the greatest Levenshtein distance to the gold standard form.7 In contrast, when there are fewer than 500 incorrectly inflected forms, we augment the set using correctly inflected forms with the lowest confidence.
Transformer model confidence (TMC) Again following Muradoglu and Hulden (2022), we train an initial inflection model on the seed training set. We use this model to make predictions and select the examples with the lowest confidence scores.
6Our model and training hyperparameters follow Liu and Hulden (2020), described in Appendix A.
7This can be thought of as maximizing the informativity of the examples.
Language model confidence scores (LMC) We train two character-level language models (LM) over lemma+MSD and wordform+MSD sequences (respectively) from the seed training set. This means that our LMs return probabilities for sequences like walk+V+PAST and walked+V+PAST. We use the LMs to select examples with low probability or, equivalently, high negative log-likelihood (NLL).8 We experiment with using NLL from either the input lemma or the predicted inflected forms (not gold forms), and term these approaches LMC(LEMMA,MSD) and LMC(WORDFORM,MSD), respectively.

heading: 4 Results and Discussion
text: Table 1 reports the average change in accuracy from the seed models for each sampling strategy. The two benchmarks provide upper and lower limits for sample selection. The baseline underperforms on average, an expected result given the Zipfian nature of language. As the sampling strategy is dependent on natural texts, the samples have less diverse lemmas and MSDs. Meanwhile, the skyline outperforms every other strategy for five of the eight languages; again, this result is unsurprising, as the UniMorph database provides highly diverse examples. However, it is nearly impossible to replicate this approach, which treats all words equally regardless of rarity, in a realistic setting.
While the WIKI UNIFORM strategy shows greater average improvements than the baseline, the results across languages are mixed9. For example, while Finnish shows a 28.8% accuracy gain, performance on Quechua decreases by 0.05%.
8This approach is inspired by the observation that novel words are often inflected based on analogy to know words (Skousen, 1990; Derwing and Skousen, 1994; Prasada and Pinker, 1993). The LMC approach aims to seek out examples which are not represented by the seed training set.
9See Table 5 for details.
It is surprising that the oracle, intended to mimic a language expert, is outperformed by the either the TMC or LMC(WORDFORM, MSD) strategies for six of the languages considered. For almost all of the languages examined, the Levenshtein distance is the primary weighing factor10. The edit distance fails to consider the diversity of vocabulary or MSD. Compound words can also skew the Levenshtein distance significantly. For example, for the Turkish compound otomatik bilet makinası (“automatic ticket machine”), if the model does not capture the space between otomatik and bilet, though characters are merely shifted to the left, the Levenshtein distance is artificially high.

heading: 4.1 Edit Diversity
text: We seek to understand the effects of the various sampling strategies by estimating the relative edit diversity for each sample. For each dataset, we enumerate the edits (insertion, deletion, or replacement of subwords) needed to transform each lexeme to the inflected word. We collect edits of the same type and subword to give an edit distribution. Using this distribution, we compute entropy, which is higher for a distribution with a more diverse set of edits, and lower when the dataset is dominated by a few frequent edits. We provide the entropy, relative to the base training set, in Figure 3.
We observe that the strategies that sample from Wikipedia (which tend to be less successful) have lower entropy on average, while the Oracle, TMC, and skyline samples (which are more successful)
10Since there are more than 500 incorrect predictions for the remaining U \ W dataset. The only exception is Georgian, with < 500 incorrect predictions.
have higher entropy. We also find correlations between lower cross-entropy with the test set and better performance (see section 4.1.1).
The distinction between the naïve UniMorph pool and the frequency stratified sampling is mirrored in the language documentation and description (LDD) community with the elicitation or naturalistic speech debate. Chelliah (2001) notes that ‘language description based solely on textual data results in patchy and incomplete descriptions’. Similarly, Evans (2008) highlights the necessity of both linguistic phenomena targeting elicitation and observed communicative events11 (often narratives, conversations, etc.).

heading: 4.1.1 Cross-entropy and performance
text: We compute the cross-entropy between the test set edit distribution and each of the sampled sets. We find that across languages, increased crossentropy, which indicates that the sampled set is more dissimilar from the test set, tends to correlate with decreased performance. For example, Figure 4 plots the performance and cross entropy for the various sampling strategies for Turkish.
This is an intuitive result, confirming the importance of sampling a training set that is similar in distribution to the target test set. We run linear regression for each language and report the slopes and R2 values.
It is clear that in most cases, reducing crossentropy by choosing a sampling strategy that approximates the test distribution is beneficial to performance. However, since the test distribution is not necessarily known in real-world active learning scenarios, this remains a difficult task to solve.
11Himmelmann (1998) distinguishes these categories further, with a third ‘Staged communicative events’. This refers to tasks that are prompted for linguistic purposes, such as a picture task.

heading: 5 Conclusion
text: Computational methods can aid language documentation and description projects by processing and analyzing recorded data. Active learning approaches can greatly aid in the rapid development of robust automated systems by focusing annotation on highly beneficial samples, but existing research on simulated AL often makes unrealistic assumptions. We compare a standard approach (skyline), where data is sampled from unrealistic linguistic resources, an approach based on naturalistic word frequencies (baseline), and a number of strategies motivated by encouraging lexical diversity. Our skyline and baseline approaches serve as analogs to elicitation and naturalistic recording.
We find that the skyline approach is difficult to beat, but as few languages have sufficient corpora with complete, diverse paradigms, we argue this approach is an unrealistic baseline for AL. Meanwhile, we find clear benefits from targeted sampling strategies, with inflection model confidence (TMC) and character LM scores (LMC(WORDFORM, MSD)) yielding the greatest improvements.

heading: 6 Limitations
text: Three of our eight languages are members of the Turkic language family. Despite our best efforts, it was not possible to have a set of languages that covered a significant range of typological features, particularly pertaining to phonology and morphology. In most cases, either the existing Wikipedia was too small or there were issues with orthography that did not map neatly with the UniMorph database. This is a limitation of the study presented
and remains an intended future rectification for the authors.
It is important to note that the style and register of Wikipedia is limited. As such, certain MSDs are underrepresented or over-represented, compared with natural speech. Our experiments use Wikipedia articles to simulate texts/recordings of language, a limited approximation of the natural setting that does not cover a broad range of genres. However, constructing a representative corpus in the language documentation context is an almost impossible endeavour.

heading: 7 Ethics Statement
text: If our results do not hold across a wide variety of languages, our suggested AL approaches may result in annotator effort that is not beneficial to the model. This would be a significant opportunity cost, particularly in the case of languages which are considered critically endangered.
Automated systems for inflection and language documentation are limited in scope and carry some degree of error. While they can greatly aid in documentation projects, they should not be used to entirely replace human annotators and linguists in the documentation, study, and preservation of languages. Particularly for Indigenous and endangered languages, care should be taken to use data and automated systems in a way consistent with the desires of the language community (Schwartz, 2022).
Finally, training models carries an unavoidable environmental cost (Bender et al., 2021). While our research uses small models, we strive to ensure the benefits outweigh these costs.

heading: A Model details
text: Across preliminary experiments and the runs listed in this paper, training took around 1,500 compute hours. We ran experiments on the UBC computing cluster and Google Colab. Models were trained with the hyperparameters listed in Table 3. Models had around 10M parameters.

heading: B Data Composition
text: Information about the composition for each language is given in Table 4.

heading: C Language-Specific Model Accuracies
text: Accuracy scores are reported in Table 5

heading: Future Work
text: This is a limitation of the study presented
and remains an intended future rectification for the authors. It is important to note that the style and register of Wikipedia is limited. As such, certain MSDs are underrepresented or over-represented, compared with natural speech. Our experiments use Wikipedia articles to simulate texts/recordings of language, a limited approximation of the natural setting that does not cover a broad range of genres. However, constructing a representative corpus in the language documentation context is an almost impossible endeavour