Morphological Inflection: A Reality Check
Year: 2023

Abstract:
Morphological inflection is a popular task in sub-word NLP with both practical and cognitive applications. For years now, state-of-theart systems have reported high, but also highly variable, performance across data sets and languages. We investigate the causes of this high performance and high variability; we find several aspects of data set creation and evaluation which systematically inflate performance and obfuscate differences between languages. To improve generalizability and reliability of results, we propose new data sampling and evaluation strategies that better reflect likely usecases. Using these new strategies, we make new observations on the generalization abilities of current inflection systems.

heading: 1 Introduction
text: Morphological inflection is a task with widereaching applications in NLP, linguistics, and cognitive science. As the reverse of lemmatization, it is a critical part of natural language generation, particularly for languages with elaborate morphological systems (Bender, 2009; Oflazer and Saraçlar, 2018). Since morphological inflection is a particular type of well-defined regular string-to-string mapping problem (Roark and Sproat, 2007; Chandlee, 2017), it is also useful for testing the properties of different neural network architectures. Within cognitive science and linguistics, computational models of inflection have a long history in arbitrating between competing theories of morphological representation and acquisition (surveyed in Pinker and Ullman, 2002; Seidenberg and Plaut, 2014), and inflection is often a focus of computational typology (Bjerva and Augenstein, 2018; Elsner et al., 2019).
However, despite the task’s popularity, standard evaluation practices have significant weaknesses. We discuss three aspects of these practices which hamper investigators’ ability to derive informative
*Denotes equal contribution
conclusions. (1) Uniform sampling, which creates unnatural train-test splits, (2) Evaluation of single data splits, which yields unstable model rankings, and (3) uncontrolled overlaps between train and test data components, which obscure diagnostic information about systems’ ability to perform morphological generalizations.

heading: 1.1 Practice 1: Uniform Sampling
text: Training and evaluation sets have been (with some exceptions) sampled uniformly by type from a corpus such as those available in the UniMorph Database (Kirov et al., 2018; McCarthy et al., 2020; Batsuren et al., 2022). While practical to implement for corpora that lack frequency information, uniform sampling is also unrealistic because morphological forms exhibit a highly skewed Zipfian distribution in any large text (Lignos and Yang, 2018). Thus, uniform sampling creates an unnatural bias towards low-frequency types. Since high frequency is correlated with irregularity across many but not all languages (Bybee, 1991; Fratini et al., 2014; Wu et al., 2019), this creates a bias towards more regular and reliable training items.
We provide two alternatives for producing realistic or challenging data sets: (1) a frequencyweighted sampling strategy to achieve a more real-
6082
istic distribution of out-of-vocabulary (OOV) lemmas and inflectional categories and better match practical use-cases or input during child language acquisition, and (2) a sampling strategy that explicitly balances OOV lemmas and inflectional categories in order to directly evaluate models’ generalization ability along these dimensions.

heading: 1.2 Practice 2: Single Data Splits
text: The current practice in inflection evaluation, employed, for example, in the SIGMORPHON, CoNLL-SIGMORPHON and SIGMORPHONUniMorph shared tasks in recent years (Cotterell et al., 2016, 2017, 2018; McCarthy et al., 2019; Vylomova et al., 2020; Pimentel et al., 2021; Kodner et al., 2022), examines different models with one particular data set that is considered representative of the language or the inflection task at hand. This data set, and therefore all evaluation, usually consists of one pre-defined train-(dev-)test split.
However, this method is problematic because it implicitly assumes that the results from a single split are informative and generalizable. In reality, this assumption is untenable, particularly when facing severe data limitation (Liu and Prud’hommeaux, 2022), as is the case for the majority of languages in the world (cf. Blasi et al., 2022): In UniMorph 4, for example, data set size varies significantly across languages, with the smallest, Manx (Celtic, IE), containing only one lemma with 14 inflected forms, and the largest, Czech (Slavic, IE) containing approximately 800,000 lemmas with 50.3 million forms. If the performance on a single split is not necessarily representative, then the original model ranking derived from the one particular data split might also not generalize well.
The concerns outlined above were demonstrated in Liu and Prud’hommeaux (2022), which investigated model generalizability in low-resource morphological segmentation. Using data from 11 languages, they provided evidence that: (1) there are major differences in the numerical performance and rankings of each evaluated model type when using different splits from the same data set, and (2) even within a single split, large performance variability can arise for each model type when it is trained using different random seeds. These findings illustrate that common methods of model evaluation can lead to largely coincidental conclusions. We extend this approach to morphological inflection by applying multiple data splits, and evaluating
variability between splits.

heading: 1.3 Practice 3: Uncontrolled Overlaps
text: The typical morphological inflection task paradigm presents (lemma, inflected form, feature set) triples during training and asks a system to predict inflected forms from (lemma, feature set) pairs during evaluation. Note that since the lemma and feature set can be combined independently, it is possible for either lemmas or feature sets that appeared during training to reappear during test without any individual triple violating train-on-test. Test pairs with OOV lemmas or feature sets require a system to generalize along different morphological dimensions. Performance is likely related to the relative rates of OOV lemmas and feature sets in the evaluation split, yet existing sampling strategies generally leave these variables uncontrolled.
We observe that uncontrolled OOV rates vary dramatically between different sampled data splits, and that uncontrolled sampling biases test sets towards “easier” items with in-vocabulary lemmas and feature sets. To remedy this, we argue that performance should be reported independently for items with each lemma/feature set overlap type regardless of sampling strategy. Furthermore, if a project’s research goal is to evaluate the generalization ability of a model, lemma/feature set overlapaware sampling should be used to ensure that a sufficient number of test items of each overlap type are present.

heading: 2 Defining Overlap
text: Morphological inflection requires generalization over two primary dimensions: to new lemmas (“If I have witnessed the 2pl imperfective subjunctive with other verbs, how do I apply that to new verb X?”) and to new inflectional categories (“If I have seen X inflected in several other categories, how do I create the 2pl imperfect subjunctive of X?”). Because of the sparsity of morphological inflections in language use (Chan, 2008), both types of generalization are necessary during language acquisition as well as deployment of computational models.
As with many linguistic phenomena, the attestation of inflected forms follows an extremely sparse and skewed long-tailed distribution, as do attested lemmas ranked by the proportions of their potential paradigms that are actually attested (paradigm saturation; PS), and inflectional categories ranked by the number of lemmas with which they oc-
cur (Chan, 2008). For example, the median PS for Spanish verbs in millions of tokens of childdirected speech is equivalent to two of its three dozen possible forms, and the 2nd person plural imperfect subjunctive only occurs with two lemmas (cf. Lignos and Yang, 2018; Kodner, 2022).
Given the importance of both types of generalization, it is necessary to evaluate both to assess the abilities of a morphological learning model. In the evaluation made popular by the SIGMORPHON shared tasks, models are asked to predict inflected forms given (lemma, feature set) pairs, where feature sets can be seen as corresponding to inflectional categories or paradigm cells. Generalization across lemmas is required when an evaluation pair contains a lemma that was out-of-vocabulary (OOV) in training, and generalization across categories is required when an evaluation pair contains a feature set that was OOV. In all, there are four logically possible licit types of evaluation pairs distinguished by their lemma and feature overlap with training. These are expressed visually in Figure 1 along with two types which are unions of the other types:
both Overlap: Both the lemma and feature set of an evaluation pair are attested in the training set (but not together in the same triple). lemmaOnly Overlap: An eval pair’s lemma is attested in training, but its feature set is novel. featsOnly Overlap: An eval pair’s feature set is attested in training, but its lemma is novel. neither Overlap: An evaluation pair is entirely unattested in training. Both its lemma and features are novel.
featsAttested: An eval pair’s feature set is attested in training (both ∪ featsOnly) featsNovel: An eval pair’s feature set is novel (lemmaOnly ∪ neither)
For a concrete illustration, consider the training and evaluation sets provided in (1)-(2). Each evaluation pair exhibits a different kind of overlap.
(1) Example Training Set t0: see seeing V;V.PTCP;PRS t1: sit sat V;PST
(2) Example Evaluation Set e0: see V;PST <-- both e1: sit V;NFIN <-- lemmaOnly e2: eat V;PST <-- featsOnly e3: run V;PRS;3;SG <-- neither
featsAttested = {e0, e2} featsNovel = {e1, e3}
Computational work in morphological inflection has generally ignored these dimensions of evaluation. In the shared task, the four overlap types were uncontrolled before 2021, which contains one partial evaluation on featsOnly ∪ neither items. But, recognition of the value of these overlap types has grown recently. Goldman et al. (2022) showed that four models consistently struggle to generalize across lemmas, concluding that test sets should avoid lemma overlap altogether. However, this proposal removes the option to contrast performance on seen and unseen lemmas. Furthermore, they did not control for or evaluate feature overlap, so both vs. lemmaOnly and featsOnly vs. neither also cannot be distinguished. (3) summarizes their partition scheme, which distinguishes two overlap types. We call these lemmaAttested (= both ∪ lemmaOnly) and lemmaNovel (= featsOnly ∪ neither).
(3) Goldman et al. (2022) Partition Types e0: sit V;PST <-- lemmaAttested e1: see V;NFIN <-- lemmaAttested e2: eat V;PST <-- lemmaNovel e3: run V;PRS;3;SG <-- lemmaNovel
The 2022 SIGMORPHON-UniMorph shared task was the first to report results on all four overlap types (both, featsOnly, lemmaOnly, neither). Every system submitted to the shared task achieved much better performance on in-vocabulary feature sets (both and featsOnly) than OOV feature sets (lemmaOnly or neither). This discrepancy even held for languages for which a model should be able to generalize: highly regular agglutinative morphology for which this type of generalization is often transparent. On the other hand, lemma attestation produced a much smaller discrepancy. Following these observations, we focus our investigation on the four logical overlap types with extra emphasis on the featsAttested vs. featsNovel dichotomy. We address agglutinative languages specifically in Section 5.3

heading: 3 Data Sources and Preparation
text: We follow prior literature in providing training and evaluation data in UniMorph’s format. Data sets were sampled from UniMorph 4 (Batsuren et al., 2022) and 3 (McCarthy et al., 2020)1 aug-
1In some cases, UniMorph 4 was found to lack highfrequency items present in UniMorph 3. For example, English verbs happen and run are present in 3 and absent in 4. For languages where we determined this to be an issue, we sampled from deduplicated UniMorph 3+4 with tags normalized to 4.
mented with frequencies from running text corpora. When possible, frequencies were drawn from childdirected speech (CDS) corpora from the CHILDES database (MacWhinney, 2000), since one possible downstream application of the morphological inflection task is contribution to the computational cognitive science of language acquisition. CHILDES lemma and morphological annotations were converted into UniMorph format and intersected with UniMorph to create frequency lists.2

heading: 3.1 Languages
text: Languages were prioritized for typological diversity and accessibility of text corpora. Quantitative summaries of our frequency+UniMorph data sets are provided in Appendix B.
Arabic (Semitic, AA): Modern Standard Arabic frequencies were drawn from the diacritized and morphologically annotated Penn Arabic Treebank (PATB; Maamouri et al., 2004) and intersected with UniMorph 4 ara ∪ ara_new. Diacritized text is a requirement because orthographic forms drawn from undiacritized text are massively morphologically ambiguous. The text in the CHILDES Arabic corpora is undiacritized and thus unusable.
German (Germanic, IE): German was drawn from the Leo Corpus (Behrens, 2006), the only morphologically annotated German corpus in CHILDES, and intersected with UniMorph 3+4. Only nouns and verbs were extracted because annotation for adjectives is inconsistent.
English (Germanic, IE): English was included because it is heavily studied despite its relatively sparse morphology. Data was extracted from all morphologically annotated CHILDES English-NA corpora and intersected with UniMorph 3+4.3 Only nouns and verbs were extracted due to inconsistent adjective annotation in both data sources.
Spanish (Romance, IE): Spanish exhibits a variety of fusional and agglutinative patterns. Data was extracted from all morphologically annotated Spanish CHILDES corpora intersected with Spanish UniMorph 3+4. Non-Spanish vocabulary was removed by intersecting with UniMorph. Only nouns and verbs were extracted.
Swahili (Bantu, Niger-Congo): Swahili morphology is highly regular and agglutinative with very large paradigms. Frequencies were drawn
2All data and code is available at https://github.com/ jkodner05/ACL2023_RealityCheck.
3A full list of utilized English and Spanish CHILDES corpora is provided in Appendix A.
from Swahili Wikipedia dump 20221201 accessed through Huggingface (Wikimedia, 2022) and intersected with UniMorph 4 swc ∪ swc.sm. In cases where mapping inflected forms to UniMorph creates ambiguity due to syncretism, frequency was divided evenly across each triple sharing the inflected form. This ensured that the frequencies of inflected forms remain consistent with Wikipedia. Intersecting with UniMorph removed the large amount of non-Swahili vocabulary in the Wikipedia text.
Turkish (Turkic): Turkish is also highly regular and agglutinative with very large paradigms. Frequencies were drawn from Turkish Wikipedia dump 20221201 accessed through Huggingface, intersected with UniMorph 4, and processed identically to Swahili.

heading: 3.2 Data Splits
text: We employed three distinct sampling strategies to generate small (400 items) and large (1600) training, small (100) and large (400) fine-tuning, development (500), and test (1000) sets for each language.4 Small training and fine-tuning are subsets of large training and fine-tuning. Each splitting strategy was applied five times with unique random seeds to produce distinct data sets.
UNIFORM: Raw UniMorph 3+4 corpora were partitioned uniformly at random. This approach is most similar to that employed by SIGMORPHON shared tasks, except for 2017 and 2022.
WEIGHTED: Identical to UNIFORM except splits were partitioned at random weighted by frequency. Small training+fine-tuning were sampled first, then additional items were sampled to create large training+fine-tuning. Training and fine-tuning sets were then split uniformly at random. Dev+test was next sampled by weight and then were separated uniformly. This frequencyweighted sampling is reminiscent of the 2017 shared task: it strongly biases the small training set towards high-frequency items and dev+test towards low-frequency items. Since most UniMorph forms do not occur in our corpora due to morphological sparsity, most triples had zero weight and were never sampled.
OVERLAPAWARE: Similar to the 2022 SIGMORPHON shared task. It enforces a maximum proportion of featsAttested pairs in the test set relative to train+fine-tuning: as close to 50% as pos-
4Swahili large train and large fine-tune contain 800 and 200 items respectively due to the limited size of UniMorph.
sible without exceeding it. This ensures that there is ample representation of each overlap type in test. It is adversarial, since featsNovel pairs are expected to be more challenging than featsAttested pairs. This process also tends to increase the proportion of lemmaOnly items in the test set. Only items with non-zero frequency were sampled.
UNIFORM produces a heavy bias towards lower frequency words. For all languages and splits, the median frequency of sampled items is actually zero: that is, the majority of sampled items were not attested in our corpora. This is a consequence of the extreme sparsity of morphological forms discussed in Section 2. As a consequence, overlap between splits from different seeds is orders of magnitude lower for UNIFORM than the other strategies. WEIGHTED achieves the expected high-frequency bias in training sets relative to test sets.
Table 1 provides average means and standard deviations for the proportion of featsAttested and featsNovel in test sets relative to small and large train. OVERLAPAWARE consistently achieves a roughly 50-50 split with low variability across languages and seeds. The other strategies bias test sets heavily towards featsAttested with high variance across languages and seeds.5

heading: 4 Experimental Setup
text: One non-neural and three neural systems were evaluated. These were chosen based on their availability and performance in recent shared tasks:
CHR-TRM (Wu et al., 2021) is a character-level transformer that was used as a baseline in 2021 and 2022. We used the hyper-parameters suggested by the original authors for small training conditions.
CLUZH-GR and CLUZH-B4 (Wehrli et al., 2022) is a character-level transducer which substantially
5See Appendix B for breakdowns by language, training size, and overlap partitions.
outperformed CHR-TRM in the 2022 shared task. The results submitted for the shared task are from an elaborate ensemble model optimized for each language. For this work, we evaluate two published variants with consistent hyper-parameters across languages, CLUZH-GR with greedy decoding and CLUZH-B4 with beam decoding, beam size = 4.
NONNEUR (Cotterell et al., 2017) has been used as a baseline in SIGMORPHON shared tasks since 2017. It heuristically extracts transformations between lemmas and inflected forms and applies a majority classifier conditioned on the associated feature sets. NONNEUR was trained on combined training and fine-tuning sets so that each architecture was exposed to the same amount of data.

heading: 5 Results
text: This section presents our analyses of the results. All evaluations report exact match accuracy. Overall accuracy refers to average accuracy on an entire evaluation set. Average overall accuracy refers to the mean of overall accuracy over all five seeds. See Appendix C for full breakdowns by language and architecture.

heading: 5.1 Effect of Training Size
text: We begin by comparing average overall accuracy for each training size. All reported analyses focus on test, but there were no observable qualitative differences in behavior between dev and test. We summarize the results in Table 2, broken down by overlap partition and sampling strategy. The large training size consistently leads to higher accuracies than small training. Across languages, the average accuracy score difference between the two training sizes is 9.52%. Taking Arabic as an illustrative example, the score difference between the two training sizes ranges from 1.74% to 19.32% depending on model type and splitting strategy, with an average of 12.05%.

heading: 5.2 Effect of Sampling Strategy
text: We next turn to measuring the effect of sampling strategy on overall accuracy. Figure 2 provides a visualization of accuracy by sampling strategy across seeds broken down by training size, language, model type. Using Arabic as an illustration, for large training, WEIGHTED sampling leads to the highest average overall accuracy across model types (77.76%), while OVERLAPAWARE sampling yields the lowest (61.06%); comparing the results from the three sampling strategies given each of the four model types, WEIGHTED consistently results in the highest accuracy for all model types except for CLUZH-B4, where UNIFORM sampling (83.84%) leads to a performance slightly better than that of WEIGHTED (83.82%). We make similar observations for small training: WEIGHTED and OVERLAPAWARE result in the highest and the lowest average overall accuracy, respectively, across model types for Arabic (68.82% vs. 47.81%). WEIGHTED sampling leads to a higher accuracy compared to the other two strategies for every model type other than CHR-TRM, where the result from UNIFORM sampling (71.90%) is again slightly higher than that of WEIGHTED (71.60%).
When considering other languages, we also find some variation. WEIGHTED sampling also yields the highest average accuracy scores across model types for Arabic, German, Spanish, and Turkish for both training sizes, except for Spanish under the large training condition with CLUZH-GR, where UNIFORM leads. In contrast, UNIFORM consistently results in the highest average accuracy on
English and Swahili for both training sizes. Across languages, the average accuracy from WEIGHTED is the highest for both large (83.75%) and small (74.22%) training sizes, followed by UNIFORM (large: 79.20%, small: 66.16%). OVERLAPAWARE always yields the lowest accuracy. These observations align with our expectations about the adversarial nature of OVERLAPAWARE, where challenging featsNovel (Table 2) constitutes a much larger proportion test set (Table 1).

heading: 5.3 Effect of Overlap
text: We now provide an analysis of accuracy scores by overlap partition. Figure 3 provides a visualization of accuracy by partition across seeds broken down by training size, language, model type. Using Arabic again as an illustration, the average accuracy across model types and sampling strategies for large training is much higher for featsAttested (77.70%) than for featsNovel (41.92%), somewhat higher accuracy is achieved for both (79.53%) than for featsOnly (77.28%), and higher accuracy is achieved for lemmaOnly (49.12%) than for neither (41.92%). This ranking is consistent across model types, sampling strategies, and training sizes. Scores from these two overlap partitions are also higher than those from lemmaOnly and neither.
These patterns hold across languages. Specifically, we observe two general tendencies. First, the accuracy averaged across model types and sampling strategies is always substantially higher for featsAttested than it is for featsNovel; the average accuracy difference between the two is
49.75% for the large training, and 48.02% for small training. This is reflected in a full breakdown by overlap type: higher accuracy is consistently achieved for both and featsOnly, than for neither and lemmaOnly. This large asymmetry corresponds to our expectations regarding the effect of feature overlap on performance.
We provide three sub-analyses to further investigate this asymmetry and compare it with the lemma-based division advocated for by (Goldman et al., 2022). First, we compute the average accuracy difference between lemmaAttested (both ∪ lemmaOnly) and lemmaNovel (featsOnly ∪ neither). The score difference between lemmaAttested and lemmaNovel is less than 2% averaged across languages for both training sizes, which is an order of magnitude smaller than the difference between featsAttested and featsNovel. This trend is consistent with the results of the 2022 SIGMORPHON shared task, which also found a much greater impact of feature set attestation than lemma attestation.
Second, we measure the correlation between the proportion of featsAttested items (number featsAttested items divided by the size of the dev or test set), and overall accuracy (average accuracy on an entire dev or test set), as well as between the proportion of lemmaAttested and overall accuracy. We used Spearman’s ρ, which assesses if there is any monotonic (not necessarily linear) relationship between the two variables.6 If ρ between
6ρ falls in the range [-1,1], where -1 is a perfect negative correlation and 1 is a perfect positive correlation.
an overlap type and overall accuracy is high, it would suggest that the distribution of overlaps is an important driver of performance. lemmaAttested shows little correlation (small: 0.01, large: -0.10). However, we find substantial positive correlations for featsAttested (small: 0.69, large: 0.68).
Third, we compute the correlation between the accuracy score of individual partitions and the overall accuracy score on UNIFORM and WEIGHTED vs. on OVERLAPAWARE. This demonstrates to what extent evaluation results based on each overlap partition resemble those captured by the overall accuracy and how it differs when overlaps are controlled during sampling. If the correlation is small, it suggests that the performance on a particular overlap partition is largely independent of the others and should be evaluated independently.
When overlaps are not explicitly controlled, correlations are particularly strong for featsAttested because this partition makes up a large majority of the test set (Table 3). These partitions are also the ones that tend to show the highest performance, which is then reflected in the overall accuracy. However, for OVERLAPAWARE, correlations are higher between overall accuracy and the challenging partitions: featsNovel, lemmaOnly, and neither. They are also higher not only for featsNovel, but also lemmaAttested, and lemmaNovel even though these overlaps were not explicitly controlled. This demonstrates that OVERLAPAWARE sampling better balances individual partitions in its overall accuracy scores and can be expected to produce
a more challenging evaluation. However, all partitions should be evaluated regardless of sampling strategy.
Up to this point, we have considered all languages in the analysis. However, whether or not it is reasonable to expect a system to achieve high accuracy on featsNovel items varies typologically. For languages with highly regular and agglutinative morphologies, such as Swahili and Turkish, each feature in a feature set roughly corresponds to a single affix in a certain order with a limited number of allomorphs. For these languages, this dimension of generalization should often be straightforward. For languages with mixed systems, like Spanish and Arabic, and languages with fusional systems like English, the individual members of a feature set often do not have direct bearing on the inflected form. For these languages, generalization to a novel feature set is sometimes impossible when it cannot be inferred from its component features. The same problem applies to lemmas with erratic stem changes or suppletion.
Thus, if a model type can generalize to novel feature sets, one would expect that the accuracy gap between featsAttested and featsNovel would be lower for Swahili and Turkish than for the other languages. However, the gaps for these are actually larger than for German or Arabic. One would also expect the correlation between the proportion of featsAttested in the data and overall accuracy to be lower for Swahili and Turkish, however this is not borne out either. These findings, provided in Table 4, reveal that current leading inflection models do not necessarily generalize well to novel feature sets even in precisely the cases where they should be able to.

heading: 5.4 Model Ranking
text: In this section, we analyze how performance varies across the four model types. We first compare
model performance based on the average overall accuracy. Averaged across the six languages, CLUZHB4 ranks among the highest, while NONNEUR consistently achieves the lowest performance.
large: CLUZH-B4 (78.32%) > CHR-TRM (78.07%) > CLUZH-GR (76.17%) > NONNEUR (65.82%) small: CLUZH-B4 (68.58%) > CLUZH-GR (67.97%) > CHRTRM (64.76%) > NONNEUR (58.97%)
Model rankings for individual languages are much more variable, especially for large training. There is not a single model ranking that holds for every language. While CLUZH-B4 yields the best performance for three languages (German, Spanish, and Turkish), CHR-TRM outperforms other model types for Arabic and Swahili, and NONNEUR leads to the highest accuracy for English. There is less variation in model rankings for small training; the same model ranking was observed for German, English, and Spanish (NONNEUR > CLUZH-B4 > CLUZH-GR > CHR-TRM). Notably, for each individual language, the model rankings were always inconsistent between the two training sizes.
Several trends emerge in model rankings by overlap partition. First, the model rankings based on the overall accuracy do not hold for the overlap partitions except for Arabic and Swahili large training. Second, within each overlap partition, model rankings are more stable across languages for small train than large. Third, on average, CLUZH-B4 outperforms the other model types on partitions with feature overlap whereas CHR-TRM leads on partitions without feature overlap. These tendencies resonate with our proposal in Section 2: future models of morphological inflection should be evaluated based on alternative metrics in addition to
overall accuracy. They also reveal difference generalization strengths across models.
When comparing performance by sampling strategy, we found lower variability for each language. For example, with UNIFORM large training, two model rankings turn out to be the most frequent, each observed in two languages. Among the models, CLUZH-B4 and CHR-TRM achieve the best performance. For small training, one model ranking holds for three out of the six languages (CLUZH-B4 > CLUZH-GR > CHR-TRM > NONNEUR). Considering both training sizes, there are no noticeable differences in terms of the most frequent model ranking across the three sampling strategies. For UNIFORM and WEIGHTED, the neural systems are always ranked among the highest for both training sizes; yet for OVERLAPAWARE with small training, NONNEUR achieves the highest performance for German, English, and Spanish.

heading: 5.5 Variability across Random Seeds
text: Analysis so far relies on accuracy scores averaged across random seeds. The final component of our analysis investigates how much variation arises due to random data sampling. Given the five random seeds for each combination of language, sampling strategy, overlap partition, and model type, we calculated the score range, which is the difference between the lowest and the highest overall accuracy, as well as the standard deviation of the accuracy scores across the seeds, which we refer to as random seed variability.
We first considered the score range for overall accuracy for each language. For large training, the mean score range spans from 4.41% for Arabic, to 8.38% for English; the mean random seed variability follows the same trend (1.73% to 3.54%). For every language, the score range and random seed variability for the large training size are consistently larger than those derived from small training. In both cases, score ranges are non-negligible.
Next, for each language, we analyze the average score range for each sampling strategy and model type separately. Comparing results from the three sampling strategies in Table 5, OVERLAPAWARE sampling consistently yields the highest score range and random seed variability. This indicates that OVERLAPAWARE, despite exhibiting the least variability in overlap partition sizes, is also the most variable in terms of model performance. This likely suggests that it is not just feature set attestation in general, but also exactly which feature sets that happen to appear in train vs. test drive performance. Finally, when looking at results for each individual model type, CLUZH-GR demonstrates the most variable performance. Its average score range (9.47% for large training, 7.94% for small) and its average random seed variability (4.03% for large training, 3.31% for small) end up being the highest.

heading: 6 Conclusions
text: We investigated the roles that sampling strategy, random seeds, and overlap types play in evaluating and analyzing the results of morphological inflection tasks and conclude that common practices leave much to be desired. We argue for frequencyweighted splitting to achieve more realistic traintest distributions and feature/lemma overlap-aware sampling for directly investigating the generalization abilities of different models. The high score range observed for overlap-aware sampling relative to other strategies suggests that which feature sets happen to appear in train vs. test play a major role in the ability of a model to generalize, though future work would need to confirm this.
Regardless of sampling strategy, evaluation items of each overlap type should be used in addition to an overall analysis. The evaluation in this work reveals that all model types under investigation struggle to generalize to unseen feature sets, even for languages where that should be possible, a fact that has been overlooked in prior studies. Finally, results drawn from one data split are unlikely to be representative, so multiple splits should be made with different random seeds and compared, particularly for shared tasks and leader boards where final model rankings matter.
Limitations
Our suggested approaches have two primary practical limitations: First, WEIGHTED sampling is
restricted to languages with available running text sources for extracting frequencies. A project on extremely low-resource languages (e.g., Liu et al., 2022) may be restricted to UNIFORM and OVERLAPAWARE sampling. Second, as the number of seeds increases, so do requirements for training time and/or computing power. A shared task, for example, might limit itself to only a few seeds in order to assure on-time submissions. Future work would benefit from a wider selection of model architectures, along with more sampling strategies, and of course a wider sample of typologically diverse languages.
Notably, this work reproduces the effect observed in the SIGMORPHON 2022 shared task (Kodner et al., 2022), which found a substantial performance hit for featsNovel relative to featsAttested, but not lemmaNovel relative to lemmaAttested. However, both this work and the shared task fail to replicate the effect observed in Goldman et al. (2022), which reports a 95% performance hit on lemmaNovel vs. lemmaAttested. This may have something to do with differences in splitting algorithms, unmeasured feature overlap in Goldman et al. (2022), or choice of model architectures.
Ethics Statement
To the best of our knowledge, all results published in this paper are accurate, and we have represented prior work fairly to the best of our abilities. All data sources are free and publicly available, except for the Penn Arabic Treebank (Maamouri et al., 2004), which is accessible through the LDC.7 No sensitive data was used which could violate individuals’ privacy or confidentiality. Authorship and acknowledgements fairly reflect contributions.

heading: Acknowledgements
text: We thank Charles Yang, Jeffrey Heinz, Mitch Marcus, and the audience at Stony Brook University ATLaC for their helpful discussion. Experiments were performed on the SeaWulf HPC cluster maintained by RCC and the Institute for Advanced Computational Science (IACS) at Stony Brook University and made possible by National Science Foundation (NSF) grant No. 1531492. The second author gratefully acknowledges funding through the IACS Graduate Research Fellowship and the NSF Gradu-
7https://catalog.ldc.upenn.edu/LDC2005T20
ate Research Fellowship Program under NSF Grant No. 2234683.

heading: A English and Spanish Data Sources
text: A.1 English The following CHILDES corpora were used to create the English data set. Utterances from speaker *CHI were excluded: Bates (Bates et al., 1991), Bliss (Bliss, 1988), Bloom (Bloom, 1970; Bloom et al., 1974), Bohannon (Bohannon III and Marquis, 1977), Braunwald (Braunwald, 1971), Brent (Brent and Siskind, 2001), Brown (Brown, 1973), Clark (Clark, 1978), Davis (Davis and MacNeilage, 1995), Demetras (Demetras, 1986, 1989), EllisWeismer (Heilmann et al., 2005), Feldman (Feldman, 1998), Garvey (Garvey and Hogan, 1973), Gathercole (Gathercole, 1986), Gelman (Gelman et al., 1998), Gillam (Gillam and Pearson, 2004), Gleason (Gleason, 1980), Hall (Hall and Tirre, 1979), Higginson (Higginson, 1985), HSLLD (Dickinson and Tabors, 2001), Kuczaj (Kuczaj II, 1977), MacWhinney (MacWhinney, 1991), McCune (McCune, 1995), Morisset (Morisset et al., 1995), Nadig (Bang and Nadig, 2015), Nelson (Nelson, 2006), NewEngland (Ninio et al.,
1994), NewmanRatner (Newman et al., 2016), Nichols-TD (Nicholas and Geers, 1997), Peters (Peters, 1987), POLER (Berl et al., 2005), Post (Demetras et al., 1986), Providence (Demuth et al., 2006), Rollins (Rollins, 2003), Sachs (Sachs and Nelson, 1983), Sawyer (Sawyer, 2013), Snow (MacWhinney and Snow, 1985), Soderstrom (Soderstrom et al., 2008), Sprott (Sprott, 1992), Suppes (Suppes, 1974), Tardif (MacWhinney, 2000), Valian (Valian, 1991), VanHouten (van Houton, 1986), VanKleeck (MacWhinney, 2000), Warren-Leubecker (WarrenLeubecker, 1982), Weist (Weist et al., 2009).
A.2 Spanish The following CHILDES corpora were used to create the Spanish data set. Utterances from speaker *CHI were excluded: Aguirre (Martínez and Altares, 2005), ColMex (MacWhinney, 2000), Fernandez/Aguado (MacWhinney, 2000), GRERLI (MacWhinney, 2000), Hess (Zimmermann, 2003), Linaza (Linaza et al., 1981), Marrero (Capelli et al., 1994), Montes (Montes, 1987), AguadoOrea/Pine (Aguado-Orea and Pine, 2015), Ornat (López Ornat, 1997), Remedi (Remedi, 2014), SerraSole (MacWhinney, 2000).

heading: B Splitting Strategy Data Summaries
text: This appendix contains Tables 6-9.

heading: C Detailed Results
text: This appendix contains Tables 10-11.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations Section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Conclusions and Limitations.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Many corpora were used, all cited in the introduction and/or Appendix A. Code and data are available through the link provided in the paper

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Citations in-line and in Appendix A

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Ethics Statement.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We processed wordlists
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Throughout the paper and in detail in Appendix. Our data sets have also been made available
C 3 Did you run computational experiments? Section 4-5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4 and Acknowledgements
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Throughout the paper

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Throughout the paper and Appendix B

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Our code is available
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: test play a major role in the ability of a model to generalize, though future work would need to confirm this. Regardless of sampling strategy, evaluation items of each overlap type should be used in addition to an overall analysis. The evaluation in this work reveals that all model types under investigation struggle to generalize to unseen feature sets, even for languages where that should be possible, a fact that has been overlooked in prior studies. Finally, results drawn from one data split are unlikely to be representative, so multiple splits should be made with different random seeds and compared, particularly for shared tasks and leader boards where final model rankings matter. Limitations
Our suggested approaches have two primary practical limitations: First, WEIGHTED sampling is
restricted to languages with available running text sources for extracting frequencies. A project on extremely low-resource languages (e. g. , Liu et al. , 2022) may be restricted to UNIFORM and OVERLAPAWARE sampling. Second, as the number of seeds increases, so do requirements for training time and/or computing power. A shared task, for example, might limit itself to only a few seeds in order to assure on-time submissions. Future work would benefit from a wider selection of model architectures, along with more sampling strategies, and of course a wider sample of typologically diverse languages. Notably, this work reproduces the effect observed in the SIGMORPHON 2022 shared task (Kodner et al. , 2022), which found a substantial performance hit for featsNovel relative to featsAttested, but not lemmaNovel relative to lemmaAttested. However, both this work and the shared task fail to replicate the effect observed in Goldman et al. (2022), which reports a 95% performance hit on lemmaNovel vs. lemmaAttested. This may have something to do with differences in splitting algorithms, unmeasured feature overlap in Goldman et al. (2022), or choice of model architectures