Lemmas Matter, But Not Like That: Predictors of Lemma-Based Generalization in Morphological Inflection
Authors: Sarah Ruth Brogden Payne, Jordan Kodner
Accepted at: acl25

Lemmas Matter, But Not Like That:
Predictors of Lemma-Based Generalization in Morphological Inflection

Sarah Payne and Jordan Kodner
Department of Linguistics & Institute for Advanced Computational Science
Stony Brook University, Stony Brook, NY, USA
{first.last}@stonybrook.edu

Abstract
Recent work has shown that overlap – whether a given lemma or feature set is attested independently in train – drives model performance on morphological inflection tasks. The impact of lemma overlap, however, is debated, with accuracy drops from 0% to 30% reported between seen and unseen test lemmas. In this paper, we introduce a novel splitting algorithm designed to investigate predictors of accuracy on seen and unseen lemmas. We find only an 11% average drop from seen to unseen test lemmas but show that the number of lemmas in train has a much stronger effect on accuracy on unseen than seen lemmas. We also show that the previously reported 30% drop is inflated due to the introduction of a near-30% drop in the number of training lemmas from the original splits to the novel splits. These results help us better understand the factors affecting morphological generalization by neural models.

1 Introduction
Morphological inflection – the task of producing an inflected form given a lemma and a set of morphosyntactic features – is a fundamental problem in NLP, with both practical and scientific implications. On the practical side, morphological inflection is a crucial tool for pipelining, particularly for low-resourced and highly-inflected languages for which end-to-end systems remain impractical or infeasible (Bender, 2009; Oflazer and Saraçlar, 2018; Wiemerslage et al., 2022; Arnett and Bergen, 2025; Rapacz and Smywiński-Pohl, 2025, i.a.). On the scientific side, morphological inflections are often complex transformations at the word and sub-word level (Chandlee, 2017). Investigating how neural systems handle these transformations, and the circumstances under which they are able to generalize them, provides a window into their linguistic generalization capabilities (Kodner et al., 2023b).

In recent years, the SIGMORPHON shared tasks on morphological inflection have reported gener-

![A schematicization of the train and test methods of Kodner et al. (2022) and Goldman et al. (2022).](page_1016_340_388_246.png)

Figure 1: A schematization of the train and test methods of Kodner et al. (2022) and Goldman et al. (2022).

ally high accuracy across many languages (Cotterell et al., 2016, 2017, 2018; McCarthy et al., 2019; Vylomova et al., 2020; Pimentel et al., 2021; Kodner et al., 2022; Goldman et al., 2023). Work seeking to understand this success has investigated predictors of model performance in terms of generalization along different dimensions. Two aspects of this are lemma overlap: whether the lemma in a test (lemma, features, inflected form) triple is also attested in train, and feature overlap, whether the feature set in a test triple is also attested in train. In both cases, entire triples do not reappear between train and test, rather the test triple contains a lemma or feature set seen in a distinct train triple. The effect of overlap is measured by comparing accuracy on test triples containing seen lemmas or feature sets to those containing the unseen ones (Goldman et al., 2022; Kodner et al., 2022); it thus probes the ability of neural models to generalize to novel lemmas or feature sets.

Previous work on lemma overlap has drawn conflicting conclusions. Goldman et al. (2022) compare model performance on the original SIGMORPHON 2020 splits – in which triples were partitioned uniformly at random – to their “lemma-splits,” in which entire paradigms are partitioned uniformly at random, and thus all lemmas occur only in train or only in test. Goldman et al. report
a 30% average decrease in accuracy from the models trained and tested on the SIGMORPHON splits to those trained and tested on their lemma-splits. However, Kodner et al. (2022) and Kodner et al. (2023b) find no performance drop on unseen lemmas. Their setup involves a single training split, with a test set containing at most 50% triples with attested features; this also results in a significant portion of test triples with unseen lemmas. Kodner et al. train a single model per language and find no difference in accuracy between seen and unseen lemmas in their test set. Both experimental designs are conceptualized in Fig. 1. Crucially, while any drop in test accuracy can be attributed directly to differences in the test items in Kodner et al.’s setup, they could be attributed either to differences in the test items or the train sets in Goldman et al.’s.

In contrast to lemma overlap, previous work on feature overlap has unequivocally shown that neural models struggle to generalize to unseen feature sets: Kodner et al. (2022) found that all systems perform better on test triples with attested feature sets than those with unattested ones, with the gap in accuracy ranging from 32% to as high as 79%. These findings were replicated by Kodner et al. (2023a), who performed a more in-depth analysis on 6 languages. Kodner et al. (2023b) created linguistically-informed language-specific probes, and found that 2 of the 3 models which they tested only succeeded on the simplest feature-based generalization, while the third failed on all probes.

In this paper, we aim to understand the causes of the conflicting results on lemma overlap and further investigate what factors impact models’ ability to generalize to novel lemmas. Because of the consistent findings regarding feature overlap, we hold it constant; our goal is to understand lemma and feature overlap separately so that future work may investigate their interaction. In our first experiment (§3), we introduce a novel splitting algorithm which fixes a single train set and ensures that 50% of test triples contain seen lemmas. We find an average accuracy drop of 11%, between those reported by Kodner et al. and Goldman et al.. More importantly, we show that the number of training lemmas has a much stronger effect on accuracy on unseen than seen lemmas. In our second experiment (§4), we replicate Goldman et al. to investigate causes of our differing results, and show that the number of lemmas in train is a stronger predictor of generalization to unseen lemmas than the number of triples.

Since Goldman et al.’s lemma-splitting held triple count roughly constant but introduced a near-30% drop in the number of lemmas from SIGMORPHON to their lemma-based splits, they exaggerated the true impact of lemma overlap. These results are a step towards a more complete and interpretable picture of the dimensions across which neural models succeed and fail at generalization.

2 Background: Defining Overlap

In morphological inflection tasks, models are exposed to triples of (lemma, feature set, inflected form) in train. During evaluation, they are given a (lemma, feature set) pair as input and expected to correctly predict the corresponding inflected form. For example, given (walk, V;PAST), a model should output walked. In iterations of the SIGMORPHON shared task before 2022, triples are partitioned uniformly at random into train or test. While this entails that no triple occurring in train will occur in test, as both Goldman et al. (2022) and Kodner et al. (2022) note, it ignores the fact that lemmas or feature sets that appear in train may reappear during test separately. Consider a simple example, adapted from Kodner et al. (2023b):

(1) Example train set:
   t0: see seeing V;V.PTCP;PRS
   t1: sit sat V;PST
(2) Example test set:
   e0: see V;PST
   e1: sit V;NFIN
   e2: eat V;PST
   e3: run V;PRS;3;SG

Though none of the triples in train (1) re-appear in test (2), the lemmas and feature sets seen in train do reappear individually. Consider e0, for which both the lemma and feature set are attested separately in train, and e3, for which neither are. One might expect it to be easier for a model trained on (1) to generate the correct result for e0 than for e3. This is the insight behind studies of overlap: test pairs with novel lemmas or novel feature sets require a system to generalize along different dimensions, and evaluation measures that control specifically for these overlaps can better probe models’ ability to perform these generalizations. Kodner et al. (2023b) define four types of overlap:

both both the lemma and feature set are attested separately in train (e0 in Example 2).
lemmaOnly only the lemma is attested in train, and the feature set is novel (e1 in Example 2).
featsOnly only the feature set is attested in train, and the lemma is novel (e2 in Example 2)
neither neither the lemma nor the feature set is attested in train (e3 in Example 2)

We define lemmaSeen to be any evaluation triple with a lemma attested in train (i.e., lemmaSeen = lemmaOnly \( \cup \) both) and lemmaNovel to be any evaluation triple with a lemma unattested in train (i.e., lemmaNovel = featsOnly \( \cup \) neither).

3 Exp. 1: The Effect of Lemma Overlap

In this section, we test the impact of lemma overlap with a novel splitting algorithm that ensures that 50% of test triples contain seen lemmas. Like Kodner et al., our setup ensures a single training set per language, but like Goldman et al., we focus on lemma overlap rather than feature set overlap.

3.1 Splitting Algorithm

Our algorithm takes in a set of TRIPLES and a set of the unique LEMMAS contained in these triples. It begins by randomly selecting half\(^1\) of the lemmas as our OVERLAPLEMMAS. It then iteratively adds more lemmas to OVERLAPLEMMAS until OVERLAPTRIPLES, the number of triples containing a lemma in OVERLAPLEMMAS, is at least equal to the sum of the train size, fine tune \(^2\) size, and half of the test size. Once OVERLAPTRIPLES is sufficiently large, we sample TRAIN and FINE TUNE from it, as well as half of our test set; the remaining half is sampled from TRIPLES \( \setminus \) OVERLAPTRIPLES. This results in a single train set and a test set where exactly half of test triples contain a lemma seen in train. This algorithm is visualized in Figure 2, and pseudocode is provided in Appendix A.\(^3\)

3.2 Languages

Following Goldman et al., we use the data from the SIGMORPHON 2020 shared task (Vylomova et al., 2020), which contains more languages than SIGMORPHON 2022. We select a 7,000 /1,000 / 2,000 train/fine-tune/test split to follow both the 70% / 10% / 20% splitting of Goldman et al. and the 7,000 training size used by Kodner et al.. To satisfy our split sizes, we retain only languages with at least 10,000 raw triples. This leaves us with 36 languages across 7 families; these are listed in Appendix A. We create 5 random (downsampled) train-test splits since the performance of ANNs on morphological tasks is known to vary across splits (Liu and Prud’hommeaux, 2022; Kodner et al., 2023b). Across all splits, feature overlap is near-ceiling, with an average of 99.5% of test triples containing attested features (\( \sigma = 1.17\% \)). Though we report the following analyses on all test triples, they do not differ significantly if done only with those test triples containing attested feature sets.

3.3 Neural Models

We select three neural models for our evaluation, spanning a diverse range of architectures:

Chr-Trm (Wu et al., 2021) is a character-level transformer used as a baseline in SIGMORPHON 2021 and 2022, and the only model used by both Goldman et al. and Kodner et al.. We use the hyperparameters suggested by the original authors for small training conditions.\(^4\)

Cluzh (Wehrli et al., 2022) is a character-level transducer used by Kodner et al., who consider two variants with consistent hyper-parameters across languages. We use the beam decoding variant, which consistently performs better in their experiments, with a beam size of 4 and the large training hyperparameters.\(^5\)

DeepSpin (Peters and Martins, 2020) is a multilingual RNN used by Goldman et al.. It is composed of 2 biLSTM encoders with bi-linear gated attention using sparsemax (Martins and Astudillo, 2016) and one unidirectional LSTM decoder.\(^6\)

We exclude CULING (Liu and Hulden, 2020) due to its similarity to Chr-Trm and Goldman et al.’s base LSTM since they do not fully report results on this model.

3.4 Results

For each architecture, language, and seed, we calculate SEEN and NOVEL accuracy on lemmaSeen

![A schematicization of our setup for Exp. 1](page_1012_186_377_181.png)

Figure 2: A schematization of our setup for Exp. 1

\footnotetext{1We also tried sampling starting from 1 lemma until the requisite number was met, and results were similar.}
\footnotetext{2Often called “validation.” We call this set “fine tune” to clearly distinguish its purpose, avoiding a common confusion, cf. (van der Goot, 2021).}
\footnotetext{3Our code splits, and neural model outputs can be found at: https://github.com/paynesa/lemmas-matter}
\footnotetext{4https://github.com/shijie-wu/neural-transducer}
\footnotetext{5https://github.com/slvnwhrl/il-reimplementation}
\footnotetext{6https://github.com/deep-spin/sigmorphon-seq2seq}
Figure 3: The ACCURACY ~ LEMMAS beta regression models for SEEN and NOVEL accuracy.

<table>
  <tr>
    <th>Model</th>
    <th>SEEN</th>
    <th>NOVEL</th>
    <th>DROP</th>
    <th>Reported</th>
  </tr>
  <tr>
    <td>CHR-TRM</td>
    <td>92.594</td>
    <td>75.786</td>
    <td>-16.808</td>
    <td>-37.00</td>
  </tr>
  <tr>
    <td>CLUZH</td>
    <td>92.138</td>
    <td>83.221</td>
    <td>-8.917</td>
    <td>—</td>
  </tr>
  <tr>
    <td>DEEPSPIN</td>
    <td>91.946</td>
    <td>83.828</td>
    <td>-8.118</td>
    <td>-14.00</td>
  </tr>
  <tr>
    <td>Macro-Avg</td>
    <td>92.226</td>
    <td>80.945</td>
    <td>-11.281</td>
    <td>-30.00</td>
  </tr>
</table>

Table 1: Accuracy on seen and unseen lemmas by neural architecture, averaged across seeds & languages.

and lemmaNovel test triples respectively; DROP is the difference between the two. The averages of these values by architecture are given in Table 1, as well as those reported by Goldman et al. (2022). Notably, our macro-averaged accuracy drop is just a third of that reported by Goldman et al.; this difference is most notable for CHR-TRM. Full results by language and architecture are in Appendix A.

Though we fix training size, the number of unique lemmas in the training data varies across languages. To investigate the effect of training lemmas on SEEN and NOVEL accuracy, we fit two beta regression models, implemented via the mcvg package in R with family = betar and the default logit linking function. We choose beta regression because our response variables are in the interval (0, 1) and follow a beta distribution (Appendix A), and we log-scale LEMMAS, our predictor, to obtain a better model fit. The fitted ACCURACY ~ LEMMAS models are given in Fig. 3. We note that NOVEL accuracy falls off much more steeply as the number of training lemmas decreases than SEEN accuracy; this is most striking for CHR-TRM.

Interestingly, the SEEN \( \beta \) coefficients are all significantly negative (\(-0.120^*\) for CHR-TRM, \(-0.130^*\) for CLUZH and \(-0.098^*\) for DEEPSPIN), suggesting that performance on seen lemmas actually *decreases* as the number of training lemmas increases. This may be a function of feature overlap: since training size is fixed, languages with more training lemmas will have smaller paradigms available in train, which may lead to a greater proportion of test triples containing novel features. Indeed, there is a significant negative relationship between the number of training lemmas and the proportion of test triples with attested features in our data (Pearson’s \( r = -0.407^* \), Spearman’s \( \rho = -0.423^* \), Kendall’s \( \tau_B = -0.332^* \)). Since feature overlap is known to play a key role in model performance (Kodner et al., 2022), this may explain the relationship we observe for SEEN accuracy. Further, the number of times a model is exposed to a single lemma in train *regardless* of feature overlap may affect its ability to generalize that lemma to other parts of its paradigm. Both of these points are closely tied to the notion of *paradigm saturation*, or the proportion of its paradigm that a lemma is attested in during train (Chan, 2008; Lignos and Yang, 2016). Future work should further investigate the interplay between paradigm saturation, lemma overlap, and feature overlap in predicting models’ generalization abilities.

In contrast, NOVEL accuracy significantly increases as a function of the number of training lemmas (\( \beta = 0.558^* \) for CHR-TRM, 0.148* for CLUZH, 0.190* for DEEPSPIN). The combination of the negative coefficients for SEEN accuracy and the positive ones for NOVEL accuracy further mean that there is a significant relationship between DROP and LEMMAS for all architectures (\( \beta = -0.524^* \) for CHR-TRM, \(-0.174^*\) for CLUZH, and \(-0.194^*\) for DEEPSPIN). Despite this significant relationship however, the overall effect of lemma overlap is still far less than that reported by Goldman et al. at only 11% on average.

4 Exp. 2: Explaining the Discrepancy

The difference between our results and those of Goldman et al. (2022) are partly explainable by the relationship between training lemmas and accuracy drop outlined above. Since we fix our train size, we exclude from our analysis languages with minis-
Figure 4: The distribution of percent differences in triple and lemma counts between the SIGMORPHON 2020 to the Goldman et al. training data.

<table>
  <tr>
    <th></th>
    <th colspan="3">SIGMORPHON</th>
    <th colspan="3">CHIR-TRM</th>
    <th colspan="3">CLUZH</th>
    <th colspan="3">DEEPSPIN</th>
  </tr>
  <tr>
    <th></th>
    <th>β</th>
    <th>AIC</th>
    <th>BIC</th>
    <th>β</th>
    <th>AIC</th>
    <th>BIC</th>
    <th>β</th>
    <th>AIC</th>
    <th>BIC</th>
    <th>β</th>
    <th>AIC</th>
    <th>BIC</th>
  </tr>
  <tr>
    <th>TRIPLES</th>
    <td>0.296</td>
    <td>-17170.90</td>
    <td>-17167.85</td>
    <td>0.270*</td>
    <td>-15843.95</td>
    <td>-15840.90</td>
    <td>0.129</td>
    <td>-17413.73</td>
    <td>-17410.68</td>
  </tr>
  <tr>
    <th>LEMMAS</th>
    <td>0.156</td>
    <td>-17056.13</td>
    <td>-17053.08</td>
    <td>0.270*</td>
    <td>-15712.94</td>
    <td>-15709.89</td>
    <td>0.129</td>
    <td>-17356.09</td>
    <td>-17353.04</td>
  </tr>
  <tr>
    <th>GOLDMAN</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <th></th>
    <th>β</th>
    <th>AIC</th>
    <th>BIC</th>
    <th>β</th>
    <th>AIC</th>
    <th>BIC</th>
    <th>β</th>
    <th>AIC</th>
    <th>BIC</th>
    <th>β</th>
    <th>AIC</th>
    <th>BIC</th>
  </tr>
  <tr>
    <th>TRIPLES</th>
    <td>0.643</td>
    <td>-3694.535</td>
    <td>-3691.483</td>
    <td>0.396</td>
    <td>-10671.66</td>
    <td>-10668.61</td>
    <td>0.286</td>
    <td>-12273.43</td>
    <td>-12270.37</td>
  </tr>
  <tr>
    <th>LEMMAS</th>
    <td>0.811*</td>
    <td>-3795.211</td>
    <td>-3795.258</td>
    <td>0.370*</td>
    <td>-10473.82</td>
    <td>-10470.76</td>
    <td>0.192</td>
    <td>-12163.35</td>
    <td>-12160.30</td>
  </tr>
</table>

Table 2: \( \beta \) coefficients (asterisks indicate \( p < 0.05 \)) and AIC & BIC values (the better predictor is bolded) for the accuracy beta regression models.

cule numbers of training lemmas; Goldman et al. include these languages in their analysis. Indeed, the median number of training lemmas in our data is 1,015 vs. just 141 for Goldman et al., despite comparable means (1,625 and 2,038, respectively).

However, since Goldman et al. train two separate models for each language (Fig. 1), the resulting differences in performance could also be attributable to differences in the training data itself. Even if Goldman et al. had used an identical splitting algorithm across the two train-test splits, the performance of morphological ANNs varies across random seeds (Liu and Prud’hommeaux, 2022, i.a.) and initializations (e.g. Corkery et al., 2019). Moreover, since their train sets were derived from separate splitting algorithms, they show substantial distributional differences which may be relevant to performance. Firstly, though Goldman et al. retain the train-test ratios from SIGMORPHON 2020, the number of triples in their train splits do not match exactly since they split on entire paradigms. They report an average decrease in training triples of 3.5%, but there is a great deal of variation (\( \sigma = 6.341\% \)), with one language (Ludic) experiencing a 47.3% decrease. What’s more, Goldman et al.’s lemma-splitting strategy results in a 28.846% (\( \sigma = 5.221\% \)) average drop in the number of training lemmas between the SIGMORPHON 2020 splits and their splits, which they do not report. The cause of this drop is intuitive: if we assume that nearly all available lemmas will appear at least once in the SIGMORPHON training data, then since Goldman et al. sample 70% for train, we expect about a 30% decrease in the number of training lemmas. Fig. 4 shows the distributions of train triple and lemma count decreases.

Implicit in the conclusions drawn by Goldman et al. (2022) is the assumption that the decrease in the number of training lemmas does not meaningfully influence their comparison, and thus that the number of training lemmas is not a predictor of model performance or performance drop. The evidence presented in §3 already suggests that this is not the case, but to further investigate this point, we replicate a subset of Goldman et al.’s results and investigate predictors of raw accuracy and accuracy drop. We show that the number of training lemmas, both measured directly and in terms of the decrease between the two training sets, has a significant effect on accuracy and is the best predictor of accuracy drop across architectures. We further show that Goldman et al.’s reported accuracy drops are substantially higher than they would have been had they held lemma counts constant.

4.1 Setup

We focus our replication on three language families: Niger-Congo, Uralic, and Romance. These were chosen for the range of SIGMORPHON data sizes: Niger-Congo data are typically quite small, while Uralic are large, and Romance contains a mix. Languages are listed in Appendix B. We retain the neural architectures from §3.3.

4.2 Accuracy and Accuracy Drop

To differentiate our results on the Goldman et al. splits from the results reported by Goldman et al., we denote our results as GOLDMAN. We calculate GOLDMAN accuracy and SIGMORPHON accuracy for each language and neural architecture, as well as the DROP in accuracy between these two. Since Goldman et al. did not fix train size in their study, we consider both TRIPLES, the number of unique training triples, and LEMMAS, the number of unique training lemmas, as possible predictors for raw accuracy. For DROP, we consider each of these predictors as measured on the GOLDMAN
data, as well as LEMMA DROP, the unscaled difference in the number of lemmas from the SIGMORPHON to the GOLDMAN training data.7 Following §3, for each data set and neural architecture, we fit a beta regression for each predictor, all log-scaled.

The fitted models for raw accuracy are in Fig. 5. Similarly to §3, we note that both TRIPLES and LEMMAS have a much stronger effect on GOLDMAN accuracy than SIGMORPHON. This is mirrored by the \( \beta \) coefficients, which are given in Table 2, along with AIC and BIC, two measures of the goodness of model fit that balance model performance, in terms of log likelihood, with model complexity. For both measures, lower values indicate a better model. Since we used a logit linking function in our beta regression model, our \( \beta \) coefficients are *log-odds interpretable*: a one-unit increase in the predictor increases the log-odds of the predicted mean proportion by \( \beta \). As such, larger values of \( \beta \) correspond to stronger (or steeper, in terms of slope) influences on accuracy. In line with Fig. 5, \( \beta \) coefficients are consistently larger for GOLDMAN than SIGMORPHON accuracy. For CHR-TRM on GOLDMAN, LEMMAS is the better predictor of accuracy, as evidenced by the significant \( \beta \) coefficients and lower AIC and BIC values. In all other cases, TRIPLES is the better predictor.

7One could also consider scaling LEMMA DIFFERENCE as a proportion decrease. However, because these scaled values are so closely clustered around the mean, there is little relation between them and DROP.

<table>
  <tr>
    <th>Model</th>
    <th>S ACC</th>
    <th>G ACC</th>
    <th>DROP</th>
    <th>Reported</th>
  </tr>
  <tr>
    <td>CHR-TRM</td>
    <td>93.451</td>
    <td>61.067</td>
    <td>-32.383</td>
    <td>-37.00</td>
  </tr>
  <tr>
    <td>DEEPSPIN</td>
    <td>94.036</td>
    <td>86.377</td>
    <td>-7.660</td>
    <td>-14.00</td>
  </tr>
  <tr>
    <td>CLUZH</td>
    <td>91.222</td>
    <td>81.790</td>
    <td>-9.432</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Macro-Avg</td>
    <td>92.903</td>
    <td>76.411</td>
    <td>-16.492</td>
    <td>-30.00</td>
  </tr>
  <tr>
    <th colspan="5">Family</th>
  </tr>
  <tr>
    <th>Family</th>
    <th>S ACC</th>
    <th>G ACC</th>
    <th>DROP</th>
    <th>Reported</th>
  </tr>
  <tr>
    <td>Niger-Congo</td>
    <td>97.470</td>
    <td>76.162</td>
    <td>-21.308</td>
    <td>-39.00</td>
  </tr>
  <tr>
    <td>Romance</td>
    <td>97.999</td>
    <td>85.478</td>
    <td>-12.521</td>
    <td>-28.00</td>
  </tr>
  <tr>
    <td>Uralic</td>
    <td>87.502</td>
    <td>72.035</td>
    <td>-15.467</td>
    <td>-23.00</td>
  </tr>
  <tr>
    <td>Macro-Avg</td>
    <td>94.323</td>
    <td>77.891</td>
    <td>-16.432</td>
    <td>-30.00</td>
  </tr>
</table>

Table 3: SIGMORPHON (S ACC) and GOLDMAN (G ACC) accuracy and DROP by architecture and language family compared to Goldman et al. reported drops.

For each dataset and neural architecture, we fit an additional beta regression model using both TRIPLES and LEMMAS as predictors. Since our two predictors are highly co-linear (\( r = 0.904, \rho = 0.891, \tau_b = 0.717 \) for SIGMORPHON; \( r = 0.892, \rho = 0.891, \tau_b = 0.722 \) for GOLDMAN), we use the better predictor as the *reference category*, so the effect of the second predictor is only measured as that *beyond* the effect of the first. An ANOVA test determines that the second predictor does not have a significant effect beyond the first for any architecture on the GOLDMAN data. For SIGMORPHON, LEMMAS has a significant effect beyond that of TRIPLES for CHR-TRM and DEEPSPIN. Full model details and outputs are in Appendix B.

We now turn from raw accuracy to accuracy DROP, which Goldman et al. reported in their original study. Table 3 lists our observed accu-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">TRIPLES</th>
    <th colspan="3">LEMMAS</th>
    <th colspan="3">LEMMAS DROP</th>
  </tr>
  <tr>
    <th>Coeff.</th>
    <th>AIC</th>
    <th>BIC</th>
    <th>Coeff.</th>
    <th>AIC</th>
    <th>BIC</th>
    <th>Coeff.</th>
    <th>AIC</th>
    <th>BIC</th>
  </tr>
  <tr>
    <td>CHR-TRM</td>
    <td>0.854*</td>
    <td>1550.002</td>
    <td>1553.055</td>
    <td>0.904*</td>
    <td><b>1016.286</b></td>
    <td><b>1019.338</b></td>
    <td>0.816*</td>
    <td>1193.401</td>
    <td>1196.454</td>
  </tr>
  <tr>
    <td>CLUZH</td>
    <td>0.239*</td>
    <td>5855.958</td>
    <td>5859.011</td>
    <td>0.287*</td>
    <td><b>5823.695</b></td>
    <td><b>5826.747</b></td>
    <td>0.300*</td>
    <td>5831.558</td>
    <td>5834.611</td>
  </tr>
  <tr>
    <td>DeepSpin</td>
    <td>0.238*</td>
    <td>7018.613</td>
    <td>7021.666</td>
    <td>0.210*</td>
    <td>7027.519</td>
    <td>7030.572</td>
    <td>0.253*</td>
    <td><b>6992.300</b></td>
    <td><b>6995.352</b></td>
  </tr>
</table>

Table 4: \( \beta \) coefficients (asterisks indicate \( p < 0.05 \)) and AIC and BIC values (the better predictor is bolded) for the three predictors of DROP.

<table>
  <tr>
    <th>Model</th>
    <th>CONSTANT</th>
    <th>DECREASE</th>
    <th>Diff</th>
  </tr>
  <tr>
    <td>CHR-TRM</td>
    <td>25.689</td>
    <td>30.585</td>
    <td>4.896</td>
  </tr>
  <tr>
    <td>DeepSpin</td>
    <td>6.681</td>
    <td>7.521</td>
    <td>0.840</td>
  </tr>
  <tr>
    <td>CLUZH</td>
    <td>7.010</td>
    <td>8.855</td>
    <td>1.845</td>
  </tr>
  <tr>
    <td>Macro-Avg.</td>
    <td>13.127</td>
    <td>15.654</td>
    <td>2.527</td>
  </tr>
</table>

Table 5: Predicted CONSTANT/DECREASE acc. drops

racy drops by neural architecture and by language family, compared to those reported by Goldman et al.. We observe an average decrease of 45% from the values reported by Goldman et al. to our setup. This discrepancy is explainable in two ways. First, CLUZH outperformed Goldman et al.’s base LSTM, which brought down average performance on unseen lemmas. Second, the families we omitted from our replication contain many languages with extremely small training sets, for which we expect there to be larger drops given Fig. 5.

We also perform a beta regression with each predictor for DROP, and find that the best predictor for all three architectures depends on lemmas: either LEMMAS or LEMMA DROP (Table 4). Given the collinearity between these predictors, however (LEMMAS and LEMMA DROP: \( r = 0.986; \rho = 0.978, \tau_b = 0.945 \)), we wish to test whether the addition of either of the other predictors significantly increases in the predictive power of the beta regression model. As for ACCURACY, we train a third beta regression model for each architecture with all three predictors, again using the best predictor as the reference category, with the other two added in order of increasing AIC. For all three architectures, an ANOVA test indicates that addition of neither the second nor third predictor led to a significant increase in model performance.

In sum, though TRIPLES is often an important predictor of raw accuracy, LEMMAS or LEMMA DROP are the key predictors of DROP, which was the original focus of Goldman et al. (2022). As such, the drop in training lemmas from the SIGMORPHON to GOLDMAN data can be expected to significantly affect the drop in accuracy between these datasets, a confound which favors the increased accuracy drop reported by Goldman et al..

4.3 Quantifying the Confound

Our results thus far demonstrate that the number of training lemmas plays a crucial role in predicting the drop in accuracy between SIGMORPHON and GOLDMAN: train sets with fewer lemmas exhibit a larger drop. As we saw above, however, the GOLDMAN setup creates train sets with fewer lemmas than their SIGMORPHON counterparts even when their size in triples is held constant. Consider, for example, a language with 1000 training lemmas in SIGMORPHON. Because accuracy will fall off for the GOLDMAN data as the number of training lemmas decreases, the drop in accuracy between the 1000-lemma SIGMORPHON sample and a 1000-lemma GOLDMAN sample will be significantly less than that between a 1000-lemma SIGMORPHON sample and the 700-lemma GOLDMAN sample predicted by the near-30% decrease in the number of training lemmas.

We can further formalize this intuition using the beta regression models from §4.2. For values of LEMMAS from 10 to 25,000, we calculate two types of accuracy drops. DECREASE represents Goldman et al.’s actual approach in which train sets averaged nearly 30% fewer lemmas than their SIGMORPHON 2020 counterparts. It is calculated by subtracting the predicted GOLDMAN accuracy on 0.7L lemmas from the predicted SIGMORPHON accuracy on L lemmas. CONSTANT drop is meant to model the results had train lemmas been held constant, and is calculated by subtracting the predicted GOLDMAN accuracy on L lemmas from the predicted SIGMORPHON accuracy on L lemmas.

Fig. 6 shows the differences between the CONSTANT and DECREASE predicted drops for each of the neural architectures as a function of the number of training LEMMAS plotted on top of the actual distribution of training lemmas in the GOLDMAN and SIGMORPHON data. Notably, the distribution of train sizes peaks at the point where Goldman et al.’s DECREASE lemma-splitting approach overpredicts accuracy drop by the greatest degree. As such, the original study’s confounds mutually reinforced one another to increase accuracy drops.
Figure 6: The difference between the CONSTANT and DECREASE predicted drop as a function of the number of training lemmas, plus the distribution of training lemmas in the SIGMORPHON and GOLDMAN data.

Table 5 gives the average predicted CONSTANT and DECREASE accuracy drops for each architecture for the actual values of training lemmas in the SIGMORPHON and GOLDMAN data. The average difference in drop is 2.5%, with the maximum (for CHR-TRM) estimated at 4.9%. This means that if Goldman et al. (2022) had held the number of training lemmas constant between their overlapping and non-overlapping splits, they would have found an effect of lemma overlap several percentage points smaller than the one they reported.

5 Discussion

The findings presented here suggest a more nuanced view of the role of lemma overlap than proposed by either Goldman et al. (2022) or Kodner et al. (2022). With our novel splitting algorithm (§3), we find that all neural architectures achieve substantially higher accuracy on seen than unseen lemmas, unlike Kodner et al., but the difference is much lower (11%), about a third of that reported by Goldman et al. reported (30%). More importantly, however, we show that the number of lemmas in train has a much stronger effect on accuracy on test triples containing unseen lemmas than seen ones.

While there are several incidental differences between our study and the two prior studies, we established one principled difference which resulted in the especially large performance drop which Goldman et al. reported: the number of lemmas in the training set (absolute or relative to the original set) is the most important predictor for accuracy drop, with larger accuracy drops occurring for languages with fewer train lemmas. Goldman et al. tested the effect of lemma overlap with two distinct training sets, one which allowed for lemma overlap, and one that both banned overlap and was on average 30% smaller in terms of lemma count; this was a confound which inflated accuracy drop. As we found in §4.3, Goldman et al.’s data was also highly skewed towards small training sizes where the impact of this confound was at its greatest.

Our own study of lemma overlap was set up similarly to Kodner et al. (2022) and Kodner et al. (2023b) in that we trained models on a single training set and tested them on distinct test sets (Fig. 2). This eliminates the potential for confounds such as the training lemma count. Unlike Kodner et al., who found no effect of lemma overlap, however, we found a substantial drop. One possible reason for this discrepancy is the role of feature attestation: in our study, we ensured a 50% seen/unseen lemma split in test. Doing so on at least 10,000 triple data sets caused feature set overlap to rise to 99.5%, thus controlling for this additional factor; indeed, our analysis does not significantly change when conducted only over test triples containing seen feature sets. Kodner et al., on the other hand, fixed feature set overlap to a maximum of 50% and did not control for lemma overlap explicitly.

It is possible that this discrepancy emerges as an artifact of the feature-based splitting algorithm or of the interaction between lemma and feature overlap. If, for example, lemma overlap has a small but significant effect for triples with seen features (what our results show) but does not play a role in triples with unseen features, this could help to explain the results of Kodner et al.. Now that we have established a better understanding of the effect of lemma overlap on triples with seen feature sets, future work should further investigate the interaction between lemma overlap and feature overlap. As discussed in §3, future work should also investigate the effect of the number of times a given lemma
or feature set has been seen in train rather than making a binary attested-unattested distinction.

The findings presented here have implications for our understanding of subword-level generalization in different classes of neural network and are thus an important contribution interpretability for these models. Intuitively, generalizing morphological inflections to unseen lemmas is a more challenging task than generalization to seen lemmas, since it requires generalization from the lemma and feature sets only. Without additional evidence from other inflected forms of the lemma, we cannot know, for example, if it follows a regular or irregular pattern.

While all models performed similarly to each other on languages with the largest numbers of training lemmas (Fig. 3), they diverged in their ability to generalize on languages with the smallest numbers of training lemmas. CHR-TRM, a character transformer trained on a single language, struggled the most to generalize to unseen lemmas on the smallest training sets, conforming with our expectations about the data hungry nature of transformers. On the other hand, DEEPSPIN, a multilingual model with a much simpler BiLSTM architecture, much more readily generalized to unseen lemmas even at the smallest training sizes and was nearly immune to performance drop.

6 Conclusion

In this paper, we have demonstrated that the number of lemmas in the training data significantly affects the performance of neural models on unseen lemmas, but not on seen lemmas. This result entails that neural models struggle to generalize to novel lemmas when trained on small data, an important step to understanding how different classes of model generalize at a subword level. Beyond just understanding how models generalize or fail to, inflection still remains an important element in NLP for morphologically rich low-resource and endangered languages, as evidenced by contributions to recent workshops on NLP for under-resourced, endangered, and indigenous language families.8

In addition to the main results concerning generalization to seen and unseen lemmas, this paper provides a methodological contribution. We find that careful experimental design lets us not only reassess top line numbers claimed in prior work (namely, Goldman et al. 2022), but also provides an explanation for the causes of previous discrepancies and allows us to identify confounds. We evaluated the role of lemma overlap while controlling for a number of potential confounds by holding the training set constant. We augmented the reporting of top-level accuracy numbers with statistical analysis modeling the role of confounding factors on performance. It is likely that these approaches will prove useful in future work assessing patterns in morphological generalization. In particular, now that a separate understanding of the roles of feature overlap and lemma overlap has been established, future work is well-positioned to investigate their interaction. While overlap has been quantified in terms of binary attestation in train, future work may also consider the number of times a given lemma or feature set is attested in order to gain a more complete picture of factors affecting generalization.

Acknowledgements

We are grateful to Sandy Abu El Adas, Mark Aronoff, Smeet Chheda, Jeff Heinz, Scott Nelson, Owen Rambow, and the anonymous reviewers for their feedback on this work. SP gratefully acknowledges support from the Institute for Advanced Computational Science Graduate Research Fellowship and the National Science Foundation Graduate Research Fellowship Program under NSF Grant No. 2234683. Experiments were performed on the SeaWulf HPC cluster maintained by RCC and IACS at Stony Brook University and made possible by NSF grant No. 1531492.

Limitations

We discuss two key limitations. Firstly, any experiment of this nature must generalize from the specific neural architectures and languages tested to draw more general claims. In §4, we discussed the effects our inclusion of CLUZH and omission of Goldman et al.’s base LSTM, but future work should investigate more neural architectures, as well as more language samples as they become available in future SIGMORPHON shared tasks. Secondly, while we have conducted an extensive comparison between our results and those of Goldman et al., we have yet to conduct a similar comparison with Kodner et al. since the latter paper also deals with feature overlap. Now that we have a better understanding of the factors affecting lemma overlap, however, we are better positioned to investigate its interaction with feature overlap, which we plan to do in future work.

8e.g., ComputEL, AmericasNLP, RAIL, ArabicNLP
References

Catherine Arnett and Benjamin Bergen. 2025. Why do language models perform worse for morphologically complex languages? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6607–6623, Abu Dhabi, UAE. Association for Computational Linguistics.

Emily M. Bender. 2009. Linguistically naïve != language independent: Why NLP needs linguistic typology. In Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?, pages 26–32, Athens, Greece. Association for Computational Linguistics.

Erwin Chan. 2008. Structures and distributions in morphological learning. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA.

Jane Chandlee. 2017. Computational locality in morphological maps. Morphology, 27(4):599–641.

Maria Corkery, Yevgen Matusevych, and Sharon Goldwater. 2019. Are we there yet? encoder-decoder neural networks as cognitive models of English past tense inflection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3868–3877, Florence, Italy. Association for Computational Linguistics.

Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. The CoNLL–SIGMORPHON 2018 shared task: Universal morphological reinflection. In Proceedings of the CoNLL–SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection, pages 1–27, Brussels. Association for Computational Linguistics.

Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. CoNLL–SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1–30, Vancouver. Association for Computational Linguistics.

Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The SIGMORPHON 2016 shared Task—Morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10–22, Berlin, Germany. Association for Computational Linguistics.

Omer Goldman, Khuyagbaatar Batsuren, Salam Khalifa, Aryaman Arora, Garrett Nicolai, Reut Tsarfaty, and Ekaterina Vylomova. 2023. SIGMORPHON–UniMorph 2023 shared task 0: Typologically diverse morphological inflection. In Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 117–125, Toronto, Canada. Association for Computational Linguistics.

Omer Goldman, David Guriel, and Reut Tsarfaty. 2022. (un)solving morphological inflection: Lemma overlap artificially inflates models’ performance. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 864–870, Dublin, Ireland. Association for Computational Linguistics.

Jordan Kodner, Salam Khalifa, Khuyagbaatar Batsuren, Hossep Dolatian, Ryan Cotterell, Faruk Akkus, Antonios Anastasopoulos, Taras Andrushko, Aryaman Arora, Nona Atanalov, Gábor Bella, Elena Budianskaya, Yustinus Ghanggo Ate, Omer Goldman, David Guriel, Simon Guriel, Silvia Guriel-Agiashvili, Witold Kieraś, Andrew Krizhanovsky, Natalia Krizhanovsky, Igor Marchenko, Magdalena Markowska, Polina Mashkovtseva, Maria Nepomniashchaya, Daria Rodionova, Karina Schiefer, Alexandra Sorova, Anastasia Yemelina, Jeremiah Young, and Ekaterina Vylomova. 2022. SIGMORPHON–UniMorph 2022 shared task 0: Generalization and typologically diverse morphological inflection. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 176–203, Seattle, Washington. Association for Computational Linguistics.

Jordan Kodner, Salam Khalifa, and Sarah Ruth Brogden Payne. 2023a. Exploring linguistic probes for morphological generalization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8933–8941, Singapore. Association for Computational Linguistics.

Jordan Kodner, Sarah Payne, Salam Khalifa, and Zoey Liu. 2023b. Morphological inflection: A reality check. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6082–6101, Toronto, Canada. Association for Computational Linguistics.

Constantine Lignos and Charles Yang. 2016. Morphology and language acquisition. The Cambridge handbook of morphology, 743764.

Ling Liu and Mans Hulden. 2020. Leveraging principal parts for morphological inflection. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 153–161, Online. Association for Computational Linguistics.

Zoey Liu and Emily Prud’hommeaux. 2022. Data-driven model generalizability in crosslinguistic low-resource morphological segmentation. Transactions of the Association for Computational Linguistics, 10:393–413.
Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning, pages 1614–1623. Proceedings of Machine Learning Research.

Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019. The SIGMORPHON 2019 shared task: Morphological analysis in context and cross-lingual transfer for inflection. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 229–244, Florence, Italy. Association for Computational Linguistics.

Kemal Oflazer and Murat Saraclar. 2018. Turkish Natural Language Processing. Theory and Applications of Natural Language Processing. Springer International Publishing, Cham, Switzerland.

Ben Peters and André F. T. Martins. 2020. One-size-fits-all multilingual models. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 63–69, Online. Association for Computational Linguistics.

Tiago Pimentel, Maria Ryskina, Sabrina J. Mielke, Shijie Wu, Eleanor Chodroff, Brian Leonard, Garrett Nicolai, Yustinus Ghanggo Ate, Salam Khalifa, Nizar Habash, Charbel El-Khaissi, Omer Goldman, Michael Gasser, William Lane, Matt Coler, Arturo Oncevay, Jaime Rafael Montoya Samame, Gema Celeste Silva Villegas, Adam Ek, Jean-Philippe Bernardy, Andrey Shcherbakov, Aziyana Bayyr-ool, Karina Sheifer, Sofya Ganieva, Matvey Plugarov, Elena Klyachko, Ali Salehi, Andrew Krizhanovsky, Natalia Krizhanovsky, Clara Vania, Sardana Ivanova, Aelita Salchak, Christopher Straughn, Zoey Liu, Jonathan North Washington, Duygu Ataman, Witold Kieraś, Marcin Woliński, Totok Suhardjanto, Niklas Stoehr, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Richard J. Hatcher, Emily Prud’hommeaux, Ritesh Kumar, Mans Hulden, Botond Barta, Dorina Lakatos, Gábor Szolnok, Judit Ács, Mohit Raj, David Yarowsky, Ryan Cotterell, Ben Ambridge, and Ekaterina Vylomova. 2021. SIGMORPHON 2021 shared task on morphological reinflection: Generalization across languages. In Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 229–259, Online. Association for Computational Linguistics.

Maciej Rapacz and Aleksander Smywiński-Pohl. 2025. Low-resource interlinear translation: Morphology-enhanced neural models for Ancient Greek. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 145–165, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Rob van der Goot. 2021. We need to talk about train-dev-test splits. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4485–4494, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Ekaterina Vylomova, Jennifer White, Elizabeth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrod, Josef Valvoda, Svetlana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky, Paula Czarnowska, Irene Nikkarinen, Andrew Krizhanovsky, Tiago Pimentel, Lucas Torroba Hennigen, Christo Kirov, Garrett Nicolai, Adina Williams, Antonios Anastasopoulos, Hilaria Cruz, Eleanor Chodroff, Ryan Cotterell, Miikka Silfverberg, and Mans Hulden. 2020. SIGMORPHON 2020 shared task 0: Typologically diverse morphological inflection. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 1–39, Online. Association for Computational Linguistics.

Silvan Wehrli, Simon Clematide, and Peter Makarov. 2022. CLUZH at SIGMORPHON 2022 shared tasks on morpheme segmentation and inflection generation. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 212–219, Seattle, Washington. Association for Computational Linguistics.

Adam Wiemerslage, Miikka Silfverberg, Changbing Yang, Arya McCarthy, Garrett Nicolai, Eliana Colunga, and Katharina Kann. 2022. Morphological processing of low-resource languages: Where we are and what’s next. In Findings of the Association for Computational Linguistics: ACL 2022, pages 988–1007, Dublin, Ireland. Association for Computational Linguistics.

Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. Applying the transformer to character-level transduction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1901–1907, Online. Association for Computational Linguistics.
A Experiment 1

Algorithm 1 Our splitting algorithm for Expt. 1
Require: LEMMAS (all lemmas in data), TRIPLES (all triples in data), trainsize, ftunsize, testsize
1: OVERLAPLEMMAS ← random(LEMMAS, ⌊0.5 × | LEMMAS |⌋)
2: OVERLAPTRIPLES ← {(L, F, I) ∈ TRIPLES | L ∈ OVERLAPLEMMAS}
3: while |OVERLAPTRIPLES| < |trainsize + ftunsize + 0.5 × testsize| do
4:   OVERLAPLEMMAS ← OVERLAPLEMMAS ∪ random(LEMMAS \ OVERLAPLEMMAS, 1)
5:   OVERLAPTRIPLES ← {(L, F, I) ∈ TRIPLES | L ∈ OVERLAPLEMMAS}
6: end while
7: TRAIN ← random(OVERLAPTRIPLES, trainsize)
8: FINETUNE ← random(OVERLAPTRIPLES \ TRAIN, ftunsize)
9: TESTO ← random(OVERLAPTRIPLES \ (TRAIN \ FINETUNE), ⌊0.5 × testsize⌋)
10: TESTN ← random(TRIPLES \ OVERLAPTRIPLES, ⌊0.5 × testsize⌋)
11: TEST ← TESTO ∪ TESTN
12: return TRAIN, FINETUNE, TEST

<table>
  <tr>
    <th>FAMILY</th>
    <th>LANGUAGE</th>
  </tr>
  <tr>
    <td rowspan="7">Germanic</td>
    <td>Old English (ang)</td>
  </tr>
  <tr>
    <td>Danish (dan)</td>
  </tr>
  <tr>
    <td>German (deu)</td>
  </tr>
  <tr>
    <td>English (eng)</td>
  </tr>
  <tr>
    <td>Icelandic (isl)</td>
  </tr>
  <tr>
    <td>Dutch (nld)</td>
  </tr>
  <tr>
    <td>Norwegian Nyorsk (nno)</td>
  </tr>
  <tr>
    <td>Norwegian Bokmal (nob)</td>
  </tr>
  <tr>
    <td>Swedish (swe)</td>
  </tr>
  <tr>
    <td rowspan="3">Indo-Aryan</td>
    <td>Hindi (hin)</td>
  </tr>
  <tr>
    <td>Sanskrit (san)</td>
  </tr>
  <tr>
    <td>Urdu (urd)</td>
  </tr>
  <tr>
    <td>Iranian</td>
    <td>Persian (fas)</td>
  </tr>
  <tr>
    <td rowspan="3">Oto-Manguecan</td>
    <td>San Pedro Amuzgos (azg)</td>
  </tr>
  <tr>
    <td>Mezquital Otomi (ote)</td>
  </tr>
  <tr>
    <td>Sierra Otomi (otm)</td>
  </tr>
  <tr>
    <td>Chichimeca-Jonaz (pei)</td>
  </tr>
  <tr>
    <td rowspan="4">Romance</td>
    <td>Catalan (cat)</td>
  </tr>
  <tr>
    <td>Middle French (frm)</td>
  </tr>
  <tr>
    <td>Galician (glg)</td>
  </tr>
  <tr>
    <td>Venetian (vec)</td>
  </tr>
  <tr>
    <td rowspan="4">Turkish</td>
    <td>Bashkir (bak)</td>
  </tr>
  <tr>
    <td>Kazakh (kaz)</td>
  </tr>
  <tr>
    <td>Turkmen (tuk)</td>
  </tr>
  <tr>
    <td>Uzbek (uzb)</td>
  </tr>
  <tr>
    <td rowspan="7">Uralic</td>
    <td>Estonian (est)</td>
  </tr>
  <tr>
    <td>Finnish (fin)</td>
  </tr>
  <tr>
    <td>Komi-Zyrian (kpv)</td>
  </tr>
  <tr>
    <td>Karelian (kr1)</td>
  </tr>
  <tr>
    <td>Moksha (mdf)</td>
  </tr>
  <tr>
    <td>Meadow Mari (mhr)</td>
  </tr>
  <tr>
    <td>Erzya (myv)</td>
  </tr>
  <tr>
    <td>Livvi (olo)</td>
  </tr>
  <tr>
    <td>Northern Sami (sme)</td>
  </tr>
  <tr>
    <td>Udmurt (udm)</td>
  </tr>
  <tr>
    <td>Veps (vep)</td>
  </tr>
</table>

Table 6: The 36 SIGMORPHON 2020 languages used in Experiment 1.

<table>
  <tr>
    <th>Family</th>
    <th>Seen</th>
    <th>Unseen</th>
    <th>Difference</th>
  </tr>
  <tr>
    <td>Germanic</td>
    <td>86.422</td>
    <td>79.230</td>
    <td>-7.193</td>
  </tr>
  <tr>
    <td>Indo-Aryan</td>
    <td>97.176</td>
    <td>79.724</td>
    <td>-17.451</td>
  </tr>
  <tr>
    <td>Iranian</td>
    <td>99.673</td>
    <td>76.447</td>
    <td>-23.227</td>
  </tr>
  <tr>
    <td>Oto-Manguecan</td>
    <td>90.339</td>
    <td>54.920</td>
    <td>-35.419</td>
  </tr>
  <tr>
    <td>Romance</td>
    <td>99.172</td>
    <td>93.600</td>
    <td>-5.572</td>
  </tr>
  <tr>
    <td>Turkic</td>
    <td>95.098</td>
    <td>83.532</td>
    <td>-11.567</td>
  </tr>
  <tr>
    <td>Uralic</td>
    <td>92.064</td>
    <td>87.013</td>
    <td>-5.052</td>
  </tr>
  <tr>
    <td>Macro-average</td>
    <td>94.278</td>
    <td>79.209</td>
    <td>-15.068</td>
  </tr>
</table>

Table 7: Experiment 1 accuracy by language family, averaged across neural architectures & seeds. This can be compared to Table 2 in Goldman et al. (2022), though the family groupings in their paper are different than those in their data and the SIGMORPHON data. Rather than attempt to replicate this grouping without sufficient information to do so, we instead report grouping by the language families in Goldman et al. (2022)'s data splits.

![Three density plots showing the distribution of Seen, Novel, and Drop accuracy for different language families](page_1012_1012_388_388.png)

Figure 7: SEEN and NOVEL accuracy and DROP are beta distributed in Experiment 1.

A.1 Beta regression models for SEEN lemmas

The following provides the full R output of the SEEN ~ LEMMAS beta regression models in Experiment 1 (\S 3.4).

CHR-TRM
Call: gam(formula = scaled_seen ~ log(train_lemmas),
  family = betar(link = "logit"),
  data = wu_res_clean)
Deviance Residuals:
  Min      1Q  Median      3Q     Max
-1.7781 -0.7678  0.0000  0.0000  0.7016
<table>
  <tr>
    <th>Lang</th>
    <th>CHR-TRM</th>
    <th>SEEN</th>
    <th>CLUZH</th>
    <th>DEEPSPIN</th>
    <th>CHR-TRM</th>
    <th>NOVEL</th>
    <th>CLUZH</th>
    <th>DEEPSPIN</th>
    <th>CHR-TRM</th>
    <th>DROP</th>
    <th>CLUZH</th>
    <th>DEEPSPIN</th>
  </tr>
  <tr>
    <td>ang</td>
    <td>72.873</td>
    <td>72.150</td>
    <td>72.631</td>
    <td>46.236</td>
    <td>49.739</td>
    <td>49.582</td>
    <td>-26.637</td>
    <td>-22.411</td>
    <td>-23.049</td>
  </tr>
  <tr>
    <td>azg</td>
    <td>93.980</td>
    <td>93.060</td>
    <td>92.480</td>
    <td>23.400</td>
    <td>45.020</td>
    <td>43.100</td>
    <td>-70.580</td>
    <td>-48.040</td>
    <td>-49.380</td>
  </tr>
  <tr>
    <td>bak</td>
    <td>99.640</td>
    <td>99.440</td>
    <td>99.600</td>
    <td>92.460</td>
    <td>93.440</td>
    <td>93.380</td>
    <td>-7.180</td>
    <td>-6.000</td>
    <td>-6.220</td>
  </tr>
  <tr>
    <td>cat</td>
    <td>98.880</td>
    <td>98.740</td>
    <td>98.500</td>
    <td>96.160</td>
    <td>96.980</td>
    <td>96.440</td>
    <td>-2.720</td>
    <td>-1.760</td>
    <td>-2.060</td>
  </tr>
  <tr>
    <td>dan</td>
    <td>71.591</td>
    <td>71.992</td>
    <td>66.814</td>
    <td>64.116</td>
    <td>66.168</td>
    <td>66.646</td>
    <td>-7.475</td>
    <td>-5.825</td>
    <td>-0.168</td>
  </tr>
  <tr>
    <td>deu</td>
    <td>94.770</td>
    <td>95.236</td>
    <td>94.516</td>
    <td>88.576</td>
    <td>89.788</td>
    <td>89.635</td>
    <td>-6.194</td>
    <td>-5.448</td>
    <td>-4.881</td>
  </tr>
  <tr>
    <td>eng</td>
    <td>94.847</td>
    <td>95.062</td>
    <td>94.868</td>
    <td>94.783</td>
    <td>94.906</td>
    <td>95.364</td>
    <td>-0.064</td>
    <td>-0.156</td>
    <td>0.496</td>
  </tr>
  <tr>
    <td>est</td>
    <td>93.980</td>
    <td>93.040</td>
    <td>93.920</td>
    <td>75.680</td>
    <td>77.760</td>
    <td>80.280</td>
    <td>-18.300</td>
    <td>-15.280</td>
    <td>-13.640</td>
  </tr>
  <tr>
    <td>fas</td>
    <td>99.900</td>
    <td>99.340</td>
    <td>99.780</td>
    <td>50.880</td>
    <td>90.020</td>
    <td>88.440</td>
    <td>-49.020</td>
    <td>-9.320</td>
    <td>-11.340</td>
  </tr>
  <tr>
    <td>fin</td>
    <td>97.443</td>
    <td>96.238</td>
    <td>96.972</td>
    <td>92.917</td>
    <td>91.703</td>
    <td>92.642</td>
    <td>-4.527</td>
    <td>-4.535</td>
    <td>-4.330</td>
  </tr>
  <tr>
    <td>frm</td>
    <td>99.480</td>
    <td>98.920</td>
    <td>99.080</td>
    <td>89.640</td>
    <td>96.460</td>
    <td>96.160</td>
    <td>-9.840</td>
    <td>-2.460</td>
    <td>-2.920</td>
  </tr>
  <tr>
    <td>gig</td>
    <td>99.580</td>
    <td>99.120</td>
    <td>99.280</td>
    <td>88.880</td>
    <td>94.820</td>
    <td>93.960</td>
    <td>-10.700</td>
    <td>-4.300</td>
    <td>-5.320</td>
  </tr>
  <tr>
    <td>hin</td>
    <td>100.000</td>
    <td>99.960</td>
    <td>100.000</td>
    <td>43.800</td>
    <td>95.440</td>
    <td>95.240</td>
    <td>-56.200</td>
    <td>-4.520</td>
    <td>-4.760</td>
  </tr>
  <tr>
    <td>isl</td>
    <td>91.883</td>
    <td>92.576</td>
    <td>90.887</td>
    <td>81.373</td>
    <td>83.005</td>
    <td>81.531</td>
    <td>-10.511</td>
    <td>-9.571</td>
    <td>-9.356</td>
  </tr>
  <tr>
    <td>kaz</td>
    <td>98.060</td>
    <td>97.580</td>
    <td>98.020</td>
    <td>39.400</td>
    <td>78.320</td>
    <td>79.100</td>
    <td>-58.660</td>
    <td>-19.260</td>
    <td>-18.920</td>
  </tr>
  <tr>
    <td>kpv</td>
    <td>95.108</td>
    <td>93.541</td>
    <td>94.138</td>
    <td>92.876</td>
    <td>91.959</td>
    <td>93.169</td>
    <td>-2.231</td>
    <td>-1.582</td>
    <td>-0.969</td>
  </tr>
  <tr>
    <td>krl</td>
    <td>97.345</td>
    <td>93.987</td>
    <td>97.185</td>
    <td>87.211</td>
    <td>84.906</td>
    <td>89.064</td>
    <td>-10.134</td>
    <td>-9.081</td>
    <td>-8.121</td>
  </tr>
  <tr>
    <td>mdf</td>
    <td>90.467</td>
    <td>87.596</td>
    <td>89.890</td>
    <td>88.012</td>
    <td>86.236</td>
    <td>87.894</td>
    <td>-2.455</td>
    <td>-1.360</td>
    <td>-1.996</td>
  </tr>
  <tr>
    <td>mhr</td>
    <td>88.850</td>
    <td>85.997</td>
    <td>86.628</td>
    <td>85.631</td>
    <td>83.434</td>
    <td>84.338</td>
    <td>-3.220</td>
    <td>-2.564</td>
    <td>-2.289</td>
  </tr>
  <tr>
    <td>myv</td>
    <td>91.041</td>
    <td>88.428</td>
    <td>90.775</td>
    <td>90.909</td>
    <td>89.744</td>
    <td>91.240</td>
    <td>-0.133</td>
    <td>1.316</td>
    <td>0.465</td>
  </tr>
  <tr>
    <td>nld</td>
    <td>98.305</td>
    <td>97.977</td>
    <td>97.958</td>
    <td>92.958</td>
    <td>92.605</td>
    <td>93.174</td>
    <td>-5.347</td>
    <td>-5.371</td>
    <td>-4.783</td>
  </tr>
  <tr>
    <td>nno</td>
    <td>87.080</td>
    <td>87.512</td>
    <td>86.896</td>
    <td>74.293</td>
    <td>74.390</td>
    <td>79.770</td>
    <td>-12.788</td>
    <td>-13.121</td>
    <td>-7.126</td>
  </tr>
  <tr>
    <td>nob</td>
    <td>73.347</td>
    <td>75.938</td>
    <td>72.768</td>
    <td>74.934</td>
    <td>74.638</td>
    <td>79.460</td>
    <td>1.586</td>
    <td>-1.300</td>
    <td>6.692</td>
  </tr>
  <tr>
    <td>olo</td>
    <td>89.789</td>
    <td>89.986</td>
    <td>90.338</td>
    <td>88.761</td>
    <td>88.098</td>
    <td>89.441</td>
    <td>-1.028</td>
    <td>-1.887</td>
    <td>-0.897</td>
  </tr>
  <tr>
    <td>ote</td>
    <td>98.498</td>
    <td>98.097</td>
    <td>98.338</td>
    <td>83.882</td>
    <td>86.339</td>
    <td>86.858</td>
    <td>-14.615</td>
    <td>-11.758</td>
    <td>-11.479</td>
  </tr>
  <tr>
    <td>otm</td>
    <td>97.177</td>
    <td>96.997</td>
    <td>96.797</td>
    <td>71.843</td>
    <td>76.480</td>
    <td>77.297</td>
    <td>-25.335</td>
    <td>-20.517</td>
    <td>-19.500</td>
  </tr>
  <tr>
    <td>pei</td>
    <td>74.180</td>
    <td>73.720</td>
    <td>70.740</td>
    <td>17.360</td>
    <td>24.200</td>
    <td>23.260</td>
    <td>-56.820</td>
    <td>-49.520</td>
    <td>-47.480</td>
  </tr>
  <tr>
    <td>san</td>
    <td>92.520</td>
    <td>92.320</td>
    <td>91.280</td>
    <td>78.180</td>
    <td>83.580</td>
    <td>82.820</td>
    <td>-14.340</td>
    <td>-8.740</td>
    <td>-8.460</td>
  </tr>
  <tr>
    <td>sme</td>
    <td>98.594</td>
    <td>97.689</td>
    <td>97.931</td>
    <td>90.784</td>
    <td>88.873</td>
    <td>89.073</td>
    <td>-7.810</td>
    <td>-8.816</td>
    <td>-8.858</td>
  </tr>
  <tr>
    <td>swe</td>
    <td>94.074</td>
    <td>94.905</td>
    <td>93.951</td>
    <td>86.758</td>
    <td>87.056</td>
    <td>87.718</td>
    <td>-7.316</td>
    <td>-7.849</td>
    <td>-6.233</td>
  </tr>
  <tr>
    <td>tuk</td>
    <td>85.760</td>
    <td>86.340</td>
    <td>84.200</td>
    <td>73.760</td>
    <td>85.600</td>
    <td>85.360</td>
    <td>-12.000</td>
    <td>-0.740</td>
    <td>1.160</td>
  </tr>
  <tr>
    <td>udm</td>
    <td>97.250</td>
    <td>95.753</td>
    <td>96.656</td>
    <td>95.822</td>
    <td>94.748</td>
    <td>95.685</td>
    <td>-1.428</td>
    <td>-1.005</td>
    <td>-0.971</td>
  </tr>
  <tr>
    <td>urd</td>
    <td>99.520</td>
    <td>99.460</td>
    <td>99.520</td>
    <td>42.340</td>
    <td>97.620</td>
    <td>98.500</td>
    <td>-57.180</td>
    <td>-1.840</td>
    <td>-1.020</td>
  </tr>
  <tr>
    <td>uzb</td>
    <td>97.340</td>
    <td>99.180</td>
    <td>96.020</td>
    <td>89.160</td>
    <td>96.780</td>
    <td>95.620</td>
    <td>-8.180</td>
    <td>-2.400</td>
    <td>-0.400</td>
  </tr>
  <tr>
    <td>vec</td>
    <td>99.580</td>
    <td>99.380</td>
    <td>99.520</td>
    <td>84.120</td>
    <td>94.660</td>
    <td>94.920</td>
    <td>-15.460</td>
    <td>-4.720</td>
    <td>-4.600</td>
  </tr>
  <tr>
    <td>vep</td>
    <td>80.656</td>
    <td>79.728</td>
    <td>81.182</td>
    <td>70.419</td>
    <td>70.456</td>
    <td>71.650</td>
    <td>-10.237</td>
    <td>-9.272</td>
    <td>-9.532</td>
  </tr>
</table>

Table 8: Full results by language and neural architecture, averaged across our 5 seeds, for Experiment 1.

(Dispersion Parameter for Beta regression family taken to be 0.1221)

Null Deviance: -56.1663 on 179 degrees of freedom
Residual Deviance: -67.2103 on 178 degrees of freedom
AIC: -440724.1

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
<table>
  <tr>
    <th></th>
    <th>Df</th>
    <th>Sum Sq</th>
    <th>Mean Sq</th>
    <th>F value</th>
    <th>Pr(&gt;F)</th>
  </tr>
  <tr>
    <td>log(train_lemmas)</td>
    <td>1</td>
    <td>0.2035</td>
    <td>0.2035</td>
    <td>8.8191</td>
    <td>0.003392 **</td>
  </tr>
  <tr>
    <td>Residuals</td>
    <td>178</td>
    <td>20.6568</td>
    <td>0.11605</td>
    <td></td>
    <td></td>
  </tr>
</table>

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(train_lemmas)
3.1273269 -0.1189933

CLUZH
Call: gam(formula = scaled_seen ~ log(train_lemmas),
family = betar(link = "logit"),
data = cluzh_res_clean)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.7758 -0.9112 0.0000 0.0000 0.6348

(Dispersion Parameter for Beta regression family taken to be 0.116)

Null Deviance: -42.169 on 179 degrees of freedom
Residual Deviance: -50.6733 on 178 degrees of freedom

AIC: -425849.3

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
<table>
  <tr>
    <th></th>
    <th>Df</th>
    <th>Sum Sq</th>
    <th>Mean Sq</th>
    <th>F value</th>
    <th>Pr(&gt;F)</th>
  </tr>
  <tr>
    <td>log(train_lemmas)</td>
    <td>1</td>
    <td>1.0235</td>
    <td>1.0235</td>
    <td>8.8191</td>
    <td>0.003392 **</td>
  </tr>
  <tr>
    <td>Residuals</td>
    <td>178</td>
    <td>20.6568</td>
    <td>0.11605</td>
    <td></td>
    <td></td>
  </tr>
</table>

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(train_lemmas)
3.1582321 -0.1301994

DEEPSPIN
Call: gam(formula = scaled_seen ~ log(train_lemmas),
family = betar(link = "logit"),
data = ds_res_clean)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.7519 -0.7813 0.0000 0.0000 0.6982

(Dispersion Parameter for Beta regression family taken to be 0.1325)

Null Deviance: -49.5337 on 179 degrees of freedom
Residual Deviance: -60.7565 on 178 degrees of freedom
AIC: -424201.9

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
A.2 Beta regression models for NOVEL lemmas

The following provides the full R output of the NOVEL ~ LEMMAS beta regression models in Experiment 1 (§3.4).

CHR-TRM
Call: gam(formula = scaled_novel ~ log(train_lemmas),
    family = betar(link = "logit"),
    data = wu_res_clean)
Deviance Residuals:
    Min   1Q Median   3Q   Max
-1.3688 -0.9631  0.0000  0.0000  0.9252

(Dispersion Parameter for Beta regression family taken to be 0.2352)

Null Deviance: 51.5501 on 179 degrees of freedom
Residual Deviance: 79.1182 on 178 degrees of freedom
AIC: -19388.42

Number of Local Scoring Iterations: 5

Anova for Parametric Effects
<table>
  <tr><th>DF</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(train_lemmas)</td><td>1</td><td>29.862</td><td>29.8618</td><td>126.96 &lt; 2.2e-16 ***</td></tr>
  <tr><td>Residuals</td><td>178</td><td>41.867</td><td>0.2352</td><td></td></tr>
</table>

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(train_lemmas)
-2.5611164   0.5577768

CLUZH
Call: gam(formula = scaled_novel ~ log(train_lemmas),
    family = betar(link = "logit"),
    data = cluzh_res_clean)
Deviance Residuals:
    Min   1Q Median   3Q   Max
-1.51904 -0.97321  0.00000  0.09055  0.81805

(Dispersion Parameter for Beta regression family taken to be 0.284)

Null Deviance: 30.1758 on 179 degrees of freedom
Residual Deviance: 34.5711 on 178 degrees of freedom
AIC: -26119.63

Number of Local Scoring Iterations: 5

Anova for Parametric Effects
<table>
  <tr><th>DF</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(train_lemmas)</td><td>1</td><td>1.888</td><td>1.88793</td><td>6.6468 0.01074 *</td></tr>
  <tr><td>Residuals</td><td>178</td><td>50.558</td><td>0.28404</td><td></td></tr>
</table>

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(train_lemmas)
0.5417008   0.1484194

DEEPSPIN
Call: gam(formula = scaled_novel ~ log(train_lemmas),
    family = betar(link = "logit"),
    data = ds_res_clean)
Deviance Residuals:
    Min   1Q Median   3Q   Max
-1.5407 -0.9962  0.0000  0.0000  0.8116

(Dispersion Parameter for Beta regression family taken to be 0.2838)

Null Deviance: 26.5794 on 179 degrees of freedom
Residual Deviance: 33.2525 on 178 degrees of freedom
AIC: -269455.6

Number of Local Scoring Iterations: 5

Anova for Parametric Effects
<table>
  <tr><th>DF</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(train_lemmas)</td><td>1</td><td>3.002</td><td>3.00205</td><td>10.579 0.001368 **</td></tr>
  <tr><td>Residuals</td><td>178</td><td>50.510</td><td>0.28377</td><td></td></tr>
</table>

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(train_lemmas)
0.3111430   0.1895828

A.3 Beta regression models for DROP

The following provides the full R output of the DROP ~ LEMMAS beta regression models in Experiment 1 (§3.4).

CHR-TRM
all: gam(formula = scaled_diff ~ log(train_lemmas),
    family = betar(link = "logit"),
    data = wu_res_clean)
Deviance Residuals:
    Min   1Q Median   3Q   Max
-0.96697 -0.83717  0.05881  0.92129  1.11801

(Dispersion Parameter for Beta regression family taken to be 0.14)

Null Deviance: 117.7257 on 179 degrees of freedom
Residual Deviance: 128.5483 on 178 degrees of freedom
AIC: 43583.62

Number of Local Scoring Iterations: 4

Anova for Parametric Effects
<table>
  <tr><th>DF</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(train_lemmas)</td><td>1</td><td>32.169</td><td>32.169</td><td>229.69 &lt; 2.2e-16 ***</td></tr>
  <tr><td>Residuals</td><td>178</td><td>24.929</td><td>0.140</td><td></td></tr>
</table>

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(train_lemmas)
2.9609546   -0.5237525

CLUZH
Call: gam(formula = scaled_diff ~ log(train_lemmas),
    family = betar(link = "logit"),
    data = cluzh_res_clean)
Deviance Residuals:
    Min   1Q Median   3Q   Max
-0.3184 -0.7647 -0.6405  0.9666  1.1398

(Dispersion Parameter for Beta regression family taken to be 0.0977)

Null Deviance: 128.5974 on 179 degrees of freedom
Residual Deviance: 131.8342 on 178 degrees of freedom
AIC: 81136.68

Number of Local Scoring Iterations: 4

Anova for Parametric Effects
<table>
  <tr><th>DF</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(train_lemmas)</td><td>1</td><td>3.5969</td><td>3.5969</td><td>36.089 1.038e-08 ***</td></tr>
  <tr><td>Residuals</td><td>178</td><td>17.7409</td><td>0.0997</td><td></td></tr>
</table>

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(train_lemmas)
0.2686925   -0.1737422

DEEPSPIN
Call: gam(formula = scaled_diff ~ log(train_lemmas),
    family = betar(link = "logit"),
    data = ds_res_clean)
Deviance Residuals:
    Min   1Q Median   3Q   Max
-0.9166 -0.7448 -0.5476  0.9549  1.1635
(Dispersion Parameter for Beta regression family taken to be 0.1021)

Null Deviance: 125.1004 on 179 degrees of freedom
Residual Deviance: 128.2078 on 178 degrees of freedom
AIC: 84275.73

Number of Local Scoring Iterations: 4

Anova for Parametric Effects
<table>
  <tr><th>DF</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(>F)</th></tr>
  <tr><td>log(train_lemmas)</td><td>1</td><td>4.3868</td><td>4.3868</td><td>42.981</td><td>5.78e-10 ***</td></tr>
  <tr><td>Residuals</td><td>178</td><td>18.1674</td><td>0.1021</td><td></td><td></td></tr>
</table>

Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(train_lemmas)
0.3609071   -0.1935463

B  Experiment 2

<table>
  <tr>
    <th>FAMILY</th>
    <th>LANGUAGE</th>
  </tr>
  <tr>
    <td rowspan="7">Niger-Congo</td>
    <td>Akan (aka)</td>
  </tr>
  <tr>
    <td>Gà (gaa)</td>
  </tr>
  <tr>
    <td>Kongo (kon)</td>
  </tr>
  <tr>
    <td>Lingala (lin)</td>
  </tr>
  <tr>
    <td>Luganda (lug)</td>
  </tr>
  <tr>
    <td>Chewa (nya)</td>
  </tr>
  <tr>
    <td>Shona (sna)</td>
  </tr>
  <tr>
    <td>Sotho (sot)</td>
  </tr>
  <tr>
    <td>Swahili (swa)</td>
  </tr>
  <tr>
    <td>Zulu (zul)</td>
  </tr>
  <tr>
    <td rowspan="7">Romance</td>
    <td>Asturian (ast)</td>
  </tr>
  <tr>
    <td>Catalan (cat)</td>
  </tr>
  <tr>
    <td>Middle French (frm)</td>
  </tr>
  <tr>
    <td>Friulian (fur)</td>
  </tr>
  <tr>
    <td>Galician (glg)</td>
  </tr>
  <tr>
    <td>Ladin (lld)</td>
  </tr>
  <tr>
    <td>Venetian (vec)</td>
  </tr>
  <tr>
    <td>Anglo-Normal (xno)</td>
  </tr>
  <tr>
    <td rowspan="10">Uralic</td>
    <td>Estonian (est)</td>
  </tr>
  <tr>
    <td>Finnish (fin)</td>
  </tr>
  <tr>
    <td>Ingrian (izh)</td>
  </tr>
  <tr>
    <td>Komi-Zyrian (kpv)</td>
  </tr>
  <tr>
    <td>Karelian (krl)</td>
  </tr>
  <tr>
    <td>Livonian (liv)</td>
  </tr>
  <tr>
    <td>Ludic (lud)</td>
  </tr>
  <tr>
    <td>Moksha (mdf)</td>
  </tr>
  <tr>
    <td>Meadow Mari (mhr)</td>
  </tr>
  <tr>
    <td>Erzya (myv)</td>
  </tr>
  <tr>
    <td>Livvi (olo)</td>
  </tr>
  <tr>
    <td>Northern Sami (sme)</td>
  </tr>
  <tr>
    <td>Udmurt (udm)</td>
  </tr>
  <tr>
    <td>Veps (vep)</td>
  </tr>
  <tr>
    <td>Votic (vot)</td>
  </tr>
  <tr>
    <td>Võro (vro)</td>
  </tr>
</table>

Table 9: The 34 languages used to replicate the findings of Goldman et al. (2022) in Experiment 2.

![Three density plots showing the distribution of SIGMORPHON and Goldman et al. test accuracy, and accuracy drop](page_1012_120_388_563.png)

Figure 8: Replicated GOLDMAN and SIGMORPHON accuracy as well as DROP are beta distributed in Experiment 2.

Min  IQR Median  3Q  Max
-1.9418 -0.7337  0.0000  0.0000  0.0000

(Dispersion Parameter for Beta regression family taken to be 0.1921)

Null Deviance: -27.6167 on 33 degrees of freedom
Residual Deviance: -15.6955 on 31 degrees of freedom
AIC: -17287.93

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
<table>
  <tr><th>DF</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(>F)</th></tr>
  <tr><td>log(SIGMORPHON_train_size)</td><td>1</td><td>1.1632</td><td>2.16325</td><td>11.2586 0.002106 **</td></tr>
  <tr><td>log(SIGMORPHON_train_lemmas)</td><td>1</td><td>0.5541</td><td>1.65420</td><td>8.6693 0.000245 **</td></tr>
  <tr><td>Residuals</td><td>31</td><td>5.9564</td><td>0.19214</td><td></td></tr>
</table>

Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(SIGMORPHON_train_size) log(SIGMORPHON_train_lemmas)
-0.5531629   0.7623518   -0.5661172

B.1 Beta regression models for SIGMORPHON
The following provides the full R output of the multi-predictor beta regression models trained to predict SIGMORPHON accuracy in Experiment 2 (\S 4.2).

CHR-TRM
Call: gam(formula = SIGMORPHON_scaled_acc ~ log(SIGMORPHON_train_size) + log(SIGMORPHON_train_lemmas), family = betar(link = "logit"), data = CHRTRM)
Deviance Residuals:

CLUZH
Call: gam(formula = SIGMORPHON_scaled_acc ~ log(SIGMORPHON_train_size) + log(SIGMORPHON_train_lemmas), family = betar(link = "logit"), data = CLUZH)
Deviance Residuals:
<table>
  <tr><th>Min</th><th>IQR</th><th>Median</th><th>3Q</th><th>Max</th></tr>
  <tr><td>-1.0624</td><td>-1.0624</td><td>0.0000</td><td>0.0000</td><td>0.4304</td></tr>
</table>

(Dispersion Parameter for Beta regression family taken to be 0.2247)

Null Deviance: -29.4498 on 33 degrees of freedom
Residual Deviance: -17.2346 on 31 degrees of freedom
<table>
  <tr>
    <th>Lang</th>
    <th>CHR-TRM</th>
    <th>GOLDMAN</th>
    <th>CLUZH</th>
    <th>DEEPSPIN</th>
    <th>CHR-TRM</th>
    <th>GOLDMAN</th>
    <th>CLUZH</th>
    <th>DEEPSPIN</th>
    <th>CHR-TRM</th>
    <th>GOLDMAN</th>
    <th>CLUZH</th>
    <th>DEEPSPIN</th>
  </tr>
  <tr>
    <td>aka</td>
    <td>100.000</td>
    <td>99.607</td>
    <td>100.000</td>
    <td>30.038</td>
    <td>99.487</td>
    <td>94.737</td>
    <td>-69.962</td>
    <td>-0.120</td>
    <td>-5.263</td>
  </tr>
  <tr>
    <td>ast</td>
    <td>99.314</td>
    <td>98.49</td>
    <td>99.726</td>
    <td>67.645</td>
    <td>95.664</td>
    <td>99.133</td>
    <td>-31.669</td>
    <td>-2.826</td>
    <td>-0.593</td>
  </tr>
  <tr>
    <td>cat</td>
    <td>99.778</td>
    <td>99.717</td>
    <td>99.778</td>
    <td>97.507</td>
    <td>97.823</td>
    <td>97.44</td>
    <td>-2.271</td>
    <td>-1.894</td>
    <td>-2.338</td>
  </tr>
  <tr>
    <td>est</td>
    <td>95.194</td>
    <td>94.317</td>
    <td>94.566</td>
    <td>79.586</td>
    <td>81.069</td>
    <td>82.214</td>
    <td>-15.608</td>
    <td>-13.248</td>
    <td>-12.351</td>
  </tr>
  <tr>
    <td>fin</td>
    <td>99.56</td>
    <td>99.746</td>
    <td>99.782</td>
    <td>94.503</td>
    <td>94.147</td>
    <td>95.156</td>
    <td>-5.057</td>
    <td>-5.6</td>
    <td>-4.626</td>
  </tr>
  <tr>
    <td>frm</td>
    <td>99.744</td>
    <td>99.488</td>
    <td>99.701</td>
    <td>94.056</td>
    <td>95.99</td>
    <td>96.921</td>
    <td>-5.688</td>
    <td>-3.498</td>
    <td>-2.781</td>
  </tr>
  <tr>
    <td>fur</td>
    <td>99.741</td>
    <td>99.353</td>
    <td>99.806</td>
    <td>70.716</td>
    <td>100.000</td>
    <td>99.936</td>
    <td>-29.025</td>
    <td>0.647</td>
    <td>0.130</td>
  </tr>
  <tr>
    <td>gaa</td>
    <td>99.408</td>
    <td>100.000</td>
    <td>100.000</td>
    <td>53.216</td>
    <td>100.000</td>
    <td>100.000</td>
    <td>-46.192</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>glg</td>
    <td>99.840</td>
    <td>99.680</td>
    <td>99.782</td>
    <td>95.861</td>
    <td>97.037</td>
    <td>97.545</td>
    <td>-3.98</td>
    <td>-2.643</td>
    <td>-2.237</td>
  </tr>
  <tr>
    <td>izh</td>
    <td>88.839</td>
    <td>79.018</td>
    <td>87.500</td>
    <td>26.484</td>
    <td>41.553</td>
    <td>51.142</td>
    <td>-62.355</td>
    <td>-37.465</td>
    <td>-36.358</td>
  </tr>
  <tr>
    <td>kon</td>
    <td>98.077</td>
    <td>98.718</td>
    <td>94.231</td>
    <td>88.125</td>
    <td>98.750</td>
    <td>95.000</td>
    <td>-9.952</td>
    <td>0.032</td>
    <td>0.769</td>
  </tr>
  <tr>
    <td>kpv</td>
    <td>96.956</td>
    <td>96.43</td>
    <td>96.805</td>
    <td>94.085</td>
    <td>93.645</td>
    <td>93.798</td>
    <td>-2.871</td>
    <td>-2.785</td>
    <td>-3.007</td>
  </tr>
  <tr>
    <td>krk</td>
    <td>99.264</td>
    <td>99.071</td>
    <td>99.475</td>
    <td>90.19</td>
    <td>88.502</td>
    <td>91.257</td>
    <td>-9.075</td>
    <td>-10.569</td>
    <td>-8.218</td>
  </tr>
  <tr>
    <td>lin</td>
    <td>100.000</td>
    <td>100.000</td>
    <td>100.000</td>
    <td>56.250</td>
    <td>100.000</td>
    <td>100.000</td>
    <td>-43.750</td>
    <td>0.000</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>liv</td>
    <td>96.509</td>
    <td>94.638</td>
    <td>95.012</td>
    <td>50.422</td>
    <td>80.141</td>
    <td>77.465</td>
    <td>-46.086</td>
    <td>-14.498</td>
    <td>-17.548</td>
  </tr>
  <tr>
    <td>lld</td>
    <td>99.586</td>
    <td>99.034</td>
    <td>99.862</td>
    <td>60.173</td>
    <td>93.75</td>
    <td>98.338</td>
    <td>-39.413</td>
    <td>-5.284</td>
    <td>-1.524</td>
  </tr>
  <tr>
    <td>lud</td>
    <td>28.049</td>
    <td>24.39</td>
    <td>50.000</td>
    <td>1.835</td>
    <td>4.128</td>
    <td>20.642</td>
    <td>-26.214</td>
    <td>-20.262</td>
    <td>-29.358</td>
  </tr>
  <tr>
    <td>lug</td>
    <td>90.583</td>
    <td>88.434</td>
    <td>90.174</td>
    <td>18.561</td>
    <td>67.153</td>
    <td>70.699</td>
    <td>-72.022</td>
    <td>-21.281</td>
    <td>-19.475</td>
  </tr>
  <tr>
    <td>mdf</td>
    <td>93.391</td>
    <td>92.403</td>
    <td>93.286</td>
    <td>91.742</td>
    <td>90.669</td>
    <td>91.489</td>
    <td>-1.649</td>
    <td>-1.734</td>
    <td>-1.797</td>
  </tr>
  <tr>
    <td>mhr</td>
    <td>92.957</td>
    <td>93.288</td>
    <td>93.511</td>
    <td>86.974</td>
    <td>85.667</td>
    <td>86.18</td>
    <td>-5.983</td>
    <td>-7.621</td>
    <td>-7.331</td>
  </tr>
  <tr>
    <td>myv</td>
    <td>94.13</td>
    <td>93.125</td>
    <td>93.758</td>
    <td>94.162</td>
    <td>93.489</td>
    <td>93.974</td>
    <td>0.033</td>
    <td>0.364</td>
    <td>0.216</td>
  </tr>
  <tr>
    <td>nya</td>
    <td>100.000</td>
    <td>99.883</td>
    <td>100.000</td>
    <td>75.858</td>
    <td>100.000</td>
    <td>100.000</td>
    <td>-24.142</td>
    <td>0.117</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>olo</td>
    <td>94.207</td>
    <td>94.726</td>
    <td>95.246</td>
    <td>91.506</td>
    <td>91.074</td>
    <td>92.516</td>
    <td>-2.701</td>
    <td>-3.653</td>
    <td>-2.730</td>
  </tr>
  <tr>
    <td>sme</td>
    <td>99.673</td>
    <td>99.537</td>
    <td>99.737</td>
    <td>94.496</td>
    <td>93.053</td>
    <td>95.537</td>
    <td>-5.177</td>
    <td>-6.484</td>
    <td>-4.200</td>
  </tr>
  <tr>
    <td>sna</td>
    <td>100.000</td>
    <td>99.781</td>
    <td>100.000</td>
    <td>26.274</td>
    <td>90.784</td>
    <td>93.726</td>
    <td>-73.726</td>
    <td>-8.996</td>
    <td>-6.275</td>
  </tr>
  <tr>
    <td>sot</td>
    <td>97.980</td>
    <td>97.980</td>
    <td>100.000</td>
    <td>0.000</td>
    <td>100.000</td>
    <td>100.000</td>
    <td>-97.980</td>
    <td>2.020</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>swa</td>
    <td>100.000</td>
    <td>100.000</td>
    <td>100.000</td>
    <td>42.959</td>
    <td>99.898</td>
    <td>100.000</td>
    <td>-57.041</td>
    <td>-0.102</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>udm</td>
    <td>98.492</td>
    <td>98.52</td>
    <td>98.682</td>
    <td>96.413</td>
    <td>95.596</td>
    <td>96.366</td>
    <td>-2.079</td>
    <td>-2.924</td>
    <td>-2.315</td>
  </tr>
  <tr>
    <td>vec</td>
    <td>99.684</td>
    <td>99.512</td>
    <td>99.570</td>
    <td>91.591</td>
    <td>97.310</td>
    <td>97.225</td>
    <td>-8.093</td>
    <td>-2.202</td>
    <td>-2.344</td>
  </tr>
  <tr>
    <td>vep</td>
    <td>83.71</td>
    <td>83.056</td>
    <td>80.728</td>
    <td>77.947</td>
    <td>77.197</td>
    <td>78.807</td>
    <td>-5.764</td>
    <td>-5.859</td>
    <td>-1.922</td>
  </tr>
  <tr>
    <td>ves</td>
    <td>85.053</td>
    <td>73.31</td>
    <td>84.698</td>
    <td>4.895</td>
    <td>32.517</td>
    <td>53.846</td>
    <td>-80.158</td>
    <td>-40.792</td>
    <td>-30.851</td>
  </tr>
  <tr>
    <td>vro</td>
    <td>59.223</td>
    <td>48.544</td>
    <td>67.961</td>
    <td>0.000</td>
    <td>13.542</td>
    <td>26.042</td>
    <td>-59.223</td>
    <td>-35.002</td>
    <td>-41.92</td>
  </tr>
  <tr>
    <td>xno</td>
    <td>96.078</td>
    <td>70.588</td>
    <td>94.118</td>
    <td>0.000</td>
    <td>15.686</td>
    <td>94.118</td>
    <td>-96.078</td>
    <td>-54.902</td>
    <td>0.000</td>
  </tr>
  <tr>
    <td>zul</td>
    <td>92.308</td>
    <td>87.18</td>
    <td>89.744</td>
    <td>32.222</td>
    <td>75.556</td>
    <td>75.556</td>
    <td>-60.085</td>
    <td>-11.624</td>
    <td>-14.188</td>
  </tr>
</table>

Table 10: Full results by language and neural architecture for Experiment 2.

AIC: -15893.34

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
<table>
  <tr>
    <th>Df</th>
    <th>Sum Sq</th>
    <th>Mean Sq</th>
    <th>F value</th>
    <th>Pr(>F)</th>
  </tr>
  <tr>
    <td>log(SIGMORPHON_train_size)</td>
    <td>1</td>
    <td>14.0506</td>
    <td>4.0506</td>
    <td>18.0264</td>
    <td>0.0001837 ***</td>
  </tr>
  <tr>
    <td>log(SIGMORPHON_train_lemmas)</td>
    <td>1</td>
    <td>0.5878</td>
    <td>0.5878</td>
    <td>2.6159</td>
    <td>0.1195308</td>
  </tr>
  <tr>
    <td>Residuals</td>
    <td>31</td>
    <td>6.9653</td>
    <td>0.2247</td>
    <td></td>
    <td></td>
  </tr>
</table>

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(SIGMORPHON_train_size) log(SIGMORPHON_train_lemmas)
-1.1303584 0.6168127 -0.3049881

DEEPSPIN
Call: gam(formula = SIGMORPHON_scaled_acc ~ log(SIGMORPHON_train_size) + log(SIGMORPHON_train_lemmas), family = betar(link = "logit"), data = DeepSpin)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.9276 -0.7471 0.0000 0.0000 0.5294

(Dispersion Parameter for Beta regression family taken to be 0.1431)

Null Deviance: -24.6582 on 33 degrees of freedom
Residual Deviance: -18.3863 on 31 degrees of freedom
AIC: -17468.71

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
<table>
  <tr>
    <th>Df</th>
    <th>Sum Sq</th>
    <th>Mean Sq</th>
    <th>F value</th>
    <th>Pr(>F)</th>
  </tr>
  <tr>
    <td>log(SIGMORPHON_train_size)</td>
    <td>1</td>
    <td>11.4196</td>
    <td>1.41962</td>
    <td>9.9226 0.003603 **</td>
  </tr>
</table>

log(SIGMORPHON_train_lemmas) 1 0.9939 0.99394 6.9472 0.012994 *
Residuals 31 4.4352 0.14307
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(SIGMORPHON_train_size) log(SIGMORPHON_train_lemmas)
0.08735762 0.59994250 -0.43981367

B.2 Beta regression models for GOLDMAN

The following provides the full R output of the multi-predictor beta regression models trained to predict GOLDMAN accuracy in Experiment 2 (§4.2).

CHR-TRM
Call: gam(formula = Goldman_scaled_acc ~ log(Goldman_train_lemmas) + log(Goldman_train_size), family = betar(link = "logit"), data = CHR-TRM)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.7181 -0.9309 0.0000 0.0000 1.3871

(Dispersion Parameter for Beta regression family taken to be 0.3214)

Null Deviance: -1.0739 on 33 degrees of freedom
Residual Deviance: 9.2018 on 31 degrees of freedom
AIC: -3878.832

Number of Local Scoring Iterations: 6

Anova for Parametric Effects
<table>
  <tr>
    <th>Df</th>
    <th>Sum Sq</th>
    <th>Mean Sq</th>
    <th>F value</th>
    <th>Pr(>F)</th>
  </tr>
  <tr>
    <td>log(Goldman_train_lemmas)</td>
    <td>1</td>
    <td>14.2329</td>
    <td>14.2329</td>
    <td>44.2893 1.932e-07 ***</td>
  </tr>
  <tr>
    <td>log(Goldman_train_size)</td>
    <td>1</td>
    <td>0.7122</td>
    <td>0.7122</td>
    <td>2.2163 0.1467</td>
  </tr>
</table>
Residuals
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept) log(Goldman_train_lemmas)  log(Goldman_train_size)
-4.5197892   0.5532991   0.2583534

CLUZH
Call: gam(formula = Goldman_scaled_acc ~ log(Goldman_train_size) + log(Goldman_train_lemmas), family = betar(link = "logit"), data = CLUZH)
Deviance Residuals:
    Min      1Q  Median      3Q     Max
-1.6752 -0.8953  0.0000  0.0000  0.7093

(Dispersion Parameter for Beta regression family taken to be 0.4671)

    Null Deviance: 22.2953 on 33 degrees of freedom
Residual Deviance: 11.0956 on 31 degrees of freedom
AIC: -10675.54

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
<table>
  <tr><th>Df</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(Goldman_train_size)</td><td>1</td><td>7.0203</td><td>7.0203</td><td>15.0303</td><td>0.0005137 ***</td></tr>
  <tr><td>log(Goldman_train_lemmas)</td><td>1</td><td>0.0050</td><td>0.0050</td><td>0.0107</td><td>0.9182216</td></tr>
  <tr><td>Residuals</td><td>31</td><td>14.4794</td><td>0.4671</td><td></td><td></td></tr>
</table>

Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept)  log(Goldman_train_size)  log(Goldman_train_lemmas)
-1.83118611   0.41435242   -0.02250918

Number of Local Scoring Iterations: 6

Anova for Parametric Effects
<table>
  <tr><th>Df</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(Goldman_train_lemmas)</td><td>1</td><td>10.9621</td><td>10.9621</td><td>56.3870</td><td>2.267e-08 ***</td></tr>
  <tr><td>log(-1 * train_lemma_diff_raw)</td><td>1</td><td>0.2824</td><td>0.2824</td><td>1.8412</td><td>0.3157</td></tr>
  <tr><td>log(Goldman_train_size)</td><td>1</td><td>0.1658</td><td>0.1658</td><td>0.8530</td><td>0.3631</td></tr>
  <tr><td>Residuals</td><td>38</td><td>5.8323</td><td>0.1944</td><td></td><td></td></tr>
</table>

Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept)  log(Goldman_train_lemmas)  log(-1 * train_lemma_diff_raw)
3.7697667   -1.2338483   0.1499576
log(Goldman_train_size)  0.1537948

CLUZH
Call: gam(formula = scaled_drop ~ log(Goldman_train_lemmas) + log(-1 * train_lemma_diff_raw) + log(Goldman_train_size), family = betar(link = "logit"), data = CLUZH)
Deviance Residuals:
    Min      1Q  Median      3Q     Max
-0.7807  0.0000  0.3350  1.0758  1.4194

(Dispersion Parameter for Beta regression family taken to be 0.1584)

    Null Deviance: -3.114 on 33 degrees of freedom
Residual Deviance: 4.0005 on 30 degrees of freedom
AIC: 6837.224

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
<table>
  <tr><th>Df</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(Goldman_train_lemmas)</td><td>1</td><td>1.2522</td><td>1.2522</td><td>15.9285</td><td>0.0009319 ***</td></tr>
  <tr><td>log(-1 * train_lemma_diff_raw)</td><td>1</td><td>0.0524</td><td>0.0524</td><td>0.3310</td><td>0.5637939</td></tr>
  <tr><td>log(Goldman_train_size)</td><td>1</td><td>0.0128</td><td>0.0128</td><td>0.8088</td><td>0.7782866</td></tr>
  <tr><td>Residuals</td><td>30</td><td>4.7511</td><td>0.15837</td><td></td><td></td></tr>
</table>

Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept)  log(Goldman_train_lemmas)  log(-1 * train_lemma_diff_raw)
-0.6390685   0.08228355   -0.26488998
log(Goldman_train_size)  0.0463225

DEEPSPIN
Call: gam(formula = Goldman_scaled_acc ~ log(Goldman_train_size) + log(Goldman_train_lemmas), family = betar(link = "logit"), data = DeepSpin)
Deviance Residuals:
    Min      1Q  Median      3Q     Max
-1.6214 -0.9821  0.0000  0.0000  0.7569

(Dispersion Parameter for Beta regression family taken to be 0.3064)

    Null Deviance: 19.4575 on 33 degrees of freedom
Residual Deviance: 12.6952 on 31 degrees of freedom
AIC: -12350.04

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
<table>
  <tr><th>Df</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(Goldman_train_size)</td><td>1</td><td>3.5682</td><td>3.5682</td><td>11.6462</td><td>0.001809 **</td></tr>
  <tr><td>log(Goldman_train_lemmas)</td><td>1</td><td>0.8896</td><td>0.8896</td><td>2.9036</td><td>0.098388 .</td></tr>
  <tr><td>Residuals</td><td>31</td><td>9.4980</td><td>0.3064</td><td></td><td></td></tr>
</table>

Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept)  log(Goldman_train_size)  log(Goldman_train_lemmas)
-1.0463182   0.5326597   -0.3004978

Number of Local Scoring Iterations: 2

Anova for Parametric Effects
<table>
  <tr><th>Df</th><th>Sum Sq</th><th>Mean Sq</th><th>F value</th><th>Pr(&gt;F)</th></tr>
  <tr><td>log(-1 * train_lemma_diff_raw)</td><td>1</td><td>11.6345</td><td>11.6345</td><td>11.3933</td><td>0.002952 **</td></tr>
  <tr><td>log(Goldman_train_size)</td><td>1</td><td>1.0176</td><td>1.0176</td><td>1.2314</td><td>0.275945</td></tr>
  <tr><td>log(Goldman_train_lemmas)</td><td>1</td><td>0.3227</td><td>0.3227</td><td>2.2495</td><td>0.14106</td></tr>
  <tr><td>Residuals</td><td>30</td><td>4.8038</td><td>0.14346</td><td></td><td></td></tr>
</table>

Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Intercept)  log(-1 * train_lemma_diff_raw)  log(Goldman_train_size)
-1.0177739   -0.5704532   -0.1261425
log(Goldman_train_lemmas)  0.4467069

B.3 Beta regression models for DROP

The following provides the full R output of the multi-predictor beta regression models trained to predict DROP from SIGMOR-PHON to GOLDMAN accuracy in Experiment 2 (§4.2).

CHR-TRM
Call: gam(formula = scaled_drop ~ log(Goldman_train_lemmas) + log(-1 * train_lemma_diff_raw) + log(Goldman_train_size), family = betar(link = "logit"), data = CHRTRM)
Deviance Residuals:
    Min      1Q  Median      3Q     Max
-1.0309  0.0000  0.6883  1.1058  2.1611

(Dispersion Parameter for Beta regression family taken to be 0.1944)

    Null Deviance: -4.2629 on 33 degrees of freedom
Residual Deviance: 20.2587 on 30 degrees of freedom
AIC: 956.8627

Reviewer paper summaries: This paper investigates morphological inflection problem about how much does lemma overlap between train and test affect performance. They propose a splitting algorithm that controls the ratio of seen/unseen lemmas, and show that the number of unique lemmas in training, not just the presence/absence of test lemmas, is a strong predictor of performance. They also re-analyze earlier findings (e.g., Goldman et al. 2022), showing that previously reported large drops in accuracy for unseen lemmas were partly due to fewer lemmas in train, not just lack of overlap. Finally, they use beta regression and statistical modeling to quantify these effects.

This paper studies how the quality of morphological inflection differs between seen lemmas that appear in the training data and unseen lemmas that were not present in the training set. The paper resolves the contradiction between previous studies of Kodner et al. and Goldman et al. by demonstrating that the performance on unseen lemmas is worse (opposed to Kodner et al.) but the decrease is less pronounced than reported by Goldman et al. The work also replicates Goldman et al. study and explain the difference in results by different experimental setups. The main conclusion of the current work is that performance on unseen lemmas crucially depends from the training data size.
Summary of strengths: The paper walks through the reasons behind conflicting prior findings and quantifies them with statistical rigor. And it makes an empirical contribution by showing that the true effect of lemma overlap is smaller than previously claimed (~11% vs. 30%).

* A thorough statistic analysis of contradictory questions with careful control of dataset parameters
* What is rare, the authors replicate the results of the work they argue with and show the difference in experiment setup
* Several inflection models and language families are covered, in contrast to previous studies
Summary of weaknesses: 1. A key concern of this paper is: While the authors fix the total number of training triples and ensure equal proportions of seen and unseen lemmas in the test set, the amount of exposure to each lemma in train is not further stratified. I feel many factors will affect the final results. For example, while the authors control for how many unique lemmas are in train, they don’t necessarily control for how many forms per lemma, or for how morphologically “typical” the seen lemmas are. Also, the paper only control the split ratio at 50% for unseen lemmas. I would wonder what the results look like in various settings.
2. Since it is a long paper, another concern I have is the conceptual novelty. While the methodologically is strong, it is a clarification and refinement of prior work rather than a completely new problem or technique.

* The paper subject is somewhat marginal
* The authors do not control the number of unseen paradigms in the test set, but this factor definitely affects performance
Comments, suggestions and typos: The pseudocode for the splitting algorithm is useful, but a visualized example would aid clarity.

* Do you have any results concerning the dependence of inflection quality from the number of seen and unseen inflection patterns? Predicting unattested word changes may be even more troublesome than generalization to unseen lemmas.
* Why the term *fine tune size* is used, is it the same as *validation set size*?
Overall assessment scores: 3

3
Metareview score: 3.5
Metareview: The authors investigate a recent topic that has arisen in morphological processing - the overselling of previous systems that reported model quality largely on test lemmas that had been observed in the training data (the Y values were not seen in training, but certain combinations of the X values had been).
Summary of reasons to publish: The most important finding seems to be that the results are more dependent on the number of distinct lemmas (and thus, I assume, on the diversity) of the input data, rather than simply a seen / unseen split.  I think that this is an interesting (and important) finding, as it can be applied to annotation efforts and data collection strategies, and is not entirely intuitive.  I think this paper can play an important role in the literature.