Results
We demonstrate the results of each strategy for all languages, considering each combination of language, subset selection strategy, and \lvert D^{Syn}_{train}\rvert , thus obtaining 35 sets of results. For each setting, we report the performance achieved over 6 different random initializations, along with their respective standard deviations. For brevity, we show the results for \lvert\hat{D}^{Syn}_{train}\rvert\in\{128,512,2048\} ; we include the expanded set of results (including \{256,1024\} ) in Appendix D .

StemCorrupt improves compositional generalization. At a high level, we find that data augmentation brings substantial benefits for compositional generalization compared to models trained solely on the gold-standard training data D_{train} . Without StemCorrupt , the initial model \mathcal{M} for every language achieves only single-digit accuracy, while their augmented counterparts perform significantly better. For instance, the best models for Georgian and Spanish achieve over 50% accuracy. These findings agree with those of Liu and Hulden ( 2022 ) who found that unaugmented Transformer models fail to generalize inflection patterns.

We also find that performance tends to increase as we add more synthetic data; the best models for every language are on the higher end of the \lvert\hat{D}^{Syn}_{train}\rvert sizes. This finding agrees with our theoretical results that the dependence between the stem ( \mathbf{Y}_{stem} ) and that of the inflectional affix ( \mathbf{Y}_{affix} ) is weakened as we add more samples from StemCorrupt (Section 3 ; Proposition 1).

Effective subsets have high diversity and predictive uncertainty .
Our analysis reveals statistically significant differences between the subset selection strategies, highlighting the effectiveness of the hybrid approaches ( UMT/EMT+Loss ) that consider both diversity and predictive uncertainty. Among the strategies tested, the UMT+Loss method outperformed all others in approximately one-third of the 35 settings examined, as indicated in Figure 5 (left). The improvements achieved by the UMT+Loss method over a random baseline were statistically significant ( p<0.05 ) according to a bootstrap percentile test (Efron and Tibshirani, 1994 ) , as shown in Figure 5 (right).
Moreover, our results also show that the EMT+Loss strategy closely followed the UMT+Loss approach, achieving the highest performance in a quarter of the cases. In contrast, the same strategies without the uncertainty component were much less effective. For instance, UMT never achieved the best performance in any combination of the languages and \lvert\hat{D}^{Syn}_{train}\rvert sizes, highlighting that selecting a diverse subset without factoring in predictive uncertainty is suboptimal.

Furthermore, selecting datapoints based solely on high predictive uncertainty without considering diversity ( HighLoss ) is an ineffective strategy, having the second lowest proportion of wins (Fig. 5 , right). Empirically, we find this may be attributed to the HighLoss strategy fixating on a particular MSD, as shown in Fig. 6 , rather than exploring the full distribution of MSDs. The figure displays the frequency of the most commonly sampled morphosyntactic description for each of UMT+Loss , Random , and HighLoss strategies. Across all languages, the HighLoss method samples the most frequent tag much more often than the Random and UMT+Loss methods. 9 9 9 The reason that uncertainty estimates are higher for a given MSD is not entirely clear. In our investigation, we found a small correlation ( \rho=0.15 ) between the morphosyntactic description frequency and uncertainty. However, there are likely other factors beyond frequency that contribute to higher uncertainty; for example, morphological fusion (Bickel and Nichols, 2013 ; Rathi et al., 2021 ) .

``Easy'' synthetic datapoints have low sample efficiency. As hypothesized, the datapoints with low uncertainty hurt performance. We attribute this to the LowLoss strategy selecting datapoints with a smaller number of substitutions to the stem. In Fig. 7 , we show that the number of edits made to the stem – as measured by Levenshtein distance between the corrupted target sequence and the uncorrupted version – is strongly correlated with the uncertainty assigned to the synthetic datapoint across all languages. Moreover, the correlation between the number of edits and uncertainty is higher than the correlation with other plausible factors driving uncertainty, namely stem length and target length. 10 10 10 Target length is known to contribute to predictive uncertainty (Eikema and Aziz, 2020 ; Kim and Linzen, 2020 ) , and stem length is a confounding factor of the levenshtein distance calculation. Overall, the lagging sample efficiency of LowLoss corroborates our theory; StemCorrupt is effective because it generates datapoints where the stem has no correspondence with the affix. LowLoss counteracts its effectiveness, as it is biased towards datapoints where spurious dependencies between the stem and affix are maintained. 11 11 11 The ineffectiveness of LowLoss also corroborates that StemCorrupt does not simply induce a “copying bias” (Remark 3 ), since then we would expect all of the subset selection methods to perform similarly.

Selecting by high predictive uncertainty worsens performance when there are stem-affix dependencies. We found that the UMT+Loss strategy improves performance for 5 out of 7 languages compared to the Random baseline. The improvement ranges from 4.8 (Georgian) to small declines of -1.9 (Turkish) and -0.9 (Finnish). The declines for Finnish and Turkish are partly due to a mismatch between the generated synthetic examples and the languages' morphophonology. StemCorrupt will generate synthetic examples that violate vowel harmony constraints between the stem and the affix. For instance, it may replace a front vowel in the stem with a back one. As a result, UMT+Loss will select such harmony-violating examples more often, since they have greater uncertainty to the initial model \mathcal{M} (Section 4.1 ), resulting in the augmented model tending to violate the harmony restrictions more often. Indeed, for Turkish, the average uncertainty for synthetic examples violating vowel harmony ( 0.46 ) is significantly higher than those that adhere to vowel harmony ( 0.39 ), as assessed by a bootstrap percentile test ( p<0.05 ). This finding also corroborates our theory from Section 3 : StemCorrupt eliminates dependencies between stems and affixes, even when the dependencies are real rather than spurious. This shortcoming of StemCorrupt is exacerbated by selecting examples with high uncertainty, as these examples are less likely to adhere to stem-affix constraints like vowel harmony.

Takeaways. Aligning with the semantic parsing literature on efficient compositional data augmentation (Oren et al., 2021 ; Bogin et al., 2022 ; Gupta et al., 2022 ) , we find that certain subsets of data are on average more efficient at eliminating spurious correlations between substructures – in the case of morphological inflection, the relevant substructures being the individual morphemes: the stem ( \mathbf{Y}_{stem} ) and affix ( \mathbf{Y}_{affix} ; Section 3 ). However, the sample-efficiency gains from strategic sampling are less dramatic than in semantic parsing (see, for example, Oren et al., 2021 ) .

We provided empirical evidence that the gains are tempered by StemCorrupt 's tendencies to violate stem-affix constraints in its synthetic training examples, such as vowel harmony constraints in Turkish. Thus, further work is needed to adapt or supplant StemCorrupt for languages where such long-range dependencies are commonplace. In doing so, the data selection strategies are likely to fetch greater gains in sample efficiency.