RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Preliminaries
[Illustration from Anastasopoulos and Neubig ( 2019 ) ]

In automatic morphological inflection, we assume access to a gold-standard dataset D_{train} with triples of \langle\mathbf{X},\mathbf{Y},\mathbf{T}\rangle , where \mathbf{X} is the character sequence of the lemma, \mathbf{T} is a morphosyntactic description (MSD), and \mathbf{Y} is the character sequence of the inflected form. 2 2 2 For example, ¡dog, dogs, N+PL¿. The goal is then to learn the distribution P(\mathbf{Y}|\mathbf{X},\mathbf{T}) over inflected forms conditioned on a lemma and MSD .

Generating a synthetic training dataset with StemCorrupt . For many languages, D_{train} is too small for models to learn and make systematic morphological generalizations. Previous work has found that generating a complementary synthetic dataset D^{Syn}_{train} using a data augmentation technique can substantially improve generalization (Anastasopoulos and Neubig, 2019 ; Silfverberg et al., 2017 , among others) .

The technique, henceforth called StemCorrupt , works as follows: We identify the aligned subsequences (of length 3 or greater) between a lemma \mathbf{X} and an inflected form \mathbf{Y} , which we denote the stem. 3 3 3 The stem can thus be discontinuous; see Fig. 2 . We then substitute some of the characters in the stem with random ones from the language's alphabet; Fig. 2 . The StemCorrupt procedure has a hyperparameter \theta . It sets the probability that a character in the stem will be substituted by a random one; a higher value of \theta (approaching 1) indicates a greater number of substitutions in the stem. 4 4 4 We use the implementation here , which sets \theta=0.5 .

How does StemCorrupt improve compositional generalization? Despite the widespread adoption of this technique for automatic morphological inflection and analysis, this fundamental question has heretofore remained unanswered. In the next section, we argue that StemCorrupt improves compositional generalization by removing correlations between inflectional affixes and the stems to which they are attached. By enforcing independence between these two substructures, StemCorrupt facilitates productive reuse of the inflectional affixes with other lexemes. That is, our theoretical argument shows that the effectiveness of the method arises from factoring the probability distribution to approximate concatenative morphology, a cross-linguistically prevalent strategy for word formation where words can be ``neatly segmented into roots and affixes'' (Haspelmath and Sims, 2013 ) . We formalize and prove this in the following section.
Proving the theorem
The proof of the theorem is straightforward with the following proposition (proved in Appendix B ).

As \lvert D^{Syn}_{train}\rvert\to\infty , the mutual information between certain pairs of random variables declines:

I(\mathbf{Y}_{stem};\mathbf{T})\to 0

I(\mathbf{Y}_{stem};\mathbf{X}_{affix})\to 0

I(\mathbf{Y}_{affix};\mathbf{Y}_{stem})\to 0

I(\mathbf{Y}_{affix};\mathbf{X}_{stem})\to 0

By the definition of \mathbf{Y}=\mathbf{Y}_{stem}\mathbf{Y}_{affix} , we have that P(\mathbf{Y}|\mathbf{X},\mathbf{T})\equiv P(\mathbf{Y}_{stem},\mathbf{Y}_{affix}|\mathbf{X},\mathbf{T}) . Then, by the chain rule of probability, we have P(\mathbf{Y}_{affix}|\mathbf{Y}_{stem},\mathbf{X},\mathbf{T})P(\mathbf{Y}_{stem}|\mathbf{X},\mathbf{T}) . We first deconstruct the second factor. By Proposition 1 (i) , we have that the second factor P(\mathbf{Y}_{stem}|\mathbf{X},\mathbf{T})=P(\mathbf{Y}_{stem}|\mathbf{X}) , since the stem is invariant with respect to the inflectional features. Then, by Proposition 1 (ii) , we have that P(\mathbf{Y}_{stem}|\mathbf{X})=P(\mathbf{Y}_{stem}|\mathbf{X}_{stem},\mathbf{X}_{affix})=P(\mathbf{Y}_{stem}|\mathbf{X}_{stem}) , since the stem is invariant with respect to the inflectional affix of the lemma.

Next, we tackle the factor P(\mathbf{Y}_{affix}|\mathbf{Y}_{stem},\mathbf{X},\mathbf{T}) . By parts (iii) and (iv) of Proposition 1, we have that this can be simplified to P(\mathbf{Y}_{affix}|\mathbf{X}_{affix},\mathbf{T}) . Taken together, we have that P(\mathbf{Y}|\mathbf{X},\mathbf{T}) can be decomposed into P(\mathbf{Y}_{affix}|\mathbf{X}_{affix},\mathbf{T})P(\mathbf{Y}_{stem}|\mathbf{X}_{stem}) .
∎

Investigating StemCorrupt 's sample efficiency . So far, we have studied the behaviour of StemCorrupt through an asymptotic argument, demonstrating that in the infinite limit of \lvert D^{Syn}_{train}\rvert , StemCorrupt enforces complete independence between stems and affixes. In doing so, it likely removes a number of spurious correlations between stems and affixes, thus providing a theoretical explanation for its attested benefit in improving compositional generalization. However, our theoretical analysis, while informative of the overall effect of StemCorrupt , says little about the sample efficiency of the method in practice.

Indeed, recent studies in semantic parsing have demonstrated that sample efficiency can be greatly increased by strategically sampling data to overcome spurious correlations that hinder compositional generalization in non-IID data splits (Oren et al., 2021 ; Bogin et al., 2022 ; Gupta et al., 2022 ) . In the following section, we examine whether strategic data selection can yield similar benefits in the context of typologically diverse morphological inflection.
Extracting sample-efficient training sets from StemCorrupt
Problem setup . We use the following problem setup from Oren et al. ( 2021 ) to investigate whether the sample-efficiency of StemCorrupt can be improved. Recall that we have a dataset D_{train} with gold triples of \langle\mathbf{X},\mathbf{Y},\mathbf{T}\rangle . Further, we have a synthesized dataset D^{Syn}_{train} where \lvert D^{Syn}_{train}\rvert\gg\lvert D_{train}\rvert . Our goal is now to select \hat{D}^{Syn}_{train}\subset D^{Syn}_{train} so that training the model on D_{train}\cup\hat{D}^{Syn}_{train} maximizes performance on a held-out compositional testing split.
Model and training
We start by training an inflection model \mathcal{M} on the gold-standard training data, denoted as D_{train} . Following Wu et al. ( 2021 ); Liu and Hulden ( 2022 ) , we employ Transformer (Vaswani et al., 2017 ) for \mathcal{M} .
We use the fairseq package (Ott et al., 2019 ) for training our models and list our hyperparameter settings in Appendix A . We conduct all of our experiments with \lvert D_{train}\rvert=100 gold-standard examples, adhering to the the low-resource setting for SIGMORPHON 2018 shared task for each language. We next describe the construction of \hat{D}^{Syn}_{train} .
Experiments and Results
Data . We use data from the UniMorph project (Batsuren et al., 2022 ) , considering typological diversity when selecting languages to include. We aim for an evaluation similar in scope to Muradoglu and Hulden ( 2022 ) . That is, broadly, we attempt to include types of languages that exhibit variation in inflectional characteristics such as inflectional synthesis of the verb, exponence, and morphological paradigm size (Haspelmath et al., 2005 ) . Our selected languages can be seen in Fig. 4 . We provide further information on the languages in Appendix C .

Obtaining a large synthetic dataset . In order to generate the large augmentation dataset D^{Syn}_{train} for every language, we generate 10,000 augmented datapoints for every language by applying StemCorrupt to their respective low datasets from SIGMORPHON 2018 (Cotterell et al., 2018 ) .

Generating a compositional generalization test set . For generating test sets, we adopt the lemma-split approach of Goldman et al. ( 2022 ) . Specifically, we use all available data from SIGMORPHON2018 for the target language, excluding any lexemes from the low setting since those were used to train the initial model \mathcal{M} (Section 4.1 ). The remaining lexemes and their associated paradigms comprise our compositional generalization test set; see Fig. 3 .

Populating \hat{D}^{Syn}_{train} . We evaluate the performance of all methods listed in Section 4 in selecting \hat{D}^{Syn}_{train} . We evaluated the performance of using \hat{D}^{Syn}_{train} of sizes ranging from 128 to 2048 examples, increasing the size by powers of 2.
Related work
Compositional data augmentation methods. In general, such methods synthesize new data points by splicing or swapping small parts from existing training data, leveraging the fact that certain constituents can be interchanged while preserving overall meaning or syntactic structure. A common approach is to swap spans between pairs of datapoints when their surrounding contexts are identical (Andreas, 2020 ; Guo et al., 2020 ; Jia and Liang, 2016 , inter alia) . Recently, Akyürek et al. ( 2021 ) extended this approach, eschewing rule-based splicing in favour of neural network-based recombination. Chen et al. ( 2023 ) review more compositional data augmentation techniques, situating them within the broader landscape of limited-data learning techniques for NLP.

Extracting high-value subsets in NLP training data. Oren et al. ( 2021 ); Bogin et al. ( 2022 ); Gupta et al. ( 2022 ) propose methods for extracting diverse sets of abstract templates to improve compositional generalization in semantic parsing. Muradoglu and Hulden ( 2022 ) train a baseline model on a small amount of data and use entropy estimates to select new data points for annotation, reducing annotation costs for morphological inflection. Swayamdipta et al. ( 2020 ) identify effective data subsets for training high-quality models for question answering, finding that small subsets of ambiguous examples perform better than randomly selected ones. Our work is also highly related to active-learning in NLP (Tamkin et al., 2022 ; Yuan et al., 2020 ; Margatina et al., 2021 , inter alia) ; however we focus on selecting synthetic rather than unlabeled datapoints, and our experiments are geared towards compositional generalization rather than IID performance.

Compositional data splits in morphological inflection. Assessing the generalization capacity of morphological inflections has proven a challenging and multifaceted problem. Relying on standard ``IID'' (Oren et al., 2021 ; Liu and Hulden, 2022 ) splits obfuscated (at least) two different manners in which inflection models fail to generalize compositionally.

First, Goldman et al. ( 2022 ) uncovered that generalizing to novel lexemes was challenging for even state of the art inflection models. Experiments by Liu and Hulden ( 2022 ) however showed that the StemCorrupt method could significantly improve generalization to novel lexemes. Our work builds on theirs by contributing to understanding the relationship between StemCorrupt and lexical compositional generalization. Specifically, we studied the structure of the probability distribution that StemCorrupt promotes (Section 3 ), and the conditions under which it succeeds (Remark 1 ) and fails (Remark 2 ).

Second, Kodner et al. ( 2023 ) showed that inflection models also fail to generalize compositionally to novel feature combinations, even with agglutinative languages that have typically have a strong one-to-one alignment between morphological features and affixes. Discovering strategies to facilitate compositional generalization in terms of novel feature combinations remains an open-area of research.
Conclusion
This paper presents a novel theoretical explanation for the effectiveness of StemCorrupt , a widely-used data augmentation method, in enhancing compositional generalization in automatic morphological inflection. By applying information-theoretic constructs, we prove that the augmented examples work to improve compositionality by eliminating dependencies between substructures in words – stems and affixes.
Building off of our theoretical analysis, we present the first exploration of whether the sample efficiency of reducing these spurious dependencies can be improved. Our results show that improved sample efficiency is achievable by selecting subsets of synthetic data reflecting high structural diversity and predictive uncertainty, but there is room for improvement – both in strategic sampling strategies and more cross-linguistically effective data augmentation strategies that can represent long distance phonological alternations.

Overall, NLP data augmentation strategies are poorly understood (Dao et al., 2019 ; Feng et al., 2021 ) and our work contributes to filling in this gap. Through our theoretical and empirical analyses, we provide insights that can inform future research on the effectiveness of data augmentation methods in improving compositional generalization.
Limitations
Theoretical analysis. We make some simplifying assumptions to facilitate our analysis in Section 3 . First, we assume that the stem between a gold-standard lemma and inflected form \mathbf{X} and \mathbf{Y} is discoverable. This is not always the case; for example, with suppletive inflected forms, the relationship between the source lemma and the target form is not systematic. Second, we assume that all characters in the stem are randomly substituted, corresponding to setting the \theta=1 for StemCorrupt . This does not correspond to how we deploy StemCorrupt ; the implementation provided by Anastasopoulos and Neubig ( 2019 ) sets \theta=0.5 and we use this value for our empirical analysis Section 5 . We believe the analysis under \theta=1 provides a valuable and accurate characterization of StemCorrupt nonetheless and can be readily extended to accommodate the 0<\theta<1 case in future work.

Empirical analysis. In our empirical analysis, we acknowledge two limitations that can be addressed in future research. First, we conduct our experiments using data collected for the SIGMORPHON 2018 shared task, which may not contain a naturalistic distribution of morphosyntactic descriptions since it was from an online database (Wiktionary). In future work, we aim to replicate our work in a more natural setting such as applications for endangered language documentation (Muradoglu and Hulden, 2022 ; Moeller et al., 2020 ) , where the morphosyntactic description distribution is likely to be more imbalanced. Second, we perform our analyses in an extremely data-constrained setting where only 100 gold-standard examples are available. In higher resourced settings, data augmentation with StemCorrupt may provide a much more limited improvement to compositional generalization; indeed the compositional generalization study of morphological inflection systems by Goldman et al. ( 2022 ) demonstrates that the disparity between IID generalization and compositional generalization largely dissipates when the model is trained on more gold-standard data.
Acknowledgements
We thank the EMNLP reviewers, the area chair, Vered Shwartz, Kat Vylomova, and Adam Wiemerslage for helpful discussion and feedback on the manuscript. We also thank Ronak D. Mehta for insightful discussion on properties of mutual information. This work was supported by an NSERC PGS-D scholarship to the first author.
Figure 1: Given a small human-annotated dataset and a
large pool of synthetic examples, we find that sampling a subset of data representing both diversity (a multitude of shapes) and high predictive uncertainty (shapes with a question mark) are on average more sample-efficient in improving compositional generalization in morphological inflection.; Given a small human-annotated dataset and a
large pool of synthetic examples, we find that sampling a subset of data representing both diversity (a multitude of shapes) and high predictive uncertainty (shapes with a question mark) are on average more sample-efficient in improving compositional generalization in morphological inflection.
Figure 2: StemCorrupt : a data augmentation method, where the stem – aligned subsequences of length 3 or greater in the input and output – is mutated by substitution with random characters from the alphabet.; StemCorrupt : a data augmentation method, where the stem – aligned subsequences of length 3 or greater in the input and output – is mutated by substitution with random characters from the alphabet.
Figure 3: Models are trained and evaluated on entirely different lexemes.; Models are trained and evaluated on entirely different lexemes.
Figure 4: Performance after training on D_{train}\cup\hat{D}^{Syn}_{train} , for varying sizes of \hat{D}^{Syn}_{train} . In the subtitle for each language, we also list the performance from training on no augmented data in parentheses ( \hat{D}^{Syn}_{train}=\emptyset ).; Performance after training on D_{train}\cup\hat{D}^{Syn}_{train} , for varying sizes of \hat{D}^{Syn}_{train} . In the subtitle for each language, we also list the performance from training on no augmented data in parentheses ( \hat{D}^{Syn}_{train}=\emptyset ).
Figure 5: Summary of subset selection strategies performances from. Left: percentage of times each strategy gets the best performance out of 35 settings (across each of the 7 languages and 5 \hat{D}^{Syn}_{train} sizes). Right: bootstrapped confidence intervals for the percentages on the left.; Summary of subset selection strategies performances from. Left: percentage of times each strategy gets the best performance out of 35 settings (across each of the 7 languages and 5 \hat{D}^{Syn}_{train} sizes). Right: bootstrapped confidence intervals for the percentages on the left.
Figure 6: The frequency of the most commonly sampled morphosyntactic description by three of subset selection methods.; The frequency of the most commonly sampled morphosyntactic description by three of subset selection methods.
Figure 7: Pearson correlations between Negative Log Likelihood and three other metrics: length of the stem, length of the target inflected form, and Levenshtein distance between the ground-truth target form and the augmented target.; Pearson correlations between Negative Log Likelihood and three other metrics: length of the stem, length of the target inflected form, and Levenshtein distance between the ground-truth target form and the augmented target.
Figure 8: Performance after training on D_{train}\cup\hat{D}^{Syn}_{train} , for varying sizes of \hat{D}^{Syn}_{train} . In the subtitle for each language, we also list the performance from training on no augmented data in parentheses ( \hat{D}^{Syn}_{train}=\emptyset ).; Performance after training on D_{train}\cup\hat{D}^{Syn}_{train} , for varying sizes of \hat{D}^{Syn}_{train} . In the subtitle for each language, we also list the performance from training on no augmented data in parentheses ( \hat{D}^{Syn}_{train}=\emptyset ).
Improvement of UMT+Loss relative to the random baseline, averaged over all possible sizes of \hat{D}^{Syn}_{train} .
Languages assessed in our experiments on assessing the sample efficiency of data augmentation. We also list their language families and number of speakers.