Understanding Compositional Data Augmentation in Typologically Diverse Morphological Inflection
Authors: Farhan Samir
Miikka Silfverberg
Venue: EMNLP 2023 Main (2023)
Keywords: morphological inflection
computational morphology
data augmentation

Abstract: Data augmentation techniques are widely used in low-resource automatic morphological inflection to address the issue of data sparsity. However, the full implications of these techniques remain poorly understood. In this study, we aim to shed light on the theoretical aspects of the data augmentation strategy StemCorrupt, a method that generates synthetic examples by randomly substituting stem characters in existing gold standard training examples. Our analysis uncovers that StemCorrupt brings about fundamental changes in the underlying data distribution, revealing  inherent compositional concatenative structure. To complement our theoretical analysis, we investigate the data-efficiency of StemCorrupt. Through evaluation across a diverse set of seven typologically distinct languages, we demonstrate that selecting a subset of datapoints with both high diversity \textit{and} high predictive uncertainty significantly enhances the data-efficiency of compared to competitive baselines. Furthermore, we explore the impact of typological features on the choice of augmentation strategy and find that languages incorporating non-concatenativity, such as morphonological alternations, derive less benefit from synthetic examples with high predictive uncertainty. We attribute this effect to phonotactic violations induced by StemCorrupt, emphasizing the need for further research to ensure optimal performance across the entire spectrum of natural language morphology.

Understanding Compositional Data Augmentation in Typologically Diverse Morphological Inflection
Farhan Samir Mikkka Silfverberg
Natural Language Processing Group, University of British Columbia fsamir(cs ubc.ca
Abstract
involve the combination of simpler parts to form complex concepts, where valid combinations are li- censed by a recursive grammar (Kratzer and Heim, 1998; Partee et al., 1984). However, domain gen- eral neural architectures often fail to generalize to new, unseen data in compositional manner, re- vealing failure in inferring the data-generating grammar (Kim and Linzen, 2020; Lake and Baroni, 2018: Wu et al., 2023). This failure hinders these models from closely approximating the productiv- ity and systematicity of human language.
Data augmentation techniques are widely used in low-resource automatic morphological in- flection to overcome data sparsity. However, the full implications of these techniques re- main poorly understood. In this study, we aim to shed light on the theoretical aspects of the prominent data augmentation strategy STEM- CORRUPT (Silfverberg al., 2017; Anasta- sopoulos and Neubig. 2019), a method that generates synthetic examples by randomly substituting stem characters in gold standard training examples. begin, we conduct an information-theoreele analysis, arguing that STEMCORRUPT improves compositional gen- eralization by eliminating spurious correlations between morphemes, specifically between the stem and the affixes. Our theoretical anal- ysis further leads us to study the sample- efficiency with which SteMCorrupt reduces these spurious correlations. Through evaluation across seven typologically distinct languages, we demonstrate that selecting subset of dat- apoints with both high diversity and high pre- dictive uncertainty significantly enhances the data-efficiency of STEMCORRUPT. However, we also explore the impact of typological fea- tures the choice of the data selection strat- egy and find that languages incorporating high degree of allomorphy and phonological alternations derive less benefit from synthetic examples with high uncertainty. We attribute this effect to phonotactic violations induced by STEMCORRUPT, emphasizing the need for fur- ther research ensure optimal performance across the entire spectrum of natural language morphology.!
Consider the task of automatic morphological inflection, where models must learn the underly- ing rules of language's morphoysyntax to pro- duce the inflectional variants for any lexeme from a large lexicon. The task is challenging: the models must efficiently induce the rules with only small dataset. Indeed, recent analysis by Goldman al. (2022) demonstrates that even of-too-attee are-art, task-specific automatic inflection state-of
models fall short of a compositional solution: they perform well in random train-test splits, but strug- gle in compositional ones where they must inflect lexemes that were unseen at training time.
Nevertheless, there is reason for optimism. Sev- eral works have shown that automatic inflection models come much closer to compositional so- lution when the human- dataset is com- plimented by synthetic data-augmentation pro- cedure (Liu and Hulden, 2022; Silfverberg et al., 2017; Anastasopoulos and Neubig. 2019; Lane and Bird, 2020; Samir and Silfverberg, 2022), where morphological affixes are identified and attached to synthetic lexemes distinct from those in the train- ing dataset (Fig. 2). However, little is understood about this prominent data augmentation method and the extent which it can improve composi- tional generalization in neural word inflection. this work, we seek to reveal the implicit assump- tions about morpheme distributions made by this rule- based augmentation scheme, and analyze the
1 Introduction
Compositional mechanisms are widely believed to be the basis for human language production and comprehension (Baroni, 2020). These mechanisms
available at https: com www sammi/ understandi ng- augmentattioooooo morphology.
Our code is

---

Lange nyght
stem
stem
Original triple lemma
a p a u
Strategically bubarrppled cielavel
Original dalasee
IT:V::G;PPFV:PT
π a p
Hallucinated
lemma
π πεpaxáuoro W;;::G;IPFV:PPS
π rÉpéxapoteς
Training set:
[Illustration from Anastasopoulos and Neubig (2019))
Figure 2: STEMCORRUPT: data augmentation method. where stem aligned subsequences length 3 or greater the input output - is mutated by substitution with random characters from the alphabet.
Figure 1: Given a small cumman-annotated dataset and a large pool of synthetic examples, we find that sampling subset of data representing both diversity (a multitude of shapes) and high predictive uncertainty (shapes with a question mark) are on average more sample-efficient in improving compositional generalization in morphological inflection.
vestigate whether it is possible to improve the sample-efficiency with which the data augmenta- tion method induces probabilistic independence between stems and affixes. Specifically, inves- tigate whether we can use a small subset of the synthetic data to add our training dataset. We find that selecting a subset that incorporates both high predictive uncertainty and high diversity (see Fig. 1) is significantly more efficient in removing correlations between stems and affixes, providing an improvement in cumple-efficiency for languages where the morphological system is largely concate- native. At the same time, in accordance with theoretical analysis, this selection strategy impairs performance for languages where phonological al- ternations are common.
effect of these assumptions on learning cross- linguistically prevalent morphological patterns.
To this end, our work presents the first theoret- ical explanation for the effectiveness of composi- tional data augmentation in morphological inflec- tion. Through an information -hoooeti analysis (Section 3), we show that this method eliminates "spurious correlations" (Gardner al., 2021) be- tween the word's 's constituent morphemes, specifi- cally between the stem (e.g., walk) and the inflec- tional affix (e.g..-ed) -ed). By removing these correla- tions, the training data distribution becomes aligned with concatenative morphology, where a word can be broken down into two independent substruc- tures: the stem (identifying the lexeme) and the inflectional affixes (specifying grammatical func- tion). This finding sheds light why the method is widely attested to improve compositional gener- alization (Liu and Hulden, 2022), as concatenative morphological distributions are cross- cross- pooss-inguustticllyy prevalent (Haspelmath and Sims, 2013).
Our work contributes to a comprehensive under- standing of prominent data augmentation method from both theoretical (Section 3) and practical standpoint (Section 4). Through analysís, we aim to inspire further research in the analysis and evaluation of existing compositional data augmentation methods (reviewed Section 6), as well as the development of novel augmentation methods that can better capture cross-linguistic di- versity in morphological patterns.
systematic
We go on to show, however, that the augmen- tation method tends towards removing all correla- tions between stems and affixes, whether spurious or not. Unfortunately, this crude representation of concatenative morphology, while reasonable in broad strokes, is violated in virtually all languages to varying degrees by long-dise onng-iistace phonological phenomena like vowel harmony and reduplication. Thus, our analysis demonstrates that while the method induces a useful approximation to concate- native morphology, there still ample room for improvement in better handling of allomorphy and phonological alternations.
2 Preliminaries
In automatic morphological inflection, assume access gold-standard dataset Dirain with triples of {X, Y, T), where X is the character sequence of the lemma, T a morphosyntactic description (MSD), and Y is the character sequence of the in- flected form. The goal is then to learn the distribu- tion P T) over inflected forms conditioned
Building on our theoretical analysis, we in-
For example, <dog, dogs, N+PL>

---

on a lemma and MSD
abesiabbss
WINDIPST/PF WNOMI22SG) NFM
trinconos
VIMPACCIИИ LINOMIINFM, 2,5G)
Generating synthetic training dataset with STEMCORRUPT. For many languages, Dtroin is too small for models to learn and make systematic morphological generalizations. Previous work has found that generating complementary synthetic dataset train using a data augmentation technique substantially improve generalization (Anasta- sopoulos and Neubig, 2019; Silfverberg et al., 2017, among others).
abestas
WIND:PRS:NO
tRincas
VIMPAACI22 LINOMENNIM,
M225G//NMM
2,PL)
abestsbamos
WIND-PST/PPV :NOM112L)
LINOMETP
train
Figure Models trained and evaluated entirely differ- ent lexemes.
The technique, henceforth called STEMCOR- RUPT, works as follows: We identify the aligned subsequences (of length 3 or greater) between a lemma X and an inflected form Y, which we de- note the stem. 3 We then substitute some of the characters in the stem with random ones from the language's alphabet; Fig. 2. The STEMCORRUFT procedure has hyperparameter 6. sets the proba- bility that a character in the stem will be substituted by a random one; higher value (approaching 1) indicates greater number of substitutions in the stem.*
focuses asymptotic behaviour, specifically, how the conditional distribution P(YiX, T) evolves
add more augmented data (1 (Dey 0o).
Theorem For all (X, Y, T) datapoints
Dtrain. assume that X and share a stem Y stem that is non-empty, and let Yaffiz be the remaining characters of Y. Let Xstem and X offfiz fix be analogously defined. Further, let D train be generated with STEMCORRUPT using 1.5 Next, consider the data- tetta-generating probability distribution P(Y T) over Syg U DO, we have that P(Y T) P(Y stoms Yaffin (X, P(YoJii Xaffix: P)P stem (atem).
How does STEMCORRUPT improve composi- tional generalization? Despite the widespread adop- tion this technique for automatic morphological inflection and analysis, this fundamental question has heretofore remained unanswered. In the next section, we argue that STEMCORRUPT improves compositional generalization by removing corre- lations between inflectional affixes and the stems to which they are attached. By enforcing indepen- dence between these two substructures, STEMCOR- RUPT facilitates productive reuse of the inflectional affixes with other lexemes. That is, our theoreti- cal argument shows that the effectiveness of the method arises from factoring the probability distri- bution to approximate concatenative morphology, a linuitiingyssyically prevalent strategy for word for- mation where words can be "neatly segmented into roots and affixes" (Haspelmath and Sims, 2013). We formalize and prove this in the following sec- tion.
Dtrain- Then, | Sym trasn
Remark (Concatenative compositionality) The augmentation method thus makes the model more effective at capturing concatenative morphologi- cal patterns, the conditional probability distri- bution becomes factorized into root generation component (P(Y steml -eeml-)) and an affix generation component (P(Y fff Crucially, this removes the potential for any spurious correlation between these two substructures."
Remark 2 (Stem -fffx dependencies). While con- catenation a liggussiigu prevalent strat- egy for inflection (Haspelmath and Sims, 2013). stems and affixes are rarely entirely independent. Therefore, enforcing complete independence be- tween these structures is an overly strong constraint that STEMCORRUPT places on the training data. The constraint is consistently violated in Turkish, for example, where front-back vowel harmony con- straints dictate that the vowels in the suffix share the same front/back feature as the initial vowel the stem. This leads to forms like "dalarn" and pre-
3 STEMCORRUPT induces compositional structure
this section, we analyze the ramifications of training on the synthetic training dataset gener- ated by STEMCORRUPT (5m and the human- annotated training dataset (Dtrain). Our analysis
'That we substitute all characters the stem.
"Additionally, this factorization is likely to reducing over- fitting: The model effectively learns to minimize the negative log-likelihood of this simplified and factorized data distribu- tion P(Y atem P(Y (-), rather than the complex joint distribation P(Y stemo Y Yfffixl-).
'The stem can thus be discontinuous; see Fig. We use the implementation here, which sets A = 0.5.

---

vents forms like "dalerin". "dalerın", or "dalarin" (Kabak, 2011). In Section 5, we show that STEM- CORRUPT regularly generates examples violating vowel harmony, and that this can undermine its ef- fectiveness. Nevertheless, the empirical success of STEMCORRUPT demonstrates that the benefits of its concatenative bias outweigh its limitations. Remark 3 ( (Comparison to previous accounts). Our analysis provides a simple yet the most accu- rate characterization of STEMCORRUPT. Previ- ous works have called STEMCORRUPT beneficial "copying bias" (Liu and Hulden, 2022: Jaidi et al., 2022) or strategy for mitigating overgeneration of common character n n-grams (Anastasopoulos and Neubig, 2019). However, our analysis demon- strates that neither of these characterizations are en- tirely accurate. First, the denotation of a "copying bias" is only suggestive of the second factor in our statement (tYYste//// s(sttem(Xstem) Xstem), and does not address the impact affix generation. In contrast, analysis shows that both stem and affix generation are affected. Furthermore, alleviating overfitting to common character sequences is also misleading. as would suggest that STEMCORRUPT serves the same purpose as standard regularization techniques like label smoothing (Müller et al., 2019).7
stem is invariant with respect to the inflectional features. Then, by Proposition (ii), we have that P(Ystem X) P(Y stem xttem Xaffin) P(Y stem X.tem), since the stem is invariant with respect to the inflectional affix of the lemma.
the factor P(Y of stemm T).. By parts (iii) and (iv) of Proposition 1, we have that this can be sim- plified P(Y af fizz Xfffin T). Taken together, we have that P(Y X T) can be decomposed into P(Y of fafs T)P(Y stom X stem)
Next,
we
tackle
Investigating STEMCORRUPT's sample effi- ciency. So far, we have studied the behaviour of STEMCORRUPT through an asymptotic argument, demonstrating that in the infinite limit of |D Sym STEMCORRUPT enforces complete independence between stems and affixes. In doing so, it likely removes a number of spurious correlations between stems and affixes, thus providing a theoretical ex- planation for its attested benefit in improving com- positional generalization. However, our theoretical analysis, while informative of the overall effect of STEMCORRUPT, says little about the sample efficiency of the method in practice.
Indeed, recent studies in semantic parsing have demonstrated that sample efficiency can be greatly increased by strategically sampling data to over- come spurious correlations that hinder composi- tional generalization non-lID data splits (Oren et al., 2021; Bogin et al., 2022; Gupta et al., 2022). In the following section, we examine whether strategic data selection can yield similar benefits in the context of typologically diverse morphological inflection.
3.1 Proving the theorem
The proof of the theorem is straightforward with the following proposition (proved in Appendix B).
Proposition 1. As Dyyn - oc, the mutual infor- mation between certain pairs of random variables declines:
(i) I(Y stemi T) +
4 Extracting sample-efficient training sets from STEMCORRUPT
(ïi I(Yatem:: stemi Xaffix) +
(ii)) a( sseem) + 0
Problem setup. We use the following prob- lem setup from Oren et al. (2021) to investi- gate whether the sample-efficiency of STEMCOR- RUPT can be improved. Recall that we have a dataset Derain with gold triples of (X, Y, T). Fur- ther, we have synthesized dataset D 5ym train where D > Dirainl- Our goal is now to select Syn
(iv) (Yoffio) X sttem) +
definition of Y Yseem Yoffian we have that P(Y[X,T) = P(Y stem: Y Yaffin X, T). Then, by the chain rule of probability, we have P(Yaffizl steem, X, T) stem X, T). We first deconstruct the second factor. By Propo- sition (i), we have that the second factor P(Y stem () = P(Ystom X), since the
Proof of Theorem By the
S train that training the model on Dtrain U U maximizes performance on a held- out compositional testing split.
"Our preliminary experiments did not support this position; compositional generalization performance was not sensitive to lahel smoothing, yet was significantly improved hy STEM- CORRUPT.
4.1 Model and training
We start by training an inflection model M on the gold standata training data, denoted as D Darain- Fol-

---

lowing Wu al. (2021); Liu and Hulden (2022), we employ Transformer (Vaswani et al., 2017) for M. use the fairseq package (Ott et al.. 2019) for training our models and list our hyperparame- ter settings in Appendix A. We conduct all of our experiments with Dtrainl = 100 gold-standard examples, adhering to the the low-resource set- ting for SIGMORPHON 2018 shared task for each language. We next describe the construction of Dyy
HIGHLoss. Next, we employ selection strat- egy that selects datapoints that have high predictive uncertainty to the initial model M. Spurious cor- relations between substructures (like stem and T; Section 3) will exist in any dataset of bounded size (Gupta et al., 2022; Gardner et al., 2021), and we conjecture that selecting high uncertainty data- points will efficiently mitigate these correlations.
We quantify the uncertainty of synthetic dat- apoint D by computing the negative log- likelihood (averaged over all tokens in the the tar- get Y) for each synthetic datapoint in D train Next, we select the synthetic datapoints with the highest uncertainty and add them to train
4.2 Subset sampling strategies
Here, we introduce a series of strategies for sam- pling from D oriented for improving compo- sitional generalization. Broadly, we focus on se- lecting subsets that reflect either high structural diversity, high predictive uncertainty, both, as these properties have been tied to improvements in compositional generalization in prior work on semantic parsing (e.g., Bogin al., 2022), an NLP research area where compositionality is well stud- ied.
To thoroughly demonstrate that incorporating predictive uncertainty is important for yielding training examples that counteract the spurious dependencies in the ground-truth training dataset, we benchmark it against another subset selection strategy LowLoss With this method, we instead select synthetic datapoints that the model finds easy, i.e., those with the lowest uncertainty scores. We hypothesize this strategy will yield less performant synthetic training datasets, as biased towards selecting datapoints that corroborate rather than counteract the spurious correlations learned by M.
Our baseline sampling method is to construct D by sampling from the large synthetic training data Syy uniformly.
RANDOM.
UNIFORM MORPHOLOCICAL TEMPLATE (UMT). With this method, we seek to improve the structural diversity in our subsampled in synthetic training dataset tramn Training diverse subset is crucial, as the SIGMORPHON 2018 shared task dataset is imbalanced in frequency of different morphosyntactic descriptions (MSDs). These imbalances can pose challenges to the model in generalizing to rarer MSDs. To incorporate greater structural diversity, we employ the templatic sampling process proposed by Oren et al. (2021). Specifically, modify the distribution over MSDs to be closer to uniform in Dyg
hybrid ap- proach containing both high struetural diversity and predictive uncertainty by combining UMT/EMT and HIGHLOSS. First, we sample an MSD (according to the MSD distribution defined by UMT/EMT) and then select the most uncertain synthetic datapoint for that T.
UMT/EMT+ Loss. Finally,
Experiments and Results
Data. We use data from the UniMorph project (Batsuren et al., 2022), considering typological diversity when selecting languages include. We aim for evaluation similar in scope Muradoglu and Hulden (2022). That is, broadly, we attempt to include types of languages that exhibit variation in inflectional characteristios such as inflectional synthesis of the verb, exponence, and morphological paradigm size (Haspelmath et al., 2005). Our selected languages can be seen in Fig. 4. We provide further information on the languages in Appendix C.
Formally, we sample without replace- ment from the following distribution: Ya () = P(T) ET where p(T is the proportion of times that appears in D Sym
train. We consider two cases: a = 0 corresponds to sampling MSDs from a uniform distribution (UMT), while
corresponds to sampling tags according to the empirical distribution over MSDs (EMT).
FFor example, the low-resource Georgian training dataset contains 9 instances of of nouns inflected for PT,; ERG, but only one instance of noun inflected for SG; ERG.
Obtaining a large synthetic dataset. In order to

---

bengeli Inp aug, 0.02)
spanish Inp sug. 0.05)
Winnen-aak- parcentages
Accuracy
Accuratt
C
0.40
6.0
HighLoss
0.15
Accuraty
turlish inc aug. 0.04)
Accurocy
arabic (no aug. 0.01)
UNT
11.1%
URE
0.2
&
35.315
0.25
6o
0.00
16.7%
finnish (no sug : 0.00)
navejo (no aug: 0.02)
0%
random
Accuracy
0.10
JVE Loos
0.06
0.50
randon
Accuraty
georgien (no aug. 0.09)
Strategy
0.5
LMTs
UNT
Hightoss
EMT
Figure 5: Summary of subset selection strategies performances from. Left: percentage times each strategy gets the best performance out of 35 settings (across each the 7 languages and 5 D for the percentages the left.
2048
EMT+Loss
Number Augmented Examples
Figure 4: Performance after training on Derain 5yn for varying sizes of D In the subtitle for cach language, we augmented data
trase sizes). Right: bootstrapped confidence intervals
also list the performance from training in parentheses (ßSge = f).
alization. At high level, we find that data augmen- tation brings substantial benefits for compositional generalization compared to models trained solely on the gold otandadaddard training data Dtrain. Without STEMCORRUPT, the initial model M for every lan- guage achieves only single-digit accuracy, while their augmented counterparts perform significantly better. For instance, the best models for Georgian and Spanish achieve over 50% accuracy. These findings agree with those of Liu and Hulden (2022) who found that unaugmented Transformer models fail to generalize inflection patterns.
generate the large augmentation dataset Syn for every language, we generate 10, 000 augmented datapoints for every language by applying STEM- CORRUPT to their respective low datasets from SIGMORPHON 2018 (Cotterell et al., 2018).
Generating compositional generalization test set. For generating test sets, we adopt the lemma-split approach of Goldman et al. (2022). Specifically, we use all available data from SIGMORPHON2018 for the target language, excluding any lexemes from the low setting since those were used to train the initial model M (Section 4.1). The remaining lexemes and their associated paradigms comprise compositional generalization test set; see Fig. 3.
We also find that performance tends to increase as we add more synthetic data; the best models for every language are on the higher end of the Shn sizes. This finding agrees with theoret- ical results that the dependence between the stem (Y stem) and that of the inflectional affix Yafftr) weakened as we add more samples from STEM- CORRUPT (Section 3; Proposition 1).
We evaluate the performance of all methods listed Section 4 selecting Syy We evaluated the performance of using DSym
Populating Syy
train
Effective subsets have high diversity and predic- tive uncertainty. Our analysis reveals statistically significant differences between the subset selection strategies, highlighting the effectiveness of the hy- brid approaches (UMT/EMT+LosS) that consider both diversity and predictive uncertainty. Among the strategies tested, the UMT+Loss method out- performed all others in approximately one-third of the 35 settings examined, as indicated in Fig- ure 5 (left). The improvements achieved by the UMT+LoSS method over random bascline were statistically significant (p 0.05) according bootstrap percentile test (Efron and Tibshirani, 1994), as shown in Figure (right). Moreover, our results also show that the EMT+LoSs strat- egy closely followed the UMTS approach,
train of sizes ranging from 128 2048 exam- ples, increasing the size by powers of 2.
5.1 Results
We demonstrate the results of each strategy for all languages, considering each combination of lan- guage, subset selection strategy, and Stm I. thus obtaining 35 sets of results. For each setting, we report the performance achieved over 6 different random initializations, along with their respective standard deviations. For brevity, we show the re- sults for trasn {128, 512, 2048); we include the expanded set of results (including (256, 1024)) in Appendix D.
STEMCORRUPT improves compositional gener-

---

Furthermore, selecting datapoints based solely on high predictive uncertainty without consider- ing diversity (HIGHL.Oss) is an ineffective strat- egy, having the second lowest proportion of wins (Fig. 5, right). Empirically, we find this may be attributed to the HIGHLOSS strategy fixating on Eoeetation with Margetie hoc
type leegth
100
target
302
Froqurtyy Modk
200-
02
102
60-
bengell
Arnkh
grorglon rauejo
turkish
teriged
anblic
sparich geetrg
Langueg:
Lenguage
Figure 6: The frequency of the most commonly sampled mor- phosyntactic description by three of subset selection methods.
Figure 7: Pearson correlations between Negative Log Likeli- hood and three other metrics: length of the stem, length of the target inflected form, and Levenshtein distance between the ground-truth target form and the augmented target.
achieving the highest performance in a quarter of the cases. In contrast, the same strategies without the uncertainty component were much less effec- tive. For instance, UMT never achieved the best performance in any combination of the languages and 155mm sizes, highlighting that selecting a di- verse subset without factoring in predictive uncer- tainty is suboptimal.
is higher than the correlation with other plausi- ble factors driving uncertainty, namely stem length and target length. 10 Overall, the lagging sample efficiency of LowLoss corroborates our theory; STEMCORRUPT is effective because it generates datapoints where the stem has no correspondence with the affix. LowLoss counteracts its effec- tiveness, as it is biased towards datapoints where spurious dependencies between the stem and affix are maintained." |I
Selecting by high predictive uncertainty worsens performance when there are stem-affix depen- dencies. We found that the UMT+LoN strategy improves performance for 5 out of 7 languages compared to the RANDOM baseline. The improve- ment ranges from 4.8 (Georgian) to small declines of - 1.9 (Turkish) and -0.9 (Finnish). The de- clines for Finnish and Turkish are partly due to a mismatch between the generated synthetic exam- ples and the languages' morphophonology. STEM- CORRUPT will generate synthetic examples that vi- olate vowel harmony constraints between the stem and the affix. For instance, it may replace a front vowel in the stem with a back one. As a result, LMS+ will select such harmony-violating examples more often, since they have greater un- certainty to the initial model M (Section 4.1), ), re- sulting in the augmented model tending to violate the harmony restrictions more often. Indeed, for Turkish, the average uncertainty for synthetic exam- ples violating vowel harmony (0.46) is significantly higher than those that adhere to vowel harmony
a particular MSD, as shown in Fig. 6, rather than exploring the full distribution of MSDs. The fig- ure displays the frequency of the most commonly sampled morphosyntactic description for each of UMT+ RANDOM, and HIGHLOSS strate- gies. Across all languages, the HIGHLOSS method samples the most frequent tag much more often than the RANDOM and UMT+LOSS methods.? 9 "Easy" synthetic datapoints have low sample ef- ficiency. As hypothesized, the datapoints with low uncertainty hurt performance. We attribute this to the LowLoss strategy selecting datapoints with smaller number of substitutions the stem. In Fig. 7, we show that the number of edits made to the stem as measured by Levenshtein dis- tance between the corrupted target sequence and the uncorrupted version - is strongly correlated with the uncertainty assigned to the synthetic data- point across all languages. Moreover, the correla- tion between the number of edits and uncertainty
'TTareet length known contribute to predictive uncer- tainty (Eikema and Aziz, 2020; Kim and Linzen, 2020), and stem length confounding factor of the levenshtein distance calculation.
"The reason that uncertainty estimates are higher for a given MSD is not entirely clear. In our investigation, we found a small correlation (p = 0.15) between the morphosyntactic description frequency and uncertainty. However, there are likely other factors beyond frequency that contribute to higher uncertainty: for example, morphological fusion (Bickel and Nichols, 2013; Rathi et al., 2021).
'The ineffectiveness of LowLoss also corroborates that STEMCORrUPT does not simply induce a "copying bius" (Re- mark 3), since then we would expect all of the subset selection methods to perform similarly.

---

Language
Improvement
ing in favour of neural network-based recombina- tion. Chen et al. (2023) review more composi- tional data augmentation techniques, situating them within the broader landscape of limited-data learn- ing techniques for NLP.
Georgian
4.8
Bengali
2.5
Spanish
1.3
Navajo
1.1
Arabic
0.4
Finnish
0.9
Turkish
-1.9
Extracting high-value subsets in NLP training data. Oren et al. (2021); Bogin et al. (2022); Gupta et al. (2022) propose methods for extracting diverse sets of abstract templates to improve compositional generalization in semantic parsing. Muradoglu and Hulden (2022) train baseline model on a small amount of data and use entropy estimates to select new data points for annotation, reducing anmotation costs for morphological inflection. Swayamdipta et al. (2020) identify effective data subsets for train- ing high-quality models for question answering, finding that small subsets of ambiguous examples perform better than randomly selected ones. Our work is also highly related to active-learning in NLP (Tamkin et al., 2022; Yuan al., 2020; Mar- gatina et al., 2021, inter alia); however we focus on selecting synthetic rather than unlabeled datapoints, and our experiments are geared towards composi- tional generalization rather than IID performance.
Table I: Improvement Los relative to random hascline, averaged over all possible sizes of De
(0.39), as assessed by a bootstrap percentile test (p < 0.05). This finding also corroborates our the- ory from Section 3: STEMCORRUPT eliminates de- pendencies between stems and affixes, even when the dependencies are real rather than spurious. This shortcoming of STEMCORRUPT is exacerbated by selecting examples with high uncertainty, as these examples are less likely to adhere to stem-affix constraints like vowel harmony.
Takeaways. Aligning with the semantic parsing literature on efficient compositional data augmen- tation (Oren et ul., 2021; Bogin et al., 2022; Gupta et al., 2022), we find that certain subsets of data are on average more efficient eliminating spurious correlations between substructures in the case morphological inflection, the relevant substruc- tures being the individual morphemes: the stem (Y stemm and affix (Y ffz Section 3). However, the sample-efficiency gains from strategic sampling are less dramatic than in semantic parsing (see, for example, Oren et al., 2021).
Compositional data splits in morphological in- flection. Assessing the generalization capacity of morphological inflections has proven challenging and multifaceted problem. Relying standard "IID" (Oren et al., 2021; Liu and Hulden, 2022) splits obfuscated (at least) two different manners in which inflection models fail to generalize com- positionally.
We provided empirical evidence that the gains are tempered by STEMCORRUPT's tendencies to violate stem-affix constraints in its synthetic train- ing examples, such as vowel harmony constraints in Turkish. Thus, further work is needed to adapt or supplant STEMCORRUPT for languages where such long-range dependencies are commonplace. In doing so, the data selection strategies are likely fetch greater gains in sample efficiency.
First, Goldman et al. (2022) uncovered that gen- eralizing to novel lexemes was challenging for even state of the art inflection models. Experiments by Liu and Hulden (2022) however showed that the STeMCORrUPT method could significantly im- prove generalization novel lexemes. Our work builds on theirs by contributing understanding the relationship between STEMCORRUPT and lexi- cal compositional generalization. Specifically, we studied the structure of the probability distribution that StemCorrupt promotes (Section 3), and the conditions under which it succeeds (Remark 1) and fails (Remark 2).
6 Related work
Compositional data augmentation methods. In general, such methods synthesize new data points by splicing or swapping small parts from exist- ing training data, leveraging the fact that certain constituents can be interchanged while preserving overall meaning syntactic structure. A common approach to swap spans between pairs of data- points when their surrounding contexts are identi- cal (Andreas, 2020; Guo et al., 2020; Jia and Liang, 2016, inter alia). Recently, Akyürek et al. (2021) extended this approach, eschewing mule-based splic-
Second, Kodner et al. (2023) showed that inflec- tion models also fail generalize compositionally to novel feature combinations, even with agglutina- tive languages that have typically have strong one- to-one alignment between morphological features and affixes. Discovering strategies to facilitate com-

---

positional generalization in terms of novel feature combinations remains an open-area of rescarch.
Empirical analysis. In our empirical analysis, we acknowledge two limitations that can be addressed in future research. First, we conduct our experi- ments using data collected for the SIGMORPHON 2018 shared task, which may not contain a natural- istic distribution of morphosyntactic descriptions since it was from online database (Wiktionary). In future work, we aim to replicate our work in a more natural setting such as applications for en- dangered language documentation (Muradoglu and Hulden, 2022; Moeller et al., 2020), where the mor- phosyntactic description distribution is likely to be more imbalanced. Second, we perform our analy- ses in an extremely data-constrained setting where only 100 gold-standard examples are available. In higher resourced settings, data augmentation with STEMCORRUPT may provide much more lim- ited improvement to compositional generalization; indeed the compositional generalization study of morphological inflection systems by Goldman et al. (2022) demonstrates that the disparity between IID generalization and compositional generalization largely dissipates when the model is trained on more gold-standard data.
7 Conclusion
This paper presents a novel theoretical explanation for the effectiveness of STEMCORRUPT, a widely- used data augmentation method, in enhancing com- positional generalization in automatic morpholog- ical inflection. By applying information-theorent constructs, we prove that the augmented exam- ples work to improve compositionality by eliminat- ing dependencies between substructures in words stems and affixes. Building off of our theoretical analysis, we present the first exploration of whether the sample efficiency of reducing these spurious dependencies can be improved. Our results show that improved sample efficiency is achievable by selecting subsets of synthetic data reflecting high structural diversity and predictive uncertainty, but there is room for improvement both in strategic sampling strategies and more cross-linguistically effective data augmentation strategies that can rep- resent long distance phonological alternations.
Overall, NLP data augmentation strategies are poorly understood (Dao et al., 2019; Feng et al., 2021) and our work contributes to filling in this gap. Through our theoretical and empirical analyses, provide insights that can inform future research on the effectiveness of data augmentation methods in improving compositional generalization.
Acknowledgements
We thank the EMNLP reviewers, the area chair, Vered Shwartz, Kat Vylomova, and Adam Wiemer- slage for helpful discussion and feedback on the manuscript. We also thank Ronak D. Mchta for insightful discussion on properties of mutual infor- mation. This work was supported by NSERC PGS-D scholarship to the first author.
8 Limitations
Theoretical analysis. We make some simplify- ing assumptions to facilitate our analysis Sec- tion 3. First, we assume that the stem between a gold-standard lemma and inflected form X and Y is discoverable. This is not always the case; for example, with suppletive inflected forms, the rela- tionship between the source lemma and the target form is not systematic. Second, we assume that all characters the stem are randomly substituted, corresponding to setting the f = for STEMCOR- RUPT. This does not correspond to how we deploy STEMCORRUPT; the implementation provided by Anastasopoulos and Neubig (2019) sets 0 = 0.5 and we use this value for empirical analysis Section We believe the analysis under 0 1 provides valuable and accurate characterization of STEMCORRUPT nonetheless and can be readily
extended to accommodate the < 0 < 1 case in future work.

---

References
ence, pages 840-855, Marseille, France. European
Language Resources Association.
Ekin Akyürek, Afra Feyza Akyürek, and Jacob Andreas. 2021. Learning to recombine and resample data for compositional generalization. In International Con- ference on Learning Representations.
Balthasar Bickel and Johanna Nichols. 2013. Expo- nence of selected inflectional formatives (v2020.3), In Matthew S. Dryer and Martin Haspelmath, edi- tors, The World Atlas of Language Structures Online. Zenodo.
Antonios Anastasopoulos and Graham Neubig. 2019. Pushing the limits of low-resource morphological in- flection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLP), pages 984-996, Hong Kong. China. Association for Com- putational Linguisties.
Ben Bogin, Shivanshu Gupta, and Jonathan Berant. 2022. Unobserved local structures make composi- tional generalization hard. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 2731-2747, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Jacob Andreas. 2020.
Compositional Data Augmentation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7556-7566, Online. Association for Computational Linguisties.
Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2023. An empirical survey of data aug- mentation for limited data learning in NLP. Transac- rions of the Association for Computational Linguis- tics, 11:1112111.
Marco Baroni. 2020.
Linguistic generalization and compositionality in modern artificial neural networks. Philosophical Transactions of the Royal Society B, Publisher: The Royal Society.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sabrina J. Mielke, Gar- rett Nicolai, Miikaa Silfverberg, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. The CONLLL-SGGORPPHON 2018 Shared Task: Uni- versal Morphological Reinflection. In Proceedings of the CONIL-SIGMORPHON 2018 Shared Task: Universal Morphological Reinftection, pages 1-27, Brussels. Association for Computational Linguistics.
375(1791):20199) 903077
Khuyagbaatar Batsuren, Omer Goldman, Salam Khal- ifa. Nizar Habash, Witold Kieraš, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yusti- nus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel EI- Khaissi Tiago Pi- mentel, Michael Gasser, William Abbott Lane, Mohit Raj. Matt Coler, Jaime Rafael Montoya Samame. Delio Siticonatzi Camaiteri, Esaú Zu- maeta Rojas, Didier López Francis, Arturo Once- vay, Juan López Bautista, Gema Celeste Silva Vil- legas, Lucas Torroba Hemnigen, Adam Ek, David Guriel, Peter Dirix, Pean-Philippe Bernardy, An- drey Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos, Roberto Zariquiey, Karina Sheifer. Sofya Ganieva, Hilaria Cruz, Ritván Karahóga, Stella Markantonatou, George Pavlidis, Matvey Plu- garyov, Elena Klyachko. Ali Salchi, Candy An- gulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia Krizhanovskaya, Elizaheth Salesky, Clara Vania, Sar- dana Ivanova, Jennifer White, Rowan Hall Maud- slay. Josef Valvoda, Ran Zmigrod, Paula Czarnowska, Irene Nikkarinen, Aelita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington. Yuval Pinter, Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stochr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers. Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young, Daria Rodionova, Anastasia Yemelina, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud hommcaux, Maria Nepomni- ashchaya, Fausto Giunchiglia. Eleanor Chodroff, Mans Hulden. Miikka Silfverberg. Arya D. Mc- Carthy, David Yarowsky, Ryan Cotterell, Reut Tsar- faty, and Ekaterina Vylomova. 2022. UniMorph 4.0: Universal Morphology. In Proceedings of the Thir- leenth Language Resources and Evaluation Confer-
Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Ré. 2019. A kernel theory of modern data augmentation. In Proceedings of the 36th International Conference on Machine Learning, ICML. 2019, volume 97 of Proceedings of Machine Learning Research, pages 1528-1537. PMLR
Bradley Efron and Robert Tibshirani. 1994. An intro- duction to the bootstrap. CRC press.
Bryan Eikema and Wilker 2020. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Proceedings of the 28th Inter- national Conference on Computational Linguistics. pages 4506-4520. Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chan- dar, Soroush Vosoughi, Teruko Mitamura, and Ed- uard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACI-IJCNLPP 2021, pages 968 988, Online. Association for Computa- tional Linguistics.
Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021. Competency problems: On finding and removing artifacts in language data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- wral Language Processing, pages 1801-1813, Online

---

and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Brenden M. Lake and Marco Baroni. 2018. General- ization without systematicity: On the compositional skills of sequence-to-squuene In Proceedings of the 35th International Conference on Machine Learning. ICML 2018, volume 80 of Proceedings of Machine Learning Research, pages 2879-2888. PMLR.
recurrent networks.
Omer Goldman, David Guriel, and Reut Tsarfaty. 2022. (Un)solving Morphological Inflection: Lemma Over- lap Artificially Inflates Models' Performance. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 864-870, Dublin, Ireland. As- sociation for Computational Linguistics.
William Lane and Steven Bird. 2020. Bootstrapping techniques for polysynthetic morphological analy- sis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6652-666 6661,, Online. Association for Computational Linguistics.
Demi Guo, Yoon Kim, and Alexander Rush. 2020. Sequence-level mixed sample data augmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5547-5552, Online. Association for Computa- tional Linguisties.
Paul M. A. Lewis. 2009. Ethnologue: languages of the world.
Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Structurally diverse sampling for sample- efficient training and comprehensive evaluation. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages (979, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Ling Liu and Mans Hulden. 2022. Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neu- ral Morphological Inflection Models. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 739-749, Dublin, Ireland. Association for Computational Linguistics.
Martin Haspelmath. Matthew S Dryer, David Gil, and Bernard Comrie. 2005. The world atlas of language structures. OUP Oxford.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. 2021. Active Learning by Ac- quiring Contrastive Examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 650-663, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Martin Haspelmath and Andrea Sims. 2013, Under- standing Morphology. Routledge.
Badr Jaidi, Utkarsh Saboo, Xihan Wu, Garrett Nico- lai, and Miikka Silfverberg. 2022. Impact of se- quence length and copying clause-level inflec- tion. In Proceedings of the The 2nd Workshop on Multi-lingual Representation Learning (MRL), pages 106-114, Abu Dhabi. United Arab Emirates (Hybrid). Association for Computational Linguisties.
Sarah Moeller, Ling Liu, Changbing Yang. Katharina Kann, and Mans Hulden. 2020. IGT2P: From in- terlinear glossed texts to paradigms. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5251-5262. Online. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 12-22, Berlin, Germany. Association for Computa- tional Linguisties.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? volume 32. Advances in Neural Information Processing Systems.
Saliha Muradoglu and Mans Hulden. 2022. Eeny. meeny, miny, moe. How to choose data for morpho- logical inflection. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 7294-7303, Abu Dhabi, United Arah Emirates. Association for Computational Lin- guistics.
Bariş Kabak. 2011. Turkish vowel harmony. The Black- well companion to phonology. pages 1-24.
Najoung Kim and Tal Linzen. 2020. COGS: A Compo- sitional Generalization Challenge Based on Semantic Interpretation. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105, Online. As- sociation for Computational Linguistics.
Inbar Oren, Jonathan Herzig, and Jonathan Berant. 2021. Finding needles in a haystack: Sampling Structurally- diverse Training Sets from Synthetic Data for Com- positional Generalization. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 10793-10809, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jordan Kodner, Sarah Payne. Salam Khalifa, and Zoey Liu. 2023. Morphological inflection: A reality check. In Annual Meeting of the Association for Computa- tional Linguistics.
Angelika Kratzer and Irene Heim. 1998. Semantics in generative grammar, volume 1185. Blackwell Oxford.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael

---

Auli. 2019. fairseq; Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics (Demon- strations), pages 48-53, Minneapolis, Minnesota. As- sociation for Computational Linguisties.
Michelle Yuan, Hsuan-Tiew Lin, and Jordan Boyd- Graber. 2020. Cold-start active learning through self- supervised language modeling. In Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7935-7948, Online. Association for Computational Linguistics.
Barbara Partee et al. 1984. Compositionality. Varieties of formal semantics, 1:281-311.
Neil Rathi, Michael Hahn, and Richard Futrell. 2021. An information-thooeee characterization of morpho- logical fusion. In Proceedings of the 2021 Confer- ence on Empirical Methods Natural Language Pro- cessing, pages 10115-10120, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Farhan Samir and Miikka Silfverberg. 2022. One wug. two wug+s transformer inflection models hallucinate affixes. In Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study En- dangered Languages, pages 31-40, Dublin, Ireland. Association for Computational Linguistics.
Miikka Silfierherg, Adam Wiemerslage, Ling Liu, and Lingshuang Jack Mao. 2017. Data augmentation for morphological reinflection. Proceedings of CoNLI SIGMORPHON 2017 Shared Task: Univer- sal Morphological Reinflection, pages 90-99, Van- couver. Association for Computational Linguisties.
Swabha Swayamdipta. Roy Schwartz, Nicholas Lourie. Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset Cartography: Map- ping and Diagnosing Datasets with Training Dynam- ics. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online. Association for Computational Linguistics.
Alex Tamkin, Dat Nguyen, Salil Deshpande, Jesse Mu, and Noah Goodman. 2022. Active learning helps pre- trained models learn the intended task. In Advances in Neural Information Processing Systems.
Joy A Thomas and Thomas M Cover. 2006. Elements of information theory. John Wiley & Sons.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. volume 30. Advances in Neural Informa- tion Processing Systems.
Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. Applying the Transformer Character-level Trans- duction. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 1901-1907, Online. Association for Computational Linguistics.
Zhengxuan Wu, Christopher D. Manning, and Christo- pher Potts. 2023. Recogs: How incidental details of a logical form overshadow an evaluation of semantic interpretation.

---

A
Transformer training details
Bengali
Indo-Aryan; 300M
Finnish
Uralie; 5.8M
Arabic
Semitic; 360M
Batch size
16
Navajo
Athabaskan; 170K
Label smoothing
0.2
Turkish
Turkic; 88M
Warmup updates
4000
Spanish
European::: 592M
Georgian
Kartvelian; 3.7M
Total updates
6000
Dropout
0.3
Table 2: Languages assessed our experiments on assessing the sample efficiency data augmentation. We also list their language families and number of speakers.
Encoder layers
Decoder layers
4
Attention dropout
0.1
Let IA := I(T; Y stem) be the mutual informa- tion between the random variables T and Y stee in the synthetic dataset troin" where T is gener- ated from P(T) (as before) and Y stem is generated from PA Y stem (T). The data augmentation algo- rithm generates the stem characters by uniformly sampling characters the language's 's alphabet. Cru- cially, this means the mutual information Ia = 0, since the value Y stem is independent of the value of T.
Adam (31, 92)
(0.9, 0.999)
B Proof of Proposition 1
We recall proposition
Proposition 1. As Deer +
oc, the mutual infor- mation between certain pairs of random variables declines:
(i) I(Y stemi T) -
Then, let I == I(T; Y stem) be the mutual information between the random variables T and Y stem Over Dtrain U Dyyn train where (T, Y Yttem) (Ysomll
(ii) I(Y stemi (affte)
(iā) o(Yoffizi Y stemm) +
(T(T), APc stom (T) + A) sYscm!T sT)
(iv) I(Yaffirs steem -
and À = | / Dtrain U Drranl By the convexity of mutual information (Theorem 2), we have that I S. Ale + (1 A)LA-
Sym
Our proof hinges the fact that mutual informa- tion I(X;Y) Y)) convex in the conditional distribu- tion P(Y when the marginal distribution P(X)
As take [D Sym train + oo, we have that Ale + (1 A)TA > 0 IG 1
1111. TA O. Thus, I is lower bounded by zero (since mutual information non- negative) and upper bounded by 0 as D D 0 (by the above argument). Thus, we have that I - 0, as desired. This proves (1).
is held constant, due to Thomas and Cover (2006).
Theorem 2 (Thomas & Cover). Let (X,Y) ~ p(z, y) p(=)p(u((x). The mutual information I(X;Y) is a concave function of p(x) for fixed plyla) and a convex function of p(y)x)) for fixed p(z).
The same argument can be applied to prove (ii), (iii), and (iv). For (ii), we let Xoffie take the place of T and repeat the argument above. For (iii), we let Y ff take the place of T. For (iv), let Y offfir take the place of T and let X take the place of Y stem-
This theorem is useful for our argument since the data augmentation algorithm results in the marginal distribution over random variables being af- fected (namely Y stem: Xstem) and other marginals staying fixed (T, Xaffiz: Y fif). This enables us to invoke the latter half of the theorem ("convex function of p(yl .r) for fixed p(z)"" and thus obtain an upper bound on the mutual information between the pairs of variables stated in proposition 1. We will argue that this upper bound will decline to 0 we take D Syr -> D0, and thus the mutual information must also decline to O.
C Language information
In Table 2, list the languages assessed in our experiments on assessing the sample efficiency of STEMCORRUPT with their language familics and estimated number of speakers (Lewis, 2009).
D Expanded results for assessing STEMCORRUPT'S sample efficiency
Proof. Let IG I(T; Y stem) be the mutual infor- mation between the random variables T and Y stem in the human annotated dataset Dirain, where T is generated from some distribution P(T) and Ystem. be generated from Pa PY(mst (T).
Here we present the expanded set of results for Section 5; see Fig. 8. The results are the same as those Fig. 4, except they also include S e

---

{256, 1024) in addition to {128, 512, 2048).

---

bengali (no 0.021
Accuracy
0.0
aug.:
Accurscy
0.0
finnish (no aug.: 0.00)
Acciracy
0.1
georgian (no 0.09)
Accurasy
0.5
0.0
aug.
Accurscy
0.2
navajo (no aug. 0.02)
Accuracy
0.2
spanish 0.05)
Accurssy
0.5
0.0
128
256
512 Number of Augmenited Examples
1024
2048
LowLoss
HighLoss Uniform
UMT
ENT
Figure Performance training on Devain 5aym list the performance from training on augmented data in parentheses (De trasn
craiis varying of D,5mm In subtitle for each language, we also 0).


Reviews:
[{"review_id": "arkfsrNHp1", "reviewer": "Reviewer_xBw7", "summary": "", "questions": "Why didn't you pick a larger training data set scenario? \nWhy analysing sample efficiency in a setting with 100 training instances is interesting? As I see no practical interest in this, maybe a theoretical argument on why this matters would be in place.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We are pleased to hear that they found our work interesting and clearly written, and we agree with their assessment of the main contributions of the paper. The review also demonstrates that there are some important clarifications to make about the experimental sections of our paper. We hope that these clarifications will address any reservations the reviewer may have had about our work. Please see below for our responses to the critiques and questions:\n\n> Experimental part of the work involved training Transformer models on 100 samples large training data. The accuracy of the baseline is often 0.\n\nWe think there is some confusion here that is important to clarify. In our problem set up (Section 4; line 314), we introduce two training data subsets: one containing gold-standard examples ($D_{train}$) and another one containing synthetic examples generated by StemCorrupt ($\\hat{D}^{Syn}_{Train}$). In the experimental part of our work, we train on $\\hat{D}^{Syn}\\_{train}\\cup D\\_{train}$. While $\\lvert D\\_{train}\\rvert = 100$, we thoroughly evaluate performance on using $\\lvert{\\hat{D}}^{Syn}\\_{train}\\rvert\\in \\{128,256,512,1024,2048\\}$. Our full training sets ($D\\_{train} \\cup \\hat{D}^{Syn}\\_{Train}$) can thus in fact be as large as $|D\\_{train}| + |\\hat{D}^{Syn}\\_{Train}| = 100+2048=2148$. \n\nThere is only one experimental setting in which we use only 100 training examples, where $\\hat{D}^{Syn}\\_{Train}=\\emptyset$.  This setting is described in Section 4.1 (line 324 of the paper). This setting is not meant to serve as a competitive baseline model. Rather, it only serves to demonstrate that a Transformer trained with no data augmentation almost entirely fails to generalize. This has been shown in previous works (Liu & Hulden, 2022; Goldman et al., 2022). We don’t however necessarily expect readers to be familiar with the experimental results of those prior works so we replicate it in our study. It’s also useful for contextualizing the rest of the experimental results presented in Section 5.1. We can clarify this in the camera-ready. \n\nIt is entirely possible that we induced the confusion about the size of the training dataset in lines 331-334, where we write that “we conduct all of our experiments using 100 gold-standard examples, adhering to the low-resource setting for SIGMORPHON 2018”. We intended to convey that it is the gold-standard set $D\\_{train}$ — not $\\hat{D}^{Syn}\\_{Train}$ — that is fixed to 100 gold-standard training examples. We can certainly clarify this in the camera-ready. \n\n> Why didn't you pick a larger training data set scenario? \n\nWe think that the explanation above may in part alleviate the reviewer’s concern of using small training datasets, as the training datasets are larger than the reviewer may have thought. (Up to $2,148$ examples in our experiments, rather than $100$). Further, we’re interested in the sample-efficiency of the data augmentation method StemCorrupt, not the sample-efficiency of the limited gold-standard training data. \n\nThe reviewer may still ask why we only used only 100 gold-standard training examples. We want to emphasize that developing neural morphological inflection systems that use only 100 gold-standard training examples is actually entirely in adherence with precedence in experimental research on low-resource morphological inflection.  \n\nConsider the seminal paper that introduced the idea of StemCorrupt (Silfverberg et al., 2017), as well as the second paper that further enhanced the method in producing state-of-the-art morphological inflection results (Anastasopolous & Neubig, 2019). Both papers were motivated by the challenge of training models that can generalize (to a reasonable degree) on extremely scarce datasets, with up to 100 gold-standard training examples. Based on this precedent, we believe our fine-grained analysis of StemCorrupt in a data-constrained setting is appropriate, as this constrained setting is when data augmentation has proven to be an essential component of the training process. \n\nNote further that the research community interested in morphology (SIGMORPHON) has had a keen interest in developing systems that work for lower-resourced languages, some of which are undergoing concerted language revitalization efforts (see Wiemerslage et al., 2022, for a review and Moeller et al., 2020 for case studies). For such languages, having only about 100 cleanly morphologically annotated word types is not at all unusual. Thus, there is highly practical interest in understanding the behaviour of data augmentation strategies in extremely data-scarce settings, which is precisely what we set out to do in the experimental section of our work.  \n\n> Why analysing sample efficiency in a setting with 100 training instances is interesting? \n\n* It is important to reiterate that we’re interested in the sample efficiency of data augmentation rather than that of the gold-standard training data. The large synthetic dataset that we subsample from has $10,000$ datapoints, not $100$ (Section 5, line 422-427). \n\n* The motivation behind our experimental sections comes from the current standard of using data augmentation in morphological inflection, which is to use thousands of synthetic datapoints. Anastasopolous & Neubig (2019) used $10,000$ synthetic datapoints for obtaining their state of the art results in low-resource inflection, while the gold-standard training data only has $100$ datapoints . We think that the disparity between the size of the gold-standard training data and the synthetic data merited the type of investigation we conducted — do we really need synthetic data set that is 100x larger than the gold-standard training set? Could the models not attain the benefits of data augmentation using a synthetic dataset that scales linearly (100s of synthetic datapoints) or even logarithmically (dozens of synthetic datapoints) in the size of the gold-standard training data? \n\n* While it’s true that this is more of a theoretical concern, as training on even 10,000 word-level examples is not a particularly computationally intensive task for modern GPUs, there is intrinsic value in understanding how important data augmentation methods like StemCorrupt scale in relation to gold-standard data. \n\n> As I see no practical interest in this, maybe a theoretical argument on why this matters would be in place.\n\nIt’s also worth noting that our experimental investigation reveals highly practical insights. First, it is important to avoid instances where the model is highly confident about the correct prediction (i.e., LowLoss strategy in Section 4.2). Our experiments show that  selecting subsets of synthetic data that appears “easy” to the model drastically underperforms every other strategy (Figure 4 and paragraph starting line 515). \n\nThis finding may suggest that we should select synthetic examples where the model has high predictive uncertainty. Indeed, this is an approach that has been evaluated and championed in adjacent active-learning research (e.g., Tamkin et al., 2022). However, our analyses with the non-concatenative phenomenon of vowel harmony in Turkish and Finnish reveal that selecting synthetic datapoints by high predictive uncertainty is a poor strategy when the data augmentation method systematically differs from the ground-truth data generating probability distribution; see the paragraph starting on Line 537 for a deeper analysis. That is, when we pick synthetic examples by high uncertainty, we’re prone to picking examples that result in the model drawing systematically incorrect inferences.\n\nThis latter argument connects to our theoretical analysis in Section 3: we must understand the underlying probabilistic model that the data augmentation method encourages. This will in turn inform our practical data subset selection strategies. Overall, we believe these are practical insights that are relevant to NLP researchers interested in limited data generalization and data augmentation at large, not only in morphological inflection.   \n\nNote further that these results might not be apparent if we were to use a higher-resourced setting (with more ground-truth training examples), since in those settings data augmentation is no longer an essential component of the system. The variation in the selection strategies would likely then be rather small and would consequently require running the experiments with a substantially larger number of random seeds in order to have sufficient statistical power to detect any effect. \n\n> If UMT - Uniform Morphological Template, what does UME mean? Shouldn't it be EMT - Empirical Morphological Template? \n\nWe thank the reviewer for this helpful suggestion; Empirical Morphological Template (EMT) is indeed a more appropriate name for the strategy and we will update this in the paper. \n\n> Line 327 - Wu et al. (2021); Liu and Hulden (2022) -> Wu et al. (2021) and Liu and Hulden (2022) Line 306 - maybe it is worth saying \"non Independent and identically distributed (IID)\" to introduce the abbreviation.\n\nWe will make these updates. \n\n*References*\n\nMoeller, S., Liu, L., Yang, C., Kann, K., & Hulden, M. (2020). IGT2P: From Interlinear Glossed Texts to Paradigms. *Conference on Empirical Methods in Natural Language Processing*.\n\nSilfverberg, M., Wiemerslage, A., Liu, L., & Mao, L.J. (2017). Data Augmentation for Morphological Reinflection. *Conference on Computational Natural Language Learning*.\n\nAnastasopoulos, A., & Neubig, G. (2019). Pushing the Limits of Low-Resource Morphological Inflection. *Conference on Empirical Methods in Natural Language Processing*.\n\nA. Tamkin, D. Nguyen, S. Deshpande, J. Mu, & N. Goodman. (2022). In *Advances in Neural Information Processing Systems* , A. H. Oh, A. Agarwal, D. Belgrave, & K. Cho (Eds.).\n\nWiemerslage, A., Silfverberg, M., Yang, C., McCarthy, A.D., Nicolai, G., Colunga, E., & Kann, K. (2022). Morphological Processing of Low-Resource Languages: Where We Are and What’s Next. *Findings*.\n\nLiu, L., & Hulden, M. (2022). Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)* (pp. 739–749). Association for Computational Linguistics.\n\nGoldman, O., Guriel, D., & Tsarfaty, R. (2021). (Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models’ Performance. *Annual Meeting of the Association for Computational Linguistics*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Work provides a formal analysis of empirically used data augmentation method.\n- The work is interesting and clearly written.", "reasons_to_reject": "- Experimental part of the work involved training Transformer models on 100 samples large training data. The accuracy of the baseline models often is 0. While the merits of this work are theoretical, having such experiments makes me wonder if any of the shown results generalize beyond supper tiny data sets with no practical use. \n- Analyzing sample efficiency in a setting in which all training samples can be presented to the model in a single batch seems to have little use.", "questions_to_authors": "Why didn't you pick a larger training data set scenario? \nWhy analysing sample efficiency in a setting with 100 training instances is interesting? As I see no practical interest in this, maybe a theoretical argument on why this matters would be in place.", "paper_topic_and_main_contributions": "This paper analyzes how a synthetic data augmentation (STEMCORRUPT) method affects transformer-based morphological inflection models trained on golden 100 training instances + augmented data. This work formally proves the intuition that STEMCORRUPT is good for concatenative morphological processes yet has reduced performance for languages with non-concatenative morphological phenomena. The paper also explores how to improve the sample efficiency of STEMCORRUPT via improved synthetic data selection.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "If UMT - Uniform Morphological Template, what does UME mean? Shouldn't it be EMT - Empirical Morphological Template? \nLine 327 - Wu et al. (2021); Liu and Hulden (2022) -> Wu et al. (2021) and Liu and Hulden (2022)\nLine 306 - maybe it is worth saying \"non Independent and identically distributed (IID)\" to introduce the abbreviation."}}, {"review_id": "55d31CW5NV", "reviewer": "Reviewer_AGZF", "summary": "", "questions": "- QA: In the HighLoss (and LowLoss) sampling, it is not clear (at least to me) whether the uncertainty is counted for the whole entry (X,Y,T) or just Y.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. We’re glad that they found it well written and that they consider it to push the boundaries of morphological inflection research, especially in low-resource inflection. We also agree with their identification of the main contributions of the work. We hope that the following responses thoroughly address their reservations about the work: \n\n> lack of MSD splits: A key evaluation for compositionality would be a split test set based on the feature tag or the Morphosyntactic descriptions (as the authors referred to it). This evaluation will highlight additional strengths or weaknesses of using synthetic data.\n\nWe thank the reviewer for bringing the paper by Kodner et al. (2023) to our attention. It is clearly related to the compositional generalization of morphological inflection systems; Kodner et al. demonstrate that generalizing to novel feature combinations (”featsNovel”) is an aspect of compositionality that current models struggle with. Given that it is a paper that studies compositionality in morphological inflection, it is clearly very related and we will cite it. \n\nHowever, we do not believe that the lack of MSD splits in our experiments is grounds for rejection, for two reasons. First, please note that this paper was very recent at the time of submission. It was posted on arxiv on **May 25th, 2023**, which was less than a month from the submission deadline and after the anonymity deadline. \n\nSecond, we don’t believe experiments on the MSD split are directly relevant to the questions we ask and the conclusions we draw in our work. There are several aspects of compositionality in morphological inflection, as clearly delineated by Kodner et al. (2023). We target one important aspect. Namely, that which was highlighted by Goldman et al. (2022) and Liu & Hulden (2022), and that Kodner et al. (2023) call  the “lemmaNovel” setting, which showed that systems struggle to generalize when asked to inflect novel lexemes. We theoretically analyzed the relationship between StemCorrupt and the “lemmaNovel” setting because of experimental results reported by Liu & Hulden (2022), which empirically demonstrated that StemCorrupt is effective for generalization in the “lemmaNovel” setting. \n\nUnlike the empirically attested relationship between StemCorrupt and the “lemmaNovel” setting, there isn’t a priori justification to believe that the synthetic data generated by StemCorrupt would be an effective technique for improving generalization in the “featsNovel” setting. The StemCorrupt algorithm does not perturb, segment, or re-combine affixes in any way that would be beneficial for improving generalization in “featsNovel”.  Since our paper is centered on the StemCorrupt method and the “lemmaNovel” setting, we don’t believe it is appropriate to include an experiment on the  “featsNovel” test split without a priori justification to expect it to garner improvements — or have any effect at all. \n\nWe will clarify in the paper that we address the compositional generalization issue that is highlighted by Goldman et al. (2022) and Liu & Hulden (2022), but there are outstanding, important issues in morphological inflection compositionality — namely, generalization in the “featsNovel” setting highlighted by Kodner et al. (2023) — that heretofore remain unaddressed and merit further study. \n\n> • highly concatenative morphology only: This was briefly addressed in the paper. Given that one of the test languages, i.e., Arabic, employs templatic morphology, it would be important to investigate the behavior of STEMCORRUPT when generating synthetic data from entries where the inflection is a change of the stem template.\n\n**Arabic's templatic morphology**\n\nThe SIGMORPHON data collection that we draw from does indeed include Arabic, a prototypical templatic morphology-exhibiting language. Prompted by this review, we further investigated the behaviour of StemCorrupt in Arabic. We find that the StemCorrupt algorithm is only partially effective for Arabic, due to its non-concatenativity. At an aggregate level, StemCorrupt brings improvements because in most cases in the Arabic SIGMORPHON 2018 dataset, the inflection comprises a concatenative transformation (i.e., add a prefix and/or suffix; e.g., مَطَارَات → مَطَارٌ). \n\nInflections requiring stem-internal changes however (e.g., كِتَابَة → أَكْتُبُ) are not benefited from StemCorrupt.  The stem discovery algorithm used in StemCorrupt does not isolate Arabic consonant roots for perturbation (like ktb), but rather entire stems (containing a mix of consonants and vowels, like كِتَاب). Since vowels in Arabic serve as exponents of morphological information, StemCorrupt thus has the undesirable effect of obfuscating the statistical dependency between non-root segments in the stem with the morphosyntactic description (MSD). \n\nWe will add the discussion of how StemCorrupt behaves with regards to Arabic stem-changing inflections to the paper. Moreover, we will add quantitative analysis concerning the combined effect of the sampling schemas described in Section 4.2 and the inability of StemCorrupt to correctly identify consonantal roots in Arabic. This analysis will follow the structure of the vowel-harmony analysis we conducted for Turkish.\n\n**Vowel harmony analyses**\n\nIt is also important to note that the paper even in its current form **does** in fact go beyond analyzing simple concatenative morphology. While we didn’t analyze the Arabic predictions in-depth specifically, we do nonetheless perform an in-depth case study on a non-concatenative morphophonological phenomenon.\n\nSpecifically, we investigate vowel harmony in Turkish, where the vowels in the suffix must adhere to the frontness of the initial vowel in the stem (lines 229 - 238; 537-567). We investigated this phenomenon carefully as it is omnipresent in the Turkish dataset and since vowel harmony is a well understood example of non-concatenative morphology (see, for example, Amrhein & Sennrich, 2021, who also consider vowel harmony as a non-concatenative phenomenon). \n\nOur investigation revealed that StemCorrupt regularly generates synthetic data that violates this vowel harmony restriction and we find evidence that it tempers the overall effectiveness of StemCorrupt, in concordance with our theoretical analysis.  \n\n> QA: In the HighLoss (and LowLoss) sampling, it is not clear (at least to me) whether the uncertainty is counted for the whole entry (X,Y,T) or just Y.\n\nWe use the conditional predictive uncertainty, i.e., just for Y. This is consistent with other recent works that investigate data selection in NLP, e.g., Tamkin et al. (2022) and Muradoglu & Hulden (2022). We will clarify this in the paper. \n\n*References*\n\nAmrhein, C., & Sennrich, R. (2021). How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology? *Conference on Empirical Methods in Natural Language Processing*.\n\nMuradoglu, S., & Hulden, M. (2022). Eeny, meeny, miny, moe. How to choose data for morphological inflection. *Conference on Empirical Methods in Natural Language Processing*.\n\nGoldman, O., Guriel, D., & Tsarfaty, R. (2021). (Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models’ Performance. *Annual Meeting of the Association for Computational Linguistics*.\n\nA. Tamkin, D. Nguyen, S. Deshpande, J. Mu, & N. Goodman. (2022). In *Advances in Neural Information Processing Systems* , A. H. Oh, A. Agarwal, D. Belgrave, & K. Cho (Eds.).\n\nLiu, L., & Hulden, M. (2022). Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)* (pp. 739–749). Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well-written and easy to follow for the most part.\n- **low-resource aid**: This work supports a technique useful for low-resource languages and varieties, especially for the morphological inflection task.\n- **pushing boundaries**: The techniques described in this work will surely encourage more researchers to revisit the task of morphological (re)inflection from a different perspective, whether training or evaluating existing models.", "reasons_to_reject": "- **lack of MSD splits**: A key evaluation for compositionality would be a split test set based on the feature tag or the Morphosyntactic descriptions (as the authors referred to it). This evaluation will highlight additional strengths or weaknesses of using synthetic data.\n- **highly concatenative morphology only**: This was briefly addressed in the paper. Given that one of the test languages, i.e., Arabic, employs templatic morphology, it would be important to investigate the behavior of _STEMCORRUPT_ when generating synthetic data from entries where the inflection is a change of the stem template. ", "questions_to_authors": "- QA: In the HighLoss (and LowLoss) sampling, it is not clear (at least to me) whether the uncertainty is counted for the whole entry (X,Y,T) or just Y.\n", "paper_topic_and_main_contributions": "In this paper, the authors investigate the efficacy of artificially augmenting training data for morphological inflection. Theoretically and empirically, the authors show that using synthetic data in training improves overall performance. They used an information-theoric framework with simplifying assumptions to show that adding synthetic data is useful. Empirically, they used multiple setups to sample synthetic examples from a synthetically generated dataset and evaluated the different training scenarios. The synthetic data is generated from a small sample of gold data from UniMorph. The experiments were conducted on 7 typologically different languages. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- As mentioned in the **Reseson To Reject** section, it has been shown that MSD based splits reveal more about the performance and ability to generalize to unseen data of morphological inflection systems, as in this work: https://aclanthology.org/2023.acl-long.335.pdf\n"}}, {"review_id": "nrZcSQDUO0", "reviewer": "Reviewer_CMbW", "summary": "", "questions": "1. One thing is unclear -- in Section 4.2 the authors introduce techniques to sample data, but I didn't quite understand how that compares with what existing approach STEMCORRECT does? Are these novel techniques or were these also used by STEMCORRECT?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree with the reviewer’s assessment of the main contributions of the paper. We are pleased that the reviewer thinks that the theoretical analysis is concordant with the results of the empirical analyses; that the reviewer thinks the paper is well written and easy to understand; and that they find the analysis of an existing, effective strategy novel and valuable.\n\n> 1. One thing is unclear -- in Section 4.2 the authors introduce techniques to sample data, but I didn't quite understand how that compares with what existing approach STEMCORRECT does? Are these novel techniques or were these also used by STEMCORRECT?\n\nEmploying sampling techniques in conjunction with StemCorrupt is a novel contribution of our work. We will clarify this in the camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a nice study into understanding the effectiveness of an existing strategy. I have not many papers focussed on that. Its good to see the effort. \n\n2. The paper is very clearly written and easy to understand.\n\n3. The authors perform extensive experimentation across 7 languages and especially report results averaged across 6 random runs. The results are well in line with what the authors prove in Section 3 that under infinite data the distribution of stems and affix are independent of each other.", "reasons_to_reject": "I don't have any major reasons, apart from the question below.", "questions_to_authors": "1. One thing is unclear -- in Section 4.2 the authors introduce techniques to sample data, but I didn't quite understand how that compares with what existing approach STEMCORRECT does? Are these novel techniques or were these also used by STEMCORRECT?", "paper_topic_and_main_contributions": "The paper proposed to better understand why the popular data-augmentation strategy employed for learning morphology inflection, is particularly effective. Specifically, they look at the STEMCORRUPT approach where affixes are automatically identified and attached to synthetic lexemes (unseen in the training data).\nBy using information-theoretic analysis they find that this data augmentation esp. helps removing spurious correlations between the stem and its affixes.\nWithin data augmentation, they introduce new sampling strategies based on uncertainty, diversity, to check which strategy helps in selecting the best subset that leads to test performance improvement.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyses a data augmentation technique, STEMCORRUPT. The method is used to produce synthetic examples for the morphological inflection task by substituting stem characters. The authors demonstrate the utility of the technique, both theoretically and empirically. The paper also explores data-efficient strategies to sample from the synthetically generated dataset. As most reviewers outline, these research directions are particularly important for low-resource language technology. The reviewers also noted that experiments are conducted on 7 typologically diverse languages, giving this work a stronger basis. Some reviewers pointed out the flaws such as small training data and the focus on concatenative morphology only but the authors addressed and clarified them very neatly.
Overall, I think the work will be a nice contribution to NLP, and low-resource technologies in particular.
Decision: Accept-Main