Improving Low-Resource Morphological Inflection via Self-Supervised Objectives
Authors: Adam Wiemerslage, Katharina von der Wense
Accepted at: acl25

Improving Low-Resource Morphological Inflection via Self-Supervised Objectives

Adam Wiemerslage1,3 and Katharina von der Wense1,2
1University of Colorado Boulder
2Johannes Gutenberg University Mainz
3Kensho Technologies
{adwi9965,katharina.kann}@colorado.edu

Abstract

Self-supervised objectives have driven major advances in NLP by leveraging large-scale unlabeled data, but such resources are scarce for many of the world’s languages. Surprisingly, they have not been explored much for character-level tasks, where smaller amounts of data have the potential to be beneficial. We investigate the effectiveness of self-supervised auxiliary tasks for morphological inflection – a character-level task highly relevant for language documentation – in extremely low-resource settings, training encoder-decoder transformers for 19 languages and 13 auxiliary objectives. Autoencoding yields the best performance when unlabeled data is very limited, while character masked language modeling (CMLM) becomes more effective as data availability increases. Though objectives with stronger inductive biases influence model predictions intuitively, they rarely outperform standard CMLM. However, sampling masks based on known morpheme boundaries consistently improves performance, highlighting a promising direction for low-resource morphological modeling.

1 Introduction

Rapid progress in natural language processing (NLP) has largely been driven by training transformer models on massive amounts of unlabeled data. However, for many of the world’s languages, such large datasets do not exist. Still there is great potential for NLP to contribute to areas like language documentation or indigenous language educational technology, where both labeled and unlabeled data is extremely sparse, e.g., via character-level tasks like text normalization, grapheme-to-phoneme conversion, or morphological inflection.

Overcoming low-resource data challenges typically involves biasing an NLP model towards a particular language, task or domain. For instance, in the morphology community, prior work has focused on implementing neural architectures with an inherent inductive bias towards particular tasks like inflection (Aharoni and Goldberg, 2017; Makarov and Clematide, 2018; Wu and Cotterell, 2019).

Instead, we take inspiration from high-resource NLP where progress has been largely data-driven, focused on training NLP models with self-supervised objectives like language modeling. We explore self-supervised objectives, based on masked language modeling (Devlin et al., 2019, MLM), that have more inductive bias towards a particular character-level task for extremely small datasets in a multitask training setup. The goal is to change the MLM objective in order to optimally leverage unlabeled data to bias the model towards a particular task. This is more interpretable and simpler to implement than adjusting the inductive bias of an architecture.

Prior work has shown that auxiliary training on unlabeled data benefits character-level tasks, but has only investigated a small subset of standard objectives. We hypothesize that this is because tasks like MLM at a typical scale are often associated with high computational cost. We highlight that for many low-resource character-level tasks efficiency is not actually a challenge. Those tasks allow us to quickly and cheaply explore variations of auxiliary tasks. We propose several novel self-supervised

![Example of the multitask training; left: inflection with the past tense tag; right: self-supervised masked language modeling. @ represents the mask token; [TASK] is a special tag for the self-supervised task.](page_1012_420_388_246.png)

Figure 1: Example of the multitask training; left: inflection with the past tense tag; right: self-supervised masked language modeling. @ represents the mask token; [TASK] is a special tag for the self-supervised task.
task objectives with two goals: (1) to improve downstream task performance, and (2) gain a better understanding of how and why self-supervised auxiliary tasks work. We focus on morphological inflection, which has been featured in recent work on auxiliary task training (Kann and Schütze, 2017; Purushothama et al., 2024). Furthermore, it is a character-level task with small models that maintains relevance even in the age of LLMs, due to its potential to aid language documentation efforts (Moeller et al., 2020; Moeller and Hulden, 2018).

We conduct experiments with an encoder-decoder transformer (Vaswani et al., 2017) featuring 19 languages, 13 different auxiliary objectives, and 6 different unlabeled datasets. On a popular inflection benchmark, we find that simply autoencoding unlabeled words, with no masking leads to the best performance when the unlabeled dataset is very small, but MLM performs best as we attain more unlabeled data. While we find evidence that MLM objectives with more inductive bias influence model predictions in intuitive ways, they typically do not outperform the typical MLM setup on average. However, we find that sampling masks according to known morpheme boundaries tends to produce the best performance, indicating a promising direction.

2 Related Work

Inductive Bias of Auxiliary Tasks Following Devlin et al. (2019), several variations of pretraining tasks have been proposed. Joshi et al. (2020) propose masking spans of text in order to train models with an inductive bias towards span selection. Raffel et al. (2020) explore pretraining objectives in an encoder-decoder setup and find that a span-denoising objective performs best. Lewis et al. (2020) propose additionally shuffling the order of sentences, leading to models particularly strong at machine translation. More recent work typically assumes a model has been pretrained with autoregressive language modeling, but explore further training like instruction tuning (Mishra et al., 2022; Wei et al., 2021), or reinforcement learning (Christiano et al., 2017; Ouyang et al., 2022) to align large language models to human preferences.

Auxiliary Tasks for Character-Level Tasks Some work has focused specifically on character-level models in the low-resource setting like we do. Xue et al. (2022) pretrain an encoder-decoder following the objective in Raffel et al. (2020), but at the byte rather than subword level. They perform well on many character-level tasks including morphological inflection. Closely related to our work, Kann and Schütze (2017) train inflection models in a multitask training setup with autoencoding as an additional objective. Purushothama et al. (2024) also explore multitask training for inflection, comparing autoencoding to denoising, and find large performance gains even when unlabeled data is sampled exclusively from the supervised training set. This implies that models benefit from the task itself, rather than simply the additional data distribution. Our work builds closely upon this idea. Whereas (Purushothama et al., 2024) shows that the self-supervised objective is impactful beyond the incorporation of new data, we explore which self-supervised objective—among several heuristics—is best on various datasets comprising different properties. Other work has specifically investigated the inductive bias of autoencoding for inflection as beneficial for copy bias (Liu and Hulden, 2022; Yang et al., 2022).

3 Morphological Inflection

Morphological inflection is a character-level sequence-to-sequence task. Such tasks are characterized by small vocabularies \( \Sigma \) as well as short source and target strings. Given a source lemma \( \ell \in \Sigma^* \), target word \( y \in \Sigma^* \), and a bundle of morphological feature tags \( t \in \tau \), the goal of morphological inflection is to learn a mapping

\[
f(\ell, t) \to y,
\]

for instance, an inflection example for generating the past tense of the verb try would look like:

\[
f(\text{try}, \text{PST}) \to \text{tried}.
\]

4 Self-Supervised Training Heuristics

We compare several heuristics for self-supervised training, which are optimized in a multitask learning setup along with inflection. Each heuristic is implemented as an MLM method, which we vary in order to change the inductive bias of the learning problem. We assume a small lexicon \( \mathcal{L} \) of words, and an even smaller supervised inflection dataset \( \mathcal{D} \) to model with an encoder-decoder.

Baselines We compare all heuristics to two baselines. First, we train exclusively on \( \mathcal{D} \), ignoring \( \mathcal{L} \) entirely. We refer to this model as baseline.
We also implement an auxiliary baseline, which in addition to the supervised task, trains on \( \mathcal{L} \) with autoencoding (AE). A self-supervised learning sample with autoencoding would be presented to the model, for instance, as:

\[
tried + [TASK] \rightarrow tried,
\]

where [TASK] is a special symbol for the self-supervised task that takes the role of the inflection tags in the supervised task. Each heuristic noising method combines an objective with a sampling strategy, and either masks or deletes characters.

Objective  We compare two objectives for MLM. The first, CMLM (Wiemerslage et al., 2023), follows the MLM method from Liu et al. (2019)—which samples 15% of tokens dynamically and 80% of the time replaces them with a special (MASK) symbol, 10% replaces them with another token from the vocabulary, or 10% leaves the token as is—except it is performed at the character level and the mask sampling rate is increased to 25%. For instance, representing the mask token with @:

\[
tr@ed + [TASK] \rightarrow tried
\]

We additionally implement the span based variant for the T5 objective from Raffel et al. (2020). This differs from CMLM in that the sampled token is always replaced with a mask, adjacent masks are merged together into a single mask, and each mask token is unique. For instance

\[
t<X>e<Y> + [TASK] \rightarrow tried,
\]

where \( <X> \) and \( <Y> \) are mask tokens. Notice that this differs from the details of Raffel et al. (2020), who additionally mask the output sequence, keeping only the dropped out tokens intact:

\[
t<X>e<Y> + [TASK] \rightarrow <X>ri<Y>d.
\]

In initial experiments, we found that this method in Example 4 severely underperformed masking only the source side as in Example 3, so we exclude it from the experiments presented in this work.

Strategy  We compare three masking strategies: iid, suffix, and prefix. For each strategy, we sample 25% of character positions in a word. Each strategy varies in the distribution over characters from which we sample characters to be masked. iid sampling is implemented with a uniform distribution over all characters. The suffix strategy explores the hypothesis that masking with a bias towards the end of a string should create training samples that resemble concatenative, suffixing inflection, which is typologically pervasive. Here, we sample from a distribution that is skewed toward the end of the sequence. We distribute 95% of the probability mass uniformly over the final 1/3 of characters, and the other 5% uniformly over the first 2/3 of characters. Finally, for the prefix strategy, which is typologically less common for inflection and not well-represented in our dataset, we do the same thing as the suffix strategy, but skew the distribution to the start of the sequence.

Character Deletion  We additionally explore deleting rather than masking characters. Intuitively, given that inflection is a sequence-to-sequence task where in typical concatenative inflection some substring is appended to the input sequence, deleting characters of a word and then predicting them simulates the target task more closely. For instance, if we sample the word ‘baked’, and apply the suffix strategy, we might produce the training sample

\[
bake \rightarrow baked.
\]

which exactly resembles a supervised sample for producing the past tense without the inflection tag.

<table>
  <tr>
    <th>Dataset</th>
    <th>Samples</th>
    <th>Types</th>
    <th>Med Len</th>
    <th>N-grams</th>
  </tr>
  <tr>
    <td>ud-1k</td>
    <td>5836.26</td>
    <td>3260.68</td>
    <td>12.66</td>
    <td>5725.79</td>
  </tr>
  <tr>
    <td>ud-200</td>
    <td>5278.37</td>
    <td>2659.11</td>
    <td>11.74</td>
    <td>4839.58</td>
  </tr>
  <tr>
    <td>ud-vnadj</td>
    <td>5073.00</td>
    <td>3297.58</td>
    <td>11.63</td>
    <td>4744.63</td>
  </tr>
  <tr>
    <td>ud-wl</td>
    <td>5263.68</td>
    <td>3155.84</td>
    <td>12.79</td>
    <td>5479.89</td>
  </tr>
  <tr>
    <td>ud-vnadj-NR</td>
    <td>4976.74</td>
    <td>4902.68</td>
    <td>12.24</td>
    <td>6273.21</td>
  </tr>
  <tr>
    <td>ud-wl-NR</td>
    <td>5132.74</td>
    <td>5059.74</td>
    <td>12.92</td>
    <td>7562.37</td>
  </tr>
</table>

Table 1: Statistics for all datasets, averaged over all languages. This includes the supervised and unlabeled data. Types is the number of unique words, Med Len is the median number of characters in each word, and N-grams is the count of all unique char. bi and trigrams.

<table>
  <tr>
    <th>Dataset</th>
    <th>Samples</th>
    <th>Types</th>
    <th>Med Len</th>
    <th>N-grams</th>
  </tr>
  <tr>
    <td>ud-1k</td>
    <td>6000.0</td>
    <td>3139.2</td>
    <td>11.7</td>
    <td>4779.8</td>
  </tr>
  <tr>
    <td>ud-200</td>
    <td>5360.0</td>
    <td>2475.4</td>
    <td>10.5</td>
    <td>4136.0</td>
  </tr>
  <tr>
    <td>ud-vnadj</td>
    <td>5360.0</td>
    <td>3264.6</td>
    <td>11.8</td>
    <td>3861.6</td>
  </tr>
  <tr>
    <td>ud-wl</td>
    <td>5360.0</td>
    <td>3013.4</td>
    <td>11.5</td>
    <td>4494.6</td>
  </tr>
  <tr>
    <td>ud-wl-NR</td>
    <td>5360.0</td>
    <td>5320.4</td>
    <td>13.2</td>
    <td>6829.6</td>
  </tr>
  <tr>
    <td>ud-vnadj-NR</td>
    <td>5360.0</td>
    <td>5320.0</td>
    <td>13.1</td>
    <td>5467.0</td>
  </tr>
  <tr>
    <td>seg</td>
    <td>5360.0</td>
    <td>5320.4</td>
    <td>13.3</td>
    <td>5645.0</td>
  </tr>
</table>

Table 2: Statistics for all datasets, averaged over the 5 languages with morphological segmentation. This includes the supervised and external data.
Note that, with deletion, T5 and CMLM result in almost the same denoising method, since merging masks into spans is irrelevant. The only difference is that CMLM sometimes masks with another character in the vocabulary (10%) or applies the identity function to a character sampled for masking (10%).

Segment Masking  We further explore if masking subword segments according to oracle morpheme boundaries results in an objective more suitable for morphological inflection. For a subset of languages, we attain gold morpheme boundaries for the unlabeled words, and query this as an oracle. For this dataset, we produce self-supervised data that samples segments, rather than characters, for masking. All details are otherwise identical to character-level masking. For instance, given the word ‘walk - ing’, the CMLM objective and suffix strategy, we will produce with 95% probability

\[
\text{walk@@@} \rightarrow \text{walking},
\]

where the segment -ing is masked.

5 Experiments

5.1 Data
We follow Purushothama et al. (2024) in evaluating on SIGMORPHON 2023 shared task data (Goldman et al., 2023) with a subset of training data, and unlabeled data sampled from universal dependencies (UD) treebanks (Zeman et al., 2023). First, we produce results on the exact dataset from Purushothama et al. (2024), comprising 19 languages, 1k supervised samples, and 5k unlabeled words, which we refer to as ud-1k. The 19 languages are the subset of SIGMORPHON 2023 data for which UD corpora are also available. In all other experiments, we simulate a lower-resource setting than that work. For each language we sample 200 supervised samples from the SIGMORPHON 2023 training set for each part of speech. This means that each dataset contains 200 – 600 supervised samples depending on which parts of speech are present in the shared task data (nouns, adjectives, or verbs). Sampling is done one paradigm at a time, ensuring a broad variety of inflection tags are attested. This results in supervised datasets similar in size to the low-resource setting studied in the SIGMORPHON 2022 shared task on inflection (Kodner et al., 2022), which sampled 700 inflection pairs. The upper bound on our supervised datasets is 600 inflection pairs if all three of verbs, nouns, and adjectives are available. We augment these supervised samples with 5k unlabeled samples from UD. The unlabeled data is either copied from Purushothama et al. (2024) (ud-200), or resampled with additional constraints. In one dataset we constrain word-length: ensuring that all unlabeled words contain at least three characters (two for Japanese) to increase the chance that the dataset contains morphologically rich words (ud-wl). We build another dataset that samples from words that are additionally tagged with a verb, noun, or adjective part of speech in the UD treebank, to further increase the likelihood of morphological productivity (ud-vnadj).

All unlabeled data so far has been sampled with replacement. Given the size of the UD corpora, this leads to several languages with many duplicate words in the 5k unlabeled samples (See Table 1). We thus also sample datasets without replacement (ud-wl-NR, ud-vnadj-NR) to increase the diversity of the dataset, leading to 5k unique words in most cases. This means that the *-NR datasets effectively increase the size of the unlabeled dataset. We sample from additional UD corpora where necessary in order to ensure that we can sample 5k unique words. This was not possible for Amharic (amh) nor Sanskrit (san), so those datasets continue to contain fewer than 5k words. Finally, we build a dataset where all unlabeled words are segmented into morphemes (seg). For this, we use the data from the SIGMORPHON 2022 morpheme segmentation shared task (Batsuren et al., 2022). This dataset contains canonical segmentation, but we require a surface segmentation. We produce surface segmentation boundaries with a heuristic algorithm presented in Appendix B. Because the overlap in languages between the two shared tasks is small, this dataset contains only 5 languages: English, Hungarian, Italian, Russian, and Spanish. This dataset is very morphologically rich, because the shared task sampled words contain many morphemes. Statistics in Table 2 show a high median word length, and a large number of unique character n-grams. The notably high number of n-grams in the ud-wl-NR dataset is likely due to noisy web-scraped data—which is pervasive in the English data—that results in rare characters and character n-grams. We expect the seg dataset to represent many inflection variations, but also many non-inflectional processes, like derivation, or, e.g., Spanish pronominal participles.
Figure 2: Results for masking v.s. deleting for all models averaged over all datasets. On the right are the segmentation variants of all models, for which there is only one dataset comprising only 5 languages.

<table>
  <tr>
    <th></th>
    <th>baseline</th>
    <th>t5-pref</th>
    <th>cmlm-pref</th>
    <th>t5-suff</th>
    <th>cmlm-suff</th>
    <th>t5-iid</th>
    <th>cmlm-iid</th>
    <th>AE</th>
  </tr>
  <tr>
    <td>ud-1k</td>
    <td>64.39*</td>
    <td>73.4</td>
    <td>69.91</td>
    <td>73.43</td>
    <td>74.07</td>
    <td>74.39</td>
    <td>74.67</td>
    <td><b>75.83</b></td>
  </tr>
  <tr>
    <td>ud-200</td>
    <td>5.16</td>
    <td>38.31</td>
    <td>41.82</td>
    <td>41.34</td>
    <td>42.76</td>
    <td>41.04</td>
    <td>42.92</td>
    <td><b>47.48</b></td>
  </tr>
  <tr>
    <td>ud-wl</td>
    <td>5.16</td>
    <td>38.91</td>
    <td>43.16</td>
    <td>43.07</td>
    <td>44.98</td>
    <td>43.05</td>
    <td>43.98</td>
    <td><b>49.55</b></td>
  </tr>
  <tr>
    <td>ud-vnadj</td>
    <td>5.16</td>
    <td>39.51</td>
    <td>41.46</td>
    <td>41.86</td>
    <td>43.39</td>
    <td>42.86</td>
    <td>42.91</td>
    <td><b>47.8</b></td>
  </tr>
  <tr>
    <td>ud-wl-NR</td>
    <td>5.16</td>
    <td>48.04</td>
    <td>48.76</td>
    <td>49.19</td>
    <td>50.51</td>
    <td>51.26</td>
    <td><b>51.68</b></td>
    <td>50.49</td>
  </tr>
  <tr>
    <td>ud-vnadj-NR</td>
    <td>5.16</td>
    <td>47.67</td>
    <td>47.88</td>
    <td>49.2</td>
    <td>49.42</td>
    <td><b>51.48</b></td>
    <td>50.46</td>
    <td>50.06</td>
  </tr>
</table>

Table 3: Accuracy for all main experiments averaged over all languages. Note that baseline results are the same because each dataset shares the exact same supervised training samples. The best performing self-supervised setup per dataset is bolded, and the 2nd best is underlined. *Taken from published results in Purushothama et al. (2024)

5.2 Model
For all experiments, we train an encoder-decoder transformer (Vaswani et al., 2017) following Wu et al. (2021). We use a multitask training setup, where both the supervised and self-supervised task can occur in the same batch, and their losses are summed. Inflection tags are included as additional tokens in the input sequence. For the self-supervised task, these are replaced with a special task tag. Exact hyperparameters are available in Appendix C. Each model is 7.4 million parameters, and trains in about 1 hour on an NVIDIA A100.

6 Results
We first investigate the impact of masking vs. deleting characters for the denoising objectives. In Figure 2 masking outperforms deleting for all training setups on average over all datasets. However, when the training heuristic has knowledge of morpheme boundaries, this is not strictly true. In this case, for both t5-suffix and t5-iid deletion is stronger. In all subsequent experiments we focus on masking, which is typically the superior strategy.

Next we present the main results in Table 3 averaged over all languages. For per-language results see the Appendix. We first reproduce the results from Purushothama et al. (2024, ud-1k) showing that autoencoding outperforms all denoising methods. The same denoising method evaluated in their work, cmlm-i id, is the best performer after autoencoding. Then we evaluate in our setting, which constrains the supervised data to a smaller training set, while maintaining the same UD dataset for self-supervision. Here, we attain the same relative result but the gap in performance between autoencoding and the best denoising setup is much larger (4.56 absolute accuracy), due to the smaller supervised dataset. When we sample the unlabeled UD data with a minimum word-length constraint (ud-wl), we see an increase in accuracy across the board, though autoencoding still performs best by a large margin. Here, however, the suffix strategy is the best denoising method. Additionally constraining data to only verbs, adjectives and nouns (ud-vnadj) actually reduces performance when compared to the ud-wl dataset. This is likely because it reduces
the diversity of the dataset due to those constraints (See: Table 1, N-grams).

Sampling Words without Replacement  Table 1 shows that all datasets so far have several fewer unique types than the total number of samples. This is because we followed Purushothama et al. (2024) by sampling words from UD with replacement, causing many duplicate words. When we resample the datasets without replacement, effectively increasing the size of the unlabeled datasets to be truly 5000 samples, there is an increase in accuracy across the board, with a substantial increase for all denoising strategies (up to 9.13 absolute accuracy). For these datasets, denoising clearly outperforms autoencoding.

Objective  Figure 3a presents a comparison of the two self-supervised objectives averaged over all languages and datasets on the left. For all strategies, CMLM outperforms T5 on average. Despite this, when combined with the best strategy (iid), the difference of these averages is rather small.

Strategy  Figure 3b presents a comparison of the three self-supervised strategies averaged over all languages and datasets. The iid strategy performs best on average for both objectives, followed closely by the suffix strategy.

Segmentation  Finally, we present results for the five segmentation languages in Table 4. When using the CMLM objective, the segmentation variant—which samples entire morphemes for masking rather than individual characters—always outperforms the corresponding setup without segmentation. However, the opposite is true for T5 – in these cases the segmentation variant under-performs. Figure 2 also shows that deleting tends to outperform masking for the T5 objective at the segment level. The best performing system in this experiment on average is cm1m-seg-iid. However, the same setup without segmentation performs only 0.58 absolute accuracy lower on average.

6.1 Language-Specific Results

We perform a small language-specific analysis focused on the self-supervised strategy, which we expect to be most relevant to language typology. We focus on the ud-vnadj-NR dataset, which we had expected to contain the most morphologically rich data of our full 19-language datasets. In Figure 4 we present the best performing variation for each strategy on this dataset. For almost all languages, the iid masking strategy performs best. There are 4 languages for which AE outperforms every denoising strategy. In three of these four languages either we sampled far fewer than 5k self-supervised words (amh, san), or the character vocabulary is extremely large (jap, amh). Both of these scenarios support the hypothesis that autoencoding is a strong objective under extreme data sparsity. It is unclear which factor explains the autoencoding performance in Armenian (hye). Despite the fact that our dataset contains no languages with typical prefix inflection, the prefix strategy also performs best in a few cases. Upon further analysis, we noticed that in the development sets, particles like pronouns are sometimes inserted before an inflected form as in the Italian example:

\[
sdebitarsi \rightarrow ti sdebitavi.
\]

When we quantify this phenomenon, we find that in 1000 development set examples, it happens most frequently in fin (284) and ita (141), followed by hye (99) and deu (71). The German (deu) instances are caused by separable verbs, and Armenian (hye) additionally exhibits this phenomenon pervasively at the end of the string. We interpret this as evidence that skewing the mask distribution does teach the model an inductive bias towards certain typological phenomenon. The fact that many languages that exhibit a strong bias towards suffixes perform best with the iid strategy, however, shows that masking uniformly is typically stronger overall.

6.2 Discussion

The main results presented in this work show a clear trend between autoencoding—which performs strongly when there is little external unlabeled data—and denoising, which performs strongly as we train on a larger set of unlabeled words. This is apparent in the trend in the number of unique types per dataset in Table 1 vis-à-vis the accuracy in Table 3. We hypothesize that this is because autoencoding learns a strong inductive bias towards copying – a common operation in morphological inflection. On the other hand, denoising encourages models to generate particular character sequences that were masked in the input. It is thus prone to overfitting to such sequences in cases of extreme data sparsity. Denoising also encourages the model to learn to generate character sequences that are potentially morphologically rich
Figure 3: Average accuracy over all ud datasets when masking.

<table>
  <tr>
    <th></th>
    <th>eng</th>
    <th>hun</th>
    <th>ita</th>
    <th>rus</th>
    <th>spa</th>
    <th>mean</th>
  </tr>
  <tr>
    <td>AE</td>
    <td>90.70</td>
    <td>52.00</td>
    <td>53.70</td>
    <td>68.40</td>
    <td>55.20</td>
    <td>64.00</td>
  </tr>
  <tr>
    <td>t5-pref</td>
    <td>88.70</td>
    <td>58.60</td>
    <td>46.90</td>
    <td>69.80</td>
    <td>63.30</td>
    <td>65.46</td>
  </tr>
  <tr>
    <td>t5-suff</td>
    <td>89.40</td>
    <td>66.60</td>
    <td>50.20</td>
    <td>67.70</td>
    <td>57.90</td>
    <td>66.36</td>
  </tr>
  <tr>
    <td>t5-iid</td>
    <td>90.10</td>
    <td><b>68.80</b></td>
    <td>47.90</td>
    <td>70.40</td>
    <td><b>68.00</b></td>
    <td>69.04</td>
  </tr>
  <tr>
    <td>cmlm-pref</td>
    <td>88.70</td>
    <td>56.90</td>
    <td>42.10</td>
    <td>69.50</td>
    <td>59.90</td>
    <td>63.42</td>
  </tr>
  <tr>
    <td>cmlm-suff</td>
    <td>90.50</td>
    <td>66.70</td>
    <td>51.90</td>
    <td>68.50</td>
    <td>61.00</td>
    <td>67.72</td>
  </tr>
  <tr>
    <td>cmlm-iid</td>
    <td>90.60</td>
    <td>67.30</td>
    <td>52.60</td>
    <td><b>73.10</b></td>
    <td>66.30</td>
    <td><b>69.98</b></td>
  </tr>
  <tr>
    <td>t5-seg-pref</td>
    <td>86.60</td>
    <td>58.00</td>
    <td>46.70</td>
    <td>67.60</td>
    <td>60.70</td>
    <td>63.92</td>
  </tr>
  <tr>
    <td>t5-seg-suff</td>
    <td>89.50</td>
    <td>58.40</td>
    <td>45.30</td>
    <td>71.60</td>
    <td>56.10</td>
    <td>64.18</td>
  </tr>
  <tr>
    <td>t5-seg-iid</td>
    <td>89.10</td>
    <td>63.70</td>
    <td>48.70</td>
    <td>71.60</td>
    <td>62.50</td>
    <td>67.12</td>
  </tr>
  <tr>
    <td>cmlm-seg-pref</td>
    <td>89.70</td>
    <td>57.60</td>
    <td><b>58.20</b></td>
    <td>68.60</td>
    <td>64.20</td>
    <td>67.66</td>
  </tr>
  <tr>
    <td>cmlm-seg-suff</td>
    <td><b>91.00</b></td>
    <td>67.00</td>
    <td>47.90</td>
    <td><b>73.20</b></td>
    <td>64.10</td>
    <td>68.64</td>
  </tr>
  <tr>
    <td>cmlm-seg-iid</td>
    <td>90.50</td>
    <td>68.30</td>
    <td>53.20</td>
    <td><b>73.20</b></td>
    <td>67.60</td>
    <td><b>70.56</b></td>
  </tr>
</table>

Table 4: Accuracy for the languages with segmentation data. -seg- indicates the masking objective sampled morphemes, rather than characters. The best model per language is bolded, and the second best is underlined

and unavailable in the source form in an inflection (e.g., an inflectional suffix). A morphological inflection model must learn to do this competently, and with sufficiently diverse data, we hypothesize that denoising objectives encourage this capability. We are not the first to hypothesize that autoencoding benefits inflection via copy bias. Liu and Hulden (2022) proposed generating synthetic words or lemmas and training inflection models to copy them. Yang et al. (2022) confirmed that this strategy strongly increases inflection performance.

However, if we consider cmlm-iid with masking as a baseline denoising method, all of our attempts to change the objective with a different inductive bias can be considered negative results. Deleting, merging span masks, and skewing the positions of the mask distribution do not lead to better results overall. One hypothesis here is simply that morphology is complicated in most languages: even if suffix concatenation, for instance, is pervasive, it does not account for all of the inflectional transformations and is thus too strong of an inductive bias. Instead, we believe that iid sampling with masks is the best heuristic because it leads to the highest variance in training data. That is, given a small unlabeled dataset, we sample each word many times during training. A uniform sampling of masks will lead to the highest variance in word positions that are masked for the same word, and thus the most variance in our dataset. This is likely a stronger learning method than any particular inductive bias.

In subsection 6.1 we presented evidence that prefix masking is beneficial to inflection datasets with pervasive prefix inflection. This suggests that the auxiliary self-supervised task does modify model behavior in typologically interpretable ways. However, the relatively weak suffix results when compared to iid suggests that inductive bias is less beneficial than simply learning the most diverse token distribution, even in a low-resource setting.
Figure 4: Language specific results for the ud-vnadj-NR dataset. Bars represent the max test accuracy of each strategy. Languages are grouped by which strategy maximized performance on this dataset.

Figure 5: The percent of trigams in predicted inflections that were copied from the lemma or from the training outputs. This compares models trained with autoencoding to cmlm-iid with masking on two different datasets.

Segment Masking  Masking morphemes rather than characters is more closely aligned to the downstream inflection task for concatenative inflections. In our experiments, we tend to see a slight positive effect from incorporating morpheme information to this end. This indicates that smarter masking can increase performance, though on average sampling characters uniformly is a very strong alternative. We observe two other results that are unique to the seg dataset: (i) morpheme masking underperforms character masking with the T5 objective (ii) deleting sometimes outperforms masking. The first result may be due to the fact that T5 on the seg dataset is most likely to have a large mismatch between the number of symbols on the source and target side. That is, an entire segment comprising many characters is replaced by a single mask symbol. For instance, if we sample the stem in the following example: <X>s + [TASK] → recovers. Instead, the CMLM objective would result in seven individual mask tokens that align with the stem. The fact that deletion still performs strongly for the T5 objective with segments (See: Figure 2), however, may warrant further investigation into the impact of masks. See Table 11 in the appendix for full results on the seg dataset with deletion.

7 Additional Analysis: Copying

In order to explore our hypothesis that autoencoding teaches the model to copy, whereas denoising teaches the model to generate from the training distribution, we quantify the extent to which models copy n-grams from the lemma, versus generate n-grams from the training data. To this end, we analyze predictions on the development set for the ud-vnadj-NR dataset used in the language-specific analysis. We compare the autoencoding model, and the cmlm-iid mask model. For each prediction, we compute the percent of character trigrams that appear in the trigrams of the input lemma in order to quantify copying. Then, we build the set of all character trigrams in the training data output forms, and compute the percent of predicted character trigrams that are attested in the training trigrams in order to quantify generating from the training distribution. In Figure 5 we compare the average percent of the two models over all languages for those predictions where the models disagree. This represents 7286 total disagreements for ud-vnadj-NR and 2511 for the seg dataset. In both datasets, autoencoding tends to copy more from the lemma and denoising tends to produce more trigrams from the training data. We interpret this as support for our hypothesis. In
the segmentation dataset, the difference in copy percent is much more substantial, and the training data percent difference is much smaller. The latter effect is likely because this morphologically rich dataset contains most n-grams that a model is likely to generate. The former effect illustrates that many AE mistakes probably come from over-copying.

8 Conclusion

In this work, we compared several self-supervised tasks performed on unlabeled data as auxiliary training objectives for morphological inflection. We found that for small sets of unlabeled words, autoencoding is a strong objective due to its inductive bias towards copying form the source string. However, tasks based on MLM perform better as the unlabeled dataset grows because it encourages the model to generate new character sequences. Other denoising objectives with more inductive bias have predictable effects, but are not necessarily better than the standard MLM with uniform masking. We believe this is because uniform masking produces the most diverse dataset. Finally, we find that providing a morpheme segment oracle leads to the best results, establishing a promising direction if segments can be approximated without supervision.

Limitations

First, this study is limited with respect to the morphological typology of the languages. Because we use UD data, we filter out the SIGMORPHON 2023 shared task languages that tend to inflect with prefixes, for instance.

We also explore a limited set of simple self-supervised heuristics. Other methods may be more effective for inflection. Our segmentation dataset contains strictly concatenative languages, and it is unclear what this means for non-concatenative morphology, which tends to be understudied in the computational morphology community.

Finally, we claim that better inflection models can be useful for language documentation efforts, but we test all methods on a standard benchmark by simulating a low-resource scenario. Languages with data needs like those that we discuss may yield different results.

Ethics Statement

This work is partly inspired by applications of language technology to language documentation and educational tools. We note that many of these applications traditionally apply to small indigenous language communities, who may not actually desire or need such NLP technology. We thus emphasize that the methods proposed here are intended to explore the development of NLP tools under severe data constraints, but do not test the feasibility of using them in a real applied setting.

Acknowledgments

We would like to thank Abteen Ebrahimi, Stéphane Aroca-Ouellette, Ananya Ganesh, and Kyle Gorman for feedback on this work. We would also like to thank Kyle Gorman for help with the alignment algorithm for turning canonical segmentation into surface segmentation.

References

Roei Aharoni and Yoav Goldberg. 2017. Morphological inflection generation with hard monotonic attention. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2004–2015.

Khuyagbaatar Batsuren, Gábor Bella, Aryaman Arora, Viktor Martinovic, Kyle Gorman, Zdeněk Žabokrtský, Amarsanaa Ganbold, Sárka Dohnalová, Magda Ševčíková, Kateřina Pelegrinová, Fausto Giunchiglia, Ryan Cotterell, and Ekaterina Vylomova. 2022. The SIGMORPHON 2022 shared task on morpheme segmentation. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 103–116, Seattle, Washington. Association for Computational Linguistics.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Omer Goldman, Khuyagbaatar Batsuren, Salam Khalifa, Aryaman Arora, Garrett Nicolai, Reut Tsarfaty, and Ekaterina Vylomova. 2023. Sigmorphon–unimorph 2023 shared task 0: Typologically diverse morphological inflection. In Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 117–125.
Kyle Gorman. 2016. Pynini: A python library for weighted finite-state grammar compilation. In Proceedings of the SIGFSS Workshop on Statistical NLP and Weighted Automata, pages 75–80.

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8:64–77.

Katharina Kann and Hinrich Schütze. 2017. Unlabeled data for morphological generation with character-based sequence-to-sequence models. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 76–81, Copenhagen, Denmark. Association for Computational Linguistics.

Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

Jordan Kodner, Salam Khalifa, Khuyagbaatar Batseren, Hossep Dolatian, Ryan Cotterell, Faruk Akkus, Antonios Anastasopoulos, Taras Andrushko, Aryaman Arora, Nona Atanalov, Gábor Bella, Elena Budianskaya, Yustinus Ghanggo Ate, Omer Goldman, David Guriel, Simon Guriel, Silvia Guriel-Agiashvili, Witold Kieraś, Andrew Krizhanovsky, Natalia Krizhanovsky, Igor Marchenko, Magdalena Markowska, Polina Mashkovtseva, Maria Nepomniashchaya, Daria Rodionova, Karina Scheifer, Alexandra Sorova, Anastasia Yemelina, Jeremiah Young, and Ekaterina Vylomova. 2022. SIGMORPHON–UniMorph 2022 shared task 0: Generalization and typologically diverse morphological inflection. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 176–203, Seattle, Washington. Association for Computational Linguistics.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880.

Ling Liu and Mans Hulden. 2022. Can a transformer pass the wug test? tuning copying bias in neural morphological inflection models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 739–749, Dublin, Ireland. Association for Computational Linguistics.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.

Peter Makarov and Simon Clematide. 2018. Imitation learning for neural morphological string transduction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2877–2882, Brussels, Belgium. Association for Computational Linguistics.

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.

Sarah Moeller and Mans Hulden. 2018. Automatic glossing in a low-resource setting for language documentation. In Proceedings of the Workshop on Computational Modeling of Polysynthetic Languages, pages 84–93, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Sarah Moeller, Ling Liu, Changbing Yang, Katharina Kann, and Mans Hulden. 2020. IGT2P: From interlinear glossed texts to paradigms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5251–5262, Online. Association for Computational Linguistics.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Abhishek Purushothama, Adam Wiemerslage, and Katharina Von Der Wense. 2024. Getting the most out of your training data: Exploring unsupervised tasks for morphological inflection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18955–18970, Miami, Florida, USA. Association for Computational Linguistics.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.

Adam Wiemerslage, Kyle Gorman, and Katharina von der Wense. 2024. Quantifying the hyperparameter sensitivity of neural networks for character-level
sequence-to-sequence tasks. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 674–689, St. Julian’s, Malta. Association for Computational Linguistics.

Adam Wiemerslage, Changbing Yang, Garrett Nicolai, Miikka Silfverberg, and Katharina Kann. 2023. An investigation of noise in morphological inflection. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3351–3365.

Shijie Wu and Ryan Cotterell. 2019. Exact hard monotonic attention for character-level transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1530–1537.

Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. Applying the transformer to character-level transduction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1901–1907. Online. Association for Computational Linguistics.

Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306.

Changbing Yang, Ruixin (Ray) Yang, Garrett Nicolai, and Miikka Silfverberg. 2022. Generalizing morphological inflection systems to unseen lemmas. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 226–235, Seattle, Washington. Association for Computational Linguistics.

Daniel Zeman, Joakim Nirve, Mitchell Abrams, Elia Ackermann, Noemi Aeppli, Hamid Aghaei, Željko Agić, Amir Ahmadi, Lars Ahrenberg, Chika Kennedy Ajede, Salih Furkan Akkurt, Gabriele Aleksandravičiūtė, Ika Alfina, Avner Algom, Khalid Alnajjar, Chiara Alzetta, Erik Andersen, Lene Antonsen, Tatsuya Aoyama, Katya Aplonova, Angelina Aquino, Carolina Aragon, Glyd Aranes, María Jesus Aranzabe, Bilge Nas Arican, Hörunn Arnardóttir, Gashaw Arutie, Jessica Naraiswari Arwidarasti, Masayuki Asahara, Katla Ásgeirsdóttir, Deniz Baran Aslan, Cengiz Asmazoğlu, Luma Ateyah, Furkan Atmaca, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Mariana Avelás, Elena Badmaeva, Keerthana Balasubramani, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, Starkábor Barkarson, Rodolfo Basile, Victoria Basmov, Colin Batchelor, John Bauer, Seyyit Talha Bedir, Shabnam Behzad, Kepa Bengoetxea, Ibrahim Benli, Yifat Ben Moshe, Gözde Berk, Riyaz Ahmad Bhat, Erica Biagetti, Eckhard Bick, Agne Bielinskiene, Kristín Bjarnadóttir, Rogier Blokland, Victoria Bobicev, Loïc Boizou, Emanuel Borges Völker, Carl Börstel, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Anouck Braggaar, António Branco, Kristina Brokaitė, Aljoscha Burchardt, Marisa Campos, Marie Candito, Bernard Caron, Gauthier Caron, Catarina Carvalheiro, Rita Carvalho, Lauren Cassidy, Maria Clara Castro, Sérgio Castro, Tatiana Cavalcanti, Gülşen Çebiroğlu Eryiğit, Flavio Massimiliano Cecchini, Giuseppe G. A. Celano, Slavomír Čépló, Neslihan Cesur, Savas Cetin, Özlem Çetinoğlu, Fabrício Chalub, Liyanage Chamila, Shweta Chauhan, Ethan Chi, Taishi Chika, Yongseok Cho, Jinho Choi, Jayeol Chun, Juyeon Chung, Alessandra T. Cignarella, Silvie Cinková, Aurélie Collomb, Çağrı Çöltekin, Miriam Connor, Daniela Corbetta, Francisco Costa, Marine Courtin, Mihaela Cristescu, Inger Löyning Dale, Philemon Daniel, Elizabeth Davidson, Leonel Figueiredo de Alencar, Mathieu Dehouck, Martina de Laurentiis, Marie-Catherine de Manneffe, Valeria de Paiva, Mehmet Oguz Derin, Elvis de Souza, Arantza Diaz de Ilarraza, Carly Dickerson, Arawinda Dinakaramani, Elisa Di Nuovo, Bamba Dione, Peter Dirix, Kaja Dobrovoljic, Adrian Doyle, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Christian Ebert, Hanne Eckhoff, Masaki Eguchi, Sandra Eiche, Marhaba Eli, Ali Elkahky, Binyam Ephrem, Olga Erina, Tomáž Erjavec, Farah Essaidi, Aline Etienne, Wograime Evelyn, Sidney Faccundes, Richard Farkas, Federica Favero, Janannat Ferdaousi, Marília Fernanda, Hector Fernandez Alcalde, Amal Fethi, Jennifer Foster, Cláudia Freitas, Kazunori Fujita, Katarina Gajdošová, Daniel Galbraith, Federica Gamba, Marcos Garcia, Moa Gärdenfors, Fabricio Ferraz Gerardi, Kim Gerdes, Luke Gessler, Filip Ginter, Gustavo Godoy, Iakes Goenaga, Koldo Gojenola, Memduh Gökirmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta Gonzalez Saavedra, Bernadeta Griciūtė, Matias Grioni, Loïc Grobel, Normunds Grūžitis, Bruno Guillaume, Céline Guillot-Barbance, Tunga Güngör, Nizar Habash, Hinrik Hafsteinsson, Jan Hajíč, Jan Hajíč jr., Mika Hämäläinen, Linh Ha My, Na-Rae Han, Muhammad Yudistira Hanifmuti, Takahiro Harada, Sam Hardwick, Kim Harris, Dag Haug, Johannes Heinecke, Oliver Hellwig, Felix Hennig, Barbora Hladká, Jaroslava Hlaváčová, Florinel Hociung, Petter Hohle, Marivel Huerta Méndez, Jena Hwang, Takumi Ikeda, Anton Karl Ingason, Radu Ion, Elena Irimia, Olájide Ishola, Artan Islamaj, Kaoru Ito, Siratun Jannat, Tomáš Jelinek, Apoorva Jha, Katharine Jiang, Anders Johansen, Hildur Jónsdóttir, Fredrik Jørgensen, Markus Juutinen, Hüner Kaşıkara, Nadezhda Kabaeva, Sylvain Kahane, Hiroshi Kanayama, Jenna Kanerva, Neslihan Kara, Ritván Karahóga, André Käsen, Tolga Kayadelen, Sarveswaran Kengatharaiyer, Václava Kettnerová, Jesse Kirchner, Elena Klementieva, Elena Klyachko, Arne Köhn, Abdulatif Köksal, Kamil Kopacewicz, Timo Korkiakangas, Mehmet Köse, Alexey Koshevoy, Natalia Kotysba, Jolanta Kovalevskaitė, Simon Kreck, Pameswari Krishnamurthy, Sandra Kübler, Adrian Kuqi, Oğuzhan Kuyrukcü, Aslı Kuzgun, Sooyoung Kwak, Kris Kyle, Veronica Laippala, Lorenzo Lambertino, Tatiana Lando, Septina Dian Larasati, Alexei Lavrentiev, John Lee, Phng Lê Hồng, Alessandro Lenci, Saran Lertpradit, Herman Leung, Maria Levena,
Lauren Levine, Cheuk Ying Li, Josie Li, Keying Li, Yixuan Li, Yuan Li, KyungTae Lim, Bruna Lima Padovani, Yi-Ju Jessica Lin, Krister Lindén, Yang Janet Liu, Nikola Ljubešić, Olga Loginova, Stefano Lusito, André Lutfi, Mikko Luukkno, Olga Lyashchekskaya, Teresa Lynn, Vivien Macketan, Menel Mahamdi, Jean Maillard, Ilya Makarchuk, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Büşra Marşan, Cătălina Mărănduc, David Mareček, Katrin Marheinecke, Stella Markantonatou, Héctor Martínez Alonso, Lorena Martín Rodríguez, André Martins, Cláudia Martins, Jan Mašek, Hiroshi Matsuda, Yuji Matsumoto, Alessandro Mazzei, Ryan McDonald, Sarah McGuinness, Gustavo Mendonça, Tatiana Merzhević, Niko Miekka, Aaron Miller, Karina Mischenkova, Anna Missilä, Cătălin Mititelu, Maria Mitrofan, Yusuke Miyao, AmirHossein Mojiri Foroushani, Judit Molnár, Amirsaeid Moloodi, Simonetta Montemagni, Amir More, Laura Moreno Romero, Giovanni Moretti, Shinsuke Mori, Tomohiko Morioika, Shigeki Moro, Bjartur Mortensen, Bohdan Moskalevskyi, Kadri Muischnek, Robert Munro, Yugo Murawaki, Kaili Müürisepp, Pinkey Nainwani, Mariam Nakhli, Juan Ignacio Navarro Horňaicek, Anna Nedoluzhko, Gunta Nesporė-Bērzkalne, Manuela Nevaci, Lng Nguyễn Thi, Huyền Nguyên Thi Minh, Yoshihiro Nikaido, Vitaly Nikolaev, Rattima Nitisoraj, Alireza Nourian, Hanna Nurmi, Stina Ojala, Atul Kr. Ojha, Hulda Óladóttir, Adéadayo Olúokun, Mai Omura, Emeka Onwuabuzia, Noam Ordan, Petya Osenova, Robert Östling, Lilja Øvrelid, Şaziye Betül Özateş, Merve Özçelik, Arzuçan Özgür, Balkız Oztürk Başaran, Teresa Paccoi, Alessio Palmero Aprosio, Anastasia Panova, Hyunjy Hayley Park, Niko Partanen, Elena Pascual, Marco Passarotti, Agnieszka Patejuk, Guilherme Paulino-Passos, Giulia Pedone, Angelika Peljak-Lapinska, Siyao Peng, Siyao Logan Peng, Rita Pereira, Silvia Pereira, Cenel-Augusto Perez, Natalia Perkova, Guy Perrier, Slav Petrov, Daria Petrova, Andrea Perveiller, Jason Phelan, Jussi Pitulainen, Yuval Pinter, Clara Pinto, Tommi A Pirinen, Emily Pitler, Magdalena Plamada, Barbara Plank, Thierry Poibeau, Larisa Pomornareva, Martin Popel, Lauma Pretkalnina, Sophie Prévost, Prokopios Prokopidis, Adam Przeźpiorkowski, Robert Pugh, Tiina Puolakainen, Sampo Pyysalo, Peng Qi, Andreia Querido, Andriela Rääbis, Alexandre Radeemaker, Mizanur Rahoman, Taraka Rama, Loganathan Ramasamy, Joana Ramos, Fam Rashel, Mohammad Sadegh Rasooli, Vinit Ravishankar, Livy Real, Petru Rebeja, Siva Reddy, Mathilde Regnault, Georg Rehm, Arij Riabi, Ivan Riabov, Michael Rießler, Erika Rimkutė, Larissa Rinaldi, Laura Rituma, Putri Rizqiyah, Luisa Rocha, Eirikur Rögnvaldsson, Ivan Roksandic, Mykhailo Romanenko, Rudolf Rosa, Valentin Rosca, Davide Rotati, Ben Rozonoyer, Olga Rudina, Jack Rueter, Kristján Rúnarsson, Shoval Sadde, Pegah Safari, Aleksis Sahala, Shadi Saleh, Alessio Salomoni, Tanja Samardžić, Stephanie Samson, Manuela Sanguinetti, Ezgi Sanıyar, Dage Särg, Marta Sartor, Mitsuya Sasaki, Baiba Saulite, Yanin Sawanakunanon, Shefali Saxena, Kevin Scannell, Salvatore Scarlata, Nathan Schneider, Sebastian Schuster, Lane Schwartz, Djamel Seddah, Wolfgang Seeker, Mojgan Seraji, Syeda Shahzadi, Mo Shen, Atsuko Shimada, Hiroyuki Shirasi, Yana Shishkina, Muh Shohibussirri, Maria Shvedova, Janine Siewert, Einar Freyr Sigurðsson, João Silva, Aline Silveira, Natalia Silveira, Sara Silveira, Maria Simi, Radu Simionescu, Katalin Simkó, Mária Šimková, Haukur Barri Simonarson, Kiril Simov, Dmitri Stitchinava, Ted Sither, Maria Skachedubova, Aaron Smith, Isabela Soares-Bastos, Per Erik Solberg, Barbara Sonnenhauser, Shafi Sourov, Rachele Sprugnoli, Vivian Stamou, Steinhóðr Steingrímsson, Antonio Stella, Abishek Stephen, Milan Straka, Emmett Strickland, Jana Strnadová, Alane Suhr, Yogi Lesmana Sulestio, Umut Sulubacak, Shingo Suzuki, Daniel Swanson, Zsolt Szántó, Chihiro Taguchi, Dima Taj, Fabio Tamburini, Mary Ann C. Tan, Takaaki Tanaka, Dipta Tanayia, Mirko Tavoni, Samson Tella, Isabelle Tellier, Marinella Testori, Guillaume Thomas, Sara Tonelli, Liisi Torga, Marsida Toska, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Utku Türk, Francis Tyers, Sveinbjörn Hórdarson, Vilhjálmur Horsteinsson, Sumire Uematsu, Roman Untilov, Zdeněk Uresová, Larraitz Uria, Hans Uszkoreit, Andrius Utka, Elena Vagnoni, Sowmya Vajjala, Socrates Vak, Rob van der Goot, Martine Vanhove, Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Uliana Vedenina, Giulia Venturi, Veronika Vince, Natalia Vlasova, Aya Wakasa, Joel C. Wallenberg, Lars Wallin, Abigail Walsh, Jonathan North Washington, Maximilian Wendt, Paul Widmer, Shira Wigderson, Sri Hartati Wijono, Seyi Williams, Mats Wirén, Christian Wittem, Tsegay Woldemariam, Tak-sum Wong, Alina Wróblewska, Mary Yako, Kayo Yamashita, Naoki Yamazaki, Chunxiao Yan, Koichi Yasuoka, Marat M. Yavrumyan, Arife Betül Yenice, Olcay Taner Yildiz, Zhuoran Yu, Arlisa Yuliawati, Zdeněk Žabokrtský, Shorouq Zahra, Amir Zeldes, He Zhou, Hanzhi Zhu, Yilun Zhu, Anna Zhuravleva, and Rayan Ziane. 2023. Universal dependencies 2.12. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Appendix

A UD Data

When we sample from UD data, we use exactly the corpora from Purushothama et al. (2024) except for two cases. For both English and Turkish, there is not sufficient data in the existing corpora to sample 5k unique words from, so we additionally add the English-EWT corpus and the Turkish-BOUN, respectively. The English-EWT contains web data, and so we ignore tokens with special sequences "@" and "www". This process is imperfect though, so we still introduce noise into our data via this corpus.

B Surface Segmentation Data

Canonically segmented data may insert or delete characters into or from a surface word form. We thus present a simple method for attaining a surface segmentation given a surface form \( s \) and its canonical segmentation \( c \). First, we build a Pynini EditTransducer (Gorman, 2016)—which is a weighted finite state transducer aligning strings—over the language’s character vocabulary \( \Sigma \). Then, we attain the shortest path from \( s \) to \( c \): the alignment that produces their minimal Levenshtein distance. Finally, we segment \( s \) (i) at any insertion that maps to a segmentation boundary in \( c \) or (ii) after any character that maps to a segmentation boundary in \( c \).

For example, given the pair \( \langle \text{chugged},\ \text{chug-ed} \rangle \), we attain the alignment

\[
\begin{array}{cccccc}
c & h & u & g & g & e & d \\
c & h & u & g & - & e & d
\end{array}
\]

Then we match on rule (ii) and segment \( s \) into \( \text{chugg-ed} \).

C Hyperparameters

Our architecture hyperparameters match Wu et al. (2021): We use a transformer encoder and decoder, each with 4 layers, embedding size 256, hidden size 1024, and 4 attention heads. We optimize with cross-entropy loss with label smoothing of 0.1. We train with Adam optimization (Kingma, 2014) with a learning rate of 1e-3, and a beta2 of .98. We train up to 800 epochs with a batch size of 400, and evaluate every 16 epochs. For the baseline experiments, we lower the batch size to 100 and train for up to 10k epochs with a patience of 400. We train with the Warmup Inverse Square Root schedule with 4k warmup steps. When masking unlabeled data, we always sample 25% of tokens for masking. All experiments are trained on NVIDIA GPUs and implemented with yoyodyne\( ^1 \) (Wiemerslage et al., 2024).

\( ^1 \)https://github.com/CUNY-CL/yoyodyne
<table>
  <tr>
    <th></th>
    <th>t5-prefix</th>
    <th>cmlm-prefix</th>
    <th>t5-suffix</th>
    <th>cmlm-suffix</th>
    <th>t5-iid</th>
    <th>cmlm-iid</th>
    <th>AE</th>
  </tr>
  <tr>
    <td>afb</td>
    <td><b>75.80</b></td>
    <td>74.50</td>
    <td>71.90</td>
    <td>73.70</td>
    <td>74.40</td>
    <td>73.30</td>
    <td>72.60</td>
  </tr>
  <tr>
    <td>amh</td>
    <td>59.90</td>
    <td>62.80</td>
    <td>61.80</td>
    <td>62.00</td>
    <td>60.60</td>
    <td><u>63.80</u></td>
    <td><b>69.20</b></td>
  </tr>
  <tr>
    <td>bel</td>
    <td>64.00</td>
    <td>63.30</td>
    <td>62.40</td>
    <td>62.40</td>
    <td>62.40</td>
    <td><u>64.90</u></td>
    <td>62.20</td>
  </tr>
  <tr>
    <td>dan</td>
    <td>81.70</td>
    <td>82.20</td>
    <td>82.90</td>
    <td><b>83.90</b></td>
    <td><u>83.50</u></td>
    <td>82.70</td>
    <td>82.30</td>
  </tr>
  <tr>
    <td>deu</td>
    <td>74.10</td>
    <td><u>77.60</u></td>
    <td>73.80</td>
    <td>77.50</td>
    <td>76.10</td>
    <td><b>79.00</b></td>
    <td>76.90</td>
  </tr>
  <tr>
    <td>eng</td>
    <td>88.10</td>
    <td><b>90.50</b></td>
    <td>89.00</td>
    <td>89.50</td>
    <td>89.40</td>
    <td>88.50</td>
    <td><u>90.10</u></td>
  </tr>
  <tr>
    <td>fin</td>
    <td><u>73.90</u></td>
    <td>72.50</td>
    <td><b>74.60</b></td>
    <td>73.20</td>
    <td><b>74.60</b></td>
    <td>70.70</td>
    <td><u>73.90</u></td>
  </tr>
  <tr>
    <td>fra</td>
    <td>75.80</td>
    <td>74.10</td>
    <td><b>76.40</b></td>
    <td><u>76.10</u></td>
    <td>73.70</td>
    <td>74.70</td>
    <td>75.00</td>
  </tr>
  <tr>
    <td>grc</td>
    <td>52.50</td>
    <td>49.60</td>
    <td>50.80</td>
    <td>47.90</td>
    <td><u>53.40</u></td>
    <td><b>55.30</b></td>
    <td>49.20</td>
  </tr>
  <tr>
    <td>heb</td>
    <td>76.94</td>
    <td>77.04</td>
    <td><u>77.84</u></td>
    <td>77.24</td>
    <td>77.64</td>
    <td><b>79.66</b></td>
    <td>76.84</td>
  </tr>
  <tr>
    <td>hun</td>
    <td>77.70</td>
    <td>3.40</td>
    <td><u>77.90</u></td>
    <td>79.50</td>
    <td><b>80.50</b></td>
    <td>77.60</td>
    <td><u>79.80</u></td>
  </tr>
  <tr>
    <td>hye</td>
    <td>91.30</td>
    <td><u>92.40</u></td>
    <td>91.60</td>
    <td>92.20</td>
    <td><b>93.30</b></td>
    <td>92.30</td>
    <td><b>93.30</b></td>
  </tr>
  <tr>
    <td>ita</td>
    <td><b>92.40</b></td>
    <td><u>92.10</u></td>
    <td>91.20</td>
    <td>90.80</td>
    <td>92.00</td>
    <td>91.00</td>
    <td>91.30</td>
  </tr>
  <tr>
    <td>jap</td>
    <td>37.80</td>
    <td>39.10</td>
    <td>41.50</td>
    <td><u>44.20</u></td>
    <td>37.80</td>
    <td><u>44.20</u></td>
    <td><b>53.60</b></td>
  </tr>
  <tr>
    <td>rus</td>
    <td>79.50</td>
    <td>80.10</td>
    <td>80.20</td>
    <td><u>82.20</u></td>
    <td><b>82.40</b></td>
    <td><u>82.20</u></td>
    <td>81.30</td>
  </tr>
  <tr>
    <td>san</td>
    <td>56.40</td>
    <td>56.10</td>
    <td>54.80</td>
    <td>56.80</td>
    <td><u>59.10</u></td>
    <td>57.10</td>
    <td><b>61.10</b></td>
  </tr>
  <tr>
    <td>sme</td>
    <td>61.00</td>
    <td>61.30</td>
    <td>61.10</td>
    <td>59.60</td>
    <td><u>66.20</u></td>
    <td>64.50</td>
    <td><b>69.20</b></td>
  </tr>
  <tr>
    <td>spa</td>
    <td>91.20</td>
    <td>91.60</td>
    <td>90.30</td>
    <td><u>92.40</u></td>
    <td>90.60</td>
    <td>92.30</td>
    <td><b>93.10</b></td>
  </tr>
  <tr>
    <td>tur</td>
    <td>84.60</td>
    <td><b>88.00</b></td>
    <td>85.10</td>
    <td>86.10</td>
    <td>85.70</td>
    <td>84.90</td>
    <td><b>89.90</b></td>
  </tr>
  <tr>
    <td>Mean</td>
    <td>73.40</td>
    <td>69.91</td>
    <td>73.43</td>
    <td>74.07</td>
    <td>74.39</td>
    <td>74.67</td>
    <td><b>75.83</b></td>
  </tr>
</table>

Table 5: All language results for the ud-1k dataset from (Purushothama et al., 2024).

<table>
  <tr>
    <th></th>
    <th>t5-prefix</th>
    <th>cmlm-prefix</th>
    <th>t5-suffix</th>
    <th>cmlm-suffix</th>
    <th>t5-iid</th>
    <th>cmlm-iid</th>
    <th>AE</th>
  </tr>
  <tr>
    <td>afb</td>
    <td>54.30</td>
    <td>53.30</td>
    <td>54.30</td>
    <td>53.10</td>
    <td><b>56.50</b></td>
    <td>52.90</td>
    <td><u>54.40</u></td>
  </tr>
  <tr>
    <td>amh</td>
    <td>26.00</td>
    <td><u>29.30</u></td>
    <td>26.40</td>
    <td><u>29.30</u></td>
    <td>26.00</td>
    <td>28.40</td>
    <td><b>37.10</b></td>
  </tr>
  <tr>
    <td>bel</td>
    <td>47.00</td>
    <td><b>50.60</b></td>
    <td>45.20</td>
    <td><u>48.60</u></td>
    <td>45.20</td>
    <td><u>48.60</u></td>
    <td>44.90</td>
  </tr>
  <tr>
    <td>dan</td>
    <td>79.00</td>
    <td><u>80.10</u></td>
    <td>79.80</td>
    <td>79.60</td>
    <td><u>79.90</u></td>
    <td>79.60</td>
    <td>79.40</td>
  </tr>
  <tr>
    <td>deu</td>
    <td>51.40</td>
    <td>55.60</td>
    <td>54.40</td>
    <td>57.90</td>
    <td><u>58.50</u></td>
    <td><b>60.50</b></td>
    <td>53.90</td>
  </tr>
  <tr>
    <td>eng</td>
    <td>28.90</td>
    <td><u>51.60</u></td>
    <td>50.90</td>
    <td>46.60</td>
    <td>35.40</td>
    <td>48.70</td>
    <td><b>75.40</b></td>
  </tr>
  <tr>
    <td>fin</td>
    <td><b>38.70</b></td>
    <td>28.40</td>
    <td>31.60</td>
    <td>33.30</td>
    <td><u>38.50</u></td>
    <td>30.00</td>
    <td>28.70</td>
  </tr>
  <tr>
    <td>fra</td>
    <td>40.40</td>
    <td><u>47.50</u></td>
    <td>46.30</td>
    <td>45.30</td>
    <td>46.80</td>
    <td>45.10</td>
    <td><b>49.70</b></td>
  </tr>
  <tr>
    <td>grc</td>
    <td>24.00</td>
    <td>19.50</td>
    <td>22.50</td>
    <td>26.00</td>
    <td><b>30.70</b></td>
    <td><u>29.20</u></td>
    <td>21.60</td>
  </tr>
  <tr>
    <td>heb</td>
    <td>14.60</td>
    <td><b>16.01</b></td>
    <td>13.29</td>
    <td>12.49</td>
    <td><u>15.11</u></td>
    <td>14.60</td>
    <td>14.30</td>
  </tr>
  <tr>
    <td>hun</td>
    <td>49.00</td>
    <td>55.30</td>
    <td><b>58.70</b></td>
    <td>56.60</td>
    <td>51.80</td>
    <td><u>58.40</u></td>
    <td>56.90</td>
  </tr>
  <tr>
    <td>hye</td>
    <td>60.90</td>
    <td>65.10</td>
    <td><b>73.10</b></td>
    <td>71.40</td>
    <td>68.90</td>
    <td>61.10</td>
    <td><u>72.40</u></td>
  </tr>
  <tr>
    <td>ita</td>
    <td>27.10</td>
    <td>37.00</td>
    <td>34.20</td>
    <td>40.10</td>
    <td>33.30</td>
    <td><u>41.40</u></td>
    <td><b>48.60</b></td>
  </tr>
  <tr>
    <td>jap</td>
    <td>0.80</td>
    <td>1.70</td>
    <td>2.30</td>
    <td><u>3.70</u></td>
    <td>2.00</td>
    <td>2.00</td>
    <td><b>21.40</b></td>
  </tr>
  <tr>
    <td>rus</td>
    <td>66.80</td>
    <td>66.00</td>
    <td>63.10</td>
    <td>66.20</td>
    <td>66.10</td>
    <td><b>72.00</b></td>
    <td>67.40</td>
  </tr>
  <tr>
    <td>san</td>
    <td>20.10</td>
    <td>23.50</td>
    <td>26.50</td>
    <td>25.50</td>
    <td>23.20</td>
    <td><u>27.40</u></td>
    <td><b>33.80</b></td>
  </tr>
  <tr>
    <td>sme</td>
    <td>29.30</td>
    <td><b>33.20</b></td>
    <td>28.70</td>
    <td>28.80</td>
    <td><u>32.10</u></td>
    <td>28.20</td>
    <td>31.90</td>
  </tr>
  <tr>
    <td>spa</td>
    <td>53.30</td>
    <td>51.10</td>
    <td>49.10</td>
    <td>54.80</td>
    <td>54.40</td>
    <td><b>57.50</b></td>
    <td><u>56.90</u></td>
  </tr>
  <tr>
    <td>tur</td>
    <td>16.20</td>
    <td>29.80</td>
    <td>25.00</td>
    <td><u>33.20</u></td>
    <td>15.40</td>
    <td>29.80</td>
    <td><b>53.40</b></td>
  </tr>
  <tr>
    <td>Mean</td>
    <td>38.31</td>
    <td>41.82</td>
    <td>41.34</td>
    <td>42.76</td>
    <td>41.04</td>
    <td><u>42.92</u></td>
    <td><b>47.48</b></td>
  </tr>
</table>

Table 6: All language results for the ud-200 dataset.
<table>
  <tr>
    <th></th>
    <th>t5-prefix</th>
    <th>cmlm-prefix</th>
    <th>t5-suffix</th>
    <th>cmlm-suffix</th>
    <th>t5-iid</th>
    <th>cmlm-iid</th>
    <th>AE</th>
  </tr>
  <tr>
    <td>afb</td>
    <td>54.90</td>
    <td>54.20</td>
    <td><b>56.20</b></td>
    <td>55.70</td>
    <td><b>56.70</b></td>
    <td>55.60</td>
    <td>54.60</td>
  </tr>
  <tr>
    <td>amh</td>
    <td>22.40</td>
    <td>26.30</td>
    <td>26.30</td>
    <td>24.30</td>
    <td>27.40</td>
    <td>25.00</td>
    <td><b>37.80</b></td>
  </tr>
  <tr>
    <td>bel</td>
    <td>48.80</td>
    <td><b>50.20</b></td>
    <td>43.10</td>
    <td>48.70</td>
    <td>47.80</td>
    <td>48.20</td>
    <td>49.30</td>
  </tr>
  <tr>
    <td>dan</td>
    <td>80.20</td>
    <td>79.40</td>
    <td>77.30</td>
    <td>78.80</td>
    <td>80.60</td>
    <td><b>80.70</b></td>
    <td>79.20</td>
  </tr>
  <tr>
    <td>deu</td>
    <td>54.10</td>
    <td>57.90</td>
    <td>52.90</td>
    <td><b>58.10</b></td>
    <td>57.90</td>
    <td><b>62.70</b></td>
    <td>52.90</td>
  </tr>
  <tr>
    <td>eng</td>
    <td>23.30</td>
    <td>44.80</td>
    <td>40.30</td>
    <td><b>45.80</b></td>
    <td>29.60</td>
    <td>36.80</td>
    <td><b>72.70</b></td>
  </tr>
  <tr>
    <td>fin</td>
    <td>36.80</td>
    <td>28.70</td>
    <td>33.70</td>
    <td>34.00</td>
    <td><b>43.50</b></td>
    <td><b>38.10</b></td>
    <td>32.20</td>
  </tr>
  <tr>
    <td>fra</td>
    <td>43.80</td>
    <td>43.80</td>
    <td>48.40</td>
    <td>49.60</td>
    <td><b>52.60</b></td>
    <td>51.50</td>
    <td><b>52.50</b></td>
  </tr>
  <tr>
    <td>grc</td>
    <td>23.30</td>
    <td>20.70</td>
    <td>21.60</td>
    <td>26.10</td>
    <td><b>26.30</b></td>
    <td><b>28.70</b></td>
    <td>16.90</td>
  </tr>
  <tr>
    <td>heb</td>
    <td>15.41</td>
    <td><b>16.72</b></td>
    <td>15.41</td>
    <td>13.09</td>
    <td><b>15.71</b></td>
    <td>13.60</td>
    <td>15.41</td>
  </tr>
  <tr>
    <td>hun</td>
    <td>53.20</td>
    <td>55.80</td>
    <td>59.80</td>
    <td><b>60.10</b></td>
    <td><b>62.70</b></td>
    <td>58.30</td>
    <td>53.70</td>
  </tr>
  <tr>
    <td>hye</td>
    <td>69.00</td>
    <td>66.20</td>
    <td>72.20</td>
    <td><b>73.40</b></td>
    <td>69.10</td>
    <td>63.40</td>
    <td><b>73.90</b></td>
  </tr>
  <tr>
    <td>ita</td>
    <td>38.80</td>
    <td>43.10</td>
    <td>42.10</td>
    <td>43.70</td>
    <td>44.00</td>
    <td><b>45.30</b></td>
    <td><b>49.70</b></td>
  </tr>
  <tr>
    <td>jap</td>
    <td>6.90</td>
    <td>9.30</td>
    <td><b>11.10</b></td>
    <td>9.20</td>
    <td>5.40</td>
    <td>7.90</td>
    <td><b>30.80</b></td>
  </tr>
  <tr>
    <td>rus</td>
    <td>64.30</td>
    <td><b>70.50</b></td>
    <td>65.00</td>
    <td>68.80</td>
    <td>67.40</td>
    <td><b>70.00</b></td>
    <td>67.90</td>
  </tr>
  <tr>
    <td>san</td>
    <td>14.00</td>
    <td><b>21.00</b></td>
    <td>15.90</td>
    <td>17.70</td>
    <td>18.90</td>
    <td>20.30</td>
    <td><b>30.50</b></td>
  </tr>
  <tr>
    <td>sme</td>
    <td>31.00</td>
    <td>31.60</td>
    <td>30.60</td>
    <td>28.30</td>
    <td>32.60</td>
    <td><b>33.50</b></td>
    <td><b>33.70</b></td>
  </tr>
  <tr>
    <td>spa</td>
    <td>52.20</td>
    <td>45.60</td>
    <td>55.10</td>
    <td><b>56.30</b></td>
    <td><b>59.70</b></td>
    <td>55.40</td>
    <td>52.20</td>
  </tr>
  <tr>
    <td>tur</td>
    <td>18.20</td>
    <td>22.00</td>
    <td>28.30</td>
    <td><b>32.80</b></td>
    <td>16.40</td>
    <td>20.30</td>
    <td><b>52.20</b></td>
  </tr>
  <tr>
    <td>Mean</td>
    <td>39.51</td>
    <td>41.46</td>
    <td>41.86</td>
    <td><b>43.39</b></td>
    <td>42.86</td>
    <td>42.91</td>
    <td><b>47.80</b></td>
  </tr>
</table>

Table 7: All language results for the ud-vnadj dataset

<table>
  <tr>
    <th></th>
    <th>t5-prefix</th>
    <th>cmlm-prefix</th>
    <th>t5-suffix</th>
    <th>cmlm-suffix</th>
    <th>t5-iid</th>
    <th>cmlm-iid</th>
    <th>AE</th>
  </tr>
  <tr>
    <td>afb</td>
    <td>54.40</td>
    <td>53.50</td>
    <td><b>57.10</b></td>
    <td>55.50</td>
    <td><b>56.30</b></td>
    <td>54.10</td>
    <td>55.20</td>
  </tr>
  <tr>
    <td>amh</td>
    <td>25.90</td>
    <td><b>34.30</b></td>
    <td>32.70</td>
    <td>33.60</td>
    <td>32.10</td>
    <td>34.00</td>
    <td><b>40.70</b></td>
  </tr>
  <tr>
    <td>bel</td>
    <td>44.70</td>
    <td><b>49.50</b></td>
    <td>42.30</td>
    <td>47.90</td>
    <td>47.50</td>
    <td>45.40</td>
    <td><b>48.10</b></td>
  </tr>
  <tr>
    <td>dan</td>
    <td>77.00</td>
    <td><b>79.70</b></td>
    <td>77.80</td>
    <td>79.00</td>
    <td>78.90</td>
    <td>78.60</td>
    <td><b>79.20</b></td>
  </tr>
  <tr>
    <td>deu</td>
    <td>52.30</td>
    <td>56.20</td>
    <td>56.40</td>
    <td>56.80</td>
    <td><b>60.10</b></td>
    <td><b>63.40</b></td>
    <td><b>60.10</b></td>
  </tr>
  <tr>
    <td>eng</td>
    <td>40.50</td>
    <td><b>58.80</b></td>
    <td>49.60</td>
    <td>52.50</td>
    <td>37.10</td>
    <td>50.60</td>
    <td><b>80.00</b></td>
  </tr>
  <tr>
    <td>fin</td>
    <td><b>38.40</b></td>
    <td>28.50</td>
    <td>32.60</td>
    <td>30.30</td>
    <td><b>38.10</b></td>
    <td>31.60</td>
    <td>34.40</td>
  </tr>
  <tr>
    <td>fra</td>
    <td>40.30</td>
    <td>45.20</td>
    <td>51.40</td>
    <td>53.50</td>
    <td><b>56.60</b></td>
    <td>54.10</td>
    <td>53.70</td>
  </tr>
  <tr>
    <td>grc</td>
    <td>25.90</td>
    <td>25.30</td>
    <td>25.60</td>
    <td><b>30.30</b></td>
    <td>28.60</td>
    <td><b>29.10</b></td>
    <td>21.90</td>
  </tr>
  <tr>
    <td>heb</td>
    <td>15.61</td>
    <td>15.31</td>
    <td>14.00</td>
    <td>13.90</td>
    <td><b>16.62</b></td>
    <td>13.60</td>
    <td><b>18.53</b></td>
  </tr>
  <tr>
    <td>hun</td>
    <td>50.20</td>
    <td><b>54.90</b></td>
    <td>53.90</td>
    <td>54.50</td>
    <td><b>59.30</b></td>
    <td>54.10</td>
    <td>53.80</td>
  </tr>
  <tr>
    <td>hye</td>
    <td>61.30</td>
    <td>64.80</td>
    <td>70.60</td>
    <td><b>77.00</b></td>
    <td>72.20</td>
    <td>70.20</td>
    <td>74.70</td>
  </tr>
  <tr>
    <td>ita</td>
    <td>32.40</td>
    <td><b>45.10</b></td>
    <td>41.60</td>
    <td>45.00</td>
    <td>38.90</td>
    <td>43.80</td>
    <td><b>46.60</b></td>
  </tr>
  <tr>
    <td>jap</td>
    <td>3.80</td>
    <td>5.50</td>
    <td><b>9.30</b></td>
    <td>9.00</td>
    <td>8.20</td>
    <td>7.80</td>
    <td><b>29.00</b></td>
  </tr>
  <tr>
    <td>rus</td>
    <td>66.00</td>
    <td>69.00</td>
    <td>65.70</td>
    <td>66.80</td>
    <td><b>69.40</b></td>
    <td>66.80</td>
    <td>68.10</td>
  </tr>
  <tr>
    <td>san</td>
    <td>17.30</td>
    <td><b>26.50</b></td>
    <td>21.70</td>
    <td>25.80</td>
    <td>18.00</td>
    <td><b>26.50</b></td>
    <td><b>33.30</b></td>
  </tr>
  <tr>
    <td>sme</td>
    <td>28.90</td>
    <td><b>31.00</b></td>
    <td>27.80</td>
    <td><b>32.10</b></td>
    <td>26.60</td>
    <td>30.80</td>
    <td><b>31.00</b></td>
  </tr>
  <tr>
    <td>spa</td>
    <td>45.80</td>
    <td>47.30</td>
    <td>51.90</td>
    <td>54.50</td>
    <td>54.10</td>
    <td><b>55.00</b></td>
    <td><b>57.10</b></td>
  </tr>
  <tr>
    <td>tur</td>
    <td>18.50</td>
    <td>29.70</td>
    <td>36.30</td>
    <td><b>36.70</b></td>
    <td>19.30</td>
    <td>26.20</td>
    <td><b>56.10</b></td>
  </tr>
  <tr>
    <td>Mean</td>
    <td>38.91</td>
    <td>43.16</td>
    <td>43.07</td>
    <td><b>44.98</b></td>
    <td>43.05</td>
    <td>43.98</td>
    <td><b>49.55</b></td>
  </tr>
</table>

Table 8: All language results for the ud-wl dataset
<table>
  <tr>
    <th></th>
    <th>t5-prefix</th>
    <th>cmlm-prefix</th>
    <th>t5-suffix</th>
    <th>cmlm-suffix</th>
    <th>t5-iid</th>
    <th>cmlm-iid</th>
    <th>AE</th>
  </tr>
  <tr><td>afb</td><td>56.50</td><td>54.50</td><td>58.30</td><td>56.80</td><td><b>59.40</b></td><td>55.70</td><td>55.00</td></tr>
  <tr><td>amh</td><td>27.90</td><td>28.50</td><td>23.70</td><td>23.30</td><td>27.90</td><td>28.80</td><td><b>38.50</b></td></tr>
  <tr><td>bel</td><td>47.60</td><td>48.50</td><td>48.40</td><td>48.50</td><td>49.60</td><td><b>49.80</b></td><td>48.70</td></tr>
  <tr><td>dan</td><td>79.10</td><td>79.90</td><td>76.70</td><td>80.70</td><td><b>81.50</b></td><td>80.40</td><td>77.90</td></tr>
  <tr><td>deu</td><td>58.50</td><td>58.80</td><td>60.50</td><td>58.90</td><td><b>63.20</b></td><td><b>62.50</b></td><td>56.50</td></tr>
  <tr><td>eng</td><td>87.20</td><td><b>89.90</b></td><td>87.30</td><td><b>89.80</b></td><td>89.30</td><td>88.80</td><td>89.10</td></tr>
  <tr><td>fin</td><td><b>42.30</b></td><td>35.70</td><td>36.10</td><td>35.30</td><td><b>41.80</b></td><td>38.80</td><td>32.90</td></tr>
  <tr><td>fra</td><td>56.50</td><td>43.10</td><td><b>60.50</b></td><td>51.30</td><td><b>57.90</b></td><td>53.80</td><td>52.00</td></tr>
  <tr><td>grc</td><td>30.00</td><td>26.00</td><td>29.20</td><td>28.00</td><td><b>32.10</b></td><td><b>31.30</b></td><td>18.10</td></tr>
  <tr><td>heb</td><td>14.70</td><td>13.70</td><td><b>17.42</b></td><td>16.21</td><td><b>18.93</b></td><td>12.99</td><td>15.41</td></tr>
  <tr><td>hun</td><td>55.60</td><td>57.00</td><td>55.30</td><td><b>62.40</b></td><td>61.70</td><td><b>62.20</b></td><td>53.70</td></tr>
  <tr><td>hye</td><td>72.70</td><td>69.60</td><td>74.40</td><td><b>76.60</b></td><td>74.00</td><td>73.40</td><td><b>78.30</b></td></tr>
  <tr><td>ita</td><td>39.50</td><td><b>48.50</b></td><td>46.60</td><td>47.20</td><td>46.70</td><td><b>48.30</b></td><td>47.30</td></tr>
  <tr><td>jap</td><td>13.70</td><td>13.40</td><td>16.70</td><td>13.50</td><td><b>18.40</b></td><td>17.10</td><td><b>42.40</b></td></tr>
  <tr><td>rus</td><td>65.90</td><td>70.20</td><td>70.30</td><td><b>70.80</b></td><td>69.20</td><td><b>71.90</b></td><td>68.30</td></tr>
  <tr><td>san</td><td>13.10</td><td><b>22.10</b></td><td>19.90</td><td>17.10</td><td>18.60</td><td>19.30</td><td><b>28.40</b></td></tr>
  <tr><td>sme</td><td>32.10</td><td>31.80</td><td>29.90</td><td><b>34.70</b></td><td>34.20</td><td><b>36.70</b></td><td>32.20</td></tr>
  <tr><td>spa</td><td>60.70</td><td>55.60</td><td>59.30</td><td>61.80</td><td><b>63.80</b></td><td><b>62.80</b></td><td>55.00</td></tr>
  <tr><td>tur</td><td>52.20</td><td>63.00</td><td>64.20</td><td><b>66.00</b></td><td><b>69.80</b></td><td>64.10</td><td>61.40</td></tr>
  <tr><td>Mean</td><td>47.67</td><td>47.88</td><td>49.20</td><td>49.42</td><td><b>51.48</b></td><td><b>50.46</b></td><td>50.06</td></tr>
</table>

Table 9: All language results for the ud-vnadj-NR dataset.

<table>
  <tr>
    <th></th>
    <th>t5-prefix</th>
    <th>cmlm-prefix</th>
    <th>t5-suffix</th>
    <th>cmlm-suffix</th>
    <th>t5-iid</th>
    <th>cmlm-iid</th>
    <th>AE</th>
  </tr>
  <tr><td>afb</td><td>55.80</td><td>52.70</td><td><b>56.60</b></td><td>56.00</td><td><b>56.40</b></td><td>54.60</td><td>55.70</td></tr>
  <tr><td>amh</td><td>28.30</td><td><b>36.10</b></td><td>33.00</td><td>34.20</td><td>31.90</td><td>35.80</td><td><b>40.40</b></td></tr>
  <tr><td>bel</td><td>49.20</td><td>49.10</td><td>47.10</td><td>49.30</td><td><b>49.40</b></td><td><b>51.10</b></td><td>48.60</td></tr>
  <tr><td>dan</td><td><b>81.40</b></td><td>80.00</td><td>81.00</td><td><b>81.40</b></td><td><b>81.30</b></td><td>81.20</td><td>79.60</td></tr>
  <tr><td>deu</td><td>56.80</td><td><b>63.10</b></td><td>56.40</td><td>61.30</td><td>61.70</td><td><b>64.60</b></td><td>57.70</td></tr>
  <tr><td>eng</td><td>87.70</td><td>87.80</td><td>86.60</td><td>88.10</td><td>88.30</td><td><b>89.50</b></td><td>89.40</td></tr>
  <tr><td>fin</td><td><b>39.30</b></td><td>32.10</td><td>34.70</td><td>33.40</td><td><b>41.90</b></td><td>36.70</td><td>33.70</td></tr>
  <tr><td>fra</td><td>55.70</td><td>55.20</td><td>51.50</td><td>53.90</td><td><b>56.20</b></td><td><b>59.60</b></td><td>51.40</td></tr>
  <tr><td>grc</td><td>28.00</td><td>28.20</td><td>28.30</td><td><b>31.70</b></td><td>30.60</td><td><b>32.10</b></td><td>22.60</td></tr>
  <tr><td>heb</td><td>18.43</td><td>15.01</td><td>17.22</td><td>16.31</td><td><b>20.95</b></td><td>17.62</td><td>17.42</td></tr>
  <tr><td>hun</td><td>56.70</td><td>52.70</td><td>53.90</td><td><b>59.80</b></td><td><b>63.10</b></td><td>58.60</td><td>53.50</td></tr>
  <tr><td>hye</td><td>76.10</td><td>71.90</td><td>76.10</td><td><b>77.30</b></td><td><b>78.30</b></td><td>73.40</td><td>75.30</td></tr>
  <tr><td>ita</td><td>36.30</td><td>43.90</td><td>46.10</td><td><b>49.00</b></td><td>46.30</td><td><b>50.00</b></td><td>48.80</td></tr>
  <tr><td>jap</td><td>11.20</td><td>11.20</td><td>14.60</td><td>13.30</td><td><b>16.80</b></td><td>16.80</td><td><b>34.70</b></td></tr>
  <tr><td>rus</td><td>65.20</td><td>68.90</td><td>68.70</td><td>67.40</td><td><b>70.80</b></td><td><b>70.20</b></td><td>69.70</td></tr>
  <tr><td>san</td><td>21.80</td><td><b>27.80</b></td><td>24.60</td><td>25.10</td><td>20.40</td><td>25.70</td><td><b>32.30</b></td></tr>
  <tr><td>sme</td><td>31.80</td><td><b>36.80</b></td><td>33.90</td><td><b>35.70</b></td><td>32.90</td><td><b>36.80</b></td><td>33.50</td></tr>
  <tr><td>spa</td><td>57.40</td><td>52.20</td><td>58.70</td><td><b>62.60</b></td><td>59.50</td><td><b>64.30</b></td><td>55.30</td></tr>
  <tr><td>tur</td><td>55.70</td><td>61.70</td><td><b>65.60</b></td><td>63.80</td><td><b>67.10</b></td><td>63.30</td><td>59.70</td></tr>
  <tr><td>Mean</td><td>48.04</td><td>48.76</td><td>49.19</td><td>50.51</td><td><b>51.26</b></td><td><b>51.68</b></td><td>50.49</td></tr>
</table>

Table 10: All language results for the ud-wl-NR dataset.
<table>
  <tr>
    <th></th>
    <th>eng</th>
    <th>hun</th>
    <th>ita</th>
    <th>rus</th>
    <th>spa</th>
    <th>Mean</th>
  </tr>
  <tr>
    <td>t5-prefix</td>
    <td>80.10</td>
    <td>46.60</td>
    <td>13.90</td>
    <td>56.50</td>
    <td>35.30</td>
    <td>46.48</td>
  </tr>
  <tr>
    <td>cmlm-prefix</td>
    <td>88.70</td>
    <td>56.20</td>
    <td>34.90</td>
    <td>63.50</td>
    <td>47.80</td>
    <td>58.22</td>
  </tr>
  <tr>
    <td>t5-suffix</td>
    <td>88.50</td>
    <td>64.50</td>
    <td>44.00</td>
    <td>60.50</td>
    <td>49.80</td>
    <td>61.46</td>
  </tr>
  <tr>
    <td>cmlm-suffix</td>
    <td>87.30</td>
    <td>62.40</td>
    <td>45.40</td>
    <td>61.50</td>
    <td>57.20</td>
    <td>62.76</td>
  </tr>
  <tr>
    <td>t5-iid</td>
    <td>75.90</td>
    <td>51.10</td>
    <td>27.40</td>
    <td>58.20</td>
    <td>41.20</td>
    <td>50.76</td>
  </tr>
  <tr>
    <td>cmlm-iid</td>
    <td>81.10</td>
    <td>58.40</td>
    <td>32.90</td>
    <td>59.20</td>
    <td>46.50</td>
    <td>55.62</td>
  </tr>
  <tr>
    <td>t5-seg-prefix</td>
    <td>85.90</td>
    <td>55.20</td>
    <td>44.90</td>
    <td>66.30</td>
    <td>58.60</td>
    <td>62.18</td>
  </tr>
  <tr>
    <td>t5-seg-suffix</td>
    <td>89.60</td>
    <td>61.60</td>
    <td>51.90</td>
    <td>69.20</td>
    <td>59.30</td>
    <td>66.32</td>
  </tr>
  <tr>
    <td>t5-seg-iid</td>
    <td>89.20</td>
    <td>66.00</td>
    <td>48.60</td>
    <td>71.60</td>
    <td>61.60</td>
    <td>67.40</td>
  </tr>
  <tr>
    <td>cmlm-seg-prefix</td>
    <td>85.00</td>
    <td>56.90</td>
    <td>58.20</td>
    <td>65.40</td>
    <td>58.40</td>
    <td>64.78</td>
  </tr>
  <tr>
    <td>cmlm-seg-suffix</td>
    <td><b>90.00</b></td>
    <td>65.60</td>
    <td>44.20</td>
    <td><b>73.20</b></td>
    <td>59.40</td>
    <td>66.48</td>
  </tr>
  <tr>
    <td>cmlm-seg-iid</td>
    <td>89.00</td>
    <td><b>66.10</b></td>
    <td>47.80</td>
    <td>69.60</td>
    <td><b>61.70</b></td>
    <td><b>66.84</b></td>
  </tr>
</table>

Table 11: All language results for the segmentation dataset when deleting rather than masking.

Reviewer paper summaries: This paper addresses the task of low-resource morphological inflection and its improvement by means of additional language modeling objectives. The authors compare several additional tasks, such as autoencoding, BERT-like masking and T5-like masking and different variations of these tasks. They show that autoencoding outperforms masked language modeling in case of sampling training data with replacement, while the opposite holds when sampling without replacement. Additionally, using morpheme-constrained masking slightly improves performance for MLM masking strategy.

This article studies several self-supervision methods to improve performance for morphological inflection over 19 languages. 
The experimental setting constrains the amount of labelled and unlabelled words to simulate a low-resource scenario. 
The authors compare two established masked language modelling objectives: character-level MLM (CMLM) and the span-based objective from T5. Besides, three masking and deletion strategies are also implemented, aside from a more informed masking strategy based on actual morpheme boundaries.
Evaluation of different sample datasets from Universal Dependencies shows that if autoencoding remains a strong baseline, there are cases where using a more appropriate denoising method leads to better results.
Summary of strengths: * a thorough comparison of multiple methods of character language modeling in the task of morphological inflecton
* consistent improvement over baseline

- The article is well written and structured, which helps given the number of possible combinations of masking objectives and strategies (and datasets).
- There are extensive experiments and analyses spanning several masking strategies and languages, as evidenced by the full results in the appendix.
- I appreciated the variety in the strategies, but specifically seeing the more linguistically motivated masking based on morpheme boundaries.
Summary of weaknesses: * most results are method-dependent. In general, it is very hard to distinguish between main results of the paper and marginal ones.
* the work seems to be incremental with respect to Purushothama et al. (2024). However, the main contribution -- the superiority of CMLM approach in case of using more unlabeled data and sampling with replacement -- is new and doesn't follow from the previous work.

- I wonder how realistic the simulated low-resource scenario is, as mentioned in the limitations. Is the amount of samples in this study really reflecting a low-resource scenario in Universal Dependency for morphological inflection? 
- The studied languages are mostly from otherwise better-resourced languages, which are not the most morphologically rich languages. The chosen languages, however, come from an article on which the authors base their work.
- It seems difficult to draw an overall conclusion from the results. We see the strength of autoencoding (baseline) and some cases where MLM is better, but there is no specific consistency in the masking objective and method.
Comments, suggestions and typos: * Please state more clearly what is your improvement over Purushothama et al. (2024)

Typos:
* l.258 *In all other experiments, we simulate a lower-resource setting than that work* -- I don't understand what it means.
* l.267-268 variety ... IS attested

- I am not sure I understood this part correctly. When you mention that you sample the dataset without replacement (section 5.1), does it really mean that both the inflected and non-inflected forms are repeated in the previous datasets, or is it just the lemma? I hence didn't understand why the number of samples and types didn't match in the last three datasets: one explanation for me is that the types concern the word lemmata, but why are there systematically 5,320 types on average in all three cases? I am also not sure I got why there are exactly 5,360 samples (on average) for all datasets except ud-1k.
- A general but minor comment I have is that I found it a bit difficult to identify datasets in the text. Could another font be used (similarly to self-supervision strategies, but differently)?
- Figure 3: the two figures would be easier to read and understand when the actual numbers are indicated, given how close the bars can be. This could be done either as a table or with the numbers in the bar, as in Figure 5.
- Do you have any hypotheses on why segmentation helps for the CMLM objective and not for the T5 objective?
- One conclusion you draw is that using a MLM strategy brings more improvement when there are more unlabelled samples. I believe this stems from your experiments with different datasets (e.g., table 3). Yet, a better way of testing this hypothesis seems to be to try with actually more unlabelled data, especially for languages for which it is possible. Did you do some experiments in that direction (even with a partial coverage of languages)?

Typos:
- l.173: from the vocabulary
- l.527: that that
- l.552: its inductive bias
Overall assessment scores: 3.5

4