SlowGuess/ACL_2020_OCR huggingface dataset:
Huiming Jin, Liwei Cai, Yihui Peng, Chen Xia
{huimingj, liweicai, yihuip, chenxia}@cs.cmu.edu
Arya D. McCarthy
arya@jhu.edu
We propose the task of unsupervised morphological paradigm completion. Given only raw text and a lemma list, the task consists of generating the morphological paradigms, i.e., all inflected forms, of the lemmas. From a natural language processing (NLP) perspective, this is a challenging unsupervised task, and high-performing systems have the potential to improve tools for low-resource languages or to assist linguistic annotators. From a cognitive science perspective, this can shed light on how children acquire morphological knowledge. We further introduce a system for the task, which generates morphological paradigms via the following steps: (i) EDIT TREE retrieval, (ii) additional lemma retrieval, (iii) paradigm size discovery, and (iv) inflection generation. We perform an evaluation on 14 typologically diverse languages. Our system outperforms trivial baselines with ease and, for some languages, even obtains a higher accuracy than minimally supervised systems.<sup>1</sup>
Morphologically rich languages express syntactic and semantic properties—like tense or case—of words through inflection, i.e., changes to the surface forms of the words. The set of all inflected forms of a lemma—the canonical form—is called its paradigm. While English does not manifest a rich inflectional morphology, Polish verbs have around a hundred different forms (Sadowska, 2012), and Archi paradigms, an extreme example, can have over 1.5 million slots (Kibrik, 1977).
Morphologically rich languages constitute a challenge for natural language processing (NLP) systems: because each lemma can take on a variety of surface forms, the frequency of each individual inflected word decreases drastically. This yields problems for speech recognition (Creutz
Katharina Kann
University of Colorado katharina.kann@colorado.edu
Figure 1: Our unsupervised paradigm completion system, which takes raw text and a lemma list as inputs. We describe it in detail in §4.
et al., 2007), parsing (Seeker and Cetinoğlu, 2015), and keyword spotting (Narasimhan et al., 2014), inter alia. For unsupervised machine translation, Guzmán et al. (2019) encounter difficulties when translating into the morphologically rich languages Nepalese and Sinhalese.
Children acquire morphological knowledge from raw utterances and, in particular, without access to explicit morphological information (Berko, 1958). Do they have an innate capacity that enables them to learn a language's morphology? Or can morphology be learned in an unsupervised fashion? This question—in addition to practical considerations like benefits for the aforementioned NLP tasks—has motivated work on unsupervised morphological analyses (Goldsmith, 2001; Creutz, 2003). To the best of our knowledge, no previous work has considered unsupervised morphological generation. However, over the last few years, there has
$^{2}$ Kann et al. (2017), which performs a zero-shot inflection experiment, uses prior information about paradigm size and related languages and, thus, cannot draw the same conclusions.
1Our implementation is available under https://github. com/cai-lw/morpho-baseline.
6696
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6696-6707
been a lot of progress on morphological generation tasks with limited amounts of supervision, in particular on morphological inflection (Cotterell et al., 2018) and paradigm completion (Kann and Schütze, 2018), which can potentially be leveraged for unsupervised solutions.
Here, we fill the gap between unsupervised morphological analysis and morphological generation with limited training data by proposing the task of unsupervised morphological paradigm completion. That is, we aim to construct and fill inflection tables exclusively from raw text and a lemma list for a known part of speech (POS), in a situation similar to those encountered by field linguists. We further present a system for the task (see Figure 1) which employs state-of-the-art methods common in NLP and computational morphology. It performs the following four steps: (i) EDIT TREE (Chrupa, 2008) retrieval (§4.1), (ii) additional lemma retrieval (§4.2), (iii) paradigm size discovery using distributional information (§4.3), and (iv) inflection generation (§4.4).
To evaluate our approach, we design a metric for unsupervised paradigm completion, best-match accuracy (§5.4), and experiment on 14 languages from 7 families. As we are tackling a novel task with no baselines in the NLP literature, we perform an extensive ablation study to demonstrate the importance of all steps in our pipeline. We further show that our system outperforms trivial baselines and, for some languages, even obtains higher accuracy than a minimally supervised system.
Morphological Generation Versions of our task with varying degrees of supervision—though never totally unsupervised—have been explored in the past. Yarowsky and Wicentowski (2000) is the previous work most similar to ours. They also assume raw text and a word list as input, but additionally require knowledge of a language's consonants and vowels, as well as canonical suffixes for each part of speech. Dreyer and Eisner (2011) assume access to seed paradigms to discover paradigms in an empirical Bayes framework. Ahlberg et al. (2015) and Hulden et al. (2014) combine information about paradigms and word frequency from corpora to perform semi-supervised paradigm completion. Our work differs from them in that we do not assume any gold paradigms to be given.
Durrett and DeNero (2013), Nicolai et al. (2015), and Faruqui et al. (2016) explore a fully supervised approach, learning morphological paradigms from large annotated inflection tables. This framework has evolved into the SIGMORPHON shared tasks on morphological inflection (Cotterell et al., 2016), which have sparked further interest in morphological generation (Kann and Schütze, 2016; Aharoni and Goldberg, 2017; Bergmanis et al., 2017; Makarov et al., 2017; Zhou and Neubig, 2017; Kann and Schütze, 2018). We integrate two systems (Cotterell et al., 2017; Makarov and Clematide, 2018b) produced for SIGMORPHON shared tasks into our framework for unsupervised morphological paradigm completion.
Morphological Analysis Most research on unsupervised systems for morphology aims at developing approaches to segment words into their smallest meaning-bearing units, called morphemes (Goldsmith, 2001; Creutz, 2003; Creutz and Lagus, 2007; Snyder and Barzilay, 2008). Unsupervised morphological paradigm completion differs from segmentation in that, besides capturing how morphology is reflected in the word form, it also requires correctly clustering transformations into paradigm slots as well as generating unobserved forms. The model by Xu et al. (2018) recovers something akin to morphological paradigms. However, those paradigms are a means to a segmentation end, and Xu et al. (2018) do not explicitly model information about the paradigm size as required for our task.
Other unsupervised approaches to learning morphological analysis and generation rely on projections between word embeddings (Soricut and Och, 2015; Narasimhan et al., 2015); however, these approaches rely on billions of words to train embeddings; at a minimum, Narasimhan et al. (2015) use 129 million word tokens of English Wikipedia. As we will describe later on (§5.1), we, in contrast, are concerned with the setting with mere thousands of sentences.
For a detailed survey of unsupervised approaches to problems in morphology, we refer the reader to Hammarström and Borin (2011).
SIGMORPHON 2020: Unsupervised Morphological Paradigm Completion After multiple shared tasks on morphological inflection starting with Cotterell et al. (2016), in 2020, SIGMORPHON (the ACL special interest group on computational morphology and phonology) is organizing
6697
its first shared task on unsupervised morphological paradigm completion. $^3$  The system presented here is the official shared task baseline system. The first other approach applicable to this shared task has been developed by Erdmann et al. (2020). Their pipeline system is similar in spirit to ours, but the individual components are different, e.g., a transformer model (Vaswani et al., 2017) is used for inflection generation.
3 Formal Task Description
Given a corpus  $\mathcal{D} = w_{1},\ldots ,w_{|\mathcal{D}|}$  with a vocabulary  $V$  of word types  $\{w_i\}$  and a lexicon  $\mathcal{L} = \{\ell_j\}$  with  $|\mathcal{L}|$  lemmas belonging to the same part of speech, the task of unsupervised morphological paradigm completion consists of generating the paradigms  $\{\pi (\ell)\}_{\ell \in \mathcal{L}}$  of the entries in the lexicon.
Following Matthews and Matthews (1972) and Aronoff (1976), we treat a paradigm as a vector of inflected forms belonging to a lemma  $\ell$ . Paradigm completion consists of predicting missing slots in the paradigm  $\pi(\ell)$ :
$$
\pi (\ell) = \left\langle f (\ell , \vec {t} _ {\gamma}) \right\rangle_ {\gamma \in \Gamma (\ell)}, \tag {1}
$$
where  $f:\Sigma^{*}\times \mathcal{T}\to \Sigma^{*}$  transforms a lemma into an inflected form,  $\vec{t}_{\gamma}\in \mathcal{T}$  is a vector of inflectional features describing paradigm slot  $\gamma$  , and  $\Gamma (\ell)$  is the set of slots in lemma  $\ell$  's paradigm. Since we only consider lemmas that belong to the same part of speech, we will use  $\Gamma$  and  $\Gamma (\ell)$  interchangeably in the following. Furthermore, we will denote  $f(\ell ,\vec{t}_{\gamma})$  as  $f_{\gamma}(\ell)$  for simplicity.
Remarks on Task Design In general, not all paradigm entries will be present in the corpus  $\mathcal{D}$ . Thus, the task requires more than a keyword search.
On another note, it is not necessary to predict the features  $\vec{t}_{\gamma}$  corresponding to each slot  $\gamma$ ; as the exact denotation of features is up to human annotators, they cannot be inferred by unsupervised systems. For our task, it is enough to predict the ordered vector  $\pi(\ell)$  of inflected forms.
Our system implicitly solves two subtasks: (i) determining the number of paradigm slots; and (ii) generating the inflected form corresponding to each paradigm slot for each lemma. It is organized as a
Figure 2: Visualization of the EDIT TREE constructed from najtrudniejszy to trudny (Chrupa, 2008).
pipeline consisting of multiple components. Our system is highly modular: individual components can be exchanged easily. In the remainder of this section, we will dedicate each subsection to one component.
4.1 Retrieval of Relevant EDIT TREES
The first component in our pipeline identifies words in the corpus  $\mathcal{D}$  which could belong to the paradigm of one of the lemmas in the lexicon  $\mathcal{L}$ . We call those words paradigm candidates. It then uses the discovered paradigm candidates to identify EDIT TREE (Chrupa, 2008) operations that correspond to valid inflections.
Paradigm Candidate Discovery For most paradigms, all participating inflected forms share some characteristics—usually substrings—which humans use to identify the paradigm any given word belongs to. Given a pair  $(\ell, w)$  of a lemma  $\ell$  and a word form  $w$ , the first step in our pipeline is to determine whether  $w$  is a paradigm candidate for lemma  $\ell$ . For example, studied is likely to be an inflected form of the English lemma study, while monkey is not. We identify paradigm candidates  $C_{\ell}$  of a lemma  $\ell$  by computing the longest common substring (LCS) between  $\ell$  and  $w$  for all words  $w$  in the vocabulary  $V$ . If the ratio between the LCS's length and the length of  $\ell$  is higher than a threshold  $\lambda_{P}$ ,  $w$  is a paradigm candidate for  $\ell$ :
$$
C _ {\ell} = \left\{w \in V \left| \frac {| \operatorname {L C S} (\ell , w) |}{| \ell |} > \lambda_ {P} \right. \right\}. \tag {2}
$$
EDIT TREE Discovery Surface form changes, which we denote as  $\psi$ , define a modification of a word's surface form. Our system employs EDIT TREES (Chrupa, 2008) to represent  $\psi$ .
Given two strings  $x_{[1\dots n]}$  and  $x_{[1\dots m]}^{\prime}$ , EDIT TREE  $(x_{[1\dots n]}, x_{[1\dots m]}^{\prime})$  is constructed by first determining the LCS between  $x$  and  $x^{\prime}$ . We then recursively model the substrings before and after the LCS. If the length of the LCS is zero, the
3https://sigmorphon.github.io/sharedtasks/2020/task2/
4We assume a discrete alphabet of symbols  $\Sigma$
6698
EDIT TREE consists of the substitution operation of the first string with the second. For example, EDIT TREE(najtrudniejszy, trudny) $^5$  could be visualized as in Figure 2, where  $\mathrm{Split}(i,j)$  represents taking the substring  $x_{[i\dots n - j]}$ .
An EDIT TREE can be applied to new input strings. The EDIT TREE in Figure 2, for example, could be applied to najappleiejszs, and the resulting string would be apples. Note that not all EDIT TREES can be applied to all strings. For example, the EDIT TREE in Figure 2 can only be applied to words starting with naj.
Our system constructs EDIT TREES from all pairs  $(\ell, w)$  of lemmas  $\ell$  and their paradigm candidates  $w$  and counts their frequencies:
$$
n _ {\psi} = \sum_ {\substack {\ell \in \mathcal {L} \\ w \in C _ {\ell}}} \mathbb {1} \left[ \operatorname {E D I T T R E E} (\ell , w) = \psi \right]. \quad (3)
$$
It then discards EDIT TREES with frequencies  $n_{\psi}$  below a threshold  $\lambda_{FC}(|\mathcal{L}|)$ , which is a function of the size of the lexicon  $\mathcal{L}$ :
$$
\lambda_ {F C} (| \mathcal {L} |) = \max  \left\{2, \phi_ {F C} \cdot | \mathcal {L} | \right\}, \tag {4}
$$
where  $\phi_{FC} \in \mathbb{R}$  is a hyperparameter. The idea is that an EDIT TREE is only valid if it can be applied to multiple given lemmas. EDIT TREES which we observe only once are always considered unreliable. Our system then retains a set of frequent surface form changes
$$
\Psi = \left\{\psi \mid n _ {\psi} \geq \lambda_ {F C} (| \mathcal {L} |) \right\} \tag {5}
$$
represented by EDIT TREES. Assuming an one-to-one mapping between surface form changes and paradigm slots (that is, that  $|\Psi| = |\Gamma|$  and that each  $\psi$  is equivalent to a particular inflection function  $f_{\gamma}$ ), we now have a first basic paradigm completion system (PCS-I), which operates by applying all suitable EDIT TREES to all lemmas in our lexicon.
Complexity and Runtime The time complexity to compute EDIT TREE  $(x_{[1\dots n]},x_{[1\dots m]}^{\prime})$  is  $O(\max^{3}\{n,m\})$ . Computing EDIT TREES can trivially be parallelized, and in practice this computation does not take much time.
4.2 Retrieval of Additional Lemmas
Since we assume a low-resource setting, our lexicon is small ( $\leq 100$  entries). However, the more
lemmas we have, the more confident we can be that the EDIT TREES retrieved by the first component of our system represent valid inflections. An intuitive method to obtain additional lemmas would be to train a lemmatizer and to generate new lemmas from words in our corpus. However, due to the limited size of our initial lemma list, such a lemmatizer would most likely not be reliable.
The second component of our system employs another method, which guarantees that additionally retrieved lemmas are valid words: It is based on the intuition that a word  $w \in V$  is likely to be a lemma if the pseudo-inflected forms of  $w$ , obtained by applying the EDIT TREES from §4.1, also appear in  $V$ . For a word  $w \in V$ , we say it is a discovered lemma if  $w \notin \mathcal{L}$  and
$$
\sum_ {\psi \in \Psi} \mathbb {1} [ \psi (w) \in V ] > \lambda_ {N L} (| \Psi |) \tag {6}
$$
$$
\lambda_ {N L} (| \Psi |) = \max  \left\{3, \phi_ {N L} \cdot | \Psi | \right\}, \tag {7}
$$
with  $\phi_{NL} \in \mathbb{R}$  being a hyperparameter. Similar to Equation 4,  $\lambda_{NL}$  depends on the number of discovered EDIT TREES, but is never smaller than 3. We set this minimum to require evidence for at least two transformations in addition to the identity.
We can now bootstrap by iteratively computing additional paradigm candidates and EDIT TREES, and then retrieving more lemmas. We denote the paradigm completion systems resulting from one and two such iterations as PCS-II-A and PCS-II-B, respectively. Since we cannot be fully confident about retrieved lemmas, we associate each additional lemma with a weight  $\theta_{\ell} = \theta_{NL}^{it}$ , where  $\theta_{NL}$  is a preset hyperparameter and  $it$  identifies the iteration in which a lemma is added, i.e.,  $it = 0$  for gold lemmas and  $it = i$  for lemmas retrieved in the  $i$ th iteration. The weights  $\theta_{\ell}$  are used in later components of our system.
4.3 Paradigm Size Discovery
Until now, we have assumed a one-to-one mapping between paradigm slots and surface form changes. However, different EDIT TREES may indeed represent the same inflection. For example, the past tense inflection of verbs in English involves multiple EDIT TREES, as shown in Figure 3.
Thus, the next step is to group surface form changes based on the paradigm slots they realize.
5Polish form-lemma pair meaning hardest and hard. Example from Chrupala (2008).
(b)
Figure 3: Visualization of the EDIT TREES representing (a) work  $\mapsto$  worked and (b) continue  $\mapsto$  continued.
4.3.1 One EDIT TREE per Lemma and Slot
Our algorithm for grouping surface form changes is based on two assumptions. First, since EDIT TREES are extracted from  $(\ell, w)$  pairs, different EDIT TREES belonging to the same paradigm slot cannot be extracted from the same lemma.<sup>6</sup> Thus:
Assumption 1 For each lemma, at most one inflected form per paradigm slot can be found in the corpus.
Formally, for a multi-to-one mapping from EDIT TREES to paradigm slots  $z:\Psi \to \Gamma$ , we define the EDIT TREE set  $\Psi_{\gamma}$  of a potential paradigm slot  $\gamma$  as  $\Psi_{\gamma} = \{\psi \mid z(\psi) = \gamma \}$ , with  $\bigcup_{\gamma}\Psi_{\gamma} = \Psi$ . Then, for any lemma  $\ell \in \mathcal{L}$  and proposed paradigm slot  $\gamma \in \Gamma$ , we have:
$$
\left| \left\{w \in V \mid \psi (\ell) = w \wedge \psi \in \Psi_ {\gamma} \right\} \right| \leq 1. \tag {8}
$$
4.3.2 One Paradigm Slot per EDIT TREE
Our second assumption is a simplification, but helps to reduce the search space during clustering:
Assumption 2 Each surface form change  $\psi \in \Psi$  belongs to exactly one paradigm slot.
Formally, we partition  $\Psi$  into disjoint subsets:
$$
\Psi_ {\gamma} \cap \Psi_ {\gamma^ {\prime}} = \emptyset \quad \forall \gamma \neq \gamma^ {\prime}. \tag {9}
$$
4.3.3 Paradigm Slot Features and Similarity
In addition to Assumptions 1 and 2, we make use of a feature function  $r(\gamma)$  and a score function  $s(r(\gamma), r(\gamma'))$ , which measures the similarity between two potential paradigm slots.
Our feature function makes the connection between paradigm slots and the instances of inflected forms in the corpus by utilizing the part-of-speech (POS) tags as context information. In our implementation, we employ an unsupervised POS tagger
Figure 4: An example of the distributionally informed feature function with window size 3 for the past tense slot (PST). stop  $\in \mathcal{L}$  and  $f_{\mathrm{PST}}(stop) = stopped$ . When the sliding window arrives to this instance of stopped,  $\vec{r}_{[\mathbb{N},\mathbb{V},\mathbb{V}]}^{\mathrm{PST}}$  is increased by 1.8
(Stratos et al., 2016) to extract tags for each word in the corpus  $\mathcal{D}$ .
This tagger assigns an anchor word, i.e., a pseudo POS tag, to each word  $w_{i}$  in the corpus by using an anchor hidden Markov model (HMM) with 8 hidden states. Our feature function counts the tag tuples within a sliding window centered on each instance of inflected forms of a potential paradigm slot.
Formally, we denote the set of lemmas that surface form change  $\psi$  is applied to as  $\mathcal{L}_{\psi}$ , the set of lemmas that express a potential paradigm slot  $\gamma$  as  $\mathcal{L}_{\gamma}$ , and the corresponding inflected forms as  $V_{\gamma}$ :
$$
\mathcal {L} _ {\psi} = \{\ell \in \mathcal {L} | \psi (\ell) = w \wedge w \in V \} \tag {10}
$$
$$
\mathcal {L} _ {\gamma} = \bigcup_ {\psi \in \Psi_ {\gamma}} \mathcal {L} _ {\psi} \tag {11}
$$
$$
V _ {\gamma} = \bigcup_ {\psi \in \Psi_ {\gamma}} \left\{\psi (\ell) \right\} _ {\ell \in \mathcal {L} _ {\psi}}. \tag {12}
$$
We further refer to the set of available POS tag labels as  $P = \{p_1, \ldots, p_8\}$ . For a corpus  $\mathcal{D} = w_1, \ldots, w_{|\mathcal{D}|}$ , a window size  $2d + 1$ , and a potential paradigm slot  $\gamma$ , our feature function is defined as:
$$
\vec {r} ^ {\gamma} = r (\gamma ; \mathcal {D}, 2 d + 1), \tag {13}
$$
where  $\vec{r}^{\gamma}$  is a vector with one dimension for each possible tag tuple sequence of length  $2\mathrm{d} + 1$ . Its values are computed as:
$$
\vec {r} ^ {\gamma} \left[ p _ {j _ {1}}, \dots , p _ {j _ {2 d + 1}} \right] = \sum_ {w _ {i} \in \mathcal {D}} \mathbb {1} \left[ w _ {i} \in V _ {\gamma} \right. \tag {14}
$$
$$
\begin{array}{l} \left. \wedge \hat {p} _ {i - d} = p _ {j _ {1}} \wedge \dots \wedge \hat {p} _ {i + d} = p _ {j _ {2 d + 1}} \right] \\ \forall \left(\hat {p} _ {j _ {1}}, \dots , \hat {p} _ {j _ {2 d + 1}}\right) \in P ^ {2 d + 1}. \\ \end{array}
$$
In practice, we initialize  $\vec{r}^{\gamma}$  to the zero vector, and iterate the sliding window over the corpus. When the central word  $w_{i}\in V_{\gamma}$ , the corresponding value of  $\vec{r}^{\gamma}$  at the POS tuple within the sliding window is incremented by 1. Figure 4 shows an example in English.
We assume that paradigm slots  $\gamma$  and  $\gamma^\prime$  are similar if the words in  $V_{\gamma}$  and  $V_{\gamma^{\prime}}$  frequently appear in
Exceptions to this do exist. However, they are rare enough that we do not expect them to hurt our algorithm.
This assumption ignores syncretism.
Note that this example is simplified. The system does not need to actually know the tags like PST and V.
similar contexts, i.e., within similar tag sequences. With the feature function defined above, our system uses cosine similarity as the score function  $s$ .
We then develop Algorithm 1 to group one-to-one mappings from surface form changes to paradigm slots into many-to-one mappings. The idea is to iteratively merge the most similar slots if this is not violating Assumption 1 until the similarity gets too low.  $\lambda_S\in (0,1)$  is a threshold parameter.
4.4 Generation
Now, one paradigm slot can be represented by multiple EDIT TREES. Our system, thus, needs to learn to apply the correct transformation for a combination of lemma and paradigm slot. However, mapping lemmas and paradigm slots to inflected forms corresponds exactly to the morphological inflection task, which has been the subject of multiple shared tasks over the last years (Cotterell et al., 2018).
Our morphological inflection models take (slot, lemma, word) tuples extracted by the previous components of our system as training data. Formally, they are trained on the training set:
$$
\left\{\left(\vec {t} _ {\gamma}, \ell , f _ {\gamma} (\ell)\right) \mid \gamma \in \Gamma \wedge \ell \in \mathcal {L} _ {\gamma} \wedge f _ {\gamma} (\ell) \in V \right\}. \tag {15}
$$
We explore two morphological inflection models from the literature.
Affix Editing The baseline system of the CoNLL-SIGMORPHON 2017 shared task $^{10}$  (Cotterell et al., 2017) is a simple approach, which is very suitable for low-resource settings. The system breaks each word into Prefix, STEM, and SUFFIX, and then stores the Prefix editing rules and the STEM+SUFFIX editing rules. At test time, it applies the longest possible Prefix and STEM+SUFFIX editing rules to the input lemma. We denote the surface form change grouping in combination with this system as PCS-III-C.