SlowGuess/ACL_2020_OCR huggingface dataset:
Alexander Erdmann³,⁰ Micha Elsner³ Shijie Wu³ Ryan Cotterell¹,² Nizar Habash³
<sup>3</sup>Ohio State University <sup>5</sup>New York University Abu Dhabi  
<sup>3</sup>Johns Hopkins University <sup>4</sup>University of Cambridge <sup>1</sup>ETH Zürich  
{ae1541, nizar.habash}@nyu.edu elsner.14@osu.edu  
shijie.wu@jhu.edu ryan.cotterell@inf.ethz.ch
This work treats the paradigm discovery problem (PDP)—the task of learning an inflectional morphological system from unannotated sentences. We formalize the PDP and develop evaluation metrics for judging systems. Using currently available resources, we construct datasets for the task. We also devise a heuristic benchmark for the PDP and report empirical results on five diverse languages. Our benchmark system first makes use of word embeddings and string similarity to cluster forms by cell and by paradigm. Then, we bootstrap a neural transducer on top of the clustered data to predict words to realize the empty paradigm slots. An error analysis of our system suggests clustering by cell across different inflection classes is the most pressing challenge for future work. Our code and data are available at https://github.com/alexerdmann/ParadigmDiscovery.
In childhood, we induce our native language's morphological system from unannotated input. For instance, we learn that ring and rang belong to the same inflectional paradigm. We also learn that rings and bangs belong to the same cell, i.e., they realize the same morphosyntactic properties 3.SG.PRES, but in different paradigms. Acquiring such paradigmatic knowledge enables us to produce unseen inflectional variants of new vocabulary items, i.e. to complete morphological paradigms. Much work has addressed this task, which Ackerman et al. (2009) call the paradigm cell filling problem (PCFP),<sup>1</sup> but few have discussed inducing paradigmatic knowledge from scratch, which we call the paradigm discovery problem (PDP).<sup>2</sup>
As an unsupervised task, the PDP poses challenges for modeling and evaluation and has yet to be attempted in its full form (Elsner et al., 2019). However, we contend there is much to be gained from formalizing and studying the PDP. There are insights for cognitive modeling to be won (Pinker, 2001; Goldwater, 2007) and intuitions on combatting sparse data for language generation (King and White, 2018) to be accrued. Unsupervised language processing also has natural applications in the documentation of endangered languages (Zamaraeva et al., 2019) where a lot of annotated data is never likely to exist. Our formalization of the PDP offers a starting point for future work on unsupervised morphological paradigm completion.
Our paper presents a concrete formalization of the PDP. Then, as a baseline for future work, we introduce a heuristic benchmark system. Our benchmark system takes an unannotated text corpus and a lexicon of words from the corpus to be analyzed. It first clusters the lexicon by cell and then by paradigm making use of distributional semantics and string similarity. Finally, it uses this clustering as silver-standard supervision to bootstrap a neural transducer (Vaswani et al., 2017) that generates the desired target inflections. That is, the model posits forms to realize unoccupied cell slots in each proposed paradigm. Even though our benchmark system models only one part of speech (POS) at a time, our framework extends to the full PDP to support future, more intricate systems. We propose two separate metrics to evaluate both the clustering of attested forms into paradigms and cells and the prediction of unseen inflected forms. Our metrics handle non-canonical morphological behavior discussed in theoretical literature (Corbett, 2005) and extend to the full PDP.
For three of the five languages we consider, our benchmark system predicts unattested inflections
one of its subtasks which Boyé and Schalchli (2019) call the paradigm cell finding problem (see §2.2).
<sup>1</sup>In the NLP literature, this task is called morphological reinflection or morphological inflection generation (Cotterell et al., 2016a); this is only a difference in nomenclature.
$^{2}$ Elsner et al. (2019) call the task the paradigm cell discovery problem; we drop cell to distinguish our task from
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7778-7790 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
of lexicon forms with accuracy within  $20\%$  of a fully supervised system. However, our analysis suggests clustering forms into cells consistently across paradigms is still a very pressing challenge.
2 Previous Work in Morphology
This section couches our work on the PDP in terms of previous trends in morphological modeling.
2.1 Unsupervised Morphology
Much work on unsupervised morphological modeling focuses on segmentation (Gaussier, 1999; Goldsmith, 2001; Creutz and Lagus, 2005; Narasimhan et al., 2015; Bergmanis and Goldwater, 2017; Xu et al., 2018). While morphological segmenters can distinguish real from spurious affixes (e.g., bring  $\neq br + ing$ ) with high accuracy, they do not attempt to solve the PDP. They do, however, reveal which forms take the same affixes (e.g., walked, talked), not which forms occupy the same cell (e.g., walked, brought). Indeed, they explicitly struggle with irregular morphology. Segmenters also cannot easily model non-concatenative phenomena like ablaut, vowel harmony and templatic processes.
Two works have proposed tasks which can be considered alternative formulations of the PDP, using either minimal or indirect supervision to bootstrap their models. We discuss each in turn. First, Dreyer and Eisner (2011) use a generative model to cluster forms into paradigms and cells with a Bayesian non-parametric mixture of weighted finite-state transducers. They present a PDP framework which, in principle, could be fully unsupervised, but their model requires a small seed of labeled data to get key information like the number of cells distinguished, making it less relevant cognitively. In contrast, our task is not directly supervised and focuses on distributional context. Second, contemporaneous to our work, Jin et al. (2020) propose a similar framework for SIGMORPHON 2020's shared task on unsupervised morphological paradigm completion. Given only a small corpus and lexicon of verbal lemmata, participating systems must propose full paradigms for each lemma. By contrast, our framework does not reveal how many paradigms should be generated, nor do we privilege a specific form as the lemma, but we do use a larger lexicon of exclusively verbal or nominal forms. Their proposed baseline uses distributional context for POS tagging and features, but does not train embeddings as the corpus is small.
2.2 Subtasks of Paradigm Discovery
A few works address subtasks of the PDP. Erdmann and Habash (2018) learn paradigm membership from raw text, but do not sort paradigms into cells. Boyé and Schalchli (2019) discuss the paradigm cell finding problem, identifying the cell (but not paradigm) realized by a given form. Lee (2015) clusters forms into cells across inflection classes. Beniamine et al. (2018) group paradigms into inflection classes, and Eskander et al. (2013) induce inflection classes and lemmata from cell labels.
2.3 The Paradigm Cell Filling Problem
The PCFP is the task of predicting unseen inflected forms given morphologically labeled input. PCFP models can guess a word's plural having only seen its singular, but the child must bootstrap morphological knowledge from scratch, first learning that singular-plural is a relevant distinction. Thus, the PDP must be at least partially solved before the PCFP can be attempted. Yet, as a supervised task, the PCFP is more easily studied, and has received much attention on its own, especially from the word-and-paradigm camp of morphological theory.
Some cognitive works suggest the PCFP cannot be too difficult for any language (Dale et al., 1998; Ackerman and Malouf, 2013, 2015; Blevins et al., 2017; Cotterell et al., 2019). Neural models can test and extend such proposals (Cotterell et al., 2018a; Silfverberg and Hulden, 2018). A related vein of work discusses how speakers inflect nonce words (Berko, 1958; Plunkett and Juola, 1999; Yang, 2015), e.g., is the past tense of sping, spinged or spung? There is a long tradition of modeling past-tense generation with neural networks (Rumelhart and McClelland, 1986; Kirov and Cotterell, 2018; Corkery et al., 2019).
On the engineering side, Durrett and DeNero (2013) inspired much recent work, which has since benefited from large inflectional datasets (Kirov et al., 2018) and advances in neural sequence modeling (Bahdanau et al., 2015). Shared tasks have drawn extra attention to the PCFP (Cotterell et al., 2016a, 2017, 2018c; McCarthy et al., 2019).
3 The Paradigm Discovery Problem
Paradigm discovery is a natural next step in computational morphology, building on related minimally or indirectly supervised works (§2.2) to bridge the gap between unsupervised traditions (§2.1) and supervised work on the PCFP (§2.3). In the PCFP,
7779
Gold Grid	cell 1	cell 2	cell 3	cell 4	cell 5
paradigm 1	«watch»	«watches»	watching	watched	watched
paradigm 2	«follow»	follows	«following»	followed	followed
paradigm 3	see	«sees»	«seeing»	«saw»	seen
Table 1: An example corpus, lexicon, and gold analyses. All lexicon entries appear in the corpus and, for our experiments, they will all share a POS, here, verb. The grid reflects all possible analyses of syncretic forms (e.g., walked, followed), even though these only occur in the corpus as PST realizations, like saw in Cell 4, not as PST.PTCP, like seen in Cell 5. Bracketed «forms» are paradigm mates of attested forms, not attested in the lexicon.
each input form is labeled with its morphosyntactic property set, i.e., the cell in the paradigm which it realizes, and its lexeme, i.e., the paradigm of related forms to which it belongs. By contrast, to solve the PDP, unlabeled input forms must be assigned cells and paradigms. This task requires learning what syntactic and semantic factors distinguish cells, what combinations of cells can co-occur in a paradigm, and what aspects of a surface form reflect its paradigm and its cell, respectively.
Table 1 provides an overview of our PDP setup. The first two rows show input data: an unannotated corpus and a lexicon of forms attested in that corpus. Given only these data, the task is to output a grid such that (i) all lexicon forms and all their (potentially unseen) inflectional variants appear in the grid, (ii) all forms appearing in the same column realize the same morphosyntactic cell, and (iii) all forms appearing in the same row belong to the same paradigm. Unattested «forms» to be generated are depicted in brackets in Table 1's gold grid, which shows the ideal output of the system.
Our setup permits multiple forms realizing the same slot, i.e., a specific cell in a specific paradigm, a single form realizing multiple slots, and unrealizable empty slots. This supports overabundance (Thornton, 2010, 2011), defectiveness (Sims, 2015), and syncretism (Blevins, 1995; Cotterell et al., 2018b). See Corbett (2005) for more on these phenomena. Experimentally, we constrain the PDP by limiting the lexicon to forms from one POS, but our formalization is more general.
3.2 Data for the PDP
For a given language and POS, we create a corpus, lexicon, and gold grid based on a Universal Depen
dencies (UD) corpus (Nivre et al., 2016). At a high level, the corpus includes raw, non-UD sentences, and UD sentences stripped of annotations. The lexicon includes all forms occurring in the UD sentences with the specified POS (potentially including variant spellings and typographical errors). The gold grid consists of full paradigms for every word which co-occurs in UD and the UniMorph lexicon (Kirov et al., 2018) with a matching lemma-cell analysis; this is similar to the corpus created by Vylomova et al. (2019). As a system does not know which lexicon forms will be evaluated in the gold grid, it must model the entire lexicon, which should contain a realistic distribution over rare words and inflection classes having been directly extracted from distributional data (Bybee, 2003; Lignos and Yang, 2018).
To ensure the gold grid is reasonably clean, we take all word-lemma-feature tuples from the UD portion of the corpus matching the specified POS and convert the features to a morphosyntactic cell identifier compatible with UniMorph representation as in McCarthy et al. (2018). Then we check which word-lemma-cell tuples also occur in UniMorph. For each unique lemma in this intersection, the full paradigm is added as a row to the gold grid. To filter typos and annotation discrepancies, we identify any overabundant slots, i.e., slots realized by multiple forms, and remove all but the most frequently attested realization in UD. While some languages permit overabundance (Thornton, 2010), it often indicates typographical or annotation errors
The cat watched me watching it. I followed the show but she had n't seen it. Let's see who follows your logic.
watching, seen, follows, watched, followed, see
$^{3}$ Aligning UniMorph and UD requires removing diacritics in (Latin and Arabic) UniMorph corpora to match UD. This can obscure some morphosyntactic distinctions but is more consistent with natural orthography in distributional data. The use of orthographic data for morphological tasks is problematic, but standard in the field, due to scarcity of phonologically transcribed data (Malouf et al., 2020).
7780
Predictions	cell 1	cell 2	cell 3	cell 4
paradigm 1	watched	watching	«watches»	«watch»
paradigm 2	followed	«following»	follows	«follow»
paradigm 3	«seed»	«seeing»	«sees»	see
paradigm 4	«seened»	«seeing»	«seens»	seen
Table 2: Toy predictions made from the corpus and lexicon in Table 1, to be evaluated against the toy gold grid. Again, bracketed «forms» are those not occurring in the lexicon.
in UD and UniMorph (Gorman et al., 2019; Malouf et al., 2020). Unlike the gold grid, the lexicon retains overabundant realizations, requiring systems to handle such phenomena.
For each language, the raw sentences used to augment the corpus add over 1 million additional words. For German and Russian, we sample sentences from OpenSubtitles (Lison and Tiedemann, 2016), for Latin, the Latin Library (Johnson et al., 2016), and for English and Arabic, Gigaword (Parker et al., 2011a,b). Supplementary sentences are preprocessed via Moses (Koehn et al., 2007) to split punctuation, and, for supported languages, clitics. Table 3 shows corpus and lexicon sizes.
A system attempting the PDP is expected to output a morphologically organized grid in which rows and columns are arbitrarily ordered, but ideally, each row corresponds to a gold paradigm and each column to a gold cell. Aligning rows to paradigms and columns to cells is non-trivial, making it difficult to simply compute accuracy over gold grid slots. Furthermore, cluster-based metrics (Rosenberg and Hirschberg, 2007) are difficult to apply as forms can appear in multiple columns or rows. Thus, we propose novel metrics that are lexical, based on analogical relationships between forms. We propose a set of PDP metrics, to measure how well organized lexicon forms are in the grid, and a set of PCFP metrics, to measure how well the system anticipates unattested inflectional variants. All metrics support non-canonical phenomena such as defective paradigms and overabundant slots.
3.3.1 PDP Metrics
A form  $f$ 's paradigm mates are all those forms that co-occur in at least one paradigm with  $f$ .  $f$ 's paradigm F-score is the harmonic mean of precision and recall of how well we predicted its paradigm mates when viewed as an information retrieval problem (Manning et al., 2008). We macro-average all forms' paradigm F-scores to compute  $F_{\mathrm{par}}$ . Qualitatively,  $F_{\mathrm{par}}$  tells us how well we clus
ter words that belong to the same paradigm. A form  $f$ 's cell mates are all those forms that co-occur in at least one cell with  $f$ .  $f$ 's cell F-score is the harmonic mean of precision and recall of how well we predicted its cell mates. As before, we macro-average all forms' cell F-scores to compute  $F_{\mathrm{cell}}$ . Qualitatively,  $F_{\mathrm{cell}}$  tells us how well we cluster words that belong to the same cell. Finally, we propose the  $F_{\mathrm{grid}}$  metric as the harmonic mean of  $F_{\mathrm{par}}$  and  $F_{\mathrm{cell}}$ .  $F_{\mathrm{grid}}$  is a single number that reflects a system's ability to cluster forms into both paradigms and cells. Because we designate separate PCFP metrics to evaluate gold grid forms not in the lexicon, we restrict  $f$ 's mates to only include forms that occur in the lexicon.
Consider the proposed grid in Table 2. There are 6 lexicon forms in the gold grid. Starting with watched, we correctly propose its only attested paradigm mate, watching. Thus, watched's paradigm F-score is  $100\%$ . For see, we propose no attested paradigm mates, but we should have proposed seen. 0 correct out of 1 true paradigm mate from 0 predictions results in an F-score of  $0\%$  for seen. We continue like this for all 6 attested forms in the gold grid and average their scores to get  $F_{\mathrm{par}}$ . As for  $F_{\mathrm{cell}}$ , we correctly predict that watched's only cell mate is followed, yielding an F-score of  $100\%$ . However, we incorrectly predict that see has a cell mate, seen, yielding an F-score of  $0\%$ ; we average each word's F-score to get  $F_{\mathrm{cell}}$ ; the harmonic mean of  $F_{\mathrm{par}}$  and  $F_{\mathrm{cell}}$  gives us  $F_{\mathrm{grid}}$ .
While  $F_{\mathrm{grid}}$  handles syncretism, overabundance, defectiveness and mismatched grid dimensions, it is exploitable by focusing exclusively on the best attested cells realized by the most unique forms, since attested cells tend to exhibit a Zipfian distribution (Blevins et al., 2017; Lignos and Yang, 2018). Exploiting  $F_{\mathrm{grid}}$  in this manner propagates errors when bootstrapping to predict unattested forms and, thus, will be punished by PCFP metrics.
3.3.2 PCFP Metrics
We cannot evaluate the PCFP as in supervised settings (Cotterell et al., 2016a) because proposed
7781
Lexicon	Corpus	UD
Types	Tokens	Tokens
Arabic	8,732	1,050,336	223,881
German	19,481	1,270,650	263,804
English	3,330	1,212,986	204,608
Latin	6,903	1,218,377	171,928
Russian	36,321	1,885,302	871,548
Table 3: Statistics regarding the input corpus and lexicon. UD tokens refers to tokens in the corpus originally extracted from UD sentences.
cells and paradigms cannot be trivially aligned to gold cells and paradigms. Instead, we create a test set by sampling 2,000 four-way analogies from the gold grid. The first and second forms must share a row, as must the third and fourth; the first three forms must be attested and the fourth unattested, e.g., watched : watching :: seen : «seeing».
From this test set and a proposed grid, we compute a strict analogy accuracy (An) metric and a more lenient lexicon expansion accuracy (LE) metric. An counts predictions as correct if all analogy directions hold in the proposed grid (i.e., watched, watching and seen, «seeing» share rows and watched, seen and watching, «seeing» share columns). LE counts predictions as correct if the unattested fourth form appears anywhere in the grid. That is, LE asks, for each gold form, if it was predicted in any slot in any paradigm.
Like the PDP metrics, our PCFP metrics support syncretism, overabundance, defectiveness, etc. One can, however, exploit them by proposing a gratuitous number of cells, paradigms, and syncretisms, increasing the likelihood of completing analogies by chance, though this will reduce  $F_{\mathrm{grid}}$ . As both PDP and PCFP metrics can be exploited independently but not jointly, we argue that both types of metrics should be considered when evaluating an unsupervised system.
4 Building a Benchmark
This section presents a benchmark system for proposing a morphologically organized grid given a corpus and lexicon. First, we cluster lexicon forms into cells. Then we cluster forms into paradigms given their fixed cell membership. To maintain tractability, clustering assumes a one-to-one mapping of forms to slots. Following cell and paradigm clustering, we predict forms to realize empty slots given one of the lexicon forms assigned to a cell in
the same paradigm. This allows forms to appear in multiple slots, but does not support overabundance, defectiveness, or multi-word inflections.
4.1 Clustering into Cells
We use a heuristic method to determine the number of cells and what lexicon forms to assign to each. Inspired by work on inductive biases in word embeddings (Pennington et al., 2014; Trask et al., 2015; Goldberg, 2016; Avraham and Goldberg, 2017; Tu et al., 2017), we train morphosyntactically biased embeddings on the corpus and use them to  $k$ -means cluster lexicon forms into cells. Following Erdmann et al. (2018), we emphasize morphosyntactically salient dimensions in embedding space by manipulating hyperparameters in fastText (Bojanowski et al., 2017). Specifically, to encourage grouping of morphologically related words, fastText computes a word's embedding as the sum of its subword embeddings for all subword sequences between 3 and 6 characters long (Schütze, 1993). We shorten this range to 2 to 4 to bias the grouping toward shared affixes rather than (usually longer) shared stems. This helps recognize that the same affix is likely to realize the same cell, e.g., watch +ed and follow +ed. We limit the context window size to 1; small windows encourage a morphosyntactic bias in embeddings (Erk, 2016).
We determine the number of cells to cluster lexicon forms into,  $k$ , via the elbow method, which progressively considers adding clusters until the reduction in dispersion levels off (Kodinariya and Makwana, 2013; Bholowalia and Kumar, 2014). Since Tibshirani et al. (2001)'s popular formalism of the method does not converge on our data, we implement a simpler technique that works in our case. We incrementally increase  $k$ , each time recording clustering dispersion,  $d_{k}$  (for consistency, we average  $d_{k}$  over 25 iterations). Starting at  $k = 2$ , we calculate dispersion deceleration as the difference between the current and previous dispersions:
$$
\operatorname {d e c e l} (k) = d _ {k - 1} - 2 \left(d _ {k}\right) + d _ {k + 1} \tag {1}
$$
Once  $\operatorname{decel}(k)$  decreases below  $\sqrt{\operatorname{decel}(2)}$ , we take the  $k^{\text{th}}$  clustering: the  $(k + 1)^{\text{th}}$  cluster did not explain enough variation in the embedding space to justify an additional morphosyntactic distinction.
4 Clustering dispersion is the squared distance of a point from its cluster's centroid, summed over all points clustered.
7782
4.2 Clustering into Paradigms
Given a clustering of lexicon forms into  $k$  cells, denoted as  $C_1, \ldots, C_k$ , we heuristically cluster each form  $f$  into a paradigm,  $\pi$ , as a function of  $f$ 's cell,  $c$ . For tractability, we assume paradigms are pairwise disjoint and no paradigm contains multiple forms from the same cell. Our algorithm greedily builds paradigms cell by cell. To gauge the quality of a candidate paradigm, we first identify its base and exponents. Following Beniamine et al. (2018), we define  $\pi$ 's base,  $b_{\pi}$ , as the longest common subsequence shared by all forms in  $\pi$ .<sup>56</sup> For each form  $f$  in  $\pi$ , we define the exponent  $x_f$  as the subsequences of  $f$  that remain after removing  $b_{\pi}$ , i.e.,  $x_f$  is a tuple of affixes. For example, if  $\pi$  contains words  $wxyxz$  and  $axx$ ,  $b_{\pi}$  is  $xx$  and the exponents are  $(<w, y, z>)$  and  $(<a)$ , respectively.<sup>7</sup> Inspired by unsupervised maximum matching in greedy tokenization (Guo, 1997; Erdmann et al., 2019), we define the following paradigm score function:
$$
\operatorname {s c o r e} (\pi) = \sum_ {\langle c, f \rangle \in \pi} \left(| b _ {\pi} | - | x _ {f} |\right) \tag {2}
$$
which scores a candidate paradigm according to the number of base characters minus the number of exponent characters; it can be negative.