SlowGuess/ACL_2020_OCR huggingface dataset:
Alexander Erdmann, $^{\dagger}$  Tom Kenter, Markus Becker and Christian Schallhart *
Google, UK †Ohio State University, USA
erdmann.6@osu.edu, {tomkenter,mabecker,schallhart}@google.com
Lexica distinguishing all morphologically related forms of each lexeme are crucial to many language technologies, yet building them is expensive. We propose Frugal Paradigm Completion, an approach that predicts all related forms in a morphological paradigm from as few manually provided forms as possible. It induces typological information during training which it uses to determine the best sources at test time. We evaluate our language-agnostic approach on 7 diverse languages. Compared to popular alternative approaches, our Frugal Paradigm Completion approach reduces manual labor by  $16 - 63\%$  and is the most robust to typological variation.
From syntactic parsing (Seeker and Kuhn, 2013) to text-to-speech (Zen et al., 2016; Wan et al., 2019), many linguistic technologies rely on accurate lexica decorated with morphological information. Yet, building such lexica requires much human effort (Buckwalter, 2002; Tadic and Fulgosi, 2003; Forsberg et al., 2006; Sagot, 2010; Eskander et al., 2013). We present a language-agnostic method for minimizing the manual labor required to add new paradigms to an existing lexicon.
Formally, let each lexicon entry, or realization, be a triple  $(\mathbb{P},\mathbb{C},f)$ .  $\mathbb{P}$  marks membership in some paradigm  $\mathbb{P}$  of morphologically related words,  $\mathbb{C}$  defines a cell in  $\mathbb{P}$  as a bundle of morphosynthetic features, and  $f$  is the form realizing  $\mathbb{C}$  in  $\mathbb{P}$ . Hence, paradigm SING can be expressed (in the UniMorph schema (Kirov et al., 2018)) as a set of realizations:  $\{(SING,NFIN,sing), (SING,3.SG.PRES,sings), \ldots\}$ .
For each paradigm to be added to the lexicon, e.g., FLY, we aim to select as few sources as pos-
sible to be manually realized, e.g.,  $\{(\text{FLY}, \text{NFIN}, \text{fly}), (\text{FLY}, \text{PST}, \text{flew})\}$  such that the forms realizing the remaining cells can be predicted, i.e., flies, flying, flown. Here, sources are manually provided realizations. Targets are realizations whose forms must be predicted from sources. Our work differs from traditional paradigm completion (Durrett and DeNero, 2013) in that sources are not given blindly, but the system must strategically select which sources it wants to be given at test time.
Paradigm completion from one source is typically non-deterministic due to multiple inflection classes realizing different exponents in some cells, e.g., suffixing  $+ed$  generates the past tense for WALK, but not for SING or FLY which are members of different classes. Hence, many works discuss paradigm completion in the context of (implicit) inflection class disambiguation (Ackerman et al., 2009; Montermini and Bonami, 2013; Beniamine et al., 2018). Finkel and Stump (2007) propose three approaches to select the fewest sources required to deterministically identify class. Yet, neural sequence models can often complete paradigms accurately from less sources without fully disambiguating inflection class (Kann and Schütze, 2016; Aharoni and Goldberg, 2017; Wu and Cotterell, 2019). See Elsner et al. (2019) for an overview of the application of neural sequence models to morphological theory.
We propose Frugal Paradigm Completion (FPC), inspired by work on inflection class disambiguation and neural sequence modeling. We train a source selection agent (SSA) to induce typological knowledge regarding the distribution of complexity in paradigms and use this to request informative source cells to be realized by an oracle. Sources are fed to a predictor to generate target forms. For each paradigm, SSA iteratively requests sources until the oracle confirms all cells have been realized correctly.
This work was carried out during the first author's internship at Google UK in 2019.
8248
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8248-8273 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
We introduce a novel metric, auto-rate, to quantify the manual labour (performed by the oracle) needed to complete each paradigm. Using this metric, we demonstrate that FPC reduces labor by  $63\%$  over predicting targets from lemmata, and  $47\%$  over predicting them from the smallest set of sources that fully disambiguates inflection class. We propose a new typology for discussing the organization of complexity in paradigms which helps explain why strategies perform better or worse on certain languages while FPC, being sensitive to typological variation, performs robustly.
After discussing related paradigm completion approaches in Section 2, we describe FPC in Section 3. Section 4 covers all data and experimental set up details. We discuss results in Section 5 and analyze FPC's behavior in Section 6.
2 Paradigm Completion Approaches
Here we discuss several paradigm completion approaches related to FPC.
Lemma-based Paradigm Completion The standard paradigm completion approach does not select sources, but assumes one source: the lemma (Dreyer and Eisner, 2011), whose distinction is ultimately arbitrary. Yet many have shown that more informative sources can be chosen (Finkel and Stump, 2007; Cotterell et al., 2017b; Kann and Schütze, 2018).
Most Informative Source For each target form to be predicted, Kann and Schütze (2018) select the source most likely to predict that form. Unlike FPC, they do not attempt to minimize the number of unique sources that must be manually realized.
Static Principal Parts To minimize sources required to fully disambiguate inflection class, Finkel and Stump (2007); Stump and Finkel (2013) propose three approaches: static, dynamic, and adaptive. In the static approach, the same sources must be used for every paradigm (these sources are referred to as principal parts in a much older pedagogical tradition dating back to ancient Rome with Varro's de lingua Latina (Grinstead, 1916; Ahern, 1990)). Cotterell et al. (2017b) train a model on static sources and attain near  $100\%$  accuracy in Latin verb paradigm completion. However, they do not consider that one paradigm may require fewer sources than another, nor that paradigm completion may require fewer sources than inflection class disambiguation.
Dynamic Principal Parts Finkel and Stump (2007)'s dynamic approach selects a minimal set of sources necessary to fully disambiguate inflection class which can be unique to that inflection class. While efficient, this is impractical in that it requires oracular knowledge of class prior to seeing any forms.
Adaptive Principal Parts Finkel and Stump (2007)'s adaptive approach, like our FPC method, chooses the same first source cell for each paradigm  $\mathbb{P}$ . Subsequent sources are selected conditional on the set of inflection classes  $\mathbb{P}$  could belong to given the sources realized so far. Hence, the number of sources required per paradigm is upper bounded by the static approach and lower bounded by the dynamic.
Our FPC approach is a neural update, inspired by their adaptive approach. While their implementation tracks viable inflection classes explicitly with rules operating on oracularly segmented affixes, we use sequence models operating on whole words to remove reliance on oracular segmentation and leverage stem-internal phonology known to correlate with inflection class (Aronoff, 1992; Dressler and Thornton, 1996; Dawdy-Hesterberg and Pierrehumbert, 2014).
3 Frugal Paradigm Completion
This section describes the interactions of the three FPC components. As illustrated in Figure 1, the predictor takes a source cell and its realizing form as input, e.g., 3.SG.PRES: *sings*, or cell 2: *form* 2 in the figure. The predictor is composed of as many sub-predictors as there are cells in the paradigm, each of which is trained to predict the entire paradigm from one source cell's realization. Cell 2 in the paradigm is grayed out in the figure, as this was provided as input so it does not have to be predicted. The predicted paradigm is evaluated by the oracle. If there are no errors, we are done. Otherwise, based on previous sources, SSA chooses a new cell to be realized by the oracle and gives it to the predictor as the next source. Because cell 3 is chosen in the figure, sub-predictor 3 will be used to predict the paradigm going forward, and cells 2 and 3 will both be grayed out. The process continues like this until all cells have been correctly predicted by at least one sub-predictor.
Crucially, during inference, each test paradigm is empty, i.e., no realization has been seen during training and no source is available to inflect from
8249
Figure 1: Schematic representation of the flow of Frugal Paradigm Completion at inference time.
a-priori. Our setup aims to minimize the number of sources which the SSA must request from the oracle (typically a human in the loop at inference time) to predict the remaining paradigm slots correctly.
3.1 Predictor
The predictor outputs a target form given its cell, a source form and the source form's cell as input. To train the predictor, for each possible source cell, we train a sub-predictor to predict every possible target form in every paradigm in the training data given the realization of that source cell in that paradigm. Details of all sequence model architectures are provided in Section 4.
3.2 Source Selection Agent
SSA's choice of a cell for a given paradigm depends on all previously selected cells for that paradigm and their corresponding forms. This allows SSA to learn, e.g., that given a previous English PST source, PST.PTCP should only be requested as a subsequent source if the PST form did not take the regular -ed suffix. Otherwise, PST.PTCP is likely to be regular and unlikely to contribute new information.
To induce such knowledge, we train SSA on an oracle policy of ideal source selections extracted from the train set (Ross et al., 2011; Ross and Bagnell, 2014; Welleck et al., 2019). To extract the oracle policy, we divide the training lexicon into two folds and train one predictor on each, allowing us to cross validate each predictor on its held out fold. For each training paradigm, we test which target forms can be correctly predicted by which source cells' sub-predictors. As shown for SING
in Figure 2, we use this information to extract minimum set covers, i.e., the fewest source cells such that the union of the subsets they predict correctly covers the entire paradigm. These covers constitute the oracle policy used to train SSA.
The minimum set cover problem is NP-complete (Lund and Yannakakis, 1994; Kuhn et al., 2005), but we approximate it in  $O(log_e|\mathbb{P}|)$  by iteratively selecting the cell whose subset most enlarges the union. We break ties by averaging predictiveness (Equation 1) over both folds, where fold  $F$  contains  $|F|$  paradigms;  $\mathbb{P}_m$ ,  $|\mathbb{P}_m|$  cells; and  $\mathrm{Acc}(\mathbb{P}_m, \mathrm{C_{trg}}, \mathrm{C_{src}})$  returns 1 if using  $\mathrm{C_{src}}$ 's realization as a source correctly predicts the form realizing cell  $\mathrm{C_{trg}}$  in paradigm  $\mathbb{P}_m$ .
$$
\begin{array}{l} \operatorname {p r e d i c t i v e n e s s} \left(\mathrm {C} _ {\mathrm {s r c}}, F\right) = \\ \frac {\sum_ {m = 1} ^ {| F |} \sum_ {j = 1} ^ {| \mathbb {P} _ {m} |} \operatorname {A c c} \left(\mathbb {P} _ {m} , \mathrm {C} _ {\mathrm {j}} , \mathrm {C} _ {\mathrm {s r c}}\right)}{\sum_ {m = 1} ^ {| F |} | \mathbb {P} _ {m} |} \tag {1} \\ \end{array}
$$
At this stage, paradigm covers are dynamic in that no single cell need be shared by all covers. Yet, when selecting the first source, SSA has no previous sources to condition on, making it impossible to predict the first cell. Thus, we get adaptive minimum set covers by designating the start cell to be that which occurs in the most dynamic covers. Then we re-approximate all covers such that each includes this cell. Finally, we rank cells within each cover by the total number of covers in which they appear. For each cell in each cover, we train SSA to predict said cell from all higher ranked cells and their realizing forms (holding out  $2\%$  of them for development).
2We train and test on a single part-of-speech for each language, so each paradigm should contain the start cell. For defective paradigms lacking said cell, we back off to the most frequent cell that exists in the paradigm.
<sup>1</sup>While we borrow the term oracle policy from Imitation Learning (Ross et al., 2011; Ross and Bagnell, 2014; Welleck et al., 2019), we mimic the oracle policy with simple sequence learning. Our analysis suggests even this may be more machinery than necessary.
Figure 2: Minimum set cover example for SING, which is {NFIN, PST}.
3.3 Oracle
The oracle represents a human-in-the-loop during inference, providing requested source realizations to the predictor and informing SSA when a paradigm is complete and accurate (Figure 1). In our implementation, the oracle does not specify which individual predictions are incorrect, but it thus must resolve any discrepancies when two sub-predictors disagree after the fact. We do not attempt to model the additional cost this incurs, as it is unclear how to combine it with the presumably more expensive cost of correcting errors, which we model instead. This is worth re-visiting in future work.
4 Experimental Details
We evaluate 4 paradigm completion approaches on 7 languages. Here we discuss implementation, data and evaluation details.
4.1 Prediction Architecture
All sequence models in all implementations of any paradigm completion approach use the Transformer architecture (Vaswani et al., 2017). Here we describe the formatting of input and outputs as well as our hyperparameters.
Input and Output Formats Following Kann and Schütze (2016), input sequences combine characters and morphosyntactic features. The following is a sample input and output for a single source FPC sub-predictor specializing in the cell NFIN:
Input:  $f l y$  out_V.PTCP out_PST
Output:  $\textit{f l o w n}$
For any inflected-form-predicting sequence model whose input is not limited to realizations of a single cell—as in, e.g., the static principal parts approach—source cell features are presupended to the input as such:
Input: in_NFIN  $f l y$  out_V.PTCP out_PST
Output:  $f$  l o w n
For multi-source sequence models, the features of each source are inserted into the input and the target features are listed after the first source. We experimented with several different multi-source representations and the Transformer performed fairly similarly with all of them.
Input: in_NFIN  $f l y$  out_V.PTCP out_PST in_PST  $f l e w$
The FPC's SSA predicts not a form, but a cell, conditional on any previously realized sources. To predict the first source, it is given nothing and will thus deterministically select the best starting cell as determined by the oracle policy (see Section 3.2). To predict any subsequent source, it conditions on the realizations of all previously requested sources for that paradigm. The following exemplifies SSA inputs and outputs when predicting the second source for paradigm FLY:
Input: in_NFIN  $f l y$
Output: in_V.PTCP in_PST
Wu et al. (2018) and others have achieved improvements by embedding morphosyntactic features separately and concatenating them to the encoder output prior to feeding it to the decoder. Our error analysis, however, suggests Transformers handle Kann and Schütze (2016)-style input well. More sophisticated feature handling may not be necessary, but should be investigated in future work.
Hyperparameters We train all Transformer models for 100 epochs in batches of 64 with 0.1 dropout probability. The final model is restored from the epoch with the highest dev accuracy. We stop early if there is no improvement for 20
8251
Train	Dev	Test
Arabic nouns	paradigms	1006	100	100
instances	24160	2260	2352
German verbs	paradigms	1031	100	100
instances	27762	2690	2692
English verbs	paradigms	2908	200	201
instances	14522	1000	1001
Russian nouns	paradigms	3289	100	100
instances	37423	1133	1137
Latin nouns	paradigms	1630	100	100
instances	19150	1180	1174
Hungarian nouns	paradigms	1405	100	100
instances	47689	3400	3383
Irish nouns	paradigms	549	100	100
instances	6460	1197	1195
Table 1: Number of paradigms and instances by split for every language and POS considered.
epochs. The only exception is during FPC cross-validation where sub-predictor models are trained for only 50 epochs with early stopping after 5 epochs without improvement. This is just to reduce computational cost as it is sufficient to induce an oracle policy. The final sub-predictor models however (those used at inference time, not those used to induce the oracle policy), are trained on the full training data set using the full 100 epochs with 20 epochs patience for early stopping. As for Transformer-specific hyperparameters, using the original notation of Vaswani et al. (2017), we set  $N = 4$ ,  $d_{model} = 128$ ,  $d_{ff} = 512$ , and  $h = 8$ , scaling down the hyperparameters recommended for machine translation as our task is less expensive (Aharoni and Goldberg, 2017; Wu et al., 2018).
4.2 Data Preparation
For every language and part of speech (POS) considered, we extract train, dev and test sets from UniMorph (Kirov et al., 2018). Each split contains full paradigms, though the cells realized in each may vary due to defectiveness (Corbett, 2005; Sims, 2015). We filter many gold errors by removing paradigms for which no realization can be attested in actual text. We use Universal Dependencies (UD) (Nivre et al., 2016) to check for attestations. We also filter overabundant realizations (multiple forms realizing one cell), keeping
only the most frequent form, as attested in UD. While some languages allow for overabundance (Thornton, 2010, 2011), in UniMorph, this often indicates a gold error.
We randomly divide paradigms into splits such that train is maximally large and dev and test contain at least 100 paradigms and 1,000 realizations. Exact quantities are displayed in Table 1. Arabic, German, English, and Russian were used for development, while Irish, Hungarian, and Latin were only evaluated after fixing hyperparameters. The languages considered represent 3 families and 4 diverse Indo-European branches. They exhibit multiple non-canonical behaviors (Corbett, 2005) and present diverse challenges from non-concatenative morphology to complex inflection class systems.
Paradigm completion is usually evaluated via exact match accuracy on held out target forms (Cotterell et al., 2016, 2017a, 2018; McCarthy et al., 2019). Yet we use as many sources as are necessary to reach  $100\%$  accuracy in predicting the remaining slots, so accuracy is not a meaningful metric for the FPC. Some theoretical works focus on the sources required to unambiguously complete a paradigm given some implicit knowledge of viable inflection classes (Finkel and Stump, 2007; Ackerman and Malouf, 2013). Yet these tend not to propose actual paradigm completion models or evaluate their decisions in ambiguous cases. To evaluate our system and bridge these traditions, we propose auto-rate:
$$
a u t o - r a t e = \frac {\sum_ {i = 1} ^ {n} a u t o \left(\mathbb {P} _ {i}\right)}{\sum_ {i = 1} ^ {n} \left| \mathbb {P} _ {i} \right|}, \tag {2}
$$
where  $auto(\mathbb{P})$  denotes the number of realizations correctly predicted while not having been provided as sources for paradigm  $\mathbb{P}$  by the oracle.
Intuitively, auto-rate is like accuracy but it counts oracularly provided sources as additional errors since both errors and sources require labor, i.e., sources require manual input and errors, post-correction. We also report manual cells per paradigm, i.e., sources plus errors. Of course, FPC resolves all errors eventually, but other systems can make errors requiring post-correction.
We compare the FPC method to three baselines. One is a variant of FPC using a random SSA.
8252
This allows us to distinguish the benefit of a smart SSA from that of simply receiving additional feedback from an oracle in the loop. Each time a source must be selected, random SSA chooses randomly without replacement. Its performance is averaged over two runs. The lemma approach baseline predicts all paradigm forms from one designated source: the lemma. Finally, for the static approach baseline, we considered two static approach implementations. The single-source implementation predicts each target from the source that is, in theory, its best predictor (Kann and Schütze, 2018). The multi-source implementation concatenates these sources, predicting each target from the concatenated input. As results are nearly identical for either implementation, we report results only for single-source—with the exception of Latin, as explained presently.
For some languages, there is little theoretical or pedagogical literature to help identify the best sources for the static approach. Our single-source static approach for Arabic nouns predicts singular and dual forms from SG;NDEF;NOM and plurals from PL;NDEF;NOM. In theory, any non-plural plus any plural should be sufficient (Brustad et al., 2005; Habash, 2010). For German verbs, we predict present and imperative forms from NFIN and past forms from IND;PST;1SG (Grebe et al., 1966). We predict English present forms from NFIN; PST and v.PTCP;PST predict themselves. For Russian nouns, Zaliznyak (1980) argues for five sources, yet Parker (2016) demonstrates that three are usually sufficient. We follow the latter, predicting all nominative or accusative forms from ACC;SG, all other singulars from INS;SG, and all other plurals from GEN;PL. In preliminary experiments, we found this to match the accuracy of the five source approach, thus achieving a higher autoreate. For Latin, we could not evaluate a single-source static implementation as it is unclear which source cell best predicts each target. The multisource static approach for Latin nouns predicts all forms from NOM;SG and GEN;SG (following the classical grammatical analyses of Varro, Priscian and the Roman *ars grammatica)). For Irish and Hungarian, we do not evaluate a static approach as we lack the requisite linguistic knowledge to determine the best sources.
Accuracy	Auto-rate	Mcpp
Dev	Test	Dev	Test	Dev	Test
Arabic nouns
Lemma	62.0	58.8	59.3	56.5	9.6	10.7
Static	95.9	99.4	89.5	93.1	2.9	2.1
Random Ag.	90.2	90.9	2.2	2.2
FPC	90.2	93.6	2.2	1.5
German verbs
Lemma	87.6	89.0	84.1	85.8	4.3	4.0
Static	94.1	96.4	86.7	88.9	3.6	3.0
Random Ag.	90.0	92.1	2.4	1.9
FPC	91.8	92.5	2.0	1.8
English verbs
Lemma	96.5	94.0	76.7	74.2	1.2	1.3
Static	99.7	98.4	39.7	38.4	3.0	3.0
Random Ag.	76.0	73.3	1.2	1.4
FPC	77.3	74.3	1.1	1.3
Russian nouns
Lemma	97.1	95.6	88.3	87.5	1.3	1.5
Static	98.4	98.3	72.6	72.3	3.2	3.2
Random Ag.	86.1	84.3	1.6	1.8
FPC	88.5	89.1	1.3	1.2
Latin nouns
Lemma	65.5	51.6	63.6	49.6	5.1	6.7
Static	97.7	96.8	80.8	79.7	2.3	2.4
Random Ag.	85.9	84.7	1.7	1.8
FPC	89.0	87.8	1.3	1.4
Hungarian nouns
Lemma	95.6	90.9	92.8	88.0	2.5	4.1
Random Ag.	95.0	94.6	1.7	1.9
FPC	95.5	95.2	1.5	1.6
Irish nouns
Lemma	63.5	66.9	56.1	59.6	5.4	5.0
Random Ag.	64.9	68.2	4.2	3.8
FPC	72.1	69.6	3.3	3.6
Table 2: Evaluation of paradigm completion approaches with metrics defined in Section 4. We do not report accuracy for FPC or its random agent variant (Random Ag.), as it is trivially  $100\%$  (see Section 4.3). Mcpp stands for Manual cells per paradigm.
As shown in Table 2, FPC always ties or beats the next best approach, while the next best approach varies by language. On average, FPC reduces labor by  $63\%$  over the lemma approach,  $47\%$  over static,  $16\%$  over random agent, and  $13\%$  over the next best approach. Its success is mainly due to (1) making predictions from fewer sources than are required for fully disambiguating inflection class and (2) receiving feedback after each source.
Surprisingly, training a sophisticated SSA does not improve much over using a random agent. We
8253