Venue: ACL 2020

# Frugal Paradigm Completion

Alexander Erdmann, Tom Kenter, Markus Becker and Christian Schallhart \({}^{*}\)

Google, UK \({}^{\dagger}\)Ohio State University, USA

erdmann.6@osu.edu, {tomkenter,mabecker,schallhart}@google.com

This work was carried out during the first author's internship at Google UK in 2019.

###### Abstract

Lexica distinguishing all morphologically related forms of each lexeme are crucial to many language technologies, yet building them is expensive. We propose _Frugal Paradigm Completion_, an approach that predicts all related forms in a morphological paradigm from as few manually provided forms as possible. It induces typological information during training which it uses to determine the best sources at test time. We evaluate our language-agnostic approach on 7 diverse languages. Compared to popular alternative approaches, our Frugal Paradigm Completion approach reduces manual labor by 16-63% and is the most robust to typological variation.

## 1 Introduction

From syntactic parsing Seeker and Kuhn (2013) to text-to-speech Zen et al. (2016); Wan et al. (2019), many linguistic technologies rely on accurate lexica decorated with morphological information. Yet, building such lexica requires much human effort Buckwalter (2002); Tadic and Fulgosi (2003); Forsberg et al. (2006); Sagot (2010); Eskandler et al. (2013). We present a language-agnostic method for minimizing the manual labor required to add new paradigms to an existing lexicon.

Formally, let each lexicon entry, or realization, be a triple (P, C, f). P marks membership in some paradigm P of morphologically related words, C defines a cell in P as a bundle of morphysnotactic features, and f is the form realizing C in P. Hence, paradigm SING can be expressed (in the UniMorph schema Kirov et al. (2018)) as a set of realizations: {(SING, NFIN, sing), (SING, 3.sg.pres, sings), ...}.

For each paradigm to be added to the lexicon, e.g., \(\mathbb{FLY}\), we aim to select as few sources as possible to be manually realized, e.g., {(Fly, nfin, fly), (Fly, pst, flew)} such that the forms realizing the remaining cells can be predicted, i.e., flies, flying, flown. Here, sources are manually provided realizations. Targets are realizations whose forms must be predicted from sources. Our work differs from traditional paradigm completion Durrett and DeNero (2013) in that sources are not given blindly, but the system must strategically select which sources it wants to be given at test time.

Paradigm completion from one source is typically non-deterministic due to multiple inflection classes realizing different exponents in some cells, e.g., suffixing +ed generates the past tense for WALK, but not for SING or FLY which are members of different classes. Hence, many works discuss paradigm completion in the context of (implicit) inflection class disambiguation Ackerman et al. (2009); Montermini and Bonami (2013); Beniamine et al. (2018). Finkel and Stump (2007) propose three approaches to select the fewest sources required to deterministically identify class. Yet, neural sequence models can often complete paradigms accurately from less sources without fully disambiguating inflection class Kann and Schutze (2016); Aharoni and Goldberg (2017); Wu and Cotterell (2019). See Elsner et al. (2019) for an overview of the application of neural sequence models to morphological theory.

We propose Frugal Paradigm Completion (FPC), inspired by work on inflection class disambiguation and neural sequence modeling. We train a _source selection agent_ (SSA) to induce typological knowledge regarding the distribution of complexity in paradigms and use this to request informative source cells to be realized by an oracle. Sources are fed to a _predictor_ to generate target forms. For each paradigm, SSA iteratively requests sources until the oracle confirms all cells have been realized correctly

We introduce a novel metric, _auto-rate_, to quantify the manual labour (performed by the oracle) needed to complete each paradigm. Using this metric, we demonstrate that FPC reduces labor by 63% over predicting targets from lemmata, and 47% over predicting them from the smallest set of sources that fully disambiguates inflection class. We propose a new typology for discussing the organization of complexity in paradigms which helps explain why strategies perform better or worse on certain languages while FPC, being sensitive to topological variation, performs robustly.

After discussing related paradigm completion approaches in Section 2, we describe FPC in Section 3. Section 4 covers all data and experimental set up details. We discuss results in Section 5 and analyze FPC's behavior in Section 6.

## 2 Paradigm Completion Approaches

Here we discuss several paradigm completion approaches related to FPC.

Lemma-based Paradigm CompletionThe standard paradigm completion approach does not select sources, but assumes one source: the lemma Dreyer and Eisner (2011), whose distinction is ultimately arbitrary. Yet many have shown that more informative sources can be chosen Finkel and Stump (2007); Cotterell et al. (2017); Kann and Schutze (2018).

Most Informative SourceFor each target form to be predicted, Kann and Schutze (2018) select the source most likely to predict that form. Unlike FPC, they do not attempt to minimize the number of unique sources that must be manually realized.

Static Principal PartsTo minimize sources required to fully disambiguate inflection class, Finkel and Stump (2007); Stump and Finkel (2013) propose three approaches: _static_, _dynamic_, and _adaptive_. In the static approach, the same sources must be used for every paradigm (these sources are referred to as principal parts in a much older pedagogical tradition dating back to ancient Rome with Varro's _de lingua latina_Grinstead (1916); Ahern (1990)). Cotterell et al. (2017) train a model on static sources and attain near 100% accuracy in Latin verb paradigm completion. However, they do not consider that one paradigm may require fewer sources than another, nor that paradigm completion may require fewer sources than inflection class disambiguation.

Dynamic Principal PartsFinkel and Stump (2007)'s dynamic approach selects a minimal set of sources necessary to fully disambiguate inflection class which can be unique to that inflection class. While efficient, this is impractical in that it requires oracular knowledge of class prior to seeing any forms.

Adaptive Principal PartsFinkel and Stump (2007)'s adaptive approach, like our FPC method, chooses the same first source cell for each paradigm \({\mathbb{P}}\). Subsequent sources are selected conditional on the set of inflection classes \({\mathbb{P}}\) could belong to given the sources realized so far. Hence, the number of sources required per paradigm is upper bounded by the static approach and lower bounded by the dynamic.

Our FPC approach is a neural update, inspired by their adaptive approach. While their implementation tracks viable inflection classes explicitly with rules operating on oracularly segmented affixes, we use sequence models operating on whole words to remove reliance on oracular segmentation and leverage stem-internal phonology known to correlate with inflection class Aronoff (1992); Dressler and Thornton (1996); Dawdy-Hesterberg and Pierrehumbert (2014).

## 3 Frugal Paradigm Completion

This section describes the interactions of the three FPC components. As illustrated in Figure 1, the predictor takes a source cell and its realizing form as input, e.g., 3.sg.pres: _sings_, or _cell 2_: _form 2_ in the figure. The predictor is composed of as many sub-predictors as there are cells in the paradigm, each of which is trained to predict the entire paradigm from one source cell's realization. Cell 2 in the paradigm is grayed out in the figure, as this was provided as input so it does not have to be predicted. The predicted paradigm is evaluated by the oracle. If there are no errors, we are done. Otherwise, based on previous sources, SSA chooses a new cell to be realized by the oracle and gives it to the predictor as the next source. Because cell 3 is chosen in the figure, sub-predictor 3 will be used to predict the paradigm going forward, and cells 2 and 3 will both be grayed out. The process continues like this until all cells have been correctly predicted by at least one sub-predictor.

Crucially, during inference, each test paradigm is empty, i.e., no realization has been seen during training and no source is available to inflect from 

of sources which the SSA must request from the oracle (typically a human in the loop at inference time) to predict the remaining paradigm slots correctly.

### Predictor

The predictor outputs a target form given its cell, a source form and the source form's cell as input. To train the predictor, for each possible source cell, we train a sub-predictor to predict every possible target form in every paradigm in the training data given the realization of that source cell in that paradigm. Details of all sequence model architectures are provided in Section 4.

### Source Selection Agent

SSA's choice of a cell for a given paradigm depends on all previously selected cells for that paradigm and their corresponding forms. This allows SSA to learn, e.g., that given a previous English pst source, pstp should only be requested as a subsequent source if the pst form did not take the regular _-ed_ suffix. Otherwise, pstp is likely to be regular and unlikely to contribute new information.

To induce such knowledge, we train SSA on an oracle policy of ideal source selections extracted from the train set Ross et al. (2011); Ross and Bagnell (2014); Welleck et al. (2019).1 To extract the oracle policy, we divide the training lexicon into two folds and train one predictor on each, allowing us to cross-validate each predictor on its held out fold. For each training paradigm, we test which target forms can be correctly predicted by which source cells' sub-predictors. As shown for \(\mathbb{SING}\) in Figure 2, we use this information to extract minimum set covers, i.e., the fewest source cells such that the union of the subsets they predict correctly covers the entire paradigm. These covers constitute the oracle policy used to train SSA.

Footnote 1: While we borrow the term _oracle policy_ from Imitation Learning Ross et al. (2011); Ross and Bagnell (2014); Welleck et al. (2019), we mimic the oracle policy with simple sequence learning. Our analysis suggests even this may be more machinery than necessary.

The minimum set cover problem is NP-complete Lund and Yannakakis (1994); Kuhn et al. (2005), but we approximate it in \(O(log_{c}|\mathbb{P}|)\) by iteratively selecting the cell whose subset most enlarges the union. We break ties by averaging predictiveness (Equation 1) over both folds, where fold \(F\) contains \(|F|\) paradigms; \(\mathbb{P}_{m}\), \(|\mathbb{P}_{m}|\) cells; and Acc\((\mathbb{P}_{m},\text{ctrg},\text{csrc})\) returns 1 if using csrc's realization as a source correctly predicts the form realizing cell ctrg in paradigm \(\mathbb{P}_{m}\).

\[prediciveness(\text{csrc},F)=\\ \frac{\sum^{|F|}_{m=1}\sum^{|\mathbb{P}_{m}|}_{j=1}\text{Acc}( \mathbb{P}_{m},\text{cj},\text{csrc})}{\sum^{|F|}_{m=1}|\mathbb{P}_{m}|} \tag{1}\]

At this stage, paradigm covers are dynamic in that no single cell need be shared by all covers. Yet, when selecting the first source, SSA has no previous sources to condition on, making it impossible to predict the first cell. Thus, we get adaptive minimum set covers by designating the _start cell_ to be that which occurs in the most dynamic covers. Then we re-approximate all covers such that each includes this cell.2 Finally, we rank cells within each cover by the total number of covers in which they appear. For each cell in each cover, we train SSA to predict said cell from all higher ranked cells and their realizing forms (holding out 2% of them for development).

Figure 1: Schematic representation of the flow of Frugal Paradigm Completion at inference time.

 



### Oracle

The oracle represents a human-in-the-loop during inference, providing requested source realizations to the predictor and informing SSA when a paradigm is complete and accurate (Figure 1). In our implementation, the oracle does not specify which individual predictions are incorrect, but it thus must resolve any discrepancies when two sub-predictors disagree after the fact. We do not attempt to model the additional cost this incurs, as it is unclear how to combine it with the presumably more expensive cost of correcting errors, which we model instead. This is worth re-visiting in future work.

## 4 Experimental Details

We evaluate 4 paradigm completion approaches on 7 languages. Here we discuss implementation, data and evaluation details.

### Prediction Architecture

All sequence models in all implementations of any paradigm completion approach use the Transformer architecture Vaswani et al. (2017). Here we describe the formatting of input and outputs as well as our hyperparameters.

Input and Output FormatsFollowing Kann and Schutze (2016), input sequences combine characters and morphosyntactic features. The following is a sample input and output for a single source FPC sub-predictor specializing in the cell nfin:

\begin{tabular}{l l} Input: & _f l y out\text

epochs. The only exception is during FPC cross-validation where sub-predictor models are trained for only 50 epochs with early stopping after 5 epochs without improvement. This is just to reduce computational cost as it is sufficient to induce an oracle policy. The final sub-predictor models however (those used at inference time, not those used to induce the oracle policy), are trained on the full training data set using the full 100 epochs with 20 epochs patience for early stopping. As for Transformer-specific hyperparameters, using the original notation of Vaswani et al. (2017), we set \(N=4\), \(d_{model}=128\), \(d_{ff}=512\), and \(h=8\), scaling down the hyperparameters recommended for machine translation as our task is less expensive Aharoni and Goldberg (2017); Wu et al. (2018).

### Data Preparation

For every language and part of speech (POS) considered, we extract train, dev and test sets from UniMorph Kirov et al. (2018). Each split contains full paradigms, though the cells realized in each may vary due to defectiveness Corbett (2005); Sims (2015). We filter many gold errors by removing paradigms for which no realization can be attested in actual text. We use Universal Dependencies (UD) Nivre et al. (2016) to check for attestations. We also filter overabundant realizations (multiple forms realizing one cell), keeping only the most frequent form, as attested in UD. While some languages allow for overabundance Thornton (2010, 2011), in UniMorph, this often indicates a gold error.

We randomly divide paradigms into splits such that train is maximally large and dev and test contain at least 100 paradigms and 1,000 realizations. Exact quantities are displayed in Table 1. Arabic, German, English, and Russian were used for development, while Irish, Hungarian, and Latin were only evaluated after fixing hyperparameters. The languages considered represent 3 families and 4 diverse Indo-European branches. They exhibit multiple _non-canonical_ behaviors Corbett (2005) and present diverse challenges from non-concatenative morphology to complex inflection class systems.

### Evaluation

Paradigm completion is usually evaluated via exact match _accuracy_ on held out target forms Cotterell et al. (2016, 2017a, 2018); McCarthy et al. (2019). Yet we use as many sources as are necessary to reach 100% accuracy in predicting the remaining slots, so accuracy is not a meaningful metric for the FPC. Some theoretical works focus on the sources required to unambiguously complete a paradigm given some implicit knowledge of viable inflection classes Finkel and Stump (2007); Ackerman and Malouf (2013). Yet these tend not to propose actual paradigm completion models or evaluate their decisions in ambiguous cases. To evaluate our system and bridge these traditions, we propose _auto-rate_:

\[\textit{auto-rate}=\frac{\sum_{i=1}^{n}\textit{auto}(\mathbb{P}_{i})}{\sum_{ i=1}^{n}|\mathbb{P}_{i}|}, \tag{2}\]

where _auto_(\(\mathbb{P}\)) denotes the number of realizations correctly predicted while not having been provided as sources for paradigm \(\mathbb{P}\) by the oracle.

Intuitively, auto-rate is like accuracy but it counts oracularly provided sources as additional errors since both errors and sources require labor, i.e., sources require manual input and errors, post-correction. We also report _manual cells per paradigm_, i.e., sources plus errors. Of course, FPC resolves all errors eventually, but other systems can make errors requiring post-correction.

### Baselines

We compare the FPC method to three baselines. One is a variant of FPC using a _random_ SSA.

\begin{table}
\begin{tabular}{r r c c c} \hline \hline  & & Train & Dev & Test \\ \hline Arabic nouns & paradigms & 1006 & 100 & 100 \\  & instances & 24160 & 2260 & 2352 \\ \hline German verbs & paradigms & 1031 & 100 & 100 \\  & instances & 27762 & 2690 & 2692 \\ \hline English verbs & paradigms & 2908 & 200 & 201 \\  & instances & 14522 & 1000 & 1001 \\ \hline Russian nouns & paradigms & 3289 & 100 & 100 \\  & instances & 37423 & 1133 & 1137 \\ \hline Latin nouns & paradigms & 1630 & 100 & 100 \\  & instances & 19150 & 1180 & 1174 \\ \hline Hungarian nouns & paradigms & 1405 & 100 & 100 \\  & instances & 47689 & 3400 & 3383 \\ \hline Irish nouns & paradigms & 549 & 100 & 100 \\  & instances & 6460 & 1197 & 1195 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Number of paradigms and instances by split for every language and POS considered.

 

This allows us to distinguish the benefit of a smart SSA from that of simply receiving additional feedback from an oracle in the loop. Each time a source must be selected, random SSA chooses randomly without replacement. Its performance is averaged over two runs. The _lemma_ approach baseline predicts all paradigm forms from one designated source: the lemma. Finally, for the _static_ approach baseline, we considered two _static_ approach implementations. The single-source implementation predicts each target from the source that is, in theory, its best predictor Kann and Schutze (2018). The multi-source implementation concatenates these sources, predicting each target from the concatenated input. As results are nearly identical for either implementation, we report results only for single-source--with the exception of Latin, as explained presently.

For some languages, there is little theoretical or pedagogical literature to help identify the best sources for the static approach. Our single-source static approach for Arabic nouns predicts singular and dual forms from sg;ndef;nom and plurals from pl;ndef;nom. In theory, any non-plural plus any plural should be sufficient Brustad et al. (2005); Habash (2010). For German verbs, we predict present and imperative forms from nfin and past forms from ind;pst;1sg Grebe et al. (1966). We predict English present forms from nfin; pst and v.ptcp;pts predict themselves. For Russian nouns, Zaliznyak (1980) argues for five sources, yet Parker (2016) demonstrates that three are usually sufficient. We follow the latter, predicting all nominative or accusative forms from acc;sg, all other singulars from ins;sg, and all other plurals from gen;pl. In preliminary experiments, we found this to match the accuracy of the five source approach, thus achieving a higher auto-rate. For Latin, we could not evaluate a single-source static implementation as it is unclear which source cell best predicts each target. The multi-source static approach for Latin nouns predicts all forms from nom;sg and gen;sg (following the classical grammatical analyses of Varro, Priscian and the Roman _ars grammatica_). For Irish and Hungarian, we do not evaluate a static approach as we lack the requisite linguistic knowledge to determine the best sources.

## 5 Results and Discussion

As shown in Table 2, FPC always ties or beats the next best approach, while the next best approach varies by language. On average, FPC reduces labor by \(63\%\) over the lemma approach, \(47\%\) over static, \(16\%\) over random agent, and \(13\%\) over the next best approach. Its success is mainly due to (1) making predictions from fewer sources than are required for fully disambiguating inflection class and (2) receiving feedback after each source.

Surprisingly, training a sophisticated SSA does not improve much over using a random agent. We

\begin{table}
\begin{tabular}{r r r r r r r} \hline \hline  & \multicolumn{2}{c}{**Accuracy**} & \multicolumn{2}{c}{**Auto-rate**} & \multicolumn{2}{c}{**Mcpp**} \\  & Dev & Test & Dev & Test & Dev & Test \\  & \multicolumn{4}{c}{**Arabic nouns**} & & & \\ \hline Lemma & 62.0 & 58.8 & 59.3 & 56.5 & 9.6 & 10.7 \\ Static & 95.9 & 99.4 & 89.5 & 93.1 & 2.9 & 2.1 \\ Random Ag. & & & **90.2** & 90.9 & **2.2** & 2.2 \\ FPC & & & **90.2** & **93.6** & **2.2** & **1.5** \\  & \multicolumn{4}{c}{**German verbs**} & & & \\ \hline Lemma & 87.6 & 89.0 & 84.1 & 85.8 & 4.3 & 4.0 \\ Static & 94.1 & 96.4 & 86.7 & 88.9 & 3.6 & 3.0 \\ Random Ag. & & & 90.0 & 92.1 & 2.4 & 1.9 \\ FPC & & & **91.8** & **92.5** & **2.0** & **1.8** \\  & \multicolumn{4}{c}{**English verbs**} & & & \\ \hline Lemma & 96.5 & 94.0 & 76.7 & 74.2 & 1.2 & **1.3** \\ Static & 99.7 & 98.4 & 39.7 & 38.4 & 3.0 & 3.0 \\ Random Ag. & & & 76.0 & 73.3 & 1.2 & 1.4 \\ FPC & & & **77.3** & **74.3** & **1.1** & **1.3** \\  & \multicolumn{4}{c}{**Russian nouns**} & & & \\ \hline Lemma & 97.1 & 95.6 & 88.3 & 87.5 & **1.3** & 1.5 \\ Static & 98.4 & 98.3 & 72.6 & 72.3 & 3.2 & 3.2 \\ Random Ag. & & & 86.1 & 84.3 & 1.6 & 1.8 \\ FPC & & & **88.5** & **89.1** & **1.3** & **1.2** \\  & \multicolumn{4}{c}{**Latin nouns**} & & & \\ \hline Lemma & 65.5 & 51.6 & 63.6 & 49.6 & 5.1 & 6.7 \\ Static & 97.7 & 96.8 & 80.8 & 79.7 & 2.3 & 2.4 \\ Random Ag. & & & 85.9 & 84.7 & 1.7 & 1.8 \\ FPC & & & **89.0** & **87.8** & **1.3** & **1.4** \\  & \multicolumn{4}{c}{**Hungarian nouns**} & & & \\ \hline Lemma & 95.6 & 90.9 & 92.8 & 88.0 & 2.5 & 4.1 \\ Random Ag. & & & 95.0 & 94.6 & 1.7 & 1.9 \\ FPC & & & **95.5** & **95.2** & **1.5** & **1.6** \\  & \multicolumn{4}{c}{**Irish nouns**} & & & \\ \hline Lemma & 63.5 & 66.9 & 56.1 & 59.6 & 5.4 & 5.0 \\ Random Ag. & & & 64.9 & 68.2 & 4.2 & 3.8 \\ FPC & & & **72.1** & **69.6** & **3.3** & **3.6** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation of paradigm completion approaches with metrics defined in Section 4. We do not report accuracy for FPC or its random agent variant (Random Ag.), as it is trivially \(100\%\) (see Section 4.3). Mcpp stands for _Manual cells per paradigm_.

 

argue this is due to an unexpectedly large margin of error in the agent's source selection task. Despite the complexity of source selection strategies required for inflection class disambiguation, FPC uses lexical frequencies to expect regularity and stem-internal clues to anticipate irregular classes, requiring a median of just one source per paradigm for all languages except under-resourced Irish. Furthermore, inspection of the source selection minimum set covers reveals that it is often the case that a paradigm can be completed correctly from any single source. This is surprising in light of the precise strategies required for completely deterministic paradigm completion in Finkel and Stump (2007)'s framework and in light of Albright (2002)'s case for the privileged status of a single form per paradigm, though in our framework with full words and full paradigms for training, it seems that many sources can often serve as _good enough_ singleton principal parts. This supports Bonami and Beniamine (2016) proposal of gradient principal part analyses.

## 6 Analysis

Here, we discuss patterns relating SSA's first and second sources chosen (Figures 3a-b and 4a-b) to the inter-predictability of cells represented by heat maps (3c and 4c). Maps display the average accuracies with which each target (column) can be predicted from each source (row). We analyze specific SSA choices and predictor errors in Arabic and Latin.

The maps (for all languages, see the Appendix) suggest complexity can be distributed within paradigms in systematically distinct ways. Ackerman and Malouf (2013) propose integrative (I-) complexity, using average conditional entropy to describe paradigmatic organization, but this has been criticized for obscuring differences in the predictability of sub-paradigm regions (Cotterell et al., 2019; Elsner et al., 2019). To remedy this, we propose a typology for measuring the extent to which I-complexity is realized via different organizational strategies, which is useful for discussing source selection strategies. Our typology describes paradigms in terms of _mutual predictability_, the correlation of a map and its transpose, and _entropy predictiveness_, the negative correlation of cells' average predictiveness (see Equation 1) and average predictability, defined here in comparable terms as:

\[predictability({\rm C}_{\tt trg},F)=\\ \frac{\sum_{m=1}^{|F|}\sum_{j=1}^{|\mathbb{P}_{m}|}{\rm Acc}( \mathbb{P}_{m},{\rm C}_{\tt trg},{\rm C}_{j})}{\sum_{m=1}^{|F|}|\mathbb{P}_{ m}|} \tag{3}\]

Intuitively, a paradigm is mutually predictable if the fact that cell a predicts cell b means that b is likely to predict a. Such paradigms often feature regions of mutually predictable cells (as in 3c), such that an optimal strategy avoids picking multiple sources from one region. For entropy predictive paradigms, if a is generally more difficult to predict than b, a is likely to be a better predictor of the remaining cells (following the information theoretic logic that surprisal is informative (Shannon, 1948; Jaeger, 2010)). For such paradigms, the optimal strategy selects the source which would have been the most difficult target to predict.

Unlike Sims (2020)'s graph-theoretic typology for describing inflection class structure, our typology is a two-dimensional description of how the optimal paradigm completion strategy is affected by underlying class structure. In this sense, our typology is complementary to hers and future work might investigate the relationship between traits in her typology and mutual predictability or entropy predictiveness. Furthermore, our typology might be updated to consider the impact of type frequency (Sims and Parker, 2016) in a framework where distributional data is available.

Figure 5 demonstrates that cross-linguistic variation is vast with respect to our typology, as some languages even exhibit negative entropy prediciveness or mutual predictability. This partly explains why non-FPC approaches perform erratically: if paradigmatic organization varies by language, source selection strategies must be able to adapt to the data.

### Arabic Error Analysis

Arabic nouns are mutually predictable (Figure 5). Any singular or dual form can predict another. Plural forms also predict each other. Yet, in general, plurals are less predictive/able (Figure 3c) due to several inflection classes varying in the plural. The _sound_ plurals take suffixes while _broken_ plural classes are realized via non-concatenative processes. For example, _xkb_, _rideb_, _rider_ from root _x___y__r__k__b_, takes the broken plural pattern ____A__, becoming _x___y__rkAb_. Yet, having heard only singular realizations, a human might posit a sound plural, i.e., _x___y__x__y__r__k__b__n__, realizing the 

more productive exponent.

SSA learns an ideal strategy, requesting a singular source (Figure 3a) and then a plural (3b). Interestingly, 6 of 18 sound feminine plurals (most frequent single class) require multiple sources and 8 of 28 broken plurals do not. Thus, the predictor does not default to regularity, but uses stem-internal phonology to anticipate irregularity. Most errors made from the first source posit a viable broken plural, just not the right one. In future work, modeling semantics can fix such errors, e.g., knowing that \({}_{\sim}^{5}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \! 



### Latin Error Analysis

Latin is not mutually predictable with moderate entropy predictiveness. SSA's choices are, at first, opaque, but Table 3 shows that acc;pl narrows the inflection class to variants of one declension. Remaining ambiguity mostly involves \(3^{rd}\) declension nominative and vocative realizations, which can usually be predicted from the preferred second source cell, voc;sg. 44 of 100 test paradigms were \(3^{rd}\) declension, which required multiple sources at the highest rate (16 of 44; \(2^{nd}\) masculine declension was next highest at 3 of 15). There was no correlation between declension and second source chosen, yet high auto-rate suggests SSA's choices may not need to condition on previously realized source forms, but only their cells.

While 77 of 100 paradigms were completed from a single source, we found paradigms requiring three sources that might be completable from two using a multi-source FPC implementation. For example, _greges_, _flocks_ realizes GREEX.acc;pl, but the predictor mistakenly posits *_grevium_ for gen;pl from this source, guessing the wrong \(3^{rd}\) declension variant. While second source voc;sg _grex_ corrects this, it obscures the underlying stem, as \(x\) can be an allophone of \(g\) or \(c\). Thus, we still get an error, *_grecum_. A multi-source predictor could avoid forgetting the underlying allophone \(g\) after seeing the second source.3 That said, multi-source FPC is not as simple as multi-source static. Heuristic sampling of training instances based on the oracle policy yields predictors that only attend to one source or make bad predictions when only given one. This is worth exploring further in future work as there is more evidence of paradigms that are difficult to handle without jointly encoding sources in the linguistic literature Corbett (2005); Bonami and Beniamine (2016).

Footnote 3: See _vex–regis_, _king_ or _parax–pacis_, _peace_, which are technically conditioned on preceding vowel quality, though there are probably not enough training examples for the model to learn that.

## 7 Conclusion

We presented Frugal Paradigm Completion, which reduces the manual labor required to expand a morphological lexicon by 16-63% over competitive approaches across 7 languages. We demonstrated that typologically distinct morphological systems require unique treatment and benefit from our SSA, that learns its strategy from data. We found that inducing this strategy is not as challenging as previously suggested Finkel and Stump (2007). Thus, SSA might be replaced with a less costly architecture while our model might be improved by conditioning on semantics and jointly decoding from a variable number of sources.

## Acknowledgments

We are indebted to helpful conversations with Shijie Wu, Ryan Cotterell, Katharina Kann, Andrea Sims, and Olivier Bonami. We would also like to acknowledge insightful feedback from Google's Word Graph and TTS teams, as well as four anonymous reviewers.

Figure 5: Integrative typology for describing paradigms by the mutual predictability of cells and the predictive power of hard-to-predict cells.

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline Cell & \(1^{st}\) & \(2^{nd}\)M & \(2^{nd}\)N & \(3^{rd}\) & \(4^{th}\)M & \(4^{th}\)N \\ \hline NOM;SG & a & us & um & _varies_ & us & \(\bar{\text{u}}\) \\ \hline GEN;SG & ae & \(\bar{\text{t}}\) & \(\bar{\text{t}}\) & is & \(\bar{\text{u}}\)s & \(\bar{\text{o}}\)s \\ \hline DAT;SG & ae & \(\bar{\text{o}}\) & \(\bar{\text{o}}\) & \(\bar{\text{t}}\) & \(\bar{\text{u}}\) & \(\bar{\text{u}}\) \\ \hline ACC;SG & am & um & um & em & um & \(\bar{\text{u}}\) \\ \hline ABL;SG & \(\bar{\text{a}}\) & \(\bar{\text{o}}\) & \(\bar{\text{o}}\) & em & \(\bar{\text{u}}\) & \(\bar{\text{u}}\) \\ \hline VOC;SG & \multicolumn{8}{c}{_generally matches NOM;SG_} \\ \hline NOM;PL & ae & \(\bar{\text{t}}\) & a & \(\bar{\text{e}}\)s & \(\bar{\text{u}}\)s & ua \\ \hline GEN;PL & arum & érum & érum & um & num & num \\ \hline DAT;PL & is & fs & fs & ibus & ibus & ibus \\ \hline ACC;PL & as &ős & a & \(\bar{\text{e}}\)s & \(\bar{\text{u}}\)s & ua \\ \hline ABL;PL & is & fs & fs & ibus & ibus & ibus \\ \hline VOC;PL & \multicolumn{8}{c}{_generally matches NOM;PL_} \\ \hline \hline  & & & & & & \\ \end{tabular}
\end{table}
Table 3: Plat of the suffixes taken for each Latin nominal decelension.





## References

* Ackerman et al. (2009) Farrell Ackerman, James P Blevins, and Robert Malouf. 2009. Parts and wholes: Implicative patterns in inflectional paradigms. _Analogy in grammar: Form and acquisition_, pages 54-82.
* Ackerman and Maluf (2013) Farrell Ackerman and Robert Maluf. 2013. Morphological organization: The low conditional entropy conjecture. _Language_, pages 429-464.
* Aharoni and Goldberg (2017) Roee Aharoni and Yoav Goldberg. 2017. Morphological inflection generation with hard monotonic attention. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2004-2015.
* Ahern (1990) Thomas Ahern. 1990. The role of linguistics in Latin instruction. _The Classical World_, pages 505-510.
* Albright (2002) Adam Albright. 2002. _The identification of bases in morphological paradigms. UCLA Ph.D._ ph.D. thesis, dissertation.
* Aronoff (1992) Mark Aronoff. 1992. Noun classes in Arapesh. In _Yearbook of Morphology 1991_, pages 21-32. Springer.
* Beniamine et al. (2018) Sacha Beniamine, Olivier Bonami, and Benoit Sagot. 2018. Inferring inflection classes with description length. _Journal of Language Modelling_, 5(3):465-525.
* Bonami and Beniamine (2016) Olivier Bonami and Sarah Beniamine. 2016. Joint predictiveness in inflectional paradigms. _Word Structure_, 9(2):156-182.
* Brustad et al. (2005) Kristen Brustad, Abbas Al-Tonsi, and Mahmoud Ali-Batal. 2005. _Al-Kitaeab Fi Ta'Aallum Al-Arabivya: A Textbook for Arabic_. Georgetown University Press.
* Buckwalter (2002) Tim Buckwalter. 2002. Arabic morphological analyzer (AraMorph). _Linguistic Data Consortium, Philadelphia_.
* Carstairs-McCarthy (1994) Andrew Carstairs-McCarthy. 1994. Inflection classes, gender, and the principle of contrast. _Language_, pages 737-788.
* Corbett (2005) Greville G Corbett. 2005. The canonical approach in typology. _Linguistic diversity and language theories_, 25:49.
* Corbett and Fraser (2000) Greville G Corbett and Norman M Fraser. 2000. Gender assignment: a typology and a model. _Systems of nominal classification_, 4:293-325.
* Cotterell et al. (2019) Ryan Cotterell, Christo Kirov, Mans Hulden, and Jason Eisner. 2019. On the complexity and typology of inflectional morphological systems. _Transactions of the Association for Computational Linguistics_, 7:327-342.
* Cotterell et al. (2018) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Arya D McCarthy, Katharina Kann, Sebastian Mielke, Garrett Nicolai, Miikka Silfverberg, et al. 2018. The conll-sigmorphon 2018 shared task: Universal morphological reinflection. _Proceedings of the CoNLL SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection_, pages 1-27.
* Cotterell et al. (2017a) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manal Faruqui, Sandra Kubler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017a. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. _CoRR_, abs/1706.09031.
* Cotterell et al. (2016) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The SIGMORPHON 2016 shared task--morphological reinflection. In _Proceedings of the Workshop of the Special Interest Group on Computational Morphology and Phonology (SIGMORPHON)_, Berlin, Germany.
* Cotterell et al. (2017b) Ryan Cotterell, John Sylak-Glassman, and Christo Kirov. 2017b. Neural graphical models over strings for principal parts morphological paradigm completion. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 759-765.
* Dawdy-Hesterberg and Brekenridge Pierrehumbert (2014) Lisa Gamand Dawdy-Hesterberg and Janet Brekenridge Pierrehumbert. 2014. Learnability and generalisation of Arabic broken plural nouns. _Language cognition and neuroscience_, 29(10):1268-1282.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Dressler and Thornton (1996) Wolfgang U Dressler and Anna M Thornton. 1996. Italian nominal inflection. _Wiener Linguistic Gazette_, pages 55-57.
* Dreyer and Eisner (2011) Markus Dreyer and Jason Eisner. 2011. Discovering morphological paradigms from plain text using a Dirichlet process mixture model. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 616-627. Association for Computational Linguistics.
* Durrett and DeNero (2013) Greg Durrett and John DeNero. 2013. Supervised learning of complete morphological paradigms. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1185-1195.
* Elsner et al. (2019) Micha Elsner, Andrea D Sims, Alexander Erdmann, Antonio Hernandez, Evan Jaffe, Lifeng Jin, Martha Booker Johnson, Shuan Karim, David L King, Luana Lamberti Nunes, et al. 2019. Modeling morphological learning, typology, and change: What can the neural sequence-to-sequence framework contribute? _Journal of Language Modelling_, 7(1):53-98.
* Erdmann and Habash (2018) Alexander Erdmann and Nizar Habash. 2018. Complementary strategies for low resourced morphological modeling. In _Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology_, pages 54-65

Ramy Eskander, Nizar Habash, and Owen Rambow. 2013. Automatic extraction of morphological lexicons from morphologically annotated corpora. In _Proceedings of the 2013 conference on empirical methods in natural language processing_, pages 1032-1043.
* Finkel and Stump (2007) Raphael Finkel and Gregory Stump. 2007. Principal parts and morphological typology. _Morphology_, 17(1):39-75.
* Forsberg et al. (2006) Markus Forsberg, Harald Hammarstrom, and Aarne Ranta. 2006. Morphological lexicon extraction from raw text data. In _International Conference on Natural Language Processing (in Finland)_, pages 488-499. Springer.
* Grebe et al. (1966) Paul Grebe, Helmut Gipper, and Gunther Drosdowski, editors. 1966. _Duden Grammatik der deutschen Gesemwartssprache_, 2., verm. und verb. aufl edition. Bibliogr. Inst., Dudnervel., Mannheim.
* Grintestad (1916) Wren Jones Grintead. 1916. _An Introduction to the Psychology and Pedagogy of High School Latin: With Special Reference to the Value of a One-year Course_, volume 1. University of Wisconsin-Madison.
* Habash (2010) Nizar Y Habash. 2010. Introduction to Arabic natural language processing. _Synthesis Lectures on Human Language Technologies_, 3(1):1-187.
* Harris (1992) James W Harris. 1992. The form classes of Spanish substructures. In _Yearbook of morphology 1991_, pages 65-88. Springer.
* Jaeger (2010) T Florian Jaeger. 2010. Redundancy and reduction: Speakers manage syntactic information density. _Cognitive psychology_, 61(1):23-62.
* Kann and Schutze (2016) Katharina Kann and Hinrich Schutze. 2016. MED: The LMU system for the SIGMORPHON 2016 shared task on morphological reinflection. In _Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_, pages 62-70.
* Kann and Schutze (2018) Katharina Kann and Hinrich Schutze. 2018. Neural transductive learning and beyond: Morphological generation in the minimal-resource setting. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3254-3264.
* Kastner (2019) Itamar Kastner. 2019. Templatic morphology as an emergent property. _Natural Language & Linguistic Theory_, 37(2):571-619.
* Kirov et al. (2018) Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sebastian J Mielke, Arya McCarthy, Sandra Kubler, et al. 2018. UniMorph 2.0: Universal morphology. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_.
* Kuhn et al. (2005) Fabian Kuhn, Pascal Von Rickenbach, Roger Wattenhofer, Emo Welzl, and Aaron Zollinger. 2005. Interference in cellular networks: The minimum membership set cover problem. In _International Computing and Combinatorics Conference_, pages 188-198. Springer.
* Lund and Yannakakis (1994) Carsten Lund and Mihalis Yannakakis. 1994. On the hardness of approximating minimization problems. _Journal of the ACM (JACM)_, 41(5):960-981.
* McCarthy et al. (2019) Arya D McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silverberg, Sebastian Mielke, Jeffrey Heinz, et al. 2019. The SIGMORPHON 2019 shared task: Crosslinguality and context in morphology. In _Proceedings of the 16th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_.
* Montermini and Bonami (2013) Fabio Montermini and Olivier Bonami. 2013. Stem spaces and predictability in verbal inflection. _Lingue e linguaggio_, 12(2):171-190.
* Nivre et al. (2016) Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_, pages 1659-1666.
* Noyer (1992) Robert Rolf Noyer. 1992. _Features, positions and affixes in autonomous morphological structure_. Ph.D. thesis, Massachusetts Institute of Technology.
* Parker (2016) Jeff Parker. 2016. _Inflectional Complexity and Cognitive Processing: An Experimental and Corpus-based Investigation of Russian Nouns_. Ph.D. thesis, Ohio State University.
* Ross and Bagnell (2014) Stephane Ross and J Andrew Bagnell. 2014. Reinforcement and imitation learning via interactive no-regret learning. _arXiv preprint arXiv:1406.5979_.
* Ross et al. (2011) Stephane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_, pages 627-635.
* Sagot (2010) Benoit Sagot. 2010. The Lefft, a freely available and large-coverage morphological and syntactic lexicon for French. In _7th international conference on Language Resources and Evaluation (LREC 2010)_.
* Seeker and Kuhn (2013) Wolfgang Seeker and Jonas Kuhn. 2013. Morphological and syntactic case in statistical dependency parsing. _Computational Linguistics_, 39(1):23-55.
* Shannon (1948) Claude Elwood Shannon. 1948. A mathematical theory of communication. _Bell system technical journal_, 27(3):379-423.
* Sims (2015) Andrea D Sims. 2015. _Inflectional defectiveness_, volume 148. Cambridge University Press.
* Sims (2020) Andrea D Sims. 2020. Inflectional networks: Graphtheoretic tools for inflectional topology. _Proceedings of the Society for Computation in Linguistics: Vol_, 3:10.
* Sims and Parker (2016) Andrea D Sims and Jeff Parker. 2016. How inflection class systems work: On the informativity of implicative structure. _Word Structure_, 9(2):215-239 

* Stump and Finkel (2013) Gregory Stump and Raphael A Finkel. 2013. _Morphological typology: From word to paradigm_, volume 138. Cambridge University Press.
* Tadic and Fulgosi (2003) Marko Tadic and Sanja Fulgosi. 2003. Building the Croatian morphological lexicon. In _Proceedings of the 2003 EACL Workshop on Morphological Processing of Slavic Languages_, pages 41-46. Association for Computational Linguistics.
* Thornton (2011) Anna Thornton. 2011. Overabundance (multiple forms realizing the same cell): a non-canonical phenomenon in Italian verb morphology.
* Thornton (2010) Anna M Thornton. 2010. Towards a typology of overabundance. In _Decembrettes 7: International Conference on Morphology, University of Toulouse_, pages 2-3.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Wan et al. (2019) Vincent Wan, Chun an Chan, Tom Kenter, Jakub Vit, and Rob Clark. 2019. CHiVE: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network. In _ICML 2019_.
* Welleck et al. (2019) Sean Welleck, Kiante Brantley, Hal Daume III, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. In _International Conference on Machine Learning_, pages 6716-6726.
* Wu and Cotterell (2019) Shijie Wu and Ryan Cotterell. 2019. Exact hard monotonic attention for character-level transduction. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1530-1537.
* Wu et al. (2018) Shijie Wu, Pamela Shapiro, and Ryan Cotterell. 2018. Hard non-monotonic attention for character-level transduction. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4425-4438.
* Wurzel (1989) Wolfgang Ullrich Wurzel. 1989. _Inflectional morphology and naturalness_, volume 9. Springer Science & Business Media.
* Zaliznyak (1980) A. A. Zaliznyak. 1980. _The grammar dictionary of the Russian language_.
* Zen et al. (2016) Heiga Zen, Yannis Agiomyrgiannakis, Niels Egberts, Fergus Henderson, and Przemyslaw Szczepaniak. 2016. Fast, compact, and high quality LSTM-RNN based statistical parametric speech synthesizers for mobile devices. In _Interspeech 2016, 17th Annual Conference of the International Speech Communication Association_.

## Appendix A Expanded Results

The figures in this appendix demonstrate the coverage after 1 and 2 sources for every language considered as well as their inter-predictability heat maps. Figures are enlarged to show all individual cells for the reader's convenience. Hence, the Arabic and Latin figures in this appendix correspond to Figures 3 and 4 in Section 6, but show more detail 

Figure 6: Coverage of Arabic target cells after SSA chooses the first two sources.

 



## Appendix A

Figure 7: Inter-predictability heat map of Arabic cells.



* [14] M. C. Bader, A. D. Krizhevsky, and S. I. Sutskever. Learning to predict the best model parameters for a large dataset. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 257-260, June 2013.
* [15] J. L. Friedman, R. G. Olshen, and P. W. Zisserman. The use of deep learning in computer vision. _arXiv preprint arXiv:1908.05810_, 2019.
* [16] T. F. Donahue, Y. Bengio, and E. Rabinovich. Deep residual learning with convolutions. In _Advances in neural information processing systems_, volume 10, pages 103-113, 2016.
* [17] H. Dong, X. Li, and Q. Weinberger. Learning representations by convolutional networks. In _NIPS_, pages 1399-1406, 2016.

[MISSING_PAGE_POST]

 


* [146] M. C. Bader, J. S. Hubert, and A. K. G. Williams, _Theoretical Computer Science_, vol. 375, no. 2, pp. 80-93, Jan. 2000.

[MISSING_PAGE_POST]

 

Figure 10: Coverage of English target cells after SSA chooses the first two sources.

 



## Appendix A

Figure 11: Inter-predictability heat map of English cells.



Figure 12: Coverage of Russian target cells after SSA chooses the first two sources.

 



## Appendix A

Figure 13: Inter-predictability heat map of Russian cells.



Figure 14: Coverage of Latin target cells after SSA chooses the first two sources.

 

* [145] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of the design of a new model for the design of a new model in the design of a new model. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 237-246, 2018.
* [146] R. G. Baker, P. H. Schultz, and T. W. L. Smith. Design of a new model for the design of a new model. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 9, pages 103-113, 2018.
* [147] E. F. DeVries, N. V. O'Sullivan, and I. M. Roth. Design of a new model for the design of a new model. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [148] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [149] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: An overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [150] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [151] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [152] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [153] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [154] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [155] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [156] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [157] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [158] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [159] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [160] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [161] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [162] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [163] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [164] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [165] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [166] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [167] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [168] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [169] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [170] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [171] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [172] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [173] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [174] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [175] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [176] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [177] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [178] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [179] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [180] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [181] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [182] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [183] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [184] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.
* [185] M. C. Bader, J. D. Kwiatkowski, and A. S. Zweig. The role of design design: an overview. In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_, volume 1, pages 103-113, 2018.

 

* [16] M. C. Bader, J. A. Harvey, and R. H. Sohn, "The role of the human brain in a social network: The role of human activity in social networks," _Journal of Human Behavior_**20**(4), no. 3, pp. 57-89, 2000.
* [17] D. G. K. Miller, "A social network for social networks," _Social Networks_**20**(4), no. 3, pp. 105-114, 2000.

[MISSING_PAGE_POST]

 

* [164] M. C. Bader, J. A. Kwiatkowski, and D. S. Schultz, _Theoretical Computer Science_**2003**, 579-684, (2004)
* [165] R. H. Geyer, P. W. Fruhwirth, and T. E. Strangs, _Computer Methods in Applied Mathematics_**2003**, 1-10, (2004)
* [166] L. J. Hall, _Computational Geometry: An Introduction to the Theory of Computing_, volume 1, Springer Verlag, New York, (1994)
* [167] V. I. Johnson, _Mathematical methods for computing systems_, volume 1, Cambridge University Press, (1994)

[MISSING_PAGE_POST]

 

Figure 18: Coverage of Irish target cells after SSA chooses the first two sources.

 

Figure 19: Inter-predictability heat map of Irish cells.