SlowGuess/ACL_2020_OCR huggingface dataset:
argue this is due to an unexpectedly large margin of error in the agent's source selection task. Despite the complexity of source selection strategies required for inflection class disambiguation, FPC uses lexical frequencies to expect regularity and stem-internal clues to anticipate irregular classes, requiring a median of just one source per paradigm for all languages except under-resourced Irish. Furthermore, inspection of the source selection minimum set covers reveals that it is often the case that a paradigm can be completed correctly from any single source. This is surprising in light of the precise strategies required for completely deterministic paradigm completion in Finkel and Stump (2007)'s framework and in light of Albright (2002)'s case for the privileged status of a single form per paradigm, though in our framework with full words and full paradigms for training, it seems that many sources can often serve as good enough singleton principal parts. This supports Bonami and Beniamine (2016) proposal of gradient principal part analyses.
Here, we discuss patterns relating SSA's first and second sources chosen (Figures 3a-b and 4a-b) to the inter-predictability of cells represented by heat maps (3c and 4c). Maps display the average accuracies with which each target (column) can be predicted from each source (row). We analyze specific SSA choices and predictor errors in Arabic and Latin.
The maps (for all languages, see the Appendix) suggest complexity can be distributed within paradigms in systematically distinct ways. Ackerman and Malouf (2013) propose integrative (I-) complexity, using average conditional entropy to describe paradigmatic organization, but this has been criticized for obscuring differences in the predictability of sub-paradigm regions (Cotterell et al., 2019; Elsner et al., 2019). To remedy this, we propose a typology for measuring the extent to which I-complexity is realized via different organizational strategies, which is useful for discussing source selection strategies. Our typology describes paradigms in terms of mutual predictability, the correlation of a map and its transpose, and entropy predictiveness, the negative correlation of cells' average predictiveness (see Equation 1) and average predictability, defined here in comparable terms as:
$$
\begin{array}{l} \operatorname {p r e d i c t a b i l i t y} \left(\mathrm {C} _ {\mathrm {t r g}}, F\right) = \\ \frac {\sum_ {m = 1} ^ {| F |} \sum_ {j = 1} ^ {| \mathbb {P} _ {m} |} \operatorname {A c c} \left(\mathbb {P} _ {m} , \mathrm {C} _ {\mathrm {t r g}} , \mathrm {C} _ {\mathrm {j}}\right)}{\sum_ {m = 1} ^ {| F |} | \mathbb {P} _ {m} |} \tag {3} \\ \end{array}
$$
Intuitively, a paradigm is mutually predictable if the fact that cell A predicts cell B means that B is likely to predict A. Such paradigms often feature regions of mutually predictable cells (as in 3c), such that an optimal strategy avoids picking multiple sources from one region. For entropy predictive paradigms, if A is generally more difficult to predict than B, A is likely to be a better predictor of the remaining cells (following the information theoretic logic that surprisal is informative (Shannon, 1948; Jaeger, 2010)). For such paradigms, the optimal strategy selects the source which would have been the most difficult target to predict.
Unlike Sims (2020)'s graph-theoretic typology for describing inflection class structure, our typology is a two-dimensional description of how the optimal paradigm completion strategy is affected by underlying class structure. In this sense, our typology is complementary to hers and future work might investigate the relationship between traits in her typology and mutual predictability or entropy predictiveness. Furthermore, our typology might be updated to consider the impact of type frequency (Sims and Parker, 2016) in a framework where distributional data is available.
Figure 5 demonstrates that cross-linguistic variation is vast with respect to our typology, as some languages even exhibit negative entropy predictiveness or mutual predictability. This partly explains why non-FPC approaches perform erratically: if paradigmatic organization varies by language, source selection strategies must be able to adapt to the data.
6.1 Arabic Error Analysis
Arabic nouns are mutually predictable (Figure 5). Any singular or dual form can predict another. Plural forms also predict each other. Yet, in general, plurals are less predictive/able (Figure 3c) due to several inflection classes varying in the plural. The sound plurals take suffixes while broken plural classes are realized via non-concatenative processes. For example,  $\wp \text{沙} \downarrow r A k b$ , rider from root  $\wp \text{沙} \downarrow r k b$ , takes the broken plural pattern  $\_ \_ A _ { - }$ , becoming  $\wp \text{沙} \downarrow r k A b$ . Yet, having heard only singular realizations, a human might posit a sound plural, i.e.,  $^{\ast}\dot{\cup}\text{沙} \downarrow r A k b w n$ , realizing the
8254
Figure 3: Arabic analysis. (a) and (b) define how likely each cell is to be a source (white), correctly predicted (gray), or an error (black) after one (a) or two (b) sources. (c) shows the predictiveness/ability of source (rows) and target (columns) cells. Darker cells are less predictive/able. For a more detailed rendering of this graphic, please see the appendix.
more productive exponent.
SSA learns an ideal strategy, requesting a singular source (Figure 3a) and then a plural (3b). Interestingly, 6 of 18 sound feminine plurals (most frequent single class) require multiple sources and 8 of 28 broken plurals do not. Thus, the predictor does not default to regularity, but uses stem-internal phonology to anticipate irregularity. Most errors made from the first source posit a viable broken plural, just not the right one. In future work, modeling semantics can fix such errors, e.g., knowing that  $\varphi \subseteq rAkb$  is animate makes plural
Figure 4: Latin analysis. (a) and (b) define how likely each cell is to be a source (white), correctly predicted (gray), or an error (black) after one (a) or two (b) sources. (c) shows the predictiveness/ability of source (rows) and target (columns) cells. Darker cells are less predictive/able. For a more detailed rendering of this graphic, please see the appendix.
$\ast \omega \mathfrak{S}_{\mathfrak{g}} r w A k b$  unlikely, as animate nouns seldom take that inflection class.
For future work, we can pre-train on raw corpora to give our model access to such information (Devlin et al., 2019). Indeed Erdmann and Habash (2018) found distributional information to benefit inflectional paradigm clustering in Arabic. Though the benefits should generalize as semantics correlates with inflection class in many languages (Wurzel, 1989; Aronoff, 1992; Harris, 1992; Noyer, 1992; Carstairs-McCarthy, 1994; Corbett and Fraser, 2000; Kastner, 2019).
8255
Figure 5: Integrative typology for describing paradigms by the mutual predictability of cells and the predictive power of hard-to-predict cells.
Cell	1st	2ndM	2ndN	3rd	4thM	4thN
NOM;SG	a	us	um	varies	us	ū
GEN;SG	ae	ī	ī	is	ūs	ūs
DAT;SG	ae	ō	ō	ī	ūi	ū
ACC;SG	am	um	um	em	um	ū
ABL;SG	ā	ō	ō	em	ū	ū
VOC;SG	generally matches NOM;SG
NOM;PL	ae	ī	a	ēs	ūs	ua
GEN;PL	arum	ōrum	ōrum	um	uum	uum
DAT;PL	is	īs	īs	ibus	ibus	ibus
ACC;PL	as	ōs	a	ēs	ūs	ua
ABL;PL	is	īs	īs	ibus	ibus	ibus
VOC;PL	generally matches NOM;PL
Table 3: Plat of the suffixes taken for each Latin nominal declension.
6.2 Latin Error Analysis
Latin is not mutually predictable with moderate entropy predictiveness. SSA's choices are, at first, opaque, but Table 3 shows that ACC;PL narrows the inflection class to variants of one declension. Remaining ambiguity mostly involves  $3^{rd}$  declension nominative and vocative realizations, which can usually be predicted from the preferred second source cell, VOC;SG. 44 of 100 test paradigms were  $3^{rd}$  declension, which required multiple sources at the highest rate (16 of 44;  $2^{nd}$  masculine declension was next highest at 3 of 15). There was no correlation between declension and second source chosen, yet high auto-rate suggests SSA's choices may not need to condition on previously realized source forms, but only their cells.
While 77 of 100 paradigms were completed
from a single source, we found paradigms requiring three sources that might be completable from two using a multi-source FPC implementation. For example,  $greg_{\mathrm{es}}$ ,  $flocks$  realizes GREX.ACC;PL, but the predictor mistakenly posits  $*grecium$  for GEN;PL from this source, guessing the wrong  $3^{rd}$  declension variant. While second source VOC;SG grex corrects this, it obscures the underlying stem, as  $x$  can be an allophone of  $g$  or  $c$ . Thus, we still get an error,  $*grecum$ . A multi-source predictor could avoid forgetting the underlying allophone  $g$  after seeing the second source. That said, multi-source FPC is not as simple as multi-source static. Heuristic sampling of training instances based on the oracle policy yields predictors that only attend to one source or make bad predictions when only given one. This is worth exploring further in future work as there is more evidence of paradigms that are difficult to handle without jointly encoding sources in the linguistic literature (Corbett, 2005; Bonami and Beniamine, 2016).
We presented Frugal Paradigm Completion, which reduces the manual labor required to expand a morphological lexicon by  $16 - 63\%$  over competitive approaches across 7 languages. We demonstrated that typologically distinct morphological systems require unique treatment and benefit from our SSA, that learns its strategy from data. We found that inducing this strategy is not as challenging as previously suggested (Finkel and Stump, 2007). Thus, SSA might be replaced with a less costly architecture while our model might be improved by conditioning on semantics and jointly decoding from a variable number of sources.
We are indebted to helpful conversations with Shijie Wu, Ryan Cotterell, Katharina Kann, Andrea Sims, and Olivier Bonami. We would also like to acknowledge insightful feedback from Google's Word Graph and TTS teams, as well as four anonymous reviewers.
See rex-regis, king or pax-pacis, peace, which are technically conditioned on preceding vowel quality, though there are probably not enough training examples for the model to learn that.
8256
Farrell Ackerman, James P Blevins, and Robert Malouf. 2009. Parts and wholes: Implicative patterns in inflectional paradigms. *Analogy in grammar: Form and acquisition*, pages 54-82.
Farrell Ackerman and Robert Malouf. 2013. Morphological organization: The low conditional entropy conjecture. Language, pages 429-464.
Roee Aharoni and Yoav Goldberg. 2017. Morphological inflection generation with hard monotonic attention. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2004-2015.
Thomas Ahern. 1990. The role of linguistics in Latin instruction. *The Classical World*, pages 505-510.
Adam Albright. 2002. The identification of bases in morphological paradigms. UCLA Ph.D. Ph.D. thesis, dissertation.
Mark Aronoff. 1992. Noun classes in Arapesh. In Yearbook of Morphology 1991, pages 21-32. Springer.
Sacha Beniamine, Olivier Bonami, and Benoit Sagot. 2018. Inferring inflection classes with description length. Journal of Language Modelling, 5(3):465-525.
Olivier Bonami and Sarah Beniamine. 2016. Joint predictiveness in inflectional paradigms. Word Structure, 9(2):156-182.
Kristen Brustad, Abbas Al-Tonsi, and Mahmoud Al-Batal. 2005. Al-Kitaab Fii Ta'Aallum Al-'Arabiyya: A Textbook for Arabic. Georgetown University Press.
Tim Buckwalter. 2002. Arabic morphological analyzer (AraMorph). Linguistic Data Consortium, Philadelphia.
Andrew Carstairs-McCarthy. 1994. Inflection classes, gender, and the principle of contrast. Language, pages 737-788.
Greville G Corbett. 2005. The canonical approach in typology. Linguistic diversity and language theories, 25:49.
Greville G Corbett and Norman M Fraser. 2000. Gender assignment: a typology and a model. Systems of nominal classification, 4:293-325.
Ryan Cotterell, Christo Kirov, Mans Hulden, and Jason Eisner. 2019. On the complexity and typology of inflectional morphological systems. Transactions of the Association for Computational Linguistics, 7:327-342.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Arya D McCarthy, Katharina Kann, Sebastian Mielke, Garrett Nicolai, Miikka Silfverberg, et al. 2018. The conll-sigmorphon 2018 shared task: Universal morphological reinflation. Proceedings of the CoNLL SIGMORPHON 2018 Shared Task: Universal Morphological Reinflation, pages 1-27.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017a. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinfection in 52 languages. CoRR, abs/1706.09031.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The SIGMORPHON 2016 shared task—morphological reinflection. In Proceedings of the Workshop of the Special Interest Group on Computational Morphology and Phonology (SIGMORPHON), Berlin, Germany.
Ryan Cotterell, John Sylak-Glassman, and Christo Kirov. 2017b. Neural graphical models over strings for principal parts morphological paradigm completion. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 759-765.
Lisa Garnand Dawdy-Hesterberg and Janet Breckenridge Pierrehumbert. 2014. Learnability and generalisation of Arabic broken plural nouns. Language, cognition and neuroscience, 29(10):1268-1282.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Wolfgang U Dressler and Anna M Thornton. 1996. Italian nominal inflection. Wiener Linguistische Gazette, pages 55-57.
Markus Dreyer and Jason Eisner. 2011. Discovering morphological paradigms from plain text using a Dirichlet process mixture model. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 616-627. Association for Computational Linguistics.
Greg Durrett and John DeNero. 2013. Supervised learning of complete morphological paradigms. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1185-1195.
Micha Elsner, Andrea D Sims, Alexander Erdmann, Antonio Hernandez, Evan Jaffe, Lifeng Jin, Martha Booker Johnson, Shuan Karim, David L King, Luana Lamberti Nunes, et al. 2019. Modeling morphological learning, typology, and change: What can the neural sequence-to-sequence framework contribute? Journal of Language Modelling, 7(1):53-98.
Alexander Erdmann and Nizar Habash. 2018. Complementary strategies for low resourced morphological modeling. In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 54-65.
Ramy Eskander, Nizar Habash, and Owen Rambow. 2013. Automatic extraction of morphological lexicons from morphologically annotated corpora. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1032-1043.
Raphael Finkel and Gregory Stump. 2007. Principal parts and morphological typology. Morphology, 17(1):39-75.
Markus Forsberg, Harald Hammarstrom, and Aarne Ranta. 2006. Morphological lexicon extraction from raw text data. In International Conference on Natural Language Processing (in Finland), pages 488-499. Springer.
Paul Grebe, Helmut Gipper, and Gunther Drosdowski, editors. 1966. Duden Grammatik der deutschen Gegenwartssprache, 2., verm. und verb. aufl edition. Bibliogr. Inst., Dudenverl., Mannheim.
Wren Jones Grinstead. 1916. An Introduction to the Psychology and Pedagogy of High School Latin: With Special Reference to the Value of a One-year Course, volume 1. University of Wisconsin-Madison.
Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1-187.
James W Harris. 1992. The form classes of Spanish substantives. In Yearbook of morphology 1991, pages 65-88. Springer.
T Florian Jaeger. 2010. Redundancy and reduction: Speakers manage syntactic information density. Cognitive psychology, 61(1):23-62.
Katharina Kann and Hinrich Schütze. 2016. MED: The LMU system for the SIGMORPHON 2016 shared task on morphological reinfection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 62-70.
Katharina Kann and Hinrich Schütze. 2018. Neural transductive learning and beyond: Morphological generation in the minimal-resource setting. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3254-3264.
Itamar Kastner. 2019. Templatic morphology as an emergent property. *Natural Language & Linguistic Theory*, 37(2):571-619.
Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sebastian J Mielke, Arya McCarthy, Sandra Kubler, et al. 2018. UniMorph 2.0: Universal morphology. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
Fabian Kuhn, Pascal Von Rickenbach, Roger Wattenhofer, Emo Welzl, and Aaron Zollinger. 2005. Interference in cellular networks: The minimum membership set cover problem. In International Computing and Combinatorics Conference, pages 188-198. Springer.
Carsten Lund and Mihalis Yannakakis. 1994. On the hardness of approximating minimization problems. Journal of the ACM (JACM), 41(5):960-981.
Arya D McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sebastian Mielke, Jeffrey Heinz, et al. 2019. The SIGMORPHON 2019 shared task: Crosslinguality and context in morphology. In Proceedings of the 16th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology.
Fabio Montermini and Olivier Bonami. 2013. Stem spaces and predictability in verbal inflection. *Linguistics linguaggio*, 12(2):171-190.
Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 1659-1666.
Robert Rolf Noyer. 1992. Features, positions and affixes in autonomous morphological structure. Ph.D. thesis, Massachusetts Institute of Technology.
Jeff Parker. 2016. Inflectional Complexity and Cognitive Processing: An Experimental and Corpus-based Investigation of Russian Nouns. Ph.D. thesis, Ohio State University.
Stephane Ross and J Andrew Bagnell. 2014. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635.
Benoit Sagot. 2010. The Leff, a freely available and large-coverage morphological and syntactic lexicon for French. In 7th international conference on Language Resources and Evaluation (LREC 2010).
Wolfgang Seeker and Jonas Kuhn. 2013. Morphological and syntactic case in statistical dependency parsing. Computational Linguistics, 39(1):23-55.
Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell system technical journal, 27(3):379-423.
Andrea D Sims. 2015. Inflectional defectiveness, volume 148. Cambridge University Press.
Andrea D Sims. 2020. Inflectional networks: Graph theoretic tools for inflectional typology. Proceedings of the Society for Computation in Linguistics: Vol, 3:10.
Andrea D Sims and Jeff Parker. 2016. How inflection class systems work: On the informativity of implicative structure. Word Structure, 9(2):215-239.
Gregory Stump and Raphael A Finkel. 2013. Morphological typology: From word to paradigm, volume 138. Cambridge University Press.
Marko Tadic and Sanja Fulgosi. 2003. Building the Croatian morphological lexicon. In Proceedings of the 2003 EACL Workshop on Morphological Processing of Slavic Languages, pages 41-46. Association for Computational Linguistics.
Anna Thornton. 2011. Overabundance (multiple forms realizing the same cell): a non-canonical phenomenon in Italian verb morphology.
Anna M Thornton. 2010. Towards a typology of overabundance. In Decembrets 7: International Conference on Morphology, University of Toulouse, pages 2-3.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Vincent Wan, Chun an Chan, Tom Kenter, Jakub Vit, and Rob Clark. 2019. CHiVE: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network. In ICML 2019.
Sean Welleck, Kianté Brantley, Hal Daumé III, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. In International Conference on Machine Learning, pages 6716-6726.
Shijie Wu and Ryan Cotterell. 2019. Exact hard monotonic attention for character-level transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1530-1537.
Shijie Wu, Pamela Shapiro, and Ryan Cotterell. 2018. Hard non-monotonic attention for character-level transduction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4425-4438.
Wolfgang Ullrich Wurzel. 1989. Inflectional morphology and naturalness, volume 9. Springer Science & Business Media.
A. A. Zaliznyak. 1980. The grammar dictionary of the Russian language.
Heiga Zen, Yannis Agiomyrgiannakis, Niels Egberts, Fergus Henderson, and Przemysław Szczepaniak. 2016. Fast, compact, and high quality LSTM-RNN based statistical parametric speech synthesizers for mobile devices. In Interspeech 2016, 17th Annual Conference of the International Speech Communication Association.
A Expanded Results
The figures in this appendix demonstrate the coverage after 1 and 2 sources for every language considered as well as their inter-predictability heat maps. Figures are enlarged to show all individual
cells for the reader's convenience. Hence, the Arabic and Latin figures in this appendix correspond to Figures 3 and 4 in Section 6, but show more detail.
8259
Arabic Nouns  $\sim 1$  Source
Arabic Nouns  $\sim 2$  Sources
Figure 6: Coverage of Arabic target cells after SSA chooses the first two sources.
Arabic Noun Inter-Predictability
Figure 7: Inter-predictability heat map of Arabic cells.
8261
Figure 8: Coverage of German target cells after SSA chooses the first two sources.
8262
Figure 9: Inter-predictability heat map of German cells.
English Verbs  $\sim 1$  Source
English Verbs  $\sim 2$  Sources
Figure 10: Coverage of English target cells after SSA chooses the first two sources.
8264
English Verb Inter-Predictability
Figure 11: Inter-predictability heat map of English cells.
8265
Russian Nouns  $\sim 1$  Source
Russian Nouns  $\sim 2$  Sources
Figure 12: Coverage of Russian target cells after SSA chooses the first two sources.
8266
Russian Noun Inter-Predictability
Figure 13: Inter-predictability heat map of Russian cells.
8267
Latin Nouns  $\sim 1$  Source
Latin Nouns  $\sim 2$  Sources
Figure 14: Coverage of Latin target cells after SSA chooses the first two sources.
8268
Latin Noun Inter-Predictability