SlowGuess/ACL_2020_OCR huggingface dataset:
Algorithm 1 then details our heuristic clustering approach. We greedily select one or zero forms from each cell to add (via the list concatenation operator  $\circ$ ) to each paradigm such that the paradigm's score is maximized. After performing a first pass of paradigm clustering with Algorithm 1, we estimate an unsmoothed probability distribution  $p(x \mid c)$  as follows: we take the number of times each exponent (tuple of affixes) realizes a cell in the output of Algorithm 1 and divide by the number of occurrences of that cell. We use this distribution  $p(x \mid c)$  to construct an exponent penalty:
$$
\begin{array}{l} \omega \left(x _ {f}, c\right) \tag {3} \\ = \left\{ \begin{array}{l l} 0 & \text {i f} \underset {x} {\operatorname {a r g m a x}} p (x \mid c) = x _ {f} \\ 2 - \frac {p (x _ {f} \mid c)}{\max  _ {x} p (x \mid c)} & \text {o t h e r w i s e} \end{array} \right. \\ \end{array}
$$
Intuitively, if an exponent is the most likely exponent in the cell to which it belongs, the penalty weight is zero and its characters are not subtracted from the score. Otherwise, the weight is in the interval [1, 2] such that each exponent character is penalized at least as harshly but no more than twice as harshly than in the first pass, according to the exponent's likelihood. We use this exponent penalty weight to define a penalized score function:
$$
\operatorname {s c o r e} _ {\omega} (\pi) = \sum_ {\langle c, f \rangle \in \pi} \left(| b _ {\pi} | - | x _ {f} | \omega \left(x _ {f}, c\right)\right) \tag {4}
$$
We then re-run Algorithm 1, swapping out score  $(\cdot)$  for  $\mathrm{score}_{\omega}(\cdot)$ , to re-cluster forms into paradigms. Empirically, we find that harsher exponent penalties—i.e., forcing weights to be greater than 1 for suboptimal exponents—lead to higher paradigm precision in this second pass. For an example, consider candidate paradigm  $[\ll\rangle, \text{watched}, \ll\rangle, \ll\rangle, \ll\rangle]$ . If we add nothing, each character of watched can be analyzed as part of the base, yielding a score of 7. What if we attempt to add watching—pre-determined to belong to column 5 during cell clustering? Candidate paradigm  $[\ll\rangle, \text{watched}, \ll\rangle, \ll\rangle, \text{watching}]$  increases the number of base characters to 10 (watch shared by 2 words), but yields a score of 5 after subtracting
5The fact that we use a subsequence, instead of a substring, means that we can handle non-concatenative morphology.
<sup>6</sup>We note that the longest common subsequence may be found with a polynomial-time dynamic program; however, there will not exist an algorithm whose runtime is polynomial in the number of strings unless  $\mathsf{P} = \mathsf{NP}$  (Maier, 1978).
<sup>7</sup>We use word start (<) and end (>) tokens to distinguish exponents; they do not count as exponent characters in eq. (2).
$^{8}$  Algorithm 1 has complexity  $\mathcal{O}(|L|^2)$  where  $|L|$  is lexicon size. In practice, to make Algorithm 1 tractable, we limit the candidates for  $f_{j}^{\prime}$  (line 8) to the  $n = 250$  forms from cell  $j$  nearest to  $f_{i}$  in pre-trained embedding space (trained via FastText with default parameters). This achieves a complexity upper bounded by  $\mathcal{O}(|L|nk)$ .
7783
the characters from both exponents,  $(ed>)$  and  $(ing>)$ . Hence, we do not get this paradigm right on our first pass, as  $5 < 7$ . Yet, after the first pass, should  $(ed>)$  and  $(ing>)$  be the most frequent exponents in the second and fifth cells, the second pass will be different. Candidate paradigm  $[\gg, \text{watched}, \ll, \gg, \ll], \text{watching}]$  is not penalized for either exponent, yielding a score of 10, thereby allowing watching to be added to the paradigm.
4.3 Reinflation
We now use the output of the clustering by cell and paradigm to bootstrap the PCFP. We use a Transformer (Vaswani et al., 2017) to predict the forms that realize empty slots. Transformer-based neural transducers constitute the state of the art for the PCFP. In Cotterell et al. (2016b)'s terms, we reinfect the target from one of the non-empty source cells in the same paradigm. We select the source from which we can most reliably reinfect the target. We quantify this reliability by calculating the accuracy with which each target cell's realizations were predicted from each source cell's realizations in our development set. For each target cell, we rank our preferred source cells according to accuracy.
To generate train and development sets, we create instances for every possible pair of realizations occurring in the same paradigm (90% train, 10% development). We pass these instances into the Transformer, flattening cells and characters into a single sequence. Neural models for reinfection often perform poorly when the training data are noisy. We mitigate this via the harsh exponent penalty weights (eq. (3)) which encourage high paradigm precision during clustering.
Table 4 shows results for two versions of our benchmark system: BENCH, as described in §4, and GOLD  $k$ , with the number of cells oracularly set to the ground truth. For reference, we also report a supervised benchmark, SUP, which assumes a gold grid as input, then solves the PCFP exactly as the benchmark does. In terms of the PDP, clustering assigns lexicon forms to paradigms (46–82%) more accurately than to cells (26–80%). Results are high for English, which has the fewest gold cells, and
Cells	Paradigms	PDP	PCFP
\(F_{cell}\)	\(F_{par}\)	\(F_{grid}\)	An	LE
Arabic nouns - 8,732 forms
SUP	27	4,283	85.9	87.0
BENCH	12.8	5,279.3	39.9	48.5	43.7	16.8	49.5
GOLD k	27	4,930.3	25.9	46.4	33.1	16.1	57.2
German nouns - 19,481 forms
SUP	8	17,018	72.2	74.9
BENCH	7.3	17,073.3	35.2	59.4	43.3	14.2	56.7
GOLD k	8	16,836.0	29.4	66.6	40.8	14.8	60.4
English verbs - 3,330 forms
SUP	5	1,801	80.4	80.7
BENCH	7.5	1,949.5	64.0	80.1	71.1	52.0	67.5
GOLD k	5	1,977.3	79.6	82.1	80.8	54.7	69.4
Latin nouns - 6,903 forms
SUP	12	3,013	80.0	88.0
BENCH	13.0	3,746.5	38.8	73.2	50.6	17.2	72.9
GOLD k	12	3,749.0	39.9	71.6	51.3	17.5	72.6
Russian nouns - 36,321 forms
SUP	14	14,502	94.7	96.8
BENCH	16.5	19,792.0	44.5	72.2	55.0	31.9	86.2
GOLD k	14	20,944.0	45.7	69.1	55.0	31.6	84.3
Table 4: PDP and PCFP results for all languages and models, averaged over 4 runs. Metrics are defined in §3.3. An refers to analogy accuracy and LE to the lexicon expansion accuracy.
lower elsewhere. In German, Latin, and Russian, our benchmark proposes nearly as many cells as GOLD  $k$ , thus performing similarly. For English, it overestimates the true number and performs worse. For Arabic, it severely underestimates  $k$  but performs better, likely due to the orthography: without diacritics, the three case distinctions become obscured in almost all instances. In general, fixing the true number of cells can be unhelpful because syncretism and the Zipfian distribution of cells creates situations where certain gold cells are too difficult to detect. Allowing the system to choose its own number of cells lets it focus on distinctions for which there is sufficient distributional evidence.
As for the PCFP, our benchmark system does well on lexicon expansion accuracy and poorly on the analogy task. While lexicon expansion accuracy (50–86% compared to 72–97% for SUP) shows that the benchmark captures meaningful inflectional trends, analogy accuracy demonstrates vast room for improvement in terms of consistently organizing cell-realizations across paradigms. English is the only language where analogy accuracy is within half of SUP's upper bound. A major reason for low analogy accuracy is that forms, despite being clustered into paradigms well, get assigned
<sup>9</sup>We use the following hyperparameters:  $N = 4$ ,  $d_{model} = 128$ ,  $d_{ff} = 512$ . Remaining hyperparameters retain their default values as specified in Vaswani et al. (2017). Our models are trained for 100 epochs in batches of 64. We stop early after 20 epochs without improvement on the development set.
7784
SG	PL
NOM	GEN	DAT	ACC	ABL	NOM	GEN	DAT	ACC	ABL	Gloss
serv-us	i	o	um	o	i	orum	is	os	is	“slave.M”
serv-a	ae	ae	am	a	ae	arum	is	as	is	“slave.F”
frat-er	ris	ri	rem	re	res	rum	ribus	res	ribus	“brother”
Table 5: Suffixal exponents for each cell in the paradigm of three Latin nouns from different inflection classes.
Cell	Interpretations	Suffix
0	ACC.SG (.51), GEN.PL (.45)	um
1	ACC.PL (.71), NOM.PL (.27)	s
2	ACC.SG (.99)	m
3	ABL.PL (.52), GEN.SG (.40)	is
4	NOM.SG (.39), ABL.SG (.36)	a
5	ABL.SG (.62), NOM.SG (.36)	o
6	GEN.SG (.46), DAT.SG (.30)	i
7	ABL.PL (.77), DAT.PL (.25)	s
8	NOM.SG (.67), ABL.SG (.22)	∅
9	ABL.SG (.936)	e
10	ABL.SG (.5), GEN.SG (.28)	e
11	NOM.SG (.87), ACC.PL (.16)	us
Table 6: System clustering of Latin nouns.
to the wrong cell, or the same gold cell gets misaligned across paradigms from different inflection classes. We discuss this phenomenon in more detail below.
5.1 Latin Noun Error Analysis
A detailed analysis of Latin nouns (also analyzed by Stump and Finkel (2015) and Beniamine et al. (2018)) reveals challenges for our system. Table 5 shows the inflectional paradigms for three Latin nouns exemplifying different inflection classes, which are mentioned throughout the analysis. In keeping with the UD standard, there are no diacritics for long vowels in the table.
One major challenge for our system is that similar affixes can mark different cells in different inflection classes, e.g. the ACC.SG of servus "slave.M" ends in um, as does the GEN.PL of frater "brother". Table 6 shows system-posed cells, the gold cells they best match to, and the longest suffix shared by  $90\%$  of their members. The system is often misled by shared affixes, e.g., cell 0 is evenly split between ACC.SG and GEN.PL, driven by the suffix um (cells 3 (is) and 4 (a) suffer from this as well). This kind of confusion could be resolved with better context modeling, as each distinct underlying cell, despite sharing a surface affix, occurs in distinct distributional contexts. We observe that
the current system often fails to make use of context to handle some misleading suffixes. However, Cell 7 correctly groups ABL.PL forms marked with both is and ibus, excluding other suffixes ending in  $s$ . Similarly, cell 8 contains NOM.SG forms with heterogeneous endings, e.g.,  $r$ ,  $ix$  and  $ns$ .
In some cases, the system misinterprets derivational processes as inflectional, combining gold paradigms. Derivational relatives servus and serva, male and female variants of "slave", are grouped into one paradigm, as are philosophos "philosopher" and philosophia "philosophy." In other cases, cell clustering errors due to shared suffixes create spurious paradigms. After falsely clustering gold paradigm mates servum (ACC.SG) and servorum (GEN.PL) into the same cell, we must assign each to separate paradigms during paradigm clustering. This suggests clustering cells and paradigms jointly might avoid error propagation in future work.
We also find that clustering errors lead to PCFP errors. For  $servus/a$ , the neural reinforcer predicts servibus in cell 8 with a suffix from the wrong inflection class, yet the slot should not be empty in the first place. The correct form, servis, is attested, but was mistakenly clustered into cell 3.
5.2 Benchmark Analysis
Table 7 evaluates variants of the benchmark to determine the contribution of several system-task components in Arabic and Latin. We consider augmenting and shrinking the corpus. We also reset the fastText hyperparameters used to achieve a morphosyntactic inductive bias to their default values (no affix or window bias) and consider two constant exponent penalty weights  $(\omega(x_{f}, c) = 1$  and  $\omega(x_{f}, c) = 0)$  instead of our heuristic weight defined in eq. (3). Finally, we consider selecting random sources for PCFP reinflation instead of identifying reliable sources. For all variants, the number of cells is fixed to the ground truth.
Corpus Size We consider either using a smaller corpus containing only the UD subset, or using a larger corpus containing 15 (Latin) or 100 (Ara
PDP	PCFP
Paradigms	Fcell	Fpar	Fgrid	An	LE
Arabic nouns - 27 cells
GOLD k	4,930.3	25.9	46.4	33.1	16.1	57.2
larger corpus	5,039.5	29.1	37.5	32.8	20.4	49.2
smaller corpus	5,004.0	18.8	37.7	24.9	9.5	42.1
no affix bias	4,860.3	21.5	47.7	29.7	16.3	43.5
no window bias	4,978.5	24.0	47.5	31.8	17.6	55.8
ω(x,c) = 1	3,685.0	34.4	28.8	5.2	35.5
ω(x,c) = 0	1,310.5	10.0	13.9	0.1	5.8
random sources	16.3	55.9
Latin nouns - 12 cells
GOLD k	3,749.0	39.9	71.6	51.3	17.5	72.6
larger corpus	3,529.5	42.8	79.1	55.5	16.2	69.9
smaller corpus	4,381.5	30.7	49.1	37.8	14.6	51.1
no affix bias	3,906.8	37.1	68.2	48.1	22.7	66.6
no window bias	3,756.5	42.0	71.2	52.8	17.9	70.9
ω(x,c) = 1	3,262.5	67.1	49.6	11.0	52.9
ω(x,c) = 0	1,333.3	26.3	31.7	0.7	7.1
random sources	16.5	72.3
Table 7: Benchmark variations demonstrating the effects of various factors, averaged over 4 runs.
bic) million words from additional supplementary sentences. As expected, performance decreases for smaller corpora, but it does not always increase for larger ones, potentially due to domain differences between UD and the supplemental sentences. Interestingly,  $F_{\mathrm{cell}}$  always increases with larger corpora, yet this can lead to worse  $F_{\mathrm{par}}$  scores, more evidence of error propagation that might be avoided with joint cell-paradigm clustering.
Embedding Morphosyntactic Biases Targeting affix embeddings by shrinking the default fastText character  $n$ -gram sizes seems to yield a much more significant effect than shrinking the context window. In Latin, small context windows can even hurt performance slightly, likely due to extremely flexible word order, where agreement is often realized over non-adjacent words.
Exponent Penalties When clustering paradigms with the penalty weight  $\omega(x, c) = 1$ , (which is equivalent to just running the first pass of paradigm clustering), we see a steep decline in performance as opposed to the proposed heuristic weighting. It is even more detrimental to not penalize exponents at all (i.e.,  $\omega(x, c) = 0$ ), but maximize the base characters in paradigms without concern for size or likelihoods of exponents. Given allomorphic variation and multiple inflection classes, we ideally want a penalty weight which is lenient to more than just the single most likely exponent, but without
supervised data, it is difficult to determine when to stop being lenient and start being harsh in a language agnostic manner. Our choice to be harsh by default proposes fewer false paradigm mates, yielding less noisy input to train the reinfection model. In a post-hoc study, we calculated GOLD  $k$  PCFP scores on pure analogies only, where the first three attested forms were assigned correctly during clustering. Pure analogy PCFP scores were still closer to GOLD  $k$ 's performance than SUP's for all languages. This suggests most of the gap between GOLD  $k$  and SUP is due to noisy training on bad clustering assignments, not impossible test instances created by bad clustering assignments. This supports our choice of harsh penalties and suggests future work might reconsider clustering decisions given the reinfection model's confidence.
Reinflection Source Selection During reinfection, feeding the Transformer random sources instead of learning the most reliable source cell for each target cell slightly hurts performance. The margin is small, though, as most paradigms have only one attested form. In preliminary experiments, we also tried jointly encoding all available sources instead of just the most reliable, but this drastically lowers performance.
We present a framework for the paradigm discovery problem, in which words attested in an unannotated corpus are analyzed according to the morphosynthetic property set they realize and the paradigm to which they belong. Additionally, unseen inflectional variants of seen forms are to be predicted. We discuss the data required to undertake this task, a benchmark for solving it, and multiple evaluation metrics. We believe our benchmark system represents a reasonable approach to solving the problem based on past work and highlights many directions for improvement, e.g. joint modeling and making better use of distributional semantic information.
The authors would like to thank the members of New York University Abu Dhabi's CAMeL Lab, Marie-Catherine de Marneffe, Eleanor Chodroff, Katharina Kann, and Markus Dreyer. We acknowledge the support of the High Performance Computing Center at New York University Abu Dhabi. Finally, we wish to thank the anonymous reviewers at EMNLP 2019 and ACL 2020 for their feedback.
7786
Farrell Ackerman, James P. Blevins, and Robert Malouf. 2009. Parts and wholes: Implicative patterns in inflectional paradigms. In *Analogy in Grammar: Form and Acquisition*, pages 54-82. Oxford University Press.
Farrell Ackerman and Robert Malouf. 2013. Morphological organization: The low conditional entropy conjecture. Language, 89(3):429-464.
Farrell Ackerman and Robert Malouf. 2015. The No Blur Principle effects as an emergent property of language systems. In Anna E. Jurgensen, Hannah Sande, Spencer Lamoureux, Kenny Baclawski, and Alison Zerbe, editors, Proceedings of the Forty-First Annual Meeting of the Berkeley Linguistics Society, pages 1-14. Berkeley Linguistics Society.
Oded Avraham and Yoav Goldberg. 2017. The interplay of semantics and morphology in word embeddings. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 422-426, Valencia, Spain. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015.
Sacha Beniamine, Olivier Bonami, and Benoit Sagot. 2018. Inferring inflection classes with description length. Journal of Language Modelling, 5(3):465-525.
Toms Bergmanis and Sharon Goldwater. 2017. From segmentation to analyses: a probabilistic model for unsupervised morphology induction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 337-346, Valencia, Spain. Association for Computational Linguistics.
Jean Berko. 1958. The child's learning of English morphology. Word, 14(2-3):150-177.
Purnima Bholowalia and Arvind Kumar. 2014. EBKM: means: A clustering technique based on elbow method and  $k$ -means in WSN. International Journal of Computer Applications, 105(9).
James P. Blevins. 1995. Syncretism and paradigmatic opposition. Linguistics and Philosophy, 18(2):113-152.
James P. Blevins, Petar Milin, and Michael Ramscar. 2017. The Zipfian paradigm cell filling problem. In *Perspectives on Morphological Organization*, pages 139-158. Brill.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with
subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Gilles Boyé and Gauvain Schalchli. 2019. Realistic data and paradigms: The paradigm cell finding problem. Morphology, 29(2):199-248.
Joan Bybee. 2003. Mechanisms of change in grammaticization: The role of frequency. In Brian D. Joseph and Richard D. Janda, editors, *The Handbook of Historical Linguistics*, pages 602-623. Blackwell.
Greville G. Corbett. 2005. The canonical approach in typology. Linguistic Diversity and Language Theories, 25:49.
Maria Corkery, Yevgen Matushevych, and Sharon Goldwater. 2019. Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3868-3877, Florence, Italy. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, Mans Hulden, and Jason Eisner. 2018a. On the diachronic stability of irregularity in inflectional morphology. arXiv preprint arXiv:1804.08262.
Ryan Cotterell, Christo Kirov, Mans Hulden, and Jason Eisner. 2019. On the complexity and typology of inflectional morphological systems. Transactions of the Association for Computational Linguistics, 7:327-342.
Ryan Cotterell, Christo Kirov, Sabrina J. Mielke, and Jason Eisner. 2018b. Unsupervised disambiguation of syncretism in inflected lexicons. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 548-553, New Orleans, Louisiana. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, and Mans Hulden. 2018c. The CoNLL-SIGMORPHON 2018 shared task: Universal morphological reinfection. In Proceedings of the CoNLL-SIGMORPHON 2018 Shared Task: Universal Morphological Reinflation, pages 1-27, Brussels. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinfection, pages 1-30, Vancouver. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016a. The SIGMORPHON 2016 shared Task—Morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10-22, Berlin, Germany. Association for Computational Linguistics.
Ryan Cotterell, Hinrich Schütze, and Jason Eisner. 2016b. Morphological smoothing and extrapolation of word embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1651-1660, Berlin, Germany. Association for Computational Linguistics.
Mathias Creutz and KristaLAGus.2005.Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0. Helsinki University of Technology.
Robert Dale, Barbara Di Eugenio, and Donia Scott. 1998. Introduction to the special issue on natural language generation. Computational Linguistics, 24(3):345-353.
Markus Dreyer and Jason Eisner. 2011. Discovering morphological paradigms from plain text using a Dirichlet process mixture model. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 616-627, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Greg Durrett and John DeNero. 2013. Supervised learning of complete morphological paradigms. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1185-1195, Atlanta, Georgia. Association for Computational Linguistics.
Micha Elsner, Andrea D. Sims, Alexander Erdmann, Antonio Hernandez, Evan Jaffe, Lifeng Jin, Martha Booker Johnson, Shuan Karim, David L. King, Luana Lamberti Nunes, et al. 2019. Modeling morphological learning, typology, and change: What can the neural sequence-to-sequence framework contribute? Journal of Language Modelling, 7(1):53-98.
Alexander Erdmann and Nizar Habash. 2018. Complementary strategies for low resourced morphological modeling. In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 54-65, Brussels, Belgium. Association for Computational Linguistics.
Alexander Erdmann, Salam Khalifa, Mai Oudah, Nizar Habash, and Houda Bouamor. 2019. A little linguistics goes a long way: Unsupervised segmentation with limited language specific guidance. In Proceedings of the 16th Workshop on Computational