SlowGuess/ACL_2020_OCR huggingface dataset:
After finding possible paradigms, we use the same method for pruning unreliable paradigms. Given an unreliable paradigm  $p = R \times M$ , the intersection of the morphological pattern set  $M$  and the set  $M_{i}$  of each reliable paradigm  $p_{i}$  is computed, i.e.  $M_{i}' = M \cap M_{i}$ , and the one with the best score, e.g.  $M_{k}'$  will be chosen as the pruned result, i.e.  $p' = R \times M_{k}'$ . Finally, the score of an intersection  $M_{i}'$  is the sum of the frequencies of all the morphological patterns in the intersection, as shown in equation 3.
$$
\operatorname {s c o r e} (M) = \sum_ {m \in M} f r e q (m) \tag {3}
$$
5.2 Generating Morphological Derivations
After the one-step roots of all the words are found, morphological derivations (e.g., sterile, sterilize, sterilizing) are automatically generated iteratively by our system as well as final segmentations (e.g., steril-iz-ing). As described in the next section, because evaluation will be based on morpheme boundaries identification, generating such a segmentation is necessary.
6.1 Settings
We compare our model with Morfessor (Virpioja et al., 2013), the most popular baseline, MorphoChain (MC) (Narasimhan et al., 2015) and its improved version, Morph-Forest (MF) (Luo et al., 2017), and ParaMA (PMA) (Xu et al., 2018). We evaluate the models with segmentation points (boundaries of morphemes), the same metric used by Narasimhan et al. (2015) and Xu et al. (2018). We run our model in two different settings. In
Lang	Train	Test	Corpus	Morphology
Aka	74K	2K	3M	pref, suf, red
Hin	487K	2K	28M	pref, suf, red
Hun	4,390K	2K	574M	pref, suf
Ind	525K	2K	19M	pref, suf, inf, red
Rus	1,485K	2K	1,068M	pref, suf
Spa	564K	2K	24M	pref, suf
Swa	224K	2K	4M	pref, suf, red, fv
Tag	13K	2K	5M	pref, suf, inf, lred, red
Tam	2,363K	2K	47M	pref, suf, red
Table 2: Number of word types for training and testing, corpus size for training word vectors (only for MorphoChain and Morph-Forest systems), and the morphological features (pref: prefixation; suf: suffixation; inf: infixation; red: full reduplication; lred: left reduplication; fv: final vowel) for each language.
the primary experiment, we run it as a fully unsupervised model (FU), assuming all possible typological features. In a secondary experiment, each language's morphological typology is provided by an oracle so that the model can only search relevant patterns per language  $(\mathrm{U} + \mathrm{T})$ . A vowel inventory is also provided so that our system can discover the vowel change rules described in Section 4.3. MC and MF are run in two different configurations, one with semantic vectors  $(+\mathrm{v})$  and the other without vectors (more comparable to Morfessor, ParaMA, and our system).
We conduct the experiments with a data set containing 9 languages from diverse language families (Mott et al., 2020). The details of the data sets including the typological features for each language and the size of corpus that is used for training word vectors are shown in Table 2. The word lists used for training are extracted from the language pack created under the DARPA LORELEI (LOw REsource Languages and Emergent Incidents) program. The gold standard data, soon to be released by LDC, is annotated only with morpheme segmentations, and no data annotation was used in training. The languages with non-Latin scripts were romanized with the tools provided in the package.
6.2 Experimental Results and Analyses
Results are presented in Figure 1. The details are shown in Table 3 and Table 4. Both our unsupervised model (FU) and model with given typology  $(\mathrm{U} + \mathrm{T})$  achieve higher average F1 than previous work by a large margin, the highest on five of nine languages, and competitive results overall on the other four. Of the two systems, the typology feature oracle provided only slightly better average performance than fully unsupervised. As expected,
Figure 1: Comparison of different systems in F1 scores on the nine languages and their average. FU and  $\mathrm{U + T}$  are our systems. FU is fully unsupervised, while  $\mathrm{U + T}$  is unsupervised except given six flags for language typology.
Morf	MC	MF	MC+v	MF+v	PMA	U+T	FU
Aka	0.633	0.650	0.646	0.498	0.530	0.530	0.680	0.679
Hin	0.258	0.359	0.346	0.494	0.505	0.586	0.432	0.398
Hun	0.407	0.532	0.533	0.622	0.619	0.532	0.554	0.551
Ind	0.532	0.499	0.497	0.561	0.622	0.469	0.686	0.682
Rus	0.347	0.492	0.493	0.427	0.450	0.458	0.493	0.490
Spa	0.250	0.498	0.502	0.051	0.034	0.405	0.473	0.472
Swa	0.432	0.430	0.409	0.202	0.189	0.343	0.512	0.533
Tag	0.525	0.484	0.470	0.430	0.439	0.411	0.587	0.566
Tam	0.237	0.293	0.291	0.341	0.336	0.396	0.446	0.426
Avg	0.402	0.471	0.465	0.403	0.414	0.459	0.541	0.533
Table 3: Experimental results in F1 measures on the nine languages including our unsupervised (FU) and oracle  $(\mathrm{U} + \mathrm{T})$  system. The best score for each language is highlighted, considering each of our systems separately against previous work.
Morf	MC	MF	MC+v	MF+v	PMA	U+T	FU
P	0.618	0.387	0.391	0.504	0.523	0.514	0.525	0.495
R	0.317	0.647	0.623	0.370	0.383	0.428	0.576	0.593
F1	0.402	0.471	0.465	0.403	0.414	0.459	0.541	0.533
Table 4: Average performance of the systems in precision, recall and F1 measures. Best result in bold, considering all systems together.
given the very low-resource setting, the vector configuration harms the performance of both MC and MF in languages such as Akan, Spanish, Swahili and Tagalog. Even though Russian has a larger corpus, the vectors still harm performance, which we believe is due to its complicated morphology that demands many examples to train reliable vectors.
While having separate patterns for each morphology type seems to improve numbers, oracle information improves results only slightly, mostly on Hindi, Tagalog, and Tamil. Interestingly, the performance on Swahili has been noticeably decreased. Based on detailed observation, this is due to our infixation search providing an unexpected benefit for Swahili, a language with no linguistic infixation, but the final vowel pattern, by allowing us to capture string-internal linguistic suffixes as
Figure 2: The performance of our model with oracle typological features  $(\mathrm{U} + \mathrm{T})$  and with only prefixation and suffixation (Pref+Suf).
in the passive suffix -w- extracted from the verb kunyang'anywa here as  $bx_{-} < w > _{-}ex$ . In all, the performance of our model in either mode is better than the other systems we tested.
To test the contribution of morphological patterns other than prefixation and suffixation, we perform an ablation study, running the system with only prefixation and suffixation enabled. The results are shown in Figure 2. First, most performance for most languages is due to prefixation and suffixation since these are predominant for most languages. However, performance decreases measurably for Tagalog, Indonesian and Hindi due to the presence of more complex morphological patterns. This shows that modeling morphological features other than prefixation and suffixation has important benefits on languages with complicated morphology.
Our system, in both its configurations, achieves the highest average performance among those tested. It has other advantages as well. Firstly, although our model is evaluated in terms of morpheme boundaries, it produces much richer structures than that.
6678
It determines how a complex word is derived from another one through a particular morphological process such as prefixation, suffixation, infixation or full or partial reduplication. In comparison, other systems including Morpho-Chain, Morph-Forest, and ParaMA only deal with prefixes and suffixes. Our experiments as shown in Figure 2 indicate that modeling morphological patterns/processes other than prefixation and suffixation are useful.
Systems that directly find morpheme boundaries such as Morfessor are not aware of the particular morphological processes that a word's derivation goes through. So for infixed words, for example, even if the morpheme boundaries are correctly identified by such systems, they will incorrectly characterize the word as containing three morphemes rather than two. Such analyses are incorrect even though they are not penalized under a boundary-based evaluation metric.
By modeling different types of morphological structures, our system can be used to study the productivity of each morphological process and thus can be used for a quantitative analysis for theoretical morphological studies in linguistics. Figure 3 shows the number of instances of each type of morphological process generated by our fully unsupervised model. Suffixation and prefixation are the most common processes. Most of our test languages exhibit more suffixation than prefixation, but Swahili has more prefixation than suffixation, as expected for a Bantu language.
Figure 3 also shows that reduplication is rarer than other affixation. However, our model does discover full and left-partial reduplication successfully in languages that exhibit it. For example, about  $1\%$  of Akan words and fewer than  $1\%$  of Indonesian, Swahili and Tagalog words were analyzed with full or partial reduplication.
Infixation is challenging to correctly identify because infixes can appear in almost any position inside a word, and therefore generate a large search space. Our unsupervised system uses infixation to represent both true morphological infixation as in Tagalog as well as word-internal agglutinative suffixation as in Swahili, Hindi, and Tamil. This hurts the performance for Hindi and Tamil, but provides a benefit for Swahili as discussed above.
Finally, our system is fast, typically completing in several minutes, similar to ParaMA. Other systems including Morfessor, MC and MF typically require several hours, or even days on longer word
Figure 3: Normalized distribution of morphological patterns discovered by our unsupervised model for each language (top) and zoomed in on less frequent patterns (bottom). SUF: suffix, PREF: prefix, INF: infix, RED: full redduplication, LRED: left partial redduplication, RRED: right partial redduplication.
lists such as for Hungarian and Russian.
In this paper, we develop a model for morphological analysis that exploits typological features to achieve the best performance on a wide range of languages. The tool is publicly available here: https://github.com/xuhongzhi/ParaMA2. This unsupervised model can be quickly and easily extended to novel languages without data annotation or expert input. Combined with the ability to process infixation and reduplication, our system improves access for geographically diverse low-resource languages. Although the evaluation is based on segmentation points, our model outputs much richer structure. It can also tell us the productivity of each morphological process and thus can obtain much deeper knowledge in terms of morphological structures of languages.
Our next step will be to attempt to automate the determination of language typology, yielding somewhat better performance with a system requiring no human intervention per language at all. Future work will aim to extend the current model to capture particularly challenging morphological patterns such as templatic non-concatenative morphology and polysynthetic composition.
6679
We thank the rest of the University of Pennsylvania's LORELEI research team for the helpful discussions. We also thank the anonymous reviewers for their valuable and constructive comments for improving our paper. This research was funded by the DARPA LORELEI program under Agreement No. HR0011-15-2-0023.
Malin Ahlberg, Mans Hulden, and Markus Forsberg. 2014. Semi-supervised learning of morphological paradigms and lexicons. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 569-578.
Erwin Chan. 2006. Learning probabilistic paradigms for morphology in a latent class model. In Proceedings of the Eighth Meeting of the ACL Special Interest Group on Computational Phonology and Morphology, pages 69-78. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Arya D McCarthy, Katharina Kann, Sebastian Mielke, Garrett Nicolai, Miikka Silfverberg, et al. 2018. The conll-sigmorphon 2018 shared task: Universal morphological reinflation. arXiv preprint arXiv:1810.07125.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kubler, David Yarowsky, Jason Eisner, et al. 2017. Conllsigmorphon 2017 shared task: Universal morphological reinflection in 52 languages. arXiv preprint arXiv:1706.09031.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The sigmorphon 2016 shared task—morphological reincarnation. In Proceedings of the 2016 Meeting of SIGMORPHON, Berlin, Germany. Association for Computational Linguistics.
Mathias Creutz and KristaLAGus. 2002. Unsupervised discovery of morphemes. In Proceedings of the ACL-02 workshop on Morphological and phonological learning-Volume 6, pages 21-30. Association for Computational Linguistics.
Mathias Creutz and Krista Lagus. 2005. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0. Helsinki University of Technology.
Mathias Creutz and Krista Lagus. 2007. Unsupervised models for morpheme segmentation and morphology learning. ACM Transactions on Speech and Language Processing (TSLP), 4(3):1-34.
Markus Dreyer and Jason Eisner. 2011. Discovering morphological paradigms from plain text using a dirichlet process mixture model. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 616-627. Association for Computational Linguistics.
Matthew S. Dryer. 2013. Prefixing vs. suffixing in inflectional morphology. In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig.
John Goldsmith. 2001. Unsupervised learning of the morphology of a natural language. Computational linguistics, 27(2):153-198.
John Goldsmith. 2006. An algorithm for the unsupervised learning of morphology. Natural Language Engineering, 12(4):353-371.
Sharon Goldwater and Mark Johnson. 2004. Priors in bayesian learning of phonological rules. In Proceedings of the 7th Meeting of the ACL Special Interest Group in Computational Phonology: Current Themes in Computational Phonology and Morphology, pages 35-42. Association for Computational Linguistics.
Constantine Lignos. 2010. Learning from unseen data. In Proceedings of the Morpho Challenge 2010 Workshop, pages 35-38.
Jiaming Luo, Karthik Narasimhan, and Regina Barzilay. 2017. Unsupervised learning of morphological forests. Transactions of the Association for Computational Linguistics, 5:353-364.
Justin Mott, Ann Bies, Stephanie Strassel, Jordan Kodner, Caitlin Richter, Hongzhi Xu, and Mitchell Marcus. 2020. Morphological segmentation for low resource languages. In International Conference on Language Resource and Evaluation (LREC).
Victor Mugari. 2013. Object marking restrictions on shona causative and applicative constructions. Southern African Linguistics and Applied Language Studies, 31(2):151-160.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. 2015. An unsupervised method for uncovering morphological chains. Transactions of the Association for Computational Linguistics, 3:157-167.
Cornelia Parkes, Alexander M. Malek, and Mitchell P. Marcus. 1998. Towards unsupervised extraction of verb paradigms from large corpora. In Proceedings of the Sixth Workshop on Very Large Corpora (COLING-ACL).
Hoifung Poon, Colin Cherry, and Kristina Toutanova. 2009. Unsupervised morphological segmentation with log-linear models. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 209-217. Association for Computational Linguistics.
Carl Rubino. 2001. Pangasinan. In Jane Garry and Carl Rubino, editors, Encyclopedia of the World's Languages: Past and Present, pages 539-542. H.W. Wilson Press, New York / Dublin.
Carl Rubino. 2013. Reduplication. In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig.
Patrick Schone and Daniel Jurafsky. 2001. Knowledge-free induction of inflectional morphologies. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, pages 1-9.
Benjamin Snyder and Regina Barzilay. 2008. Unsupervised multilingual learning for morphological segmentation. In Proceedings of the 46th Annual Meeting on Association for Computational Linguistics, pages 737-745.
Radu Soricut and Franz Och. 2015. Unsupervised morphology induction using word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1627-1637.
Richard William Sproat. 1992. Morphology and computation. MIT press.
Gregory T Stump. 2001. Inflectional morphology: A theory of paradigm structure, volume 93. Cambridge University Press.
Sami Virpioja, Peter Smit, Stig-Arne Gronroos, Mikko Kurimo, et al. 2013. Morfessor 2.0: Python implementation and extensions for morfessor baseline.
Hongzhi Xu, Mitchell Marcus, Charles Yang, and Lyle Ungar. 2018. Unsupervised morphology learning with statistical paradigms. In Proceedings of the 27th International Conference on Computational Linguistics, pages 44-54.
6681