SlowGuess/ACL_2020_OCR huggingface dataset:
Our initial experiment with the base data set achieved  $100\%$  coverage and  $68.3\%$  accuracy on the test set. When confronted by the same problem, Moeller et al. (2018) decided to collapse ambiguous tags into an underspecified meta-tag. For example, for the Kunwinjku data, we might collapse the four tags above into [3pl.3pl.nonpast]. However, doing so results in a potential loss of information. Given the wider sentential context, the pronominal could be possibly be disambiguated, so long as the distinction is preserved and all equally-valid analyses are returned.
Further, as Schwartz et al. (2019) point out, in the Yupik language it is possible for this ambiguity to exist across other categories which are not easily collapsed. In Kunwinjku, an example of this would be the pronominals [1sg.2.past] and [3sg.past] which differ in terms of number and valency, and yet share the same null surface form. Their differences are such that they can not be easily collapsed into a single meta-tag. Therefore we do not penalize the model for producing any variation of equally valid analyses given the surface form, and for each model we adjust the evaluation for syncretism in a post-processing step.
b i k a n j ng u n e n g  $\rightarrow$  [ 3sg . 3Hsg . PST] [ BPIN ] ng u [ PP ]
6657
All of the neural models outperform the FST in terms of accuracy and coverage (Fig. 8). However, the FST is more precise, and this may be useful in certain application contexts. The best model is Base+halluc+resample, which improves on the FST by 10.3 percentage points. On the morph-level, we see that the neural models containing the hallucinated reduplication data outperform the base neural model (Fig. 9).
Acc	Cov	Precision
FST	84.4	88.5	95.4
Base	89.1	100	89.1
Base+halluc	93.7	100	93.7
Base+halluc+resample	94.7	100	94.7
Figure 8: All-or-nothing accuracy and coverage of the three morphological analyzer models
Precision	Recall	F1
Base	88.8	89.9	89.0
Base+halluc	91.6	92.6	91.8
Base+halluc+resample	93.7	93.6	93.4
We posited that the difficulties encountered by the FST model—namely reduplication, out-of-vocabulary items, and spelling variation—could be at least partially addressed by training a neural model on character and tag sequences, and hallucinating instances of reduplication into the training set. For the most part, this held true, as we see gains across all error classes (cf. Sec. 3.2). Here we report performance with respect to the three largest error classes: reduplication, OOV verbs, and OOV nouns.
6.1 Reduplication
As expected, neither the FST nor the Base neural model succeeds in recognizing reduplication. It would be impossible, as the REDUP tag does not appear in either of their vocabularies.
The Base+halluc model's performance gain over the Base model can be accounted for entirely by the fact that it achieved  $100\%$  recall of reduplicative structure. Precision, on the other hand was  $57.9\%$ . Looking at the errors, we find that the imprecise predictions were all applied to instances about which the system was already wrong in previous
Figure 9: Morph-level performance of shallow neural sequence models. Macro P/R/F1 across all morph tags.
Unseen Verbs	Base+halluc+resample	✓/×
wobekkang	[GIN]bekka	×
ngakahbanjminj	[GIN][REDUP]me	×
ngarrukkendi	dukkendi	✓
kamenyime	[GIN]yime	×
yimalngdarrkiddi	darrke[PERSIST]	×
ngamdollkang	[DIR][GIN]ka	×
dolkkang	[GIN]ka	×
karrukmirri	dukmirri	✓
ngurrimirndemornnamerren	mornname	✓
Unseen GIN/BPIN/ASP	Base+halluc+resample	✓/×
kannjilmgarnbom	[GIN]	×
yibenkangemarnbom	[REDUP]	×
kankangemurrngrayekwong	[GIN]	×
kankangemurrngrayekwong	[BPIN]	✓
kankangemurrngrayekwong	[REDUP]	×
kankangemarnbom	[REDUP]	×
ngarribangmemarnbuyi	[BPIN]	×
yimalngdarrkiddi	[GIN][REDUP]	×
Figure 10: Column 1 shows the list of verbs and nouns (in bold) which are are unseen in the FST lexicon. Column 2 is the Base neural model's prediction covering the character sequence corresponding to the unseen item. Column 3 indicates whether the neural model's analysis of the morph is correct.
models, meaning that the impact of reduplicative hallucination between models was only positive. In the Base+halluc+resample model, recall of reduplicative structure was also  $100\%$ , and precision increased slightly to  $58.8\%$ .
6.2 Discovering New Lexical Items
The neural models correctly identify some unseen verb stems, but still show room for improvement. We observe a tendency across all neural models to predict verb stems which have been seen in training, and which are also a substring of the observed unknown root. For example, the training set does not contain any verbs with the root  $\text{dolkka}$ , but it shows up 3 times in the test set. The analyses of all  $\text{dolkka}$ -rooted verbs were the same in both the Base+halluc and Base+halluc+resample models: they propose  $\text{ka}$ , a known root from the training set, and presume  $\text{dolk-}$  to be an incorporeal  $\text{noun}^2$ . Figure 10 shows a sample of OOV verb stems and nouns from the test set. In the unseen verbs table, this behavior of preferring previously observed verb stems is the cause of error in every case.
Further difficulty comes in distinguishing between general (GIN) and body-part (BPIN) incorporated noun classes. The low rate of success in positing unknown incorporated nouns is, in
2Possibly by virtue of its orthographic proximity to  $bolk-$  a common general incorporeal noun which means "land."
6658
large part, attributed to the fact that the large GIN and open BPIN classes often occur adjacent to each other and to the root. The neural model has difficulty making useful predictions when multiple morphs in this region are previously unobserved.
Overall, the Base+halluc+resample model correctly posited  $33\%$  of unseen stems, and  $12.5\%$  of unseen nouns from the FST error analyses.
6.3 Impact of distributional information
This technique to approximate distributional information led to a small improvement in overall accuracy, and in tag-level P/R/F1. We had expected that this information might help the neural models learn something about the relative frequencies of GINs or BPINs, which could help make decisions about how to draw the boundary between unseen stems and unseen incorporated nominals. Instead, we saw distributive information helped to disambiguate the boundaries between morph classes with fewer members.
One representative example is the case of yikimang, whose root is kimang. Before resample, the neural models interpret the yi- as the comitative prefix yi-, and injects a spurious COM tag into the analysis. After resample, it correctly omits the COM tag, interpreting yi- as the 2nd person singular pronominal. In the unfiltered FST-generated training data, COM occurs in  $53\%$  of instances. In the resampled data, it occurs in  $22\%$  of instances. When all morph labels are equally likely to occur, the model is just as likely to predict any morph label compatible with the character sequence. Resampling the training data according to a more realistic distribution leads to stronger morph transition priors, which tip the scale in favor of the analysis with a more likely tag sequence.
We have shown that complex features of polysynthetic morphology, such as reduplication and distributional morphotactic information, can be simulated in the dataset and used to train a robust neural morphological analyzer for a polysynthetic language. In particular, we showed that a robust neural model can be bootstrapped in a relatively short space of time from an incomplete FST.
This work represents a successful first iteration of a process whereby the morphological model can be continually improved. Indeed, the concept of
bootstrapping a model implies an iterative development story where much of the scaffolding used in early efforts will eventually fall away. For example, once the bootstrapped model has been used to tag verbs containing reduplication, we can confirm the model's high-confidence predictions and retrain. In this second iteration, we may find that we no longer need to hallucinate reduplication because it is sufficiently represented in the new training set. Similarly, once we have applied the complete neural model to a corpus of natural text, we will no longer need to approximate distributional information. For researchers developing robust morphological analyzers for low resource, morphologically complex languages, this work represents a template of model development which is well-suited for the context.
Producing a viable morphological analyzer is the first step towards building improved dictionary search interfaces, spell-checking tools, and computer-assisted language learning applications for communities who speak low-resource languages. The pattern of training robust systems on data that has been augmented by the knowledge captured in symbolic systems could be applied to areas outside of morphological analysis, and is a promising avenue of future exploration.
We are grateful for the support of the Warddeken Rangers of West Arnhem. This work was covered by a research permit from the Northern Land Council, and was sponsored by the Australian government through a PhD scholarship, and grants from the Australian Research Council and the Indigenous Language and Arts Program. We are grateful to four anonymous reviewers for their feedback on an earlier version of this paper.
6659
Antonios Anastasopoulos and Graham Neubig. 2019. Pushing the limits of low-resource morphological inflection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 984-996, Hong Kong, China. Association for Computational Linguistics.
Vasilisa Andriyanets and Francis Tyers. 2018. A prototype finite-state morphological analyser for Chukchi. In Proceedings of the Workshop on Computational Modeling of Polysynthetic Languages, pages 31-40, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Antti Arppe, Christopher Cox, Mans Hulden, Jordan Lachler, Sjur N Moshagen, Miikka Silfverberg, and Trond Trosterud. 2017. Computational Modeling of Verbs in Dene Languages: The Case of Tsuut'ina. Working Papers in Athabaskan (Dene) Languages, pages 51-69.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Brett Baker and Mark Harvey. 2003. Word Structure in Australian Languages. Australian Journal of Linguistics, 23:3-33.
Kenneth R Beesley and Lauri Karttunen. 2003. Finite-state morphology: Xerox tools and techniques. CSLI, Stanford.
Bininj Kunwok Language Project. 2019. Bininj Kunwok: kunwok dja mankarre kadberre--our language, our culture. https://bininjkunwok.org.au/. Accessed: 2019-10-10.
Ronald Cardenas and Daniel Zeman. 2018. A morphological analyzer for shipibo-konibo. In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 131-139.
Emily Chen and Lane Schwartz. 2018. A morphological analyzer for St. Lawrence Island/Central Siberian Yupik. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation.
Christopher Culy. 1985. The complexity of the vocabulary of Bambara. In *The Formal Complexity of Natural Language*, pages 349–357. Springer.
Mark Dras, François Lareau, Benjamin Borschinger, Robert Dale, Yasaman Motazedi, Owen C Rambow, Myfany Turpin, and Morgan Elizabeth Ulinski. 2012. Complex predicates in arrernte.
Steven Etherington and Narelle Etherington. 1998. Kunwinjku Kunwok: A Short Introduction to Kunwinjku Language and Society: with Extra Notes on Gundjeihmi. Gunbalanya: Kunwinjku Language Centre.
Nicholas Evans. 2003. A Pan-dialectal Grammar of Bininj Gun-Wok (Arnhem Land): Mayali, Kunwinjku and Kune. Pacific Linguistics. Australian National University.
Mans Hulden. 2009. Foma: a finite-state compiler and library. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 29-32. Association for Computational Linguistics.
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, et al. 2018. Marian: Fast neural machine translation in c++. arXiv preprint arXiv:1804.00344.
Ronald M Kaplan and Martin Kay. 1994. Regular models of phonological rule systems. Computational linguistics, 20(3):331-378.
Jordan Lachler, Lene Antonsen, Trond Trosterud, Sjur Moshagen, and Antti Arppe. 2018. Modeling Northern Haida verb morphology. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation.
William Lane and Steven Bird. 2019. Towards A Robust Morphological Analyzer for Kunwinjku. In Proceedings of ALTA 2019. None yet.
Krister Lindén, Erik Axelson, Senka Drobac, Sam Hardwick, Juha Kuokkala, Jyrki Niemi, Tommi A Pirinen, and Miikka Silfverberg. 2013. Hfst—a system for creating nlp tools. In International workshop on systems and frameworks for computational morphology, pages 53-71. Springer.
Patrick Littell. 2018. Finite-state morphology for kwak'wala: A phonological approach. In Proceedings of the Workshop on Computational Modeling of Polysynthetic Languages, pages 21-30, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Patrick Littell, Anna Kazantseva, Roland Kuhn, Aidan Pine, Antti Arppe, Christopher Cox, and Marie Odile Junker. 2018. Indigenous language technologies in canada: Assessment, challenges, and successes. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2620-2632.
Jeffrey Micher. 2017. Improving coverage of an inuktitut morphological analyzer using a segmental recurrent neural network. In Proceedings of the 2nd Workshop on the Use of Computational Methods in the Study of Endangered Languages, pages 101-106.
Sarah Moeller, Ghazaleh Kazeminejad, Andrew Cowell, and Mans Hulden. 2018. A neural morphological analyzer for arapaho verbs learned from a finite state transducer. In Proceedings of the Workshop on Computational Modeling of Polysynthetic Languages, pages 12-20.
Brian Roark, Richard Sproat, and Richard William Sproat. 2007. Computational approaches to morphology and syntax, volume 4. Oxford University Press.
Carl Rubino. 2005. Reduplication: Form, function and distribution. Studies on reduplication, pages 11-29.
Lane Schwartz, Emily Chen, Benjamin Hunt, and Sylvia Schreiner. 2019. Bootstrapping a Neural Morphological Analyzer for St. Lawrence Island Yupik from a Finite-State Transducer. In Proceedings of the Workshop on Computational Methods for Endangered Languages, pages 87-96.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891.
Miikka Silfverberg, Adam Wiemerslage, Ling Liu, and Lingshuang Jack Mao. 2017. Data augmentation for morphological reinflation. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinfection, pages 90-99, Vancouver. Association for Computational Linguistics.
We provide the MarianNMT configuration settings used for all neural models in this work.
6661