Question: The methods in the paper share the use of what model in a generative modeling framework?
Evidence: Subword segmentation has become a standard preprocessing step in many neural approaches to natural language processing (NLP) tasks, e.g Neural Machine Translation (NMT) BIBREF0 and Automatic Speech Recognition (ASR) BIBREF1. Word level modeling suffers from sparse statistics, issues with Out-of-Vocabulary (OOV) words, and heavy computational cost due to a large vocabulary. Word level modeling is particularly unsuitable for morphologically rich languages, but subwords are commonly used for other languages as well. Subword segmentation is best suited for languages with agglutinative morphology.
While rule-based morphological segmentation systems can achieve high quality, the large amount of human effort needed makes the approach problematic, particularly for low-resource languages. The systems are language dependent, necessitating use of multiple tools in multilingual setups. As a fast, cheap and effective alternative, data-driven segmentation can be learned in a completely unsupervised manner from raw corpora. Unsupervised morphological segmentation saw much research interest until the early 2010's; for a survey on the methods, see hammarstrom2011unsupervised. Semi-supervised segmentation with already small amounts of annotated training data was found to improve the accuracy significantly when compared to a linguistic segmentation; see ruokolainen2016comparative for a survey. While this line of research has been continued in supervised and more grammatically oriented tasks BIBREF2, the more recent work on unsupervised segmentation is less focused on approximating a linguistically motivated segmentation. Instead, the aim has been to tune subword segmentations for particular applications. For example, the simple substitution dictionary based Byte Pair Encoding segmentation algorithm BIBREF3, first proposed for NMT by sennrich2015neural, has become a standard in the field. Especially in the case of multilingual models, training a single language-independent subword segmentation method is preferable to linguistic segmentation BIBREF4.
In this study, we compare three existing and one novel subword segmentation method, all sharing the use of a unigram language model in a generative modeling framework. The previously published methods are Morfessor Baseline BIBREF5, Greedy Unigram Likelihood BIBREF6, and SentencePiece BIBREF7. The new Morfessor variant proposed in this work is called Morfessor EM+Prune.
The contributions of this article are
a better training algorithm for Morfessor Baseline, with reduction of search error during training, and improved segmentation quality for English, Finnish and Turkish;
comparing four similar segmentation methods, including a close look at the SentencePiece reference implementation, highlighting details omitted from the original article BIBREF7;
and showing that the proposed Morfessor EM+Prune with particular hyper-parameters yields SentencePiece.
In this study, we compare three existing and one novel subword segmentation method, all sharing the use of a unigram language model in a generative modeling framework.
Answer: A unigram language model.