Multimodal, Multilingual Grapheme-to-Phoneme Conversion for Low-Resource Languages

Grapheme-to-phoneme conversion (g2p) is the task of predicting the pronunciation of words from their orthographic representation.Historically, g2p systems were transition-or rulebased, making generalization beyond a monolingual (high resource) domain impractical.Recently, neural architectures have enabled multilingual systems to generalize widely; however, all systems to date have been trained only on spelling-pronunciation pairs.We hypothesize that the sequences of IPA characters used to represent pronunciation do not capture its full nuance, especially when cleaned to facilitate machine learning.We leverage audio data as an auxiliary modality in a multi-task training process to learn a more optimal intermediate representation of source graphemes; this is the first multimodal model proposed for multilingual g2p.Our approach is highly effective: on our in-domain test set, our multimodal model reduces phoneme error rate to 2.46%, a more than 65% decrease compared to our implementation of a unimodal spellingpronunciation model-which itself achieves state-of-the-art results on the Wiktionary test set.The advantages of the multimodal model generalize to wholly unseen languages, reducing phoneme error rate on our out-of-domain test set to 6.39% from the unimodal 8.21%, a more than 20% relative decrease.Furthermore, our training and test sets are composed primarily of low-resource languages, demonstrating that our multimodal approach remains useful when training data are constrained.* * Equal contribution ogy which has to convert between these two representations: namely text-to-speech and speechrecognition systems.Very early grapheme to phoneme systems were monolingual and often restricted to English due to dataset availability (Weide, 1998;Kingsbury et al., 1997;Sejnowski, 1987).These early systems were designed to address the problem of intra-language discrepancies through rule based transition systems.These systems required painstaking tailoring to individual languages, and their performance was largely limited to that language's domain.Recent work has extended finite state automata constructed in this way for high resource languages to very similar low resource languages by applying distance metrics and linguistic expertise (Deri and Knight, 2016), but this approach is limited in application and performance.Relieving some of the burden of technical expertise, statistical methods surpassed rule-based ones, with emphasis on joint sequence modeling (Chen, 2003;Bisani and Ney, 2008;Jiampojamarn et al., 2007).These methods improved performance, but they mandate explicit training alignments.This can be avoided by using neural attentional models, as in Toshniwal and Livescu (2016).Their work makes clear the parallel between this sequence prediction task and more traditional machine translation; this parallel inspires the model proposed in Peters et al. ( 2017), which, motivated by similarities in vocabularies, spellings, writing systems, and phonemic inventories between low and high resource languages, applies multilingual MT techniques to train a massively multilingual g2p system.This application is effective, but it-like all work on this task before it-neglects perhaps the most rich source of information on pronunciation available: speech data.All existing grapheme to phoneme systems have been trained on spelling-