Query: low resource language translation

title: The Source Target Domain Mismatch Problem in Machine Translation
abstract: While we live in an increasingly interconnected world, different places still exhibit strikingly different cultures and many events we experience in our every day life pertain only to the specific place we live in. As a result, people often talk about different things in different parts of the world. In this work we study the effect of local context in machine translation and postulate that this causes the domains of the source and target language to greatly mismatch. We first formalize the concept of source target domain mismatch, propose a metric to quantify it, and provide empirical evidence for its existence. We conclude with an empirical study of how source target domain mismatch affects training of machine translation systems on low resource languages. While this may severely affect back translation, the degradation can be alleviated by combining back translation with self training and by increasing the amount of target side monolingual data.

title: AmericasNLI: Evaluating Zero shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low resource Languages
abstract: Pretrained multilingual models are able to perform cross lingual transfer in a zero shot setting, even for languages unseen during pretraining. However, prior work evaluating performance on unseen languages has largely been limited to low level, syntactic tasks, and it remains unclear if zero shot learning of high level, semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, an extension of XNLI (Conneau et al. 2018) to 10 indigenous languages of the Americas. We conduct experiments with XLM R, testing multiple zero shot and translation based approaches. Additionally, we explore model adaptation via continued pretraining and provide an analysis of the dataset by considering hypothesis only models. We find that XLM R's zero shot performance is poor for all 10 languages, with an average performance of 38.62% Continued pretraining offers improvements, with an average accuracy of 44.05% Surprisingly, training on poorly translated data by far outperforms all other methods with an accuracy of 48.72%

title: A Survey on Recent Approaches for Natural Language Processing in Low Resource Scenarios
abstract: Deep neural networks and huge language models are becoming omnipresent in natural language applications. As they are known for requiring large amounts of training data, there is a growing body of work to improve the performance in low resource settings. Motivated by the recent fundamental changes towards neural models and the popular pre train and fine tune paradigm, we survey promising approaches for low resource natural language processing. After a discussion about the different dimensions of data availability, we give a structured overview of methods that enable learning when training data is sparse. This includes mechanisms to create additional labeled data like data augmentation and distant supervision as well as transfer learning settings that reduce the need for target supervision. A goal of our survey is to explain how these methods differ in their requirements as understanding them is essential for choosing a technique suited for a specific low resource setting. Further key aspects of this work are to highlight open issues and to outline promising directions for future research.

title: Self Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarios
abstract: Unsupervised neural machine translation (UNMT) that relies solely on massive monolingual corpora has achieved remarkable results in several translation tasks. However, in real world scenarios, massive monolingual corpora do not exist for some extremely low resource languages such as Estonian, and UNMT systems usually perform poorly when there is not adequate training corpus for one language. In this paper, we first define and analyze the unbalanced training data scenario for UNMT. Based on this scenario, we propose UNMT self training mechanisms to train a robust UNMT system and improve its performance in this case. Experimental results on several language pairs show that the proposed methods substantially outperform conventional UNMT systems.

title: Multilingual Offensive Language Identification for Low resource Languages
abstract: Offensive content is pervasive in social media and a reason for concern to companies and government organizations. Several studies have been recently published investigating methods to detect the various forms of such content (e.g. hate speech, cyberbulling, and cyberaggression) The clear majority of these studies deal with English partially because most annotated datasets available contain English data. In this paper, we take advantage of available English datasets by applying cross lingual contextual word embeddings and transfer learning to make predictions in low resource languages. We project predictions on comparable data in Arabic, Bengali, Danish, Greek, Hindi, Spanish, and Turkish. We report results of 0.8415 F1 macro for Bengali in TRAC 2 shared task [23] 0.8532 F1 macro for Danish and 0.8701 F1 macro for Greek in OffensEval 2020  0.8568 F1 macro for Hindi in HASOC 2019 shared task [27] and 0.7513 F1 macro for Spanish in in SemEval 2019 Task 5 (HatEval)  showing that our approach compares favourably to the best systems submitted to recent shared tasks on these three languages. Additionally, we report competitive performance on Arabic, and Turkish using the training and development sets of OffensEval 2020 shared task. The results for all languages confirm the robustness of cross lingual contextual embeddings and transfer learning for this task.

title: Revisiting Low Resource Status of Indian Languages in Machine Translation
abstract: Indian language machine translation performance is hampered due to the lack of large scale multi lingual sentence aligned corpora and robust benchmarks. Through this paper, we provide and analyse an automated framework to obtain such a corpus for Indian language neural machine translation (NMT) systems. Our pipeline consists of a baseline NMT system, a retrieval module, and an alignment module that is used to work with publicly available websites such as press releases by the government. The main contribution towards this effort is to obtain an incremental method that uses the above pipeline to iteratively improve the size of the corpus as well as improve each of the components of our system. Through our work, we also evaluate the design choices such as the choice of pivoting language and the effect of iterative incremental increase in corpus size. Our work in addition to providing an automated framework also results in generating a relatively larger corpus as compared to existing corpora that are available for Indian languages. This corpus helps us obtain substantially improved results on the publicly available WAT evaluation benchmark and other standard evaluation benchmarks.

title: Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages
abstract: Unsupervised translation has reached impressive performance on resource rich language pairs such as English French and English German. However, early studies have shown that in more realistic settings involving low resource, rare languages, unsupervised translation performs poorly, achieving less than 3.0 BLEU. In this work, we show that multilinguality is critical to making unsupervised systems practical for low resource settings. In particular, we present a single model for 5 low resource languages (Gujarati, Kazakh, Nepali, Sinhala, and Turkish) to and from English directions, which leverages monolingual and auxiliary parallel data from other high resource language pairs via a three stage training scheme. We outperform all current state of the art unsupervised baselines for these languages, achieving gains of up to 14.4 BLEU. Additionally, we outperform strong supervised baselines for various language pairs as well as match the performance of the current state of the art supervised model for Nepali English. We conduct a series of ablation studies to establish the robustness of our model under different degrees of data quality, as well as to analyze the factors which led to the superior performance of the proposed approach over traditional unsupervised models.

title: From bilingual to multilingual neural based machine translation by incremental training
abstract: A common intermediate language representation in neural machine translation can be used to extend bilingual systems by incremental training. We propose a new architecture based on introducing an interlingual loss as an additional training objective. By adding and forcing this interlingual loss, we can train multiple encoders and decoders for each language, sharing among them a common intermediate representation. Translation results on the low resource tasks (Turkish English and Kazakh English tasks) show a BLEU improvement of up to 2.8 points. However, results on a larger dataset (Russian English and Kazakh English) show BLEU losses of a similar amount. While our system provides improvements only for the low resource tasks in terms of translation quality, our system is capable of quickly deploying new language pairs without the need to retrain the rest of the system, which may be a game changer in some situations. Specifically, what is most relevant regarding our architecture is that it is capable of: reducing the number of production systems, with respect to the number of languages, from quadratic to linear; incrementally adding a new language to the system without retraining the languages already there; and allowing for translations from the new language to all the others present in the system.

title: When Word Embeddings Become Endangered
abstract: Big languages such as English and Finnish have many natural language processing (NLP) resources and models, but this is not the case for low resourced and endangered languages as such resources are so scarce despite the great advantages they would provide for the language communities. The most common types of resources available for low resourced and endangered languages are translation dictionaries and universal dependencies. In this paper, we present a method for constructing word embeddings for endangered languages using existing word embeddings of different resource rich languages and the translation dictionaries of resource poor languages. Thereafter, the embeddings are fine tuned using the sentences in the universal dependencies and aligned to match the semantic spaces of the big languages; resulting in crosslingual embeddings. The endangered languages we work with here are Erzya, Moksha, Komi Zyrian and Skolt Sami. Furthermore, we build a universal sentiment analysis model for all the languages that are part of this study, whether endangered or not, by utilizing cross lingual word embeddings. The evaluation conducted shows that our word embeddings for endangered languages are well aligned with the resource rich languages, and they are suitable for training task specific models as demonstrated by our sentiment analysis model which achieved a high accuracy. All our cross lingual word embeddings and the sentiment analysis model have been released openly via an easy to use Python library.

title: Low Resource Machine Translation for Low Resource Languages: Leveraging Comparable Data, Code Switching and Compute Resources
abstract: We conduct an empirical study of unsupervised neural machine translation (NMT) for truly low resource languages, exploring the case when both parallel training data and compute resource are lacking, reflecting the reality of most of the world's languages and the researchers working on these languages. We propose a simple and scalable method to improve unsupervised NMT, showing how adding comparable data mined using a bilingual dictionary along with modest additional compute resource to train the model can significantly improve its performance. We also demonstrate how the use of the dictionary to code switch monolingual data to create more comparable data can further improve performance. With this weak supervision, our best method achieves BLEU scores that improve over supervised results for English Gujarati +18.88) English Kazakh +5.84) and English Somali +1.16) showing the promise of weakly supervised NMT for many low resource languages with modest compute resource in the world. To the best of our knowledge, our work is the first to quantitatively showcase the impact of different modest compute resource in low resource NMT.