GenAI4ELab/papercli-papers huggingface dataset:
Decouple knowledge from paramters for plug-and-play language modeling
Authors: Xin Cheng; Yankai Lin; Xiuying Chen; Dongyan Zhao; Rui Yan
Venue: ACL 2023 (Findings of the Association for Computational Linguistics: ACL 2023)

Abstract: Pre-trained language models (PLM) have made impressive results in a wide range of NLP tasks and it has been revealed that one of the key factors to their success is the parameters of these models implicitly learn various types of knowledge in the pre-training corpus. However, encoding knowledge implicitly in the model parameters has two fundamental drawbacks. First, the knowledge is neither editable nor scalable once the model is trained, which is especially problematic in that knowledge is consistently evolving. Second, it lacks interpretability and prevents us from understanding what kind of knowledge PLM needs to solve a certain task. In this paper, we introduce {pasted macro ‘MODEL’}, a pre-training model with differentiable plug-in memory (DPM). The key intuition behind is to decouple the knowledge storage from model parameters with an editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the {pasted macro ‘MEMORY’}. We conduct extensive experiments under various settings to justify this design choice. In domain adaptation setting, {pasted macro ‘MODEL’} could be easily adapted to different domains with pluggable in-domain memory—obtaining 3.95 F1 improvements across four domains, without any in-domain training. {pasted macro ‘MODEL’} could also keep absorbing new knowledge after pre-training is done by knowledge updating operation in the {pasted macro ‘MEMORY’} without re-training. Finally, we show that by incorporating training samples into {pasted macro ‘MEMORY’} with knowledge prompting, {pasted macro ‘MODEL’} could further be improved by the instruction of in-task knowledge.
Use of NLP in the Context of Belief states of Ethnic Minorities in Latin America
Authors: Olga Kellert; Mahmud Zaman
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: The major goal of our study is to test methods in NLP in the domain of health care education related to Covid-19 of vulnerable groups such as indigenous people from Latin America. In order to achieve this goal, we asked participants in a survey questionnaire to provide answers about health related topics. We used these answers to measure the health education status ofour participants. In this paper, we summarize the results from our NLP-application on the participants’ answers. In the first experiment, we use embeddings-based tools to measure the semantic similarity between participants’ answers and “expert” or “reference” answers. In the second experiment, we use synonym-based methods to classify answers under topics. We compare the results from both experiments with human annotations. Our results show that the tested NLP-methods reach a significantly lower accuracy score than human annotations in both experiments. We explain this difference by the assumption that human annotators are much better in pragmatic inferencing necessary to classify the semantic similarity and topic classification of answers.
Neural Machine Translation through Active Learning on low-resource languages: The case of Spanish to Mapudungun
Authors: María Begoña Pendas; Andrés Carvallo; Carlos Aspillaga
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: Active learning is an algorithmic approach that strategically selects a subset of examples for labeling, with the goal of reducing workload and required resources. Previous research has applied active learning to Neural Machine Translation (NMT) for high-resource or well-represented languages, achieving significant reductions in manual labor. In this study, we explore the application of active learning for NMT in the context of Mapudungun, a low-resource language spoken by the Mapuche community in South America. Mapudungun was chosen due to the limited number of fluent speakers and the pressing need to provide access to content predominantly available in widely represented languages. We assess both model-dependent and model-agnostic active learning strategies for NMT between Spanish and Mapudungun in both directions, demonstrating that we can achieve over 40% reduction in manual translation workload in both cases.
Understanding Native Language Identification for Brazilian Indigenous Languages
Authors: Paulo Cavalin; Pedro Domingues; Julio Nogima; Claudio Pinhanez
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: We investigate native language identification (LangID) for Brazilian Indigenous Languages (BILs), using the Bible as training data. Our research extends from previous work, by presenting two analyses on the generalization of Bible-based LangID in non-biblical data. First, with newly collected non-biblical datasets, we show that such a LangID can still provide quite reasonable accuracy in languages for which there are more established writing standards, such as Guarani Mbya and Kaigang, but there can be a quite drastic drop in accuracy depending on the language. Then, we applied the LangID on a large set of texts, about 13M sentences from the Portuguese Wikipedia, towards understanding the difficulty factors may come out of such task in practice. The main outcome is that the lack of handling other American indigenous languages can affect considerably the precision for BILs, suggesting the need of a joint effort with related languages from the Americas.
Codex to corpus: Exploring annotation and processing for an open and extensible machine-readable edition of the Florentine Codex
Authors: Francis Tyers; Robert Pugh; Valery Berthoud F.
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: This paper describes an ongoing effort to create, from the original hand-written text, a machine-readable, linguistically-annotated, and easily-searchable corpus of the Nahuatl portion of the Florentine Codex, a 16th century Mesoamerican manuscript written in Nahuatl and Spanish. The Codex consists of 12 books and over 300,000 tokens. We describe the process of annotating 3 of these books, the steps of text preprocessing undertaken, our approach to efficient manual processing and annotation, and some of the challenges faced along the way. We also report on a set of experiments evaluating our ability to automate the text processing tasks to aid in the remaining annotation effort, and find the results promising despite the relatively low volume of training data. Finally, we briefly present a real use case from the humanities that would benefit from the searchable, linguistically annotated corpus we describe.
Developing finite-state language technology for Maya
Authors: Robert Pugh; Francis Tyers; Quetzil Castañeda
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: We describe a suite of finite-state language technologies for Maya, a Mayan language spoken in Mexico. At the core is a computational model of Maya morphology and phonology using a finite-state transducer. This model results in a morphological analyzer and a morphologically-informed spell-checker. All of these technologies are designed for use as both a pedagogical reading/writing aid for L2 learners and as a general language processing tool capable of supporting much of the natural variation in written Maya. We discuss the relevant features of Maya morphosyntax and orthography, and then outline the implementation details of the analyzer. To conclude, we present a longer-term vision for these tools and their use by both native speakers and learners.
Modelling the Reduplicating Lushootseed Morphology with an FST and LSTM
Authors: Jack Rueter; Mika Hämäläinen; Khalid Alnajjar
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: In this paper, we present an FST based approach for conducting morphological analysis, lemmatization and generation of Lushootseed words. Furthermore, we use the FST to generate training data for an LSTM based neural model and train this model to do morphological analysis. The neural model reaches a 71.9% accuracy on the test data. Furthermore, we discuss reduplication types in the Lushootseed language forms. The approach involves the use of both attested instances of reduplication and bare stems for applying a variety of reduplications to, as it is unclear just how much variation can be attributed to the individual speakers and authors of the source materials. That is, there may be areal factors that can be aligned with certain types of reduplication and their frequencies.
Fine-tuning Sentence-RoBERTa to Construct Word Embeddings for Low-resource Languages from Bilingual Dictionaries
Authors: Diego Bear; Paul Cook
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: Conventional approaches to learning word embeddings (Mikolov et al., 2013; Pennington et al., 2014) are limited to relatively few languages with sufficiently large training corpora. To address this limitation, we propose an alternative approach to deriving word embeddings for Wolastoqey and Mi’kmaq that leverages definitions from a bilingual dictionary. More specifically, following Bear and Cook (2022), we experiment with encoding English definitions of Wolastoqey and Mi’kmaq words into vector representations using English sequence representation models. For this, we consider using and finetuning sentence-RoBERTa models (Reimers and Gurevych, 2019). We evaluate our word embeddings using a similar methodology to that of Bear and Cook using evaluations based on word classification, clustering and reverse dictionary search. We additionally construct word embeddings for higher-resource languages English, German and Spanishusing our methods and evaluate our embeddings on existing word-similarity datasets. Our findings indicate that our word embedding methods can be used to produce meaningful vector representations for low-resource languages such as Wolastoqey and Mi’kmaq and for higher-resource languages.
Identification of Dialect for Eastern and Southwestern Ojibwe Words Using a Small Corpus
Authors: Kalvin Hartwig; Evan Lucas; Timothy Havens
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: The Ojibwe language has several dialects that vary to some degree in both spoken and written form. We present a method of using support vector machines to classify two different dialects (Eastern and Southwestern Ojibwe) using a very small corpus of text. Classification accuracy at the sentence level is 90% across a five-fold cross validation and 72% when the sentence-trained model is applied to a data set of individual words. Our code and the word level data set are released openly on Github at [link to be inserted for final version, working demonstration notebook uploaded with paper].
Enriching Wayúunaiki-Spanish Neural Machine Translation with Linguistic Information
Authors: Nora Graichen; Josef Van Genabith; Cristina España-bonet
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: We present the first neural machine translation system for the low-resource language pair Wayúunaiki–Spanish and explore strategies to inject linguistic knowledge into the model to improve translation quality. We explore a wide range of methods and combine complementary approaches. Results indicate that incorporating linguistic information through linguistically motivated subword segmentation, factored models, and pretrained embeddings helps the system to generate improved translations, with the segmentation contributing most. In order to evaluate translation quality in a general domain and go beyond the available religious domain data, we gather and make publicly available a new test set and supplementary material. Although translation quality as measured with automatic metrics is low, we hope these resources will facilitate and support further research on Wayúunaiki.
Towards the First Named Entity Recognition of Inuktitut for an Improved Machine Translation
Authors: Ngoc Tan Le; Soumia Kasdi; Fatiha Sadat
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: Named Entity Recognition is a crucial step to ensure good quality performance of several Natural Language Processing applications and tools, including machine translation and information retrieval. Moreover, it is considered as a fundamental module of many Natural Language Understanding tasks such as question-answering systems. This paper presents a first study on NER for an under-represented Indigenous Inuit language of Canada, Inuktitut, which lacks linguistic resources and large labeled data. Our proposed NER model for Inuktitut is built by transferring linguistic characteristics from English to Inuktitut, based on either rules or bilingual word embeddings. We provide an empirical study based on a comparison with the state of the art models and as well as intrinsic and extrinsic evaluations. In terms of Recall, Precision and F-score, the obtained results show the effectiveness of the proposed NER methods. Furthermore, it improved the performance of Inuktitut-English Neural Machine Translation.
Parallel Corpus for Indigenous Language Translation: Spanish-Mazatec and Spanish-Mixtec
Authors: Atnafu Lambebo Tonja; Christian Maldonado-sifuentes; David Alejandro Mendoza Castillo; Olga Kolesnikova; Noé Castro-Sánchez; Grigori Sidorov; Alexander Gelbukh
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: In this paper, we present a parallel Spanish- Mazatec and Spanish-Mixtec corpus for machine translation (MT) tasks, where Mazatec and Mixtec are two indigenous Mexican languages. We evaluated the usability of the collected corpus using three different approaches: transformer, transfer learning, and fine-tuning pre-trained multilingual MT models. Fine-tuning the Facebook m2m100-48 model outperformed the other approaches, with BLEU scores of 12.09 and 22.25 for Mazatec-Spanish and Spanish-Mazatec translations, respectively, and 16.75 and 22.15 for Mixtec-Spanish and Spanish-Mixtec translations, respectively. The results indicate that translation performance is influenced by the dataset size (9,799 sentences in Mazatec and 13,235 sentences in Mixtec) and is more effective when indigenous languages are used as target languages. The findings emphasize the importance of creating parallel corpora for indigenous languages and fine-tuning models for low-resource translation tasks. Future research will investigate zero-shot and few-shot learning approaches to further improve translation performance in low-resource settings.
A finite-state morphological analyser for Highland Puebla Nahuatl
Authors: Robert Pugh; Francis Tyers
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: This paper describes the development of a free/open-source finite-state morphologicaltransducer for Highland Puebla Nahuatl, a Uto-Aztecan language spoken in and around the stateof Puebla in Mexico. The finite-state toolkit used for the work is the Helsinki Finite-StateToolkit (HFST); we use the lexc formalism for modelling the morphotactics and twol formal-ism for modelling morphophonological alternations. An evaluation is presented which showsthat the transducer has a reasonable coveragearound 90%on freely-available corpora of the language, and high precisionover 95%on a manually verified test set
Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction
Authors: Manuel Mager; Rajat Bhatnagar; Graham Neubig; Ngoc Thang Vu; Katharina Kann
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: Neural models have drastically advanced state of the art for machine translation (MT) between high-resource languages. Traditionally, these models rely on large amounts of training data, but many language pairs lack these resources. However, an important part of the languages in the world do not have this amount of data. Most languages from the Americas are among them, having a limited amount of parallel and monolingual data, if any. Here, we present an introduction to the interested reader to the basic challenges, concepts, and techniques that involve the creation of MT systems for these languages. Finally, we discuss the recent advances and findings and open questions, product of an increased interest of the NLP community in these languages.
Community consultation and the development of an online Akuzipik-English dictionary
Authors: Benjamin Hunt; Lane Schwartz; Sylvia Schreiner; Emily Chen
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: In this paper, we present a new online dictionary of Akuzipik, an Indigenous language of St. Lawrence Island (Alaska) and Chukotka (Russia).We discuss community desires for strengthening language use in the community and in educational settings, and present specific features of an online dictionary designed to serve these community goals.
Finding words that aren’t there: Using word embeddings to improve dictionary search for low-resource languages
Authors: Antti Arppe; Andrew Neitsch; Daniel Dacanay; Jolene Poulin; Daniel Hieber; Atticus Harrigan
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: Modern machine learning techniques have produced many impressive results in language technology, but these techniques generally require an amount of training data that is many orders of magnitude greater than what exists for low-resource languages in general, and endangered ones in particular. However, dictionary definitions in a comparatively much more well-resourced majority language can provide a link between low-resource languages and machine learning models trained on massive amounts of majority-language data. By leveraging a pre-trained English word embedding to compute sentence embeddings for definitions in bilingual dictionaries for four Indigenous languages spoken in North America, Plains Cree (nhiyawwin), Arapaho (Hinno’itit), Northern Haida (Xaad Kl), and Tsuut’ina (Tst’n), we have obtained promising results for dictionary search. Not only are the search results in the majority language of the definitions more relevant, but they can be semantically relevant in ways not achievable with classic information retrieval techniques: users can perform successful searches for words that do not occur at all in the dictionary. These techniques are directly applicable to any bilingual dictionary providing translations between a high- and low-resource language.
Enhancing Spanish-Quechua Machine Translation with Pre-Trained Models and Diverse Data Sources: LCT-EHU at AmericasNLP Shared Task
Authors: Nouman Ahmed; Natalia Flechas Manrique; Antonije Petrović
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: We present the LCT-EHU submission to the AmericasNLP 2023 low-resource machine translation shared task. We focus on the Spanish-Quechua language pair and explore the usage of different approaches: (1) Obtain new parallel corpora from the literature and legal domains, (2) Compare a high-resource Spanish-English pre-trained MT model with a Spanish-Finnish pre-trained model (with Finnish being chosen as a target language due to its morphological similarity to Quechua), and (3) Explore additional techniques such as copied corpus and back-translation. Overall, we show that the Spanish-Finnish pre-trained model outperforms other setups, while low-quality synthetic data reduces the performance.
ChatGPT is not a good indigenous translator
Authors: David Stap; Ali Araabi
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: This report investigates the continuous challenges of Machine Translation (MT) systems on indigenous and extremely low-resource language pairs. Despite the notable achievements of Large Language Models (LLMs) that excel in various tasks, their applicability to low-resource languages remains questionable. In this study, we leveraged the AmericasNLP competition to evaluate the translation performance of different systems for Spanish to 11 indigenous languages from South America. Our team, LTLAmsterdam, submitted a total of four systems including GPT-4, a bilingual model, fine-tuned M2M100, and a combination of fine-tuned M2M100 with $k$NN-MT. We found that even large language models like GPT-4 are not well-suited for extremely low-resource languages. Our results suggest that fine-tuning M2M100 models can offer significantly better performance for extremely low-resource translation.
Few-shot Spanish-Aymara Machine Translation Using English-Aymara Lexicon
Authors: Nat Gillin; Brian Gummibaerhausen
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: This paper presents the experiments to train a Spanish-Aymara machine translation model for the AmericasNLP 2023 Machine Translation shared task. We included the English-Aymara GlobalVoices corpus and an English-Aymara lexicon to train the model and limit our training resources to train the model in a \textit{few-shot} manner.
PlayGround Low Resource Machine Translation System for the 2023 AmericasNLP Shared Task
Authors: Tianrui Gu; Kaie Chen; Siqi Ouyang; Lei Li
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: This paper presents PlayGround’s submission to the AmericasNLP 2023 shared task on machine translation (MT) into indigenous languages. We finetuned NLLB-600M, a multilingual MT model pre-trained on Flores-200, on 10 low-resource language directions and examined the effectiveness of weight averaging and back translation. Our experiments showed that weight averaging, on average, led to a 0.0169 improvement in the ChrF++ score. Additionally, we found that back translation resulted in a 0.008 improvement in the ChrF++ score.
Four Approaches to Low-Resource Multilingual NMT: The Helsinki Submission to the AmericasNLP 2023 Shared Task
Authors: Ona De Gibert; Raúl Vázquez; Mikko Aulamo; Yves Scherrer; Sami Virpioja; Jörg Tiedemann
Venue: ACL 2023 (Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP))

Abstract: The Helsinki-NLP team participated in the AmericasNLP 2023 Shared Task with 6 submissions for all 11 language pairs arising from 4 different multilingual systems. We provide a detailed look at the work that went into collecting and preprocessing the data that led to our submissions. We explore various setups for multilingual Neural Machine Translation (NMT), namely knowledge distillation and transfer learning, multilingual NMT including a high-resource language (English), language-specific fine-tuning, and multilingual NMT exclusively using low-resource data. Our multilingual Model B ranks first in 4 out of the 11 language pairs.