Building NLP Systems for Two Resource-Scarce Indigenous Languages : Mapudungun and Quechua

By adopting a “first-things-first” approach we overcome a number of challenges inherent in developing NLP Systems for resourcescarce languages. By first gathering the necessary corpora and lexicons we are then enabled to build, for Mapudungun, a spellingcorrector, morphological analyzer, and two Mapudungun-Spanish machine translation systems; and for Quechua, a morphological analyzer as well as a rule-based Quechua-Spanish machine translation system. We have also worked on languages (Hebrew, Hindi, Dutch, and Chinese) for which resources such as lexicons and morphological analyzers were available. The flexibility of the AVENUE project architecture allows us to take a different approach as needed in each case. This paper describes the Mapudungun and Quechua systems. 1. The AVENUE Project The long-term goal of the AVENUE project at CMU is to facilitate machine translation for a larger percentage of the world’s languages by reducing the cost and time of producing MT systems. There are a number of radically different ways to approach MT. Each of these methods of accomplishing machine translation has a different set of strengths and weaknesses and each requires different resources to build. The AVENUE approach combines these different types of MT in one “omnivorous” system that will “eat” whatever resources are available to produce the highest quality MT possible given the resources. If a parallel corpus is available in electronic form, we can use example-based machine translation (EBMT) (Brown et al., 2003; Brown, 2000), or Statistical machine translation (SMT). If native speakers are available with training in computational linguistics, a human-engineered set of rules can be developed. Finally, if neither a corpus nor a human computational linguist is available, AVENUE uses a newly developed machine learning technique (Probst, 2005) to learn translation rules from data that is elicited from native speakers. As detailed in the remainder of this paper, the particular resources that the AVENUE project produced facilitated developing an EBMT and a humancoded rule-based MT system for Mapudungun, and a hand-built rule-based MT system for Quechua. Automatic rule learning has been applied experimentally for several other language pairs: Hindi-to-English (Lavie et al. 2003) and Hebrew-to-English (Lavie et al. 2004). The AVENUE project as a whole consists of six main modules (Figure 1), which are used in different combinations for different languages: 1) elicitation of a word aligned parallel corpus (Levin et al. in press); 2) automatic learning of translation rules (Probst, 2005) and morphological rules (Monson et al. 2004); 3) the run time MT system for performing source to target language translation based on transfer rules; 4) the EBMT system (Brown, 1997); 5) a statistical “decoder” for selecting the most likely translation from the available alternatives; and 6) a module that allows a user to interactively correct translations and automatically refines the translation rules (Font Llitjós et al. 2005a). 2. AVENUE and Indigenous Languages of the Western Hemisphere Over the past six years the AVENUE project at the Language Technologies Institute at Carnegie Mellon University has worked with native informants and the government of Chile to produce a variety of natural language processing (NLP) tools for Mapudungun, an indigenous South American language spoken by less than 1 million people in Chile and Argentina. During the final year and a half of this time, the AVENUE team has also been developing tools for Quechua, spoken by approximately 10 million people in Peru, Bolivia, Ecuador, South of Colombia, and northern Argentina. Electronic resources for both Quechua and Mapudungun are scarce. At the time the AVENUE team started working on Mapudungun, even simple natural language tools such as morphological analyzers or spelling correctors did not exist. In fact, there were few electronic resources from which such natural language tools might be built. There were no standard Mapudungun text or speech corpora, or lexicons, and no parsed treebanks. The text that does exist is in a variety of competing orthographic formats. More resources exist for Quechua but they are still far from what is needed to build a complete MT system. In addition to these practical challenges facing construction of natural language systems for Mapudungun and Quechua, there are also theoretical and human factor challenges. Both Mapudungun and Quechua pose unique challenges from a linguistic theory perspective, since they have complex agglutinative morphological structures. In addition Mapudungun is polysynthetic, incorporating objects into the verb of a sentence. Agglutination and polysynthesis are both properties that the majority languages, for which most natural language resources have been built, do not possess. Human factors also pose a particular challenge for these two languages. Namely, there is a scarcity of people trained in computational linguistics who