Learning-From-Mistakes Prompting for Indigenous Language Translation (2024)

Abstract: Using large language models, this paper presents techniques to improve extremely lowresourced indigenous language translations. Our approaches are grounded in the use of (1) the presence of a datastore consisting of a limited number of parallel translation examples, (2) the inherent capabilities of LLMs like GPT-3.5, and (3) a word-level translation dictionary. We harness the potential of LLMs and in-context learning techniques in such a setting for using LLM as universal translators for extremely low-resourced languages. Our methodology hinge on utilizing LLMs as language compilers for selected language pairs, hypothesizing that they could internalize syntactic structures to facilitate accurate translation. We introduce three techniques: KNNPrompting with Retrieved Prompting Context, Chain-of-Thought Prompting, and Learningfrom-Mistakes Prompting, with the last method addressing past errors. The evaluation results suggest that, even with limited corpora, LLMs, when paired with proper prompting, can effectively translate extremely low-resource languages.

heading: 1 Introduction
text: In recent years, LLMs have showcased astonishing capabilities in the realm of natural language processing, particularly in tasks like language translation (Zhu et al., 2023), text generation (Yuan et al., 2022), and contextual understanding (Behnia et al., 2022). The robust functionality of these models has led us to reconsider their potential role in indigenous language translation. In our pursuit to facilitate translations from Chinese to Taiwanese indigenous languages, we leverage the power of LLMs, buttressed by three foundational pillars: the presence of a datastore consisting of a limited number of parallel translation examples, the inherent capabilities of LLMs like GPT-3.5, and the integration of a word-level translation dictionary.
In this paper, we delineate three translation methodologies that build upon each other in a cumulative fashion. Each method represents a layer in our stratified approach, starting from leveraging contextual similarity in KNN-Prompting with Retrieved Prompting Context (RPC) to harnessing the didactic potential of Chain of Thought (CoT) Prompting, and culminating in the Learning-fromMistakes (LFM) Prompting technique that incorporates feedback mechanisms for continuous improvement. Figure 1 provides an overview of our methodologies, illustrating a step-by-step translation enhancement process designed for the Taiwanese indigenous language context.
This paper is structured as follows: In Section 2, we review the literature and discuss the position of this study. In Section 3, we explore the CoT Prompting methodology, followed by an indepth analysis of the LFM Prompting approach. In Section 4, we report the evaluation results. Through empirical evaluation and expert reviews, we demonstrate the effectiveness of the proposed methodologies.

heading: 2 Related Work
text: LLMs have exhibited excellent performance in language translation tasks, particularly evident in well-represented source languages like English and Chinese. Despite significant strides in translation performance for these languages, there remains a notable gap in the exploration of LLMs for low-resourced languages or those that have not been pre-trained. This aspect represents an underexplored area within the domain of research.

heading: 2.1 Low Resource Translation with LLM
text: LLMs’ effectiveness on various task is primarily attributed to two main properties. Firstly, incontext learning (Brown et al., 2020; Lester et al., 2021) allows the model to learn to solve specific problems by providing a small number of exam-
146
ples within the input context. The second one is the ability to follow the instruction (Ouyang et al., 2022; Mishra et al., 2021; Wei et al., 2021), instruction-tuned LLMs can be guiding to solve new task based on text instruction just as the scenario they were trained. Recently, some research has focused on enhancing these instruction following LLMs through incontext learning, (Nguyen et al., 2023; Ahuja et al., 2023) explores the generation of unsupervised fewshot demonstrations to enhance translation effectiveness in low-resource scenarios. Additionally, (Yao et al., 2023) utilized cultural awareness to optimize alignment in different languages, further augmenting the translation performance of LLMs. It is noteworthy that, the mentioned works above have focused on low-resource data for LLMs in languages that have been encountered during the pre-training phase. In contrast, our emphasis lies in a scenario where the model has not been previously trained in this specific language. In contrast to conventional approaches, we refrain from training parameters on limited parallel corpora (Gu et al., 2018; Lalrempuii and Soni, 2023). Instead, our goal is to leverage the understanding and reasoning capabilities of LLMs, coupled with the provided data, to accomplish translation tasks for previously unseen languages. In summary, to the best of our knowledge, no study has delved into the challenges and applications of utilizing LLMs for languages that have not been encountered before.

heading: 2.2 Indigenous Language Translation
text: In the context of preserving and revitalizing indigenous languages, the work by (Zheng et al.,
2022) stands as a notable contribution. Zheng and colleagues introduce the Amis-Mandarin dataset, which includes a parallel corpus comprising 5,751 Amis and Mandarin sentences. This dataset is of particular relevance to our research on translating Chinese sentences into Taiwanese indigenous languages. The Amis-Mandarin dataset provides a valuable resource for studying indigenous language translation. It aligns with the objectives of our study, as it offers a substantial parallel corpus, a fundamental component for training and evaluating translation models. Our research similarly leverages parallel corpora, although we focus on the translation of Chinese into various indigenous languages, including but not limited to Amis. In this study, we conduct experiments on six different indigenous languages. Furthermore, (Zheng et al., 2022) compile a comprehensive dictionary containing 7,800 unique Amis words and phrases, each accompanied by its Mandarin definition. This lexical resource enhances the utility of their dataset for translation tasks. In our research, we assume the existence of a similar dictionary, emphasizing the importance of word-level translation between Chinese and Taiwanese indigenous languages.
Stap and Araabi (2023) evaluates the translation performance of different systems for Spanish to 11 indigenous languages from South America. The authors find that LLMs like ChatGPT are not yet good at translating into indigenous languages. This is likely due to a number of factors, including the lack of training data for indigenous languages, the complex grammar and sentence structure of indigenous languages, and the difficulty of capturing the nuances of indigenous culture in translation.

heading: 2.3 Unveiling LLMs’ Proficiency in Tool Usage
text: In recent research, (Schick et al., 2023) discovered that LLMs exhibit the ability to discern how to employ tools provided by users, including external data. They adeptly combine this external information with their own knowledge to effectively address problem -solving tasks. These investigations delve into the mechanisms of CoT (Inaba et al., 2023) and Self-instruction (Yang et al., 2023) approaches, exploring how these methodologies assist LLMs in comprehending questions and utilizing the tools at their disposal. Additionally, there has been the development of question- answering datasets, such as ToolQA (Zhuang et al., 2023; Inaba et al., 2023), which aimed at faithfully evaluating the ability of LLMs to use external tools for question-answering. Inspired by these explorations into the understanding and application capabilities of LLMs, we take a similar approach in our method design. We offer KNN examples and word-by-word translation as tools for LLMs to improve their language translation abilities.

heading: 2.4 Position of Our Paper
text: This research stands at the intersection of multiple areas, addressing the challenges of translating into low-resource indigenous languages using LLMs like ChatGPT. While prior works have explored low-resource translation and indigenous language preservation, our study distinguishes itself in two key aspects:
1. Languages Unseen in Pre-training: Unlike previous research that has primarily focused on low-resource data for LLMs in languages encountered during pre-training, our work emphasizes the scenario where the model has not been previously trained in the specific target language. We tackle the challenge of translating into languages that lack representation in themodel’s training data, making our approach more versatile and applicable to a broader range of indigenous languages.
2. Few-Shot Prompting Techniques: Our research pioneers the application of few-shot prompting techniques to enhance translation capabilities for indigenous languages. We introduce innovative methods, including KNNPrompting with RPC, CoT Prompting, and
LFM Prompting, tailored to leverage LLMs’ inherent understanding and reasoning abilities. These techniques empower LLMs to effectively tackle low-resource language translation tasks, even when working with limited parallel corpora.
In summary, our paper bridges the gap between LLMs and low-resource indigenous language translation, offering practical and innovative solutions for preserving and revitalizing endangered languages. By exploring the potential of these models in an uncharted linguistic landscape, we provide a fresh perspective and a promising direction for future research in this domain. For clarity, we also compare the related work in Table 1.

heading: 3 Methodology
text: Problem Setting and Assumptions The primary objective of this research is to enable the translation of Chinese sentences into Taiwanese indigenous languages through the utilization of LLMs. In pursuit of this goal, wemake the following assumptions for the methods proposed in this study:
• Datastore of Parallel Corpora: Our first assumption centers on the availability of a datastore with limited translation examples. Within this datastore, each data entry comprises a pair of sentences: a source sentence in Chinese (the language intended for translation) and a corresponding target sentence in the specific Taiwanese indigenous language. This resource forms the backbone for our translation, facilitating the alignment of linguistic patterns and meanings.
• Large Language Models: The cornerstone of our translation methods is the utilization of large pre-trained language models, exemplified by GPT-3.5, as the primary translation engines.
• Dictionary Existence: In addition to the aforementioned resources, we introduce an-
other assumption: the existence of a dictionary that spans word-level translations. This dictionary encompasses translations between indigenous language words and their corresponding Chinese counterparts.
Figure 1 outlines our study’s methods for enhancing translation in a cumulative manner. The KNN-Prompting with RPC method forms the base, merging contextually similar sentences and word translations to inform the LLM’s understanding of grammar and context. The CoT Prompting adds CoT demonstrations, showing RPC integration for effective translation. The LFM Prompting expands upon these with a feedback loop, leveraging previous translation errors to refine outcomes. This progressive strategy not only enhances LLM’s translation proficiency but also promotes continual learning and accuracy improvement.

heading: 3.1 KNN-Prompting with Retrieved Prompting Context (RPC)
text: We investigate the application of few-shot learning through the KNN-Prompting concept, as discussed in the works of Shi et al. (Shi et al., 2022) and Xu et al. (Xu et al., 2023). Our approach not only leverages contextually similar examples but also incorporates individual translations for each word in the source language. The methodology unfolds in the following manner:
• When taskedwith translating a sentence s, our method initiates by constructing a Retrieved Prompting Context (RPC) for s. This context includes:
– k examples that are contextually analogous to s, selected based on their similarity.
– Translations for each word in s, sourced from a comprehensive dictionary.
• For instances where direct word equivalents are unavailable, we employ the BERT-basechinese model as an embedding tool. This model aids in computing similarities to identify the most appropriate substitute words.
• The core principle of our method is to enable the LLM to assimilate the grammatical norms and sentence constructs of the target language. It achieves this through the analysis of the k
examples, thereby learning to organize the individually translated words into coherent and grammatically consistent sentences.
For a practical illustration of this process, please see Figure 2, which provides a concrete example of the RPC in action. We also show an example for prompting in Table 5.

heading: 3.2 CoT Prompting
text: In this methodology, we harness the CoT strategy, as delineated by Wei et al. (Wei et al., 2022), to guide the LLM in effectively utilizing the RPC for translating a given sentence s. Specifically, this approach involves the following steps:
• When presented with a sentence s for translation, accompanied by KNN-RPC-prompting inputs (i.e., k contextually similar examples and individual word translations), we further augment the LLM’s input with q CoT demonstrations. These demonstrations are designed to illustrate how to use the provided RPC to formulate the final translated sentences.
• An instance showcasing two CoT demonstrations is illustrated in Figure 3. It is important to note that these CoT demonstrations are integrated with the KNN-RPC-prompting inputs to serve as comprehensive prompting material for the LLM.
• For detailed examples of this prompting structure, please refer to Table 6 in the Appendix.
The overarching aim of this methodology is to empower the LLM with an understanding of the grammatical rules and the proficiency to fully leverage the RPC, including both the retrieved sentences and individual word translations, for producing coherent and accurate translations.

heading: 3.3 Learning-from-Mistakes (LFM) Prompting
text: LFM Prompting is a two-stage approach aimed at enhancing the quality of translations. This method leverages the result from the CoTKNN-Prompting and incorporates a feedback mechanism by conducting trial translation to refine the translation based on past translation errors. Themethodworks in the following phases:
• Phase 1: Trial Translation with CoT PromptingWhen given a sentence s to translate, we start by retrieving q contextually similar sentence pairs from the data store. Each pair (sqi , tqi) consists of a Chinese sentence sqi and its corresponding indigenous sentence tqi . For each sqi , we employ CoT KNNPrompting (the method introduced in Section 3.2) to translate it, resulting in t̂qi . At this stage, we have (sqi , tqi , t̂qi). Our approach involves using these results as examples for the LLM to learn from its translation errors and make improvements.
• Phase 2: Learning from Past Mistakes The second phase of LFM Prompting introduces a crucial element: the incorporation of past translation errors. Specifically, we treat (sqi , tqi , t̂qi) from the Phase 1 as LFM examples. In this phase, we present the LLM with a set of such examples, alongside the translation t̂ generated by using CoT KNN-Prompting to translate s. The language model is tasked with refining t̂ by considering the error examples in translation. It uses the provided examples of mistranslations to correct and improve the initial translation t̂, aligning it more closely with the correct target language structure and meaning.
Furthermore, Figure 4 provides a visual representation of the entire architecture’s workflow, illustrating the sequential processes outlined above. We also show a prompting example in Table 7.

heading: 4 Experiment

heading: 4.1 Model Usage
text: Utilizing the GPT-3.5-turbo-16k-0613 version with a temperature setting of 0, we employ Sentence BERT (Reimers and Gurevych, 2019) as the embedding model to retrieve k-nearest neighbor sentences. The similarity between sentences is computed using cosine similarity.

heading: 4.2 Data Sets
text: We use the learning materials for various indigenous languages from the ’Klokah’ website 1 provided by the Foundation for the Research and Development of Indigenous Languages in Taiwan as our evaluation corpora. Each indigenous group consists of 450 sentences with corresponding Chinese translations and a dictionary of 1000 words (single word translation). For each language, we divide this dataset into two parts:
• Test Data - A random selection of 100 sentences was used to evaluate the translation performance of various methods.
• Reference Data - The remaining 350 sentences and all dictionaries were used as reference materials for the LLM translation.
1https://web.klokah.tw/

heading: 4.3 Evaluation Results

heading: 4.3.1 Automatic Score
text: We’ve employed the GPT-3.5-turbo as our foundational language model for translation tests. Initially, we opted for Southern Amis, an indigenous language, as our primary focus, evaluating translation accuracy using the standard BLEU (Papineni et al., 2002) and chrF++ (Popović, 2017) metrics. As depicted in Table 2, the zero-shot translation results indicate the model’s limitations in effectively translating this language in the absence of reference data, reflected in BLEU scores nearing zero. However, introducing 20-shot reference data prompts the model to engage in (Agrawal et al., 2022) in-context learning, resulting in a marginal improvement in BLEU scores. This highlights the potential of few-shot learning. Furthermore, from the results in Table 2, we
can observe that using KNN-Prompting by retrieving contextual-relevant examples improves translation quality. We can also observe that utilizing the Chain-of-Thought strategy to guide the LLM also brings an improvement in translation quality, with an increase in BLEU scores from 1 to 3. We also report experiment results with Coastal Amis, Wanda Tayal, Siji Tayal, and Duda Seediq languages. The results are shown in Tables 3. When comparing models, we use BLEU3 as the main performance metric, as BLEU3 considers 3- gram matches, offering a more holistic view of the quality of translations, particularly in terms of fluency and coherence. In terms of BLEU3 scores, CoT Prompting consistently surpasses the base KNN-Prompting across all languages. This points towards the importance of capturing longer sequences and understanding the grammatical flow of the indigenous languages. We can also see the performance boost when we employ the LFM strategy with CoT in the compared languages.

heading: 4.3.2 Qualitive Review by Language Expert
text: In Table 4 in the appendix, we present the results of an evaluation by a Coastal Amis language expert, assessing translations from Chinese into Coastal Amis using various methods. This offers insights into the effectiveness of these translation strategies and helps us understand their impact on translation quality.
• Initially, the expert demonstrates a preference for translations produced by the LFMmethod, highlighting its contribution to linguistic precision and affirming the significant role of the LFM phase in enhancing translation quality.
• In the second dialogue, the expert’s endorsement of the COT and LFM method suggests that its incorporation can refine the LLM’s understanding and conveyance of the target language’s nuances.
• There is an identified need for improvement in translating sentence structures, particularly with time adverbs such as ” 非常” (very), ” 很” (very), ”最” (most), and the placement of temporal terms like ”今天” (today), ”明天” (tomorrow) at the sentence’s end. The LFM method is anticipated to guide the LLM in learning and internalizing these linguistic patterns, thereby refining the translations. Challenges such as dictionary absences are addressed by seeking synonyms, for instance, substituting ” 專為” (specially designed for) with ”最” (most), and ”南邊” (south) with ” 藍色” (blue). We posit that expanding the dictionary will mitigate such issues, further enhancing translation fidelity.
Overall, the expert’s reviews imply that the translation approach integrating the LFM strategy
tends to yield more precise and culturally attuned translations. This suggests that for LLMs translating less-resourced languages, a strategy amalgamating error feedback with accumulative learning might prove more effective. These insights bolster the methodologies delineated in our paper, positing that a stratified and iterative enhancement approach can substantially uplift translation quality, particularly for languages with constrained structural and lexical resources.

heading: 5 Conclusion
text: This study delves into the capabilities of LLMs in translating indigenous languages. Despite a limited datastore of parallel translations, our introduced methodologies: KNN-Prompting with RPC, CoT Prompting, and LFM Prompting demonstrate effectiveness in harnessing LLMs for this task. Emphasizing our technical contribution, empirical results highlight the superior performance of the CoT Prompting and LFM strategy over the compared baseline, signifying its adeptness at capturing intricate linguistic nuances and offering an advanced approach to preserving linguistic diversity.

heading: 6 Limitations
text: The strength of this framework lies in its capacity to translate less common, niche languages with a limited number of examples. Nevertheless, several challenges were encountered during the experiments. For example, in the case of the Southern Amis language, the term ‘we‘ can be translated as ‘kami‘ or ‘niyam,‘ among other options. Determining whether these terms carry subtle distinctions in meaning or are interchangeable necessitates the expertise of native speakers. Moreover, the use of the BLEU metric provides only one standardized answer, which may not consistently align with the actual context. Furthermore, within the LFM context, structural or grammatical corrections are solely guided by prior examples, as the language model itself lacks the capability for independent reasoning and adjustment. Therefore, achieving significant breakthroughs in effectiveness remains a challenge. Finally, while our methods have demonstrated datadriven enhancements, they do not fully address the issue of insufficient few-shot data resulting in inconsistent translation outcomes. Further research and innovation are essential in addressing this matter.

heading: Acknowledgement
text: This work is supported by NSTC 112-2634-F005-002-project Smart Sustainable New Agriculture Research Center (SMARTer), NSTC Taiwan Project under grant 112-2221-E-005-075-MY3, and Ministry of Education, Taiwan.

heading: Future Work
text: By exploring the potential of these models in an uncharted linguistic landscape, we provide a fresh perspective and a promising direction for future research in this domain. For clarity, we also compare the related work in Table 1