Improving Low-Resource Machine Translation for Formosan Languages Using Bilingual Lexical Resources (2024)

Abstract: This paper investigates how machine translation for low-resource languages can be improved by incorporating information from bilingual lexicons during the training process for mainly translation between Mandarin and Formosan languages, which are all moribund or critically endangered, and we also show that our techniques work for translation between Spanish and Nahuatl, a language pair consisting of languages from completely different language families. About 70% of the approximately 7,000 languages of the world have data in the form of lexicons, a valuable resource for improving low-resource language translation. We collect a dataset of parallel data and bilingual lexicons between Mandarin and 16 different Formosan languages and examine mainly three different approaches: (1) simply using lexical data as additional parallel data, (2) generating pseudo-parallel sentence data to use during training by replacing words in the original parallel sentence data using the lexicon, and (3) a combination of (1) and (2). All three approaches give us gains in both BLEU scores and CHRF scores, and we found that (3) provided the most gains, followed by (1) and then (2), which we observed for both translation between Mandarin and the Formosan languages and Spanish-Nahuatl. With technique (3), we saw an average increase of 5.55 in BLEU scores and 10.33 in CHRF scores.

heading: 1 Introduction
text: Evidence shows that neural machine translation (NMT) systems do not work well with languages that lack parallel data (Koehn and Knowles, 2017), and typically, such machine translation systems require millions of parallel sentences as training data, which are only available for a limited number of language pairs (Haddow et al., 2022). However, there are several low-resource languages with a significant number of speakers, such as Burmese and (Haddow et al., 2022), highlighting the need
for more attention to be paid to the development of machine translation systems for these languages. For example, while there are nearly 280 million English-French parallel sentences in OPUS (Tiedemann, 2012), there are only about 700,000 English-Burmese parallel sentences, even though Burmese (Myanmar) has tens of millions of speakers (Haddow et al., 2022). Furthermore, it is important to improve machine translation for all languages, even those with few speakers, to help provide fair access to technology.
Though parallel data are only available for a limited number of language pairs, bilingual lexicons are an often overlooked resource that are more readily available and less expensive to produce (Haddow et al., 2022). Language documentation efforts typically produce bilingual lexicons first, and about 70% of the approximately 7,000 languages of the world have such lexicons (Wang et al., 2022). They may be advantageous in that they can provide information about infrequent terms that may not appear in parallel data, but they also are not as helpful as parallel data in that they do not give information about how a word may be translated differently depending on the context (Haddow et al., 2022). However, despite the lack of contextual information, bilingual lexicons still provide additional valuable information that machine translation systems can use.
This paper explores how bilingual lexicons can be used as a complement to parallel data in the context of low-resource machine translation for Asian languages. In contrast to prior work which so far has mostly used replacement-based techniques which leverage bilingual lexicons by generating pseudo-parallel data, in this paper we propose to simply use lexical data as additional parallel data for training. To test our idea, we introduce a dataset containing parallel sentences and bilingual lexicons for 16 Formosan languages, which have a great deal of diversity in linguistic struc-
11248
ture, which we use to propose an extensive empirical framework. Concretely, we aim to answer the following research questions: (RQ1) Can adding lexical information as parallel data improve translation quality?, (RQ2) How much lexical information is needed to improve translation quality?, (RQ3) How does using lexical information compare with using parallel data?, (RQ4) How does using lexical information compare with pseudoparallel data?
Our results demonstrate that adding lexical information during the training process can produce significant improvements in translation quality for real-world, low-resource language pairs, using Formosan languages and Mandarin as an example. We also examine the relationship between improvements in translation quality and the amount of additional lexical information provided, and compare the effect of using lexical information with the effect of using additional parallel sentence data. Additionally, we also demonstrate that our methods improve translation quality for one non-Asian language pair, Spanish-Nahuatl, suggesting that our technique may generalize to other language pairs.

heading: 2 Related Work
text: Bilingual lexicons have proven useful in aiding a large a variety of downstream tasks such as semantic role labeling (van der Plas et al., 2011), sentence alignment (Fernando et al., 2023), and partof-speech tagging (Wang et al., 2022; Täckström et al., 2013). Wang et al. (2022) explored the use of bilingual lexicons in three tasks: named entity recognition, part-of-speech tagging, and dependency parsing. The authors used bilingual lexicons to synthesize data and were able to use these synthesized data to achieve significant improvements in performance in all three of these tasks across 19 under-represented languages. These synthesized data were made by using bilingual lexicons to create synthetic sentences in the target language through word-to-word translation. Word order and morphological differences between the source and target languages were not considered in this process, so the synthetic data may be quite different from true data in that language (Wang et al., 2022).
Bilingual lexicons have also been used to improve machine translation. Despite the fact that approximately 70% of the world’s languages have
documentation in the form of bilingual lexicons (Wang et al., 2022), techniques involving bilingual lexicons in machine translation typically involve artificially generated lexicons.
Dinu et al. (2019) used terms extracted from two English-German lexicons and inserted these terms after their corresponding source words to improve translation quality. Prior to this work, most work on neural machine translation used constrained decoding to integrate terminology, but Dinu et al. (2019) used a black-box generic neural machine translation architecture that is directly trained to use an external lexicon provided during run-time, achieving increases of 0.2 to 0.9 in BLEU scores.
A more recent multilingual pre-trained method called mRASP (Lin et al., 2020) also makes use of lexicons to improve translation, as demonstrated by experiments on 42 translation directions, including both low-resource and high-resource languages. In this method, a lexicon composed of unsupervised word alignments generated by MUSE (Conneau et al., 2017) is used to substitute words in the source sentence during pre-training. This is done to bring semantically similar words from different languages closer in the representation space (Haddow et al., 2022). Through a set of analytical experiments, the authors also showed that alignment information does help bridge any gaps between languages and improve translation quality.
AFROMT (Reid et al., 2021) is a machine translation benchmark for eight African languages, and the authors of this benchmark demonstrated that by extracting a dictionary from parallel corpora using eflomal1 (Östling and Tiedemann, 2016) and taking word alignments that appear over 20 times to produce a bilingual lexicon, this lexicon can be used to produce code-switched monolingual data to aid in training machine translation models. The authors also experimented with iteratively creating pseudo-monolingual data in low-resource languages.
Most recently, Jones et al. (2023) showed that using bilingual lexicons to substitute source sentence words for their translations to create code-switched data and prepending translations of source words to source sentences are two techniques that can improve machine translation quality for low-resource and unsupervised languages. The authors used bilingual lexicons from PanLex
1https://github.com/robertostling/eflomal/
(Kamholz et al., 2014) to do this. Our work shows that there may be an even simpler approach to integrating lexical information when training neural machine translation systems. This makes use of existing bilingual lexicons without the need for inferring a bilingual lexicon from parallel data.

heading: 3 Data
text: To explore whether lexicons can be simply used as additional parallel data in a low-resource setting in the context of Asian languages, we believe it is important to experiment with a diverse set of languages. To that end, for this paper we collect data from 16 different Formosan languages, a geographic grouping of Austronesian languages indigenous to Taiwan, which have a great deal of diversity in linguistic structure (Li, 2008) despite all being from a relatively small geographic area.
To obtain this data, we downloaded lexicon entries and parallel sentences from an online dictionary (原住民族語言線上辭典)2 published by the Indigenous Languages Research and Development Foundation (原住民族語言研究發展基金 會), an organization based in Taiwan dedicated to research on Taiwan’s indigenous languages. We specifically consider the following languages: Amis, Atayal, Bunun, Kanakanavu, Kavalan, Paiwan, Puyuma, Rukai, Saaroa, Saisiyat, Sakizaya, Seediq, Thao, Truku, Tsou, and Yami. The dictionaries consist of words, phrases, and example sentences relating to daily life from each of these languages with their Mandarin equivalents. We manually verified the data don’t contain personal information or offensive content, and to the best of our knowledge, our use of these data is compatible with the original access conditions.
While these data are available in an online dictionary format that can be searched, they were not immediately ready for computational use. One can download parts of the dictionary as PDF or ODT files, but neither format is easy to use due to how the lexicon entries are organized. Therefore, we downloaded PDFs from the website of the online dictionary, converted them into HTML using PDFMiner3, and extracted data using Beautiful Soup4. Each Formosan word/phrase and its Mandarin dictionary entry were extracted along
2https://e-dictionary.ilrdf.org.tw/index.htm 3https://github.com/pdfminer/pdfminer.six 4https://www.crummy.com/software/
BeautifulSoup/
with parallel sentence data included in this dictionary. These data are summarized in Table 1 and Table 13. For each language pair, we shuffled the sentences and took 80% of the parallel sentence data to use as our training set, 10% of the data to use as our development set, and the remaining 10% of the data to use as our test set. Our dataset can be found at https://github.com/francisdzheng/ formosan-mandarin.
Our studied Formosan languages are all written using the Latin script, unlike Mandarin Chinese. Though there are a small number of Formosan languages, their linguistic features are diverse (Li, 2008). For example, word order, focus systems, auxiliaries, numerals, personal pronouns, compounding, affixation, and phonology all demonstrate great diversity across the Formosan languages (Li, 2008). Most Formosan languages are verb-initial, but some also have an SVO (subject-verb-object) order and use different word orders depending on the situation. Thao and Atayal demonstrate little compounding, while Tsou and Bunun demonstrate rich compounding in their vocabularies. Additionally, while some Formosan languages distinguish between human and nonhuman numerals, some also do not. Some Formosan languages require auxiliaries, while others do not. We hope these examples, though limited, help illustrate how linguistically diverse the Formosan languages are.
One key point is that all of our studied languages are moribund or critically endangered. For
example, Thao was documented to have only four native speakers according to Ethnologue in 2022. Yet some other languages, such as Amis, had roughly 30,000 speakers as of 2015 (Kuo, 2015). This is still, however, a much smaller number of speakers than the most commonly spoken languages in the world, such as English and Spanish, and to that extent, data for Formosan languages are extremely lacking.
It is also important to note that existing literature on these languages written in English may conflate the Truku language and Seediq language. This is due to the fact that the Truku people (also known as the Taroko people) were previously grouped with the Seediq people by the Taiwanese government. Truku, in fact, has been described as a dialect of Seediq (Lee et al., 2011), which means that some literature may use Seediq as a general term to refer to languages including Truku. The two languages also share the same ISO 639-3 code, trv, which may cause further confusion. In this paper, Truku refers to what’s known as Tàilǔgéyǔ (太魯閣語) in Mandarin, and Seediq refers to what’s known as Sàidékè-yǔ (賽德克語) in Mandarin.
Additionally, to test how our ideas work with an externally created lexicon, we used a SeediqMandarin lexicon of 1,556 entries from a SeediqMandarin dictionary5 by Temi Nawi (曾瑞琳). This lexicon was kindly provided to us by Darryl Sterk.
Finally, to explore how our ideas work with nonFormosan languages, we obtained parallel data and a lexicon for Spanish and Nahuatl, a language native to Mexico. Parallel data was obtained from AmericasNLP6 and originally comes from Axotol (Gutierrez-Vasques et al., 2016), a SpanishNahuatl parallel corpus. This dataset consists of 16,145 sentences in the training set, 672 sentences in the development set, and 1,003 sentences in the test set. A Nahuatl-Spanish lexicon containing 10,888 lexicon entries was obtained from AULEX7. This lexicon consists of entries from three dictionaries: Diccionario español-nahuatl by Manuel Rodríguez Villegas, Diccionario náhu-
5Dictionary title in original language: 賽德克語辭典 (Published by國家文化藝術基金會)
6https://github.com/AmericasNLP/ americasnlp2021/tree/main/data/nahuatl-spanish, https://github.com/AmericasNLP/americasnlp2021/ blob/main/test_data/test.nah
7https://aulex.org/nahuatl/descarga.php? archivo=nau-es
atl de la huasteca veracruzana - español by Marcelino Hernández Beatriz, and Diccionario náhuatl-español by Francisco Xavier Clavijero and Sybille de Pury.

heading: 4 Experimental Setup
text: As described in Section 2, bilingual lexicons have played a significant role in some of the latest developments in machine translation. In contrast to existing work, we propose to simply treat bilingual lexicons as additional parallel data to explicitly inform the model of translations of different words.
Preprocessing Data were tokenized using the unigram (Kudo, 2018) implementation of SentencePiece (Kudo and Richardson, 2018). A vocabulary size of 8,000 and a character coverage rate of 0.99 were used to train our SentencePiece model.
Models Our Transformer models use six encoder and decoder layers with four attention heads each, a hidden dimension of 512, and a feedforward size of 1024, and a learning rate of 0.0005. Our model was optimized using Adam (Kingma and Ba, 2015) with hyperparameters β = (0.9, 0.98) and ϵ = 10−8. A dropout rate of 0.3 and a weight decay of 0.0001 were used for regularization. Each model for each language pair and direction was trained for two hours on three Quadro RTX 8000 GPUs.
Evaluation Using the SACREBLEU library8 (Post, 2018), we evaluated the translations outputted by our models with detokenized BLEU (Papineni et al., 2002; Post, 2018) on the test data from our parallel sentence data. We also used CHRF (Popović, 2015) to measure performance at the character level since most Austronesian languages are agglutinative (Blust, 2013), meaning they make extensive use of affixes and are morphologically rich. For example, some languages such as Thao (Blust, 2003) and Tsou (Tsuchida, 1990) have hundreds of prefixes.

heading: 5 Results and Analysis
text: We begin by creating a baseline by training models using only the parallel sentence data and one in which the model was trained using both the parallel sentence data and bilingual lexicon. To incorporate the bilingual lexicon during the training process, we added the lexicon entries in the source lan-
8https://github.com/mjpost/sacrebleu
to Mandarin from Mandarin Baseline + Lexicon Baseline + Lexicon
guage to the parallel data in the source language and the lexicon entries in the target language to the parallel data in the target language, essentially treating the lexicon entries as additional parallel data.
Table 2 summarizes our results in terms of BLEU and CHRF scores, clearly allowing us to decide whether lexical information can simply be added to our parallel data to improve translation quality. In the table, the “Baseline” columns contain results achieved using only parallel sentence data as training data, and the “+ Lexicon” columns contain results achieved using both parallel sentences and the bilingual lexicon as training data. Overall, we see an average increase of 3.37 in BLEU scores and 6.06 in CHRF scores after adding
lexical data during training. Every language pair demonstrated an increase in translation quality regardless of translation direction (to Mandarin and from Mandarin) and evaluation method (BLEU and CHRF). On average, when translating from Formosan languages to Mandarin, we saw an increase of 2.90 in BLEU scores and 5.98 in CHRF scores. When translating from Mandarin to Formosan languages, we saw an average increase of 3.84 in BLEU scores and 6.15 in CHRF scores. Improvements ranged from 0.97 to 7.65 for BLEU scores and 0.99 to 20.02 for CHRF scores.
Having established the effectiveness of our technique, we present a sensitivity study regarding the amount of parallel and lexical data used to train our model, in an attempt to provide insights to
help answer RQ2 and RQ3. To this end, we experiment with adding different amounts of the lexicon (in increments of 1,000 words) to the parallel sentence data to see the effect on translation quality. Because we did not have additional parallel data for many of these languages, we explored the third question by simply using 1,000 of the original parallel sentences to create a baseline for translation and set aside another 1,000 parallel sentences to use as test data. Then, we compared the effect of adding parallel sentences (in increments of 1,000) and the effect of adding lexical entries (in increments of 1,000) on translation quality. For this experiment, we used translation be-
tween Kanakanavu and Mandarin and translation between Tsou and Mandarin as examples.
Tables 3 and 4 summarize our sensitivity study results, showing how much lexical information is needed to attain performance improvements. We see that across all languages and directions in translation, there is a general trend of increases in translation quality as we add more lexicon entries. This is also indicated by the R2 values shown in the tables, which indicate the correlation between the translation quality achieved and number of lexicon entries used during training.
Results for our sensitivity study regarding parallel data can be found in Tables 5 and 6 for translation between Mandarin and Kanakanavu and Ta-
bles 7 and 8 for translation between Mandarin and Tsou. Overall, we observe that more gains in translation quality were achieved by adding parallel sentences compared to adding lexicon entries. For example, for translation from Kanakanavu to Mandarin, adding 4,000 lexicon entries to the training data increased translation quality by 3.11 BLEU points, whereas adding 4,000 parallel sentences to the training data increased translation quality by 6.22 BLEU points.

heading: 5.1 Comparison with prior work
text: As mentioned earlier, previous work has shown that pseudo-parallel data generation approaches (Imankulova et al., 2017; Wang et al., 2022) are very effective in improving translation results for low resource languages. One method of creating such pseudo data is by using a lexicon to replace words in the existing data (Wang et al., 2022). To show the effectiveness of our models and help answer RQ4, we follow Wang et al. (2022) and proceed to create pseudo-parallel data by using the lexicon to replace words in the parallel data. Words were only replaced in the Formosan language data due to tokenization difficulties in the Mandarin data, and we also experimented with different rates of replacement to figure out the optimal rate of replacement. Models were then trained with both the existing parallel data and pseudo-parallel data together (effectively doubling the amount of parallel sentence data). For this experiment, we used translation between Mandarin and six of our Formosan languages, Amis, Bunun, Kanakanavu, Paiwan, Sakizaya, and Seediq, due to computational budget limitations. Additionally, we combine our techniques and train models using the existing parallel data, our pseudo-parallel data, and the lexicons all together.
Table 9 shows the translation quality achieved using pseudo-parallel data in addition to the original parallel data during the training process (indicated in the “+ PPD” rows). For comparison, the translation quality achieved using only the original parallel data and the translation quality achieved using a combination of the original parallel data and lexicon (“+ Lex.”) are also shown.
We experimented with pseudo-parallel data created using different rates of replacement, 10%, 20%, 30%, 40%, 60%, and 80%, and found that a 20% rate of replacement produced the best results in most cases. Table 9 displays the best results we achieved across different models trained with the
pseudo-parallel data created using different rates of placement. In all cases, using a combination of pseudo-parallel data and lexicon entries proved to produce higher translation quality than using just the lexicon entries as additional data during training.

heading: 5.2 Using an externally created lexicon
text: Though the overlap between the lexicon entries and parallel sentences is also important in evaluating the impact of lexical resources on translation quality, this overlap was difficult to define
and examine. For example, in Mandarin Chinese, the definition of a word is vague as nearly each Chinese character is morphologically significant with its own meaning, so the overlap can be defined in multiple ways. Additionally, due to the low-resource nature of the language pairs examined in this paper, we could not find many additional lexicons with which to experiment. We did, however, find one externally created lexicon for Seediq-Mandarin, one with much fewer entries that could also be found in our parallel sentences, to use in our experiments.
To see how our techniques work with an externally created lexicon, one that was created separately from the parallel data, we used a SeediqMandarin lexicon of 1,556 entries described in 3. Table 10 displays the translation quality achieved by adding this lexicon and pseudo-parallel data using this lexicon. By adding this lexicon and pseudo-parallel data made using this lexicon to the original parallel data during the training process, we achieved increases of 0.51 BLEU points and 2.68 CHRF points when translating from Seediq to Mandarin and increases of 1.19 BLEU points and 1.61 CHRF points when translating from Mandarin to Seediq. Though these gains are not as great as the translation quality achieved by using the original lexicon, these results still demonstrate that our techniques work. We believe the smaller gains may be attributable to the smaller size of this external lexicon, which contains 1,556 entries which compares unfavorably to ours with 6,723 entries. Though less than half of the Formosan words in this externally lexicon could be found in the parallel sentences, unlike our lexicon, whose Formosan entries nearly all showed up in our parallel sentences, its addition to the training data still produced gains in translation quality. More studies, however, are needed to more clearly define the overlap between lexicons and parallel sentences, especially for Mandarin Chinese, and examine its effect on how adding lexical data can improve translation quality.

heading: 5.3 Non-Asian language pair
text: Finally, to explore how our techniques work with a non-Asian language pair, we considered translation between Spanish and Nahuatl. Adding a lexicon during the training process gave us gains of 0.79 BLEU points and 5.18 CHRF points when translating from Nahuatl to Spanish and gains of 1.48 BLEU points and 11.86 CHRF points when
translating from Spanish to Nahuatl. Adding pseudo-parallel data generated with a 20% rate of replacement in addition to this lexicon gave us gains of 1.03 BLEU points and 6.67 CHRF points when translating from Nahuatl to Spanish and gains of 1.98 BLEU points and 21.89 CHRF points when translating from Spanish to Nahuatl. For comparison, the best score achieved in the AmericasNLP 2023 shared task (Ebrahimi et al., 2023) on translation from Spanish to Nahuatl using the same dataset was a BLEU score of 2.33 and a CHRF score of 27.25. This was achieved through extending, training, and combining various adaptations of NLLB-200 (Costa-jussà et al., 2022) and utilizing additional data from other sources including constitutions, handbooks, news articles, and back-translations produced from monolingual data (Gow-Smith and Sánchez Villegas, 2023). Though we did not achieve as high of a BLEU score, we achieved a slightly higher CHRF score using our simple method of using a lexicon and pseudoparallel data created from it during training.

heading: 5.4 Multilingual models
text: Finally, we also explored how models would perform if pretrained on all the Formosan data available to us. Though the Formosan languages are linguistically quite diverse, they are still part of the same language family, and we wanted to see if there were any possibilities for the model to learn from these languages all together. Table 12 displays the translation quality achieved by these multilingual models. On average, the BLEU scores from the model pretrained on all the Formosan languages were 0.15 higher, and the CHRF scores were 0.17 points lower than the models that were not pretrained on anything when finetuning using only parallel sentences. When finetuning using both parallel sentences and the lexicon, the BLEU scores from the model pretrained on all the Formosan languages were 0.07 higher, and the CHRF scores were 0.22 points lower than the models that
were not pretrained on anything on average. The BLEU scores indicated minimal improvements in quality, and the CHRF scores indicated small decreases in quality. Thus, using multilingual models did not provide any conclusive improvements. This may be due to the fact that the Formosan languages are, in fact, too diverse for the model to learn any meaningful linguistic features from other languages that would help in translation for any given language pair. Additionally, we think our results suggest that the amount of Formosan language data available was also simply too little for cross-lingual pretraining to be meaningful.

heading: 6 Conclusion
text: About 70% of the approximately 7,000 languages of the world have data in the form of lexicons (Wang et al., 2022), and these data are an incredibly valuable resource that can be used to improve low-resource machine translation. When parallel sentence data is limited for languages, it may be difficult for models to learn the translations of different words, and bilingual lexicons can be used to more explicitly teach which words correspond to one another.
By adding lexical data during the training process, we achieved an average gain of 3.37 in BLEU scores and 6.06 in CHRF scores for translation between Mandarin and 16 Formosan languages, whose pairings can all be considered low-resource. These Formosan languages demonstrate consid-
erable variety in linguistic features and are considered the most diverse set of Austronesian languages (Li, 2008). By using existing lexicons and adding them to parallel data, we demonstrate how existing bilingual lexicons, a relatively inexpensive source of data, can be added as parallel data to improve translation quality for low-resource language pairs. We also show that adding more lexical data to parallel data during training tends to lead to higher translation quality. Though adding lexical data is not as effective as adding more parallel data, lexical data are a much cheaper source of data that still generate significant improvements to model output.
When using these lexicons to create pseudoparallel data, and using both the lexicon entries and this pseudo-parallel data during the training process, we achieved an average increase of 5.55 in BLEU scores and 10.33 in CHRF scores. Adding lexical data was more effective than adding pseudo-parallel data, but adding both lexical data and pseudo-parallel data produced the most gains in translation quality. Additionally, we demonstrate that these techniques also improve translation quality for one non-Asian low-resource language pair, Spanish-Nahuatl. Motivated by these results, we intend to continue exploring how these techniques work for other low-resource language pairs in the world. In the future, we’d also like to have humans evaluate our model outputs, though this remains a challenge due to the lowresource nature of the languages involved.
Limitations
Because the languages that we work with in this paper are extremely low-resource, it was not feasible to find native speakers of the 16 Formosan languages to check the output of our models. The significance of the BLEU and CHRF gains may be more clear by focusing on a smaller set of language pairs to make finding native speakers more manageable and describing the issues in translation output with the help of these native speakers.
We also wanted to explore how externally created lexicons work with these techniques for more language pairs other than Seediq-Mandarin, but we were not able to find any additional bilingual lexicons for other language pairs. This may be resolved by examining paper resources that may not yet be available digitally.
Additionally, we cannot ascertain how well our results generalize to other languages. Formosan languages have a considerable amount of linguistic variation, and our techniques work for SpanishNahuatl translation, but we were unable to experiment with and demonstrate results for other language pairs.

heading: A Train-Dev-Test Split

heading: Future Work
text: In the future, we’d also like to have humans evaluate our model outputs, though this remains a challenge due to the lowresource nature of the languages involved. Limitations
Because the languages that we work with in this paper are extremely low-resource, it was not feasible to find native speakers of the 16 Formosan languages to check the output of our models. The significance of the BLEU and CHRF gains may be more clear by focusing on a smaller set of language pairs to make finding native speakers more manageable and describing the issues in translation output with the help of these native speakers. We also wanted to explore how externally created lexicons work with these techniques for more language pairs other than Seediq-Mandarin, but we were not able to find any additional bilingual lexicons for other language pairs. This may be resolved by examining paper resources that may not yet be available digitally. Additionally, we cannot ascertain how well our results generalize to other languages. Formosan languages have a considerable amount of linguistic variation, and our techniques work for SpanishNahuatl translation, but we were unable to experiment with and demonstrate results for other language pairs