PlayGround Low Resource Machine Translation System for the 2023 AmericasNLP Shared Task
Year: 2023

Abstract:
This paper presents PlayGround’s submission to the AmericasNLP 2023 shared task on machine translation (MT) into indigenous languages. We finetuned NLLB-600M, a multilingual MT model pre-trained on Flores-200, on 10 low-resource language directions and examined the effectiveness of weight averaging and back translation. Our experiments showed that weight averaging, on average, led to a 0.0169 improvement in the ChrF++ score. Additionally, we found that back translation resulted in a 0.008 improvement in the ChrF++ score.

heading: 1 Introduction
text: We participated in the AmericasNLP 2023 (Ebrahimi et al., 2023) shared task with the goal of advancing previous studies (Mager et al., 2021) on indigenous American languages. The task is to translate Spanish into 10 indigenous languages, including Ashaninka, Aymara, Bribri, Guarani, Hñähñu, Nahuatl, Quechua, Raramuri, ShipiboKonibo, and Wixarika. Additionally, there was another language, Chatino1, for which we did not participate in.
We started with the monolingual and bilingual data from Mager et al. (2021) and finetuned NLLB600M, a multilingual pre-trained MT model from Meta’s No Language Left Behind (NLLB) project (NLLBTeam et al., 2022) both bilingually and multilingually. On top of that, we employed weight averaging and back translation. For back translation, we additionally filtered the back translated sentence pairs to improve the data quality.
We demonstrate that training on model weights averaged from multiple checkpoints improves translation quality, as indicated by a 0.0169 increase in the ChrF++ score on average, without requiring additional computation resources. Additionally, we found that back translation can enhance translation
1https://scholarworks.iu.edu/dspace/handle/ 2022/21028
quality for low-resource languages, although it is sensitive to the quality of synthetic data. To address this, we introduced a data filtering technique to improve the quality of synthetic data. With filtered back translation, our system achieved an average improvement of 0.008 in the ChrF++ score. Furthermore, our study reveals that multilingual finetuning achieves comparable translation quality to bilingual fine-tuning for low-resource languages.
We selected the bilingual model with weight averaging and back translation as our final submission. The implementation of this study is available in our Git repository2.

heading: 2 Methods

heading: 2.1 Data
text: We adopted the data preparation method described by the University of Helsinki’s submission to AmericasNLP 2021 (Vázquez et al., 2021) for our system. The details of the dataset can be found in Table 1. Our model training utilized the filtered parallel data (referred to as parallel data), which consisted of the training data provided by the organizers as well as additional data collected by the University of Helsinki (Vázquez et al., 2021). In order to generate synthetic parallel data (referred to as synthetic data), we employed monolingual data and applied back translation techniques (refer to Section 2.3). The development data was used for model selection purposes.

heading: 2.2 Pre-trained Model
text: Our models are based on the NLLB-600M Seq2Seq pre-training scheme introduced by the NLLB team (NLLBTeam et al., 2022). For tokenization, we utilize the SentencePiece tokenizer (Kudo and Richardson, 2018), following the NLLB configuration. The NLLB model was initially trained on
2https://github.com/KaieChen/ameircasnlp2023
173
the Flores-200 dataset, which consists of Aymara, Guarani, Quechua, and Spanish.

heading: 2.3 Fine-tuned Models
text: We fine-tune NLLB-600M using the data mentioned in Table 1. For both X-to-Spanish and Spanish-to-X directions, we fine-tune NLLB-600M using filtered parallel data in both bilingual and multilingual way. This produces 20 bilingual models and 2 multilingual models.
We leverage the above X-to-Spanish models to generate back translated data to enrich the training corpus. Then we further fine-tune the Spanish-to-X models with parallel dataset extended with back translated sentence pairs.
The final models are obtained with weight averaging since the training can be unstable with insufficient data.

heading: 2.3.1 Back Translation
text: In order to make use of monolingual data in indigenous languages, we employed back translation. Specifically, we froze the decoder layers of NLLB model and performed fine-tuning of an Xto-Spanish model using parallel data. Then, we utilized this model to generate synthetic sentences.
Data filtering: Synthetic sentences may contain noise. To address this issue, we implement a data filter to select a subset of synthetic sentences that will expand the original parallel dataset (Ranathunga et al., 2023). In our task, we initially fine-tuned a Spanish-to-X model using the parallel data. Subsequently, we evaluated this model on the synthetic sentences and selected the top N samples with the lowest cross-entropy loss. The value of N
is determined by the following:
N = min(|Ypar|, |Ysyn|) (1)
where |Ypar| represents the number of segments in the parallel dataset, and |Ysyn| represents the number of segments in the synthetic dataset.
Finally, we combined the selected synthetic data with the parallel data and proceeded to perform additional fine-tuning of the NLLB model.

heading: 2.3.2 Weight Averaging
text: Studies have shown that averaging the weights of multiple finetuned models can enhance accuracy (Wortsman et al., 2022). In our training approach, the weights of the next epoch are trained based on the average of the model weights from the previous K epochs. For inference, we compute the final model by averaging the model weights from the last K epochs. The model can be defined as follows:
NLLB(x; Θt) = NLLB(x; 1
K
K∑
k=1
Θt−k) (2)
where Θt represents the model parameters at epoch t.
This technique shares similarities with training different models using various hyperparameters (Wortsman et al., 2022; Xu et al., 2020). However, as we only need to train a single model, this technique can be particularly efficient for large language models. The effectiveness of this approach is further discussed in Section 3.

heading: 2.3.3 Hyperparameters
text: In the fine-tuning process, we froze the encoder layers of the NLLB model, considering its prior training on a vast amount of Spanish sentences. We optimized the model using AdamW (Loshchilov and Hutter, 2017) with hyperparameters β = (0.9, 0.999), ϵ = 10−6. We employed a learning rate of 3×10−4 for a total of 10, 000 iterations. For regularization, we utilized the same dropout rate as the original NLLB model and a weight decay of 0.01. Furthermore, for weight averaging, we set the value of K to be 5.

heading: 2.4 Evaluation
text: We report the results using ChrF++ (Popović, 2017), following the evaluation script3 provided by the AmericasNLP 2023 shared task. ChrF++
3https://github.com/AmericasNLP/ americasnlp2023
captures the character-level performance, making it particularly suitable for evaluating the polysynthetic properties observed in many indigenous languages (Zheng et al., 2021).

heading: 3 Results
text: The results are presented in Table 2 for both the development and test datasets. Our Bi++ model demonstrates improvements in four languages: Hñähñu, Aymara, Asháninka, and Quechua, compared to the Baseline model provided by the organizer. In general, the trends in results for the development and training datasets are similar, except for Rarámuri and Bribri. This discrepancy may be attributed to the test dataset containing more unknown tokens, to which our model is sensitive.
Previous study (Mager et al., 2021) has primarily focused on fine-tuning bilingual machine translation models. However, the results from our Multi++ and Bi++ models demonstrate the promising potential of multilingual fine-tuning (Tang et al., 2020). On average, the ChrF++ score for Multi++ is only 0.0012 lower than that of Bi++.
We also compared the effectiveness of weight averaging and back translation. Weight averaging improved translations for all target languages. On average, Multi+ achieved a ChrF++ score that was 0.0169 higher than Multi. These results indicate that our simple technique can enhance low-resource machine translation without requiring additional computational resources.
However, the impact of back translation varied across languages, as observed in the results for Multi+ and Multi++. On average, the implementation of back translation resulted in a 0.008 im-
provement in the ChrF++ metric. For Wixarika and Aymara, there was a slight drop in the ChrF++ scores after back translation. Despite performing data filtering, the quality of synthetic data largely depends on the performance of the X-to-Spanish model.
In summary, our fine-tuning technique has shown improvements in performance. However, with further refinements and design enhancements, there is potential for our model to achieve higher levels of performance.

heading: 4 Conclusion
text: In this paper, we presented our submission to the AmericasNLP 2023 shared task. Our system utilized the NLLB-600M pre-trained model to translate Spanish into 10 indigenous languages. We also investigated the potential of multilingual translation models, which showed promising results. Additionally, we found that averaging model weights from previous epochs proved to be an efficient and effective approach. While back translation demonstrated performance improvements, further methods are necessary to address noisy data. These findings highlight the positive outcomes of our study and provide valuable insights for future advancements in low-resource machine translation techniques.

heading: Future Work
text: These findings highlight the positive outcomes of our study and provide valuable insights for future advancements in low-resource machine translation techniques