Rule-Based, Neural and LLM Back-Translation: Comparative Insights from a Variant of Ladin (2024)

Abstract: This paper explores the impact of different back-translation approaches on machine translation for Ladin, specifically the Val Badia variant. Given the limited amount of parallel data available for this language (only 18k Ladin–Italian sentence pairs), we investigate the performance of a multilingual neural machine translation model fine-tuned for Ladin– Italian. In addition to the available authentic data, we synthesise further translations by using three different models: a fine-tuned neural model, a rule-based system developed specifically for this language pair, and a large language model. Our experiments show that all approaches achieve comparable translation quality in this low-resource scenario, yet round-trip translations highlight differences in model performance.

heading: 1 Introduction
text: In recent years, a variety of methods have been developed to apply neural machine translation (NMT) also in low-resource scenarios (Shi et al., 2022; Haddow et al., 2022; Ranathunga et al., 2023). The back-translation technique has shown to be particularly effective in such settings (Sennrich et al., 2016; Edunov et al., 2018), offering the potential for substantial improvements in translation quality.
This work investigates the influence of the backtranslation model selection for a low-resource language. We do this, by comparing the results obtained by fine-tuning a pre-trained multilingual NMT model using synthesised translations generated by (i) a NMT system fine-tuned on the available parallel data, (ii) a rule-based machine translation (RBMT) system developed for the specific language pair, and (iii) a large language model (LLM) prompted to translate given texts, accompanied by 8 exemplary samples.
The quality of the synthesised data, which in turn is determined by the underlying models used
to generate it, matters (Burlot and Yvon, 2018). In our case, the synthesised translations originate from three models based on a different paradigm. Thus, the synthesised data is characterised by the specific strengths and weaknesses of the respective paradigms.
Rule-based systems are robust and computationally lightweight, but may face challenges in dealing with ambiguity. Moreover, they lag behind at the grammatical level. Neural models show a high ability to adapt to provided texts, but perform less well when confronted with out-of-domain data (Shen et al., 2021). In contrast, language model-based approaches (LLMs) are praised for their ability to produce fluent, coherent texts, but they are prone to hallucinations (Rawte et al., 2023). It is therefore an interesting question to investigate how this affects the quality of the NMT models trained on this data. This comparative analysis sheds light on the nuanced contrasts inherent in the different MT methods.
Our results show that in this low-resource scenario the back-translation model does not have a significant impact, and the performance of the models converges to similar results in terms of BLEU/chrF++ points. This assertion is supported by an empirical analysis carried out on the Val Badia variant of Ladin. Our main contributions are:
• we are the first to explore MT for Ladin in general, with a specific focus on the Val Badia variant,
• we compare an RBMT-, an NMT- and an LLM-based back-translation, providing insights into the efficacy of the methods for Ladin,
• we establish baseline results and make the test data, the RBMT system, as well as the bestperforming models publicly available
128
In Section 2 we describe our data collection and corpus creation process. In Section 3 we present the three different methods used for the back-translation of monolingual Ladin data into Italian. Section 4 gives an overview of the conducted experiments and Section 5 presents the obtained results. Section 6 discusses related work and similar approaches. In Section 7, we summarize and discuss future work.
The Ladin Language Ladin is an officially recognised minority language, and thus taught in schools, used in the media, and employed in public administration. For this reason, an effective machine translation system could make a significant contribution to facilitating and supporting communication in this language. However, Ladin is still an unexplored language in the field of machine translation. Indeed, nearly no parallel data1 is publicly available for this language, except for a few hundred samples on OPUS (Tiedemann, 2012). This language, spoken by around 30,000 people in the northern Italian Dolomite regions, exhibits significant diversity across its five main variants (Val Badia, Gherdëina, Fascia, Fodom, Anpezo), each shaped uniquely by its development in different valleys. This diversity is not only evident in the spoken language but has also resulted in distinct standards for written communication. The first author of the paper originates from the Val Badia and is a native speaker of Ladin. Therefore, in this work we concentrate on the standard written language of this valley. In the rest of the paper, we will use lvb as language code to refer to this variant of Ladin, and ita for Italian.

heading: 2 Data
text: This section gives an overview of the linguistic resources available for the Ladin language and describes the method employed to collect data for the specific Val Badia variant of Ladin.

heading: 2.1 Available Resources
text: Publicly accessible parallel data for Ladin is scarce. The Open Parallel Corpus (Tiedemann, 2012) e.g. lists 1543 Ladin–German, 220 Ladin–Italian, and 81 Ladin–English sentences. However, these texts are mainly specific to the variants of Gherdëina and Fassa and were not disseminated by public institutions. For our experiments, we were provided
1In a machine readable format.
with the archive of the weekly newspaper La Usc di Ladins2 and a digitised version of the dictionary Ladin Val Badia – Italian (Moling et al., 2016). From these data sources we extracted monolingual texts as well as a small dataset of parallel sentences. We furthermore used the dictionary as the basis for implementing a RBMT system. The collection of other parallel texts is time-consuming and has therefore been left for future work.

heading: 2.2 Parallel Data
text: The Ladin (Val Badia) – Italian dictionary (Moling et al., 2016) contains, alongside the word entries, also sentences that illustrate their usage. For these sentences the corresponding Italian translation is also given. We have collected this data to create our training dataset, which contains a total of 18, 139 sentences. These sentences are basic and short because they were created specifically to illustrate the use of words and phrases. The average length is 23.43 and 25.69 characters for Ladin and Italian respectively. This dataset has been publicly released.3

heading: 2.3 Ladin Monolingual Data
text: The Ladin newspaper La Usc di Ladins, digitally archived since 2012, provides an extensive dataset of monolingual texts. These texts are published in five different variants, each corresponding to one of the five Ladin valleys. We extracted these texts from the PDF documents and segmented them into individual sentences using the NLTK library (Bird et al., 2009), specifically setting Italian as the language to accommodate Ladin. In total, we accumulated 1, 937, 608 sentences. These sentences had to be categorised by variant, as described below.
Variant Classification In order to train a variant classifier, labeled training data is essential. However, the monolingual data from the newspaper PDFs lacked these labels. Therefore, we collected the texts from the newspaper’s website.4 Here, the article excerpts are categorized according to their origin valleys and the corresponding language variants, allowing us to create a labeled dataset.
We gathered a corpus of 7, 766 article excerpts with a total of 42, 745 individual sentences for training. These sentences were then split into training (comprising 75% of the sentences) and test data
2https://www.lausc.it/ 3https://www.doi.org/10.57967/hf/1878 4https://www.lausc.it
(the remaining 25%). Using the 2, 500 most frequent 3-gram characters as features, we trained an XGBoost variant-classifier (Chen and Guestrin, 2016). On the test data, our classifier achieved 94.48% accuracy in classifying these 5 labels.
The resulting model was used to predict the variant of each of the 1, 937, 608 sentences in the monolingual dataset. Table 1 reports the respective number of classifications (and the total number of characters) for each variant. 746,704 sentences were classified as val-badia and were considered for further processing.
Data Preparation Because of the spelling reform in 2015, we further processed the sentences classified as val-badia to exclude any with words that are no longer valid. To do this, we used the implementation of our RBMT system which is explained in more detail in Section 3.2. We used the system to identify unknown words and tried to adapt them to the new spelling according to certain rules. Sentences where this was not possible were left out. This process ensured that the filtered sentences fully adhered to the new spelling, which also facilitated the rule-based translations. We collected a total of 274,665 sentences (≈ 31% of the extracted sentences) which constitute the monolingual Ladin data we used in our experiments. Among the unused sentences, ≈ 100k contain only one unknown word/typo so there would be still potential to acquire additional data if additional time were spent analysing and preparing these texts.

heading: 2.4 Italian Monolingual Data
text: As monolingual data for Italian, we used the ELRC-CORDIS_News dataset5 from OPUS (Tiedemann, 2012), which contains 123,691 Italian sentences.

heading: 2.5 Test Data
text: This section introduces the three test sets on which the models were evaluated. This test data differs
5https://elrc-share.eu/
considerably from the training data, so that it can be considered out-of-domain data.
Testset 1 This dataset includes the statute of the Stiftung Südtiroler Sparkasse, a nonprofit foundation dedicated to supporting and promoting various initiatives and projects, primarily within the province of Bolzano. The document is rich in formal and legal terminology. It contains 424 sentences6.
Testset 2 This dataset is a festive compendium of the history of the region associated with this language (Kager, 2022). It combines historical narratives with legal and administrative statements. The result is a mixture of stylistic elements and lexical domains. It contains 833 sentences7.
Testset 3 This dataset delves into the literary realm with the classic story of Pinocchio (Collodi, 2017), a text rich in narrative prose, dialogue and idiomatic expressions, challenging the models with its creative and figurative language. It contains 1563 sentences8.

heading: 3 Back-translation Strategies
text: The so-called back-translation, first introduced in Sennrich et al. (2016), refers to the process of automatic translation of monolingual texts in the target language to the source language. This method of enriching additional training data in the sourceto-target translation direction (where the target side remains authentic) has proven to be particularly effective and is particularly valuable in low-resource scenarios. In this section we present the three different back-translation strategies used in our research to translate monolingual Ladin texts into Italian.

heading: 3.1 Neural MT
text: There is evidence that low-resource languages benefit from multilingual models (Aharoni et al., 2019). For this reason, we opted to utilise a pre-trained, multilingual model, specifically the Helsinki-NLP/opus-mt-ine-ine9 model available from the Hugging Face Model Hub, as our
6https://huggingface.co/datasets/sfrontull/ stiftungsparkasse-lld_valbadia-ita
7https://huggingface.co/datasets/sfrontull/ autonomia-lld_valbadia-ita
8https://huggingface.co/datasets/sfrontull/ pinocchio-lld_valbadia-ita
9https://huggingface.co/Helsinki-NLP/ opus-mt-ine-ine
base model. This model, which is part of OPUSMT (Tiedemann and Thottingal, 2020), was trained to translate between 135 Indo-European languages, to which Ladin and Italian also belong.
The Marian MT model, configured for Helsinki-NLP/opus-mt-ine-ine, features 6 encoder and 6 decoder layers, each with 8 attention heads and a feed-forward dimension of 2048. The model employs a beam search size of 6, a dropout rate of 0.1, and an embedding size of 512. It shares embeddings between the encoder and decoder.
We fine-tuned this model for the two translation directions lvb → ita and ita → lvb on the available authentic training data. We trained a single model for both directions by using the tags »ita« for lvb → ita and »lld_Latn« 10 for the opposite direction as prefixes of the source text. In the rest of the paper, we refer to this fine-tuned model as N1. For fine-tuning, we utilized the AdamW optimizer11 with the defaults settings. The fine-tuning greatly improves the model in both translation directions, as the scores reported in Table 4 and 3 show. This demonstrates that the data is reliable and that the model adapts well.

heading: 3.2 Rule-based MT
text: For low-resource languages, RBMT frameworks offer a crucial advantage: leveraging linguistic expertise to overcome the limitations of datadriven methods (Khanna et al., 2021). Considering the similar sentence structure and composition of Ladin and Italian (they are both Romance languages), it can be assumed that a rule-based MT system can also perform well without excessive structural transfer work. The available Ladin Val Badia-Italian dictionary served as the foundation for the rule-based MT system we developed in Apertium (Forcada and Tyers, 2016) for this language pair.
This dictionary provides, in addition to the individual words and word translations, also a list of all inflected forms for each lemma. To effectively utilise this dictionary within our translation system, we mapped the lexicographical data to paradigms within the framework of Apertium (monodix format). Specifically, we created 742 paradigms for
10We (re)used the tag »lld_Latn« because it is listed as a valid target language ID, as few Ladin texts were already included in the training of this model.
11https://huggingface.co/docs/transformers/ v4.41.0/en/main_classes/optimizer_schedules# transformers.AdamW
a total of 19,034 lemmas. This extensive set includes multiple lexical categories: 597 adverbs, 3,366 adjectives, 11,496 nouns, 162 pronouns, and 2,439 verbs. Additionally, we incorporated proper nouns, short phrases, and wordgrams that were identified during the monolingual text extraction process. The resulting bilingual dictionary contains a total of 30,468 entries. The integration with Apertium was facilitated by connecting to and reusing the pre-existing module for Italian12. The Ladin module13 and the Ladin–Italian14 module can be found on GitHub. In the rest of the paper, we refer to this RBMT system as R1.
According to aq-covtest15 R1 has a coverage of 96.66% on Testset 1, 95.81% on Testset 2 and 95.90% on Testset 3. However, since we did not develop disambiguation modules, we designed the system to select the first suggestion in cases of morphological and lexical ambiguity, which can sometimes result in incorrect choices that may distort the meaning of the texts. To counteract this, and to further enhance the rule-based translation system, we extracted the 900 most common word n-grams from the texts and added their corresponding translations as entries to the bilingual dictionary.
In addition to the data, we have also included 13 1-level structural transfer rules to avoid common errors. For example, in Ladin, the word pa is used to emphasize a question. In Italian, however, there is no corresponding word for this purpose. We have therefore developed a rule to exclude this word from the translation. The other rules include gender correction, dealing with reflexive verbs and prepositions.

heading: 3.3 MT with a Large Language Model
text: LLMs have shown remarkable capability in understanding and generating human-like text across various languages and domains (Brown et al., 2020). However, their performance in MT tasks exhibits significant variability across languages, especially when comparing high-resource languages to lowresource languages (Robinson et al., 2023). We explore the utilisation of a LLM, specifically GPT3.5 Turbo (OpenAI, 2024), to generate translations from Ladin to Italian. The process involved leverag-
12https://github.com/apertium/apertium-ita 13https://github.com/schtailmuel/
apertium-lld-ita 14https://github.com/schtailmuel/apertium-lld 15https://wikis.swarthmore.edu/ling073/ Apertium-quality
ing the advanced capabilities of the LLM, accessed through the gpt-3.5-turbo-0125 API endpoint. In the rest of the paper, we refer to this LLM as L1.
To enhance throughput and reduce the number of API requests, we generated the translation of 16 Ladin texts in a single request. We provided a set of 8 example translations in JSON format, randomly selected from the available authentic training data and instructed the LLM to generate translations for 16 Ladin texts, which were also provided as a JSON dictionary, with empty Italian translations. Listing 1 (Appendix A) showcases an exemplary prompt.
With this prompting approach, we translated the entire monolingual Ladin corpus into Italian. By providing the exemplary translations as JSON, we were able to reduce the failure rate (invalid/incomplete answers). The extent to which these examples also helped with the translation itself remains open. The entire process spanned approximately 100 hours, with an average processing time of around 22 seconds per request.

heading: 4 Experiments
text: We used the opus-mt-ine-ine16 model as base model for the experiments. In the rest of the paper, we use BM to refer to this model. We fine-tuned BM with the various data sets using the Transformers library (Wolf et al., 2020), specifically leveraging the Seq2SeqTrainer module. We always trained a single model for both directions using the corresponding prefixes.
We configured the training to process batches of 16 samples, and restricted the input and output sequences to a maximum of 128 tokens to ensure manageable computation loads. The models were evaluated each 16,000 steps. As a stopping criterion, we used three consecutive evaluations resulting in an improvement of less than 0.2 chrF points on the validation set. For training, we utilised an NVIDIA TITAN RTX graphics card with 24 GB. In total, we have trained 15 models:
• Model N1: BM fine-tuned with the available parallel data consisting of 18,139 sentences.
• Models N2/R2/L2: BM fine-tuned with authentic data and Ladin monolingual data backtranslated (BT) to Italian using N1/R1/L1 respectively.
16https://huggingface.co/Helsinki-NLP/ opus-mt-ine-ine
• Models N3/R3/L3: This iteration extends the training base of N2/R2/L2 by integrating Italian monolingual data that has been translated into Ladin utilising N2/R2/L2 respectively.
• Models N4/R4/L4: BM fine-tuned with same training data as N3/R3/L3 models, but with Ladin and Italian monolingual data backtranslated with N3/R3/L3 model.
• Models N5/R5/L5: This iteration extends the training base of N4/R4/L4 by adding also the forward-translations (FT) as training data.
• Models A1/A2: A1 was trained on the combined training data used to trained N4, R4 and L4. In A2 we additionally included the forward-translations into the training data.
We refer to the models N1, . . . , N5 that were trained with NMT backtranslated data as N-models. Analogously, we use the term R-models and Lmodels to refer to RBMT and LLM models, respectively.
Models N4/R4/L4 illustrate the gains achieved through iterative back-translation (Hoang et al., 2018). Additionally, models N5/R5/L5 demonstrate potential improvements achievable with synthetically generated forward-translation data.
We evaluated these models on the 3 test sets presented in Section 2.5. The results are presented and analysed in the following section.

heading: 5 Results and Discussion
text: The results of the various experiments conducted are presented in Table 3 and Table 4, where the SacreBLEU (Post, 2018) and chrF++ (Popović, 2015) scores for different models and test sets are detailed. To facilitate comparison, the best scores for each approach have been underlined, and the overall best scores for each testset are highlighted in bold.
Additionally, as recommended in Edunov et al. (2020), in Table 2 we report the mean perplexity values for the Italian translations generated by the different models to complement BLEU’s emphasis on adequacy. Perplexity measures how well a language model can predict the next word in a sequence based on the preceding words. Lower perplexity means that the model is more confident and accurate in its predictions, indicating that it can better reproduce the structure and patterns of the language it generates. Therefore, we present the mean perplexity values obtained from GPT-2 (Radford et al., 2019) , computed using the implementation available from Hugging Face17.
17https://huggingface.co/spaces/ evaluate-metric/perplexity
Several findings can be deduced from these results, and will be discussed below. In general, there is evidence that augmenting the training data with monolingual data through back-translation is effective. N1 shows that fine-tuning the model with only authentic training data substantially improves the results in both directions (compared to BM) in terms of BLEU/chrF++ points. This shows on the one hand that the training is effective and on the other hand that the available data is adequate. However, it is also evident that the model generates less fluent text, as indicated by the perplexity scores which increase for this model.
The results reveal a progression in difficulty among the test sets, where Testset 3 emerges as the most challenging one. On this test set, all approaches achieve similar low scores, suggesting the presented approach may face limitations with more complex texts.
In the translation direction lvb → ita, the best results were achieved by combining the different back-translations, as model A2 results indicate. This emphasises the importance of a broad and diversified dataset. Remarkably, the A2 model is also competitive in the reverse translation direction (ita
to lvb), although it does not achieve the best results.
A comparison of the models N1, R1 and L1 suggests that the LLM generates more fluent texts (low perplexity) but perhaps does not always accurately reproduce the meaning, as attested by the performance on Testset 3 (low perplexity but also low BLEU score). In this assessment, the RBMT system R1 also performs better than the fine-tuned NMT model N1. One of the reasons for the high perplexity values of N1 is that this model tends to hallucinate because it has been fine-tuned with a small data set. However, this does not seem to affect the performance as the models trained on this data do not perform considerably worse.
The performance of the LLM varies significantly, with pronounced differences between the three test sets in both directions of translation. The significant difference observed between Testset 1 and Testset 2 in the translation direction from lvb → ita cannot be seen in the R- and N-models. It remains unclear to what extent the LLM benefits from the given examples in the prompt. However, by providing an example, the propensity for errors was minimised, resulting in fewer mistakes during execution. Even though LLMs are not (yet) suitable for
generating texts in low-resource languages out-ofthe-box (see performance of L1 in Table 4), Ladin and low-resource languages in general could benefit from this technology. Our experiments show that models trained on back-translations from L1 performed best on Testset 1 and Testset 2 in the translation direction ita → lvb.
The inclusion of forward translations in the training data did not consistently improve the models, with the exception of the R-models for lvb → ita. This suggests that these synthesised texts introduce too much noise. However, model A2 was able to benefit from this data in the translation direction lvb → ita. Filtering this data could slightly improve the model.
As the models achieve similar scores on the test data, we also examined the quality of round-trip translations to gain additional insights. For this, we used 10k sentences from the monolingual Ladin and Italian data (which were also used for training, hence the high scores), translated them into the other language and then back-translated them. This concept of so-called round-trip translation is a suitable evaluation method (Zhuo et al., 2023). We used the R4/N4/L4 models for this purpose, ap-
plying one model A for one direction and the same or a different model B for the opposite direction. Table 5 shows the obtained results. It can be clearly seen that the results are worse when a different model is used for the reverse translation. This shows that although the models achieve similar results with the test data, they work differently. The R4 model proves to be the most stable here, as its translations can be back-translated well by all three models. For other combinations, a high variance can be observed.
The translation models N418, R419, and L420 have been released on Hugging Face, making them accessible for further research and application.

heading: 6 Related Work
text: Data augmentation such as back-translation (Sennrich et al., 2016; Hoang et al., 2018) and transfer learning (Zoph et al., 2016) are established strategies to improve MT systems. These concepts are discussed in Haddow et al. (2022); Ranathunga et al. (2023), with a focus on low-resource scenarios. The fact that the synthesised data plays a critical role in the quality of the systems trained on it, as it also introduces a certain degree of noise, was discussed extensively in Edunov et al. (2018); Xu et al. (2022). It was shown that tagging synthetic data can be beneficial in the training process (Caswell et al., 2019). In our work we do not apply advanced techniques to differentiate synthetic from real translations in training. The fact that RBMT systems can still be valuable for lowresource languages and can even help to achieve better results was also demonstrated for Northern
18https://doi.org/10.57967/HF/2695 19https://doi.org/10.57967/HF/2693 20https://doi.org/10.57967/HF/2694
Sámi (Aulamo et al., 2021) . In our experiments, we could also observe this in the translation direction lvb → ita. This could be due to the ability of the RBMT system to provide general knowledge that is not available in the relatively limited parallel training datasets (Aulamo et al., 2021). The use of LLMs for MT and different prompting techniques was investigated in Zhang et al. (2023) and their performance in the machine translation of low-resource languages has already been analysed in Moslem et al. (2023). Even if they struggle to generate texts in low-resource languages (Robinson et al., 2023), it has already been claimed that they can contribute to advances in machine translation of such languages. Our work is an example of how LLMs can be used in machine translation of a low-resource language; however, further prompt engineering is needed to make better use of such models.

heading: 7 Conclusion
text: In this work, we conducted a detailed comparison of RBMT, NMT and LLMs for back-translation in a low-resource scenario. We have tested various back-translation approaches and evaluated them for a previously unexplored language in the field of machine translation.
Our current methodology involved the exclusion of numerous Ladin monolingual sentences. However, this filtering would be less important for the translation direction lvb → ita. This previously discarded data could be re-incorporated to improve the performance of the models in this particular translation direction.
The round-trip translation scores indicate that the initial back-translation with the RBMT system leads to more robust models. Improving the ambiguity resolution of this rule-based translation system could lead to even better results.
The simplicity of the prompts used to feed the LLMs provides a further starting point for investigations. In particular, the question arises as to whether the results can be improved by further prompt engineering, e.g., by including the meaning for the distinct words occurring in a text using the available dictionary. Investigating the effects of prompt optimisation could provide new insights into maximising the efficiency of LLMs in machine translation, especially in low-resource scenarios.
We plan to address these research questions in our future work.

heading: Acknowledgements
text: This research was conducted in collaboration with the Ladin Cultural Institute Micurà de Rü and funded by the Regione Autonoma Trentino-Alto Adige/Südtirol.

heading: A Prompt template
text: I’ll give you samples for the translation from Ladin to Italian:
{ "translations": [
{ "Ladin": "scrí sües minunghes", "Italian": "scrivere le proprie opinioni" }, {
"Ladin": "mëte la secunda", "Italian": "mettere la seconda"
}, {
"Ladin": ’"zessa, i á prescia!"’, "Italian": ’"scansati, ho fretta!"’
}, {
"Ladin": "passé ia le rü", "Italian": "oltrepassare il fiume"
}, ... {
"Ladin": "chësc liber é to", "Italian": "questo libro è tuo"
}, ]
}
Please generate the translation of each of the 16 entries in the given dictionary, where the translations are empty. Return the same JSON dictionary where the values for Italian are filled:
{ "translations": [ { "Ladin": "Sperun da salvé almanco val’, dijun:", "Italian": ""
}, { "Ladin": "Ince tröc toponims y cognoms ladins desmostra che l’identité ladina é coliada
ala natöra y ala cultura da munt", "Italian": ""
}, { "Ladin": "De profesciun este pech... co este pa rové pro chësc laur?", "Italian": "" }, ... { "Ladin": "I dormi n pü’ domisdé y spo ciamó val’ ora dan mesanöt", "Italian": ""
} ]
}
Listing 1: Prompt template used to obtain the Italian translations from the LLM.

heading: Future Work
text: We plan to address these research questions in our future work