The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilities (2024)

Abstract: Fine-tuning large language models (LLMs) for machine translation has shown improvements in overall translation quality. However, it is unclear what is the impact of fine-tuning on desirable LLM behaviors that are not present in neural machine translation models, such as steerability, inherent document-level translation abilities, and the ability to produce less literal translations. We perform an extensive translation evaluation on the LLaMA and Falcon family of models with model size ranging from 7 billion up to 65 billion parameters. Our results show that while fine-tuning improves the general translation quality of LLMs, several abilities degrade. In particular, we observe a decline in the ability to perform formality steering, to produce technical translations through few-shot examples, and to perform documentlevel translation. On the other hand, we observe that the model produces less literal translations after fine-tuning on parallel data. We show that by including monolingual data as part of the fine-tuning data we can maintain the abilities while simultaneously enhancing overall translation quality. Our findings emphasize the need for fine-tuning strategies that preserve the benefits of LLMs for machine translation.

heading: 1 Introduction
text: Recent work has highlighted a range of qualitative advantages that large language models (LLMs) hold over Neural Machine Translation (NMT) models. One significant advantage is the controllability of style and language variety which can be achieved through prompting and in-context learning (Brown et al., 2020; Garcia et al., 2023; Agrawal et al., 2023). LLMs also exhibit inherent document-level translation abilities (Wang et al., 2023; Karpinska and Iyyer, 2023). Another advantage is their ability to produce less literal translations (Raunak et al., 2023). Finally, LLMs have been shown to have
∗Work done during an internship at Amazon.
better performance in handling difficult linguistic phenomena such as idioms and ambiguous expressions (Neubig, 2023). Taken together, LLMs are surpassing NMT models in terms of versatility.
Recent studies have demonstrated that finetuning LLMs on parallel data further improves their translations as measured by metrics that reflect overall quality (such as COMET) (Li et al., 2023; Yang et al., 2023; Zeng et al., 2023). However, relying on general translation quality metrics and generic test sets does not fully capture the nuanced abilities of LLMs in machine translation. This oversight raises questions about the retention of LLM-specific advantages — such as controllability, document-level translation proficiency, and the production of less literal translations — after fine-tuning on parallel data. While it is clear that general machine translation quality improves through fine-tuning, there is a risk that LLMs lose their unique strengths due to catastrophic forgetting (McCloskey and Cohen, 1989; Ratcliff, 1990; Luo et al., 2023). Determining the extent of this risk and comparing the effect of various fine-tuning strategies in preserving the qualitative benefits of LLMs remains an important yet unresolved question.
We investigate how qualitative advantages of LLMs change when fine-tuning on parallel data. We consider LLaMA and Falcon models, with parameter counts ranging from 7 billion up to 65 billion. The LLM properties we investigate are general translation quality, formality steerability, non-literalness in idiom translations, performance on specialized domains, and performance on document-level input which requires contextualisation of ambiguous tokens. We compare two fine-tuning strategies for varying data sizes (89K up to 1.4M) in six translation directions. Our main findings and contributions are:
• We show that while fine-tuning LLMs on parallel data enhances overall translation quality as measured by COMET, it simultane-
6189
ously leads to a decline in important attributes. Even when only using 18k fine-tuning samples we observe degradations in formality steering, technical translation through fewshot examples, and contextualization capabilities required for document-level translation. In general, we find that using larger data sets for fine-tuning data results in more severe degradations, and these trends are consistent across all tested model scales and architectures. The exception we observe is in the ability to produce less literal translations, which improves in fine-tuning.
• We show that incorporating a mix of monolingual and parallel data during fine-tuning can preserve abilities of LLMs. Overall translation quality is enhanced to a greater extent compared to fine-tuning on parallel data alone.
• We introduce a novel evaluation dataset, IdiomsInCtx-MT1, to measure non-literalness performance. To our knowledge, it is the first dataset that consists of idiomatic expressions in context and their human-written translations. It covers 2 language pairs with 3 translation directions.
Our findings highlight the importance of creating fine-tuning approaches that enhance general translation quality while also preserving the distinctive capabilities of LLMs for machine translation.

heading: 2 Related Work
text: Advantages of LLMs for MT Several studies have investigated the use of LLMs for translation. Generally, current LLMs show strong performance for most language pairs, but lag behind NMT systems when translating into low-resource languages (Zhu et al., 2023a; Stap and Araabi, 2023; Robinson et al., 2023; Kocmi et al., 2023). In addition to strong performance, LLMs exhibit certain abilities that are relevant for translation. NMT systems show a bias towards generating text that is overrepresented in the data, such as language varieties (Riley et al., 2023) and formality (Rippeth et al., 2022), whereas LLMs can easily be controlled for this bias using examples (Garcia et al., 2023). In addition, examples can be supplied to improve general LLM translation quality via in-context learning
1We release the dataset at https://github.com/ amazon-science/idioms-incontext-mt
(Agrawal et al., 2022; Moslem et al., 2023a). NMT models are often unable to translate idioms accurately and generate literal translations (Dankers et al., 2022). LLMs produce less literal outputs compared to NMT models, particularly for sentences that contain idiomatic expressions (Vilar et al., 2023; Raunak et al., 2023). NMT models are trained on sentence level, and thus do not take into account document context. LLMs outperform NMT models for document translation in general domains (such as news and social media) (Wang et al., 2023), as well as in more specialised domains such as literature (Karpinska and Iyyer, 2023).
Finetuning LLMs for MT There are multiple strategies for fine-tuning LLMs for machine translation. One approach makes use of either a small set of high-quality human-written translations or a set of translation instructions for fine-tuning (Li et al., 2023; Zeng et al., 2023; Jiao et al., 2023). Another line of work makes use of more traditional machine translation data: parallel data from the web, which is orders of magnitude larger compared to what is used in fine-tuning (Yang et al., 2023; Alves et al., 2023; Zhang et al., 2023; Zhu et al., 2023b). These strategies are focused on improving general machine translation quality, but it remains unclear what happens to other abilities that are relevant for translation. We investigate the effect of fine-tuning on relevant abilities for translation using publicly available models.

heading: 3 Fine-tuning LLMs on parallel data

heading: 3.1 Experimental Setup
text: Models We use the LLaMA (Touvron et al., 2023) 7B, 13B, 30B, 65B, and Falcon (Almazrouei et al., 2023) 7B and 40B language models.
Optimisation We refer the reader to Appendix A for full details on optimisation, hyperparameters, and instruction formats.
Language directions We consider the language directions German (de), Russian (ru), and Chinese (zh) into and out of English (en).
Human-written training data Following Xu et al. (2023), we use human-written translations from WMT17 to WMT20, resulting in 89K training examples that are evenly distributed across the language directions we consider. On this dataset, we perform full fine-tuning for models up to 40B and QLoRA fine-tuning for the LLaMA 65B model.
Web-scraped training data Additionally we train models on general domain OPUS (Tiedemann, 2012) data from the News-Commentary, WikiTitles, and ParaCrawl (Bañón et al., 2020) corpora. To ensure that the resulting data is above an acceptable quality threshold we perform data filtering using an internal Quality Estimation (QE) model, which has a similar architecture as COMETKiwi (Rei et al., 2022) and is based on the InfoXLMLarge pretrained multilingual encoder (Chi et al., 2021). We train our sentence-level QE model on a large internal dataset of human annotations for more than 12 languages, where each translation is rated between 1 (completely random) and 6 (perfect translation).
We use a subset of 1.4M sentence pairs of this filtered data that is evenly distributed across language directions for training. We fine-tune LLaMA models up to 40B parameters on this dataset but leave out larger models because of the high computational cost.
Evaluation data and metrics For evaluation, we consider the following test sets:
• WMT22 To evaluate general machine translation quality, we use WMT22 (Kocmi et al., 2022) test sets consisting of news, e-commerce, social, and conversational domains. We evaluate all language directions on this test set in a 0-shot setting. Following the recommendation of Kocmi et al. (2021), we report COMET scores2 (Rei et al., 2020). We do not report BLEU scores (Papineni et al., 2002), since this metric is known to have a poor correlation with human judgments (Mathur et al., 2020; Freitag et al., 2021; Kocmi et al., 2021; Freitag et al., 2022).
• CoCoA-MT To evaluate formality steering ability of LLMs, we make use of the CoCoAMT (Nadejde et al., 2022) dataset. It consists of 600 test sentences with English source and contrastive target sentences consisting of a formal and informal translation. We use German as the target language and report both COMET and accuracy based on the ratio of correctly predicted formality forms. We use 5-shot examples to bias the formality of outputs.3
2model: unbabel/wmt22-comet-da 3We also experimented with steering formality through prompting, but the results were inferior to using 5-shot exam-
• Law, Medical, and TICO-19 To evaluate the in-context learning ability on technical domains, we consider the Law and Medical test sets from Aharoni and Goldberg (2020). We consider 5-shot inputs. We evaluate on German ↔ English and report COMET scores. In addition we evaluate technical abilities on TICO-19 (Anastasopoulos et al., 2020), which consists of translations in the COVID-19 domain. We evaluate on Russian ↔ English and Chinese ↔ English.
• ctxpro We evaluate 0-shot performance on longer inputs that includes sentences that require context to be disambiguated correctly by including ctxpro (Wicks and Post, 2023). We consider the animacy ambiguity type in German → English and Russian → English. 4 While English makes no gender distinction for inanimate objects, some other languages such as Russian do. The ambiguous animacy examples in the ctxpro dataset require corresponding document-level context for correct disambiguation. We subsample the test set to 2K examples per language direction. The average number of sentences per input is 10.16± 1.27. We report the generative accuracy score, which measures the accuracy of the contextualization.
• IdiomsInCtx-MT We introduce a novel dataset consisting of idiomatic expressions in context and their human-written translations.5 The dataset comprises 2 language pairs: German and Russian paired with English. For German, the opposite translation directions are also included. Current idiom datasets stem from potentially noisy, web-extracted sources (Fadaee et al., 2018), machine-generated translations (Tang, 2022), or are monolingual (Haagsma et al., 2020). In contrast, we use professional translators to create a high-quality evaluation benchmark. Idiomatic expressions and their context sentences were sourced in the respective source language and translated to the target language
ples. 4We also experimented with different ambiguity types (auxiliary and gender in out-of-English directions). However, the resulting translations were often incomplete, containing only a subset of the total number of sentences, making it impossible to effectively evaluate decontextualization abilities.
5The dataset will be available shortly.
by professional translators. In addition, the dataset contains annotations of the source and target idiomatic expressions for each segment. This enables running targeted evaluation on the 0-shot translations of the idiomatic expressions in addition to general quality metrics. We evaluate on English→German, German→English and Russian→English using test splits of 1000 segments. We report COMET, LitTER (Baziotis et al., 2023) and MWEScore, an internal multi-reference version of Score_mwe (Zaninello and Birch,
2020).
Additionally, we considered style transfer and constrained generation, such as prompting a language model to use custom terminology, as additional translation capabilities. However, our final choices were influenced by the availability of suitable test sets and the applicability of targeted automatic metrics, leading us to not pursue these options.

heading: 3.2 Results
text: General translation quality improves Results on WMT22 for models trained on human-written translations are summarized in Figure 1. Consistent with expectations, we observe that fine-tuning generally improves the translation quality, and larger models generally perform better. Using 89K parallel examples does not always yield better results compared to smaller datasets. For most language directions, we notice an initial increase in translation quality, followed by a slight decline. The most notable increases are found for the out-of-English directions. In contrast, when models are fine-tuned on a more extensive dataset (up to 1.4 million examples) sourced from the web (refer to Appendix B, Figure 7), translation quality continues to improve with the addition of more data. We hypothesize that this difference stems from the domain-specific composition of the human-written WMT training data, which is exclusively news content. In contrast, the OPUS dataset is more diverse and includes multiple domains. This diversity better reflects domain composition of the WMT22 test set, which has content from news, e-commerce, social media and
conversational domains. The improvements are significantly more marked in translations from other languages into English. A comparative analysis of the best-performing checkpoints of LLaMA models (7B, 13B, 30B) fine-tuned on human-written and OPUS data across 6 translation directions shows a clear trend. In 15 out of 18 cases, models finetuned on the larger OPUS dataset achieved superior results (refer to Appendix B, Table 4).
Formality steering ability degrades We show results for formality steering using 5-shot examples for models trained on human-written data in Figure 2. Notably, the base models exhibit strong performance for this task; for instance, the LLaMA-7b model achieves an accuracy of 0.982 in identifying informal markers. However, fine-tuning on only 18K examples results in a decline of this ability to 0.862, even though German-English COMET on WMT22 continues to improve up to 36K examples. The degradation is more pronounced with informal markers, which is likely attributable to the formal bias inherent in the WMT22 training data. Fine-tuning on more data further degrades formality steering capabilities: there is a significant negative correlation (Spearman’s ρ = −0.46, p < 0.001) between formal and informal marker prediction and dataset size. We find that COMET stays relatively constant (see Appendix B Figure ??), despite some fluctuations, indicating that it does not adequately capture formality markers. As a result, the accuracy scores correlate very weakly (ρ = 0.16, p < 0.001) with COMET scores, which suggests that comprehensive evaluation of LLMs for machine translation benefits from task-specific metrics.
Further, when examining models trained on OPUS data (see Appendix B Figure 9), we find that increasing the amount of fine-tuning data up to 1.4M parallel sentences further exacerbates the degradation. Again, we observe a significant negative correlation (ρ = −0.58, p < 0.001) between accuracy and dataset size for OPUS-trained models, reinforcing the conclusion that larger parallel datasets during fine-tuning adversely affect formality steering capabilities.
Performance on technical domains degrades Next, we evaluate the model capabilities of doing technical translations given 5-shot examples. Results on human-written training data for a subset of the domains and directions are shown in
Figure 3. In most cases, performance starts to degrade after only 18K examples. For example, LLaMA-7b scores 0.8308 COMET on TICO English-Russian, whereas after fine-tuning on 18K examples COMET is 0.8234, a degradation of 0.0074. Results on the other domains and directions show a similar trend (Appendix B Figure 10). The COMET scores correlate negatively with datastore size (ρ = −0.27, p < 0.001), indicating that fine-tuning on more data results in larger degradations. When inspecting models trained on OPUS data, we observe consistent conclusions: few-shot technical domain translation capabilities degrade, and the amount of degradation is dependent on the dataset size (see Appendix B Figure 11).
Contextualization ability on document-level input degrades Our analysis extends to the animacy contextualization accuracy of document-level input. We show results for models trained on human-written data in Figure 4. Mirroring the trend observed in formality steering, a clear degradation in contextualization accuracy is noted upon fine-tuning the models on parallel data. Again, we observe that fine-tuning on only 18K examples results in a decline of this ability, even though general translation quality on WMT continues to improve. For example, Falcon-40b scores 0.91 before finetuning, which degrades to 0.85 after 18K examples. The decline can be summarized by a negative correlation between accuracy and the size of the dataset used for fine-tuning (ρ = −0.55, p < 0.001), indicating that contextualization abilities further degrade when fine-tuning on more data. This trend is not exclusive to WMT data. A similar pattern emerges when analyzing models trained on smaller human-written translation datasets. As detailed in Appendix B, Figure 12, we observe a consistent reduction in contextualization accuracy with the addition of more training data, again supported by a negative correlation (ρ = −0.49, p < 0.001).
Performance on idioms remains stable or improves We evaluate the quality of idiomatic translations on the IdiomsInCtx-MT test sets. Since idiomatic translation is inherently difficult, we focus our analysis on the strongest model, LLaMA-65b. Figure 5 shows COMET, LitTER (lower is better) and MWEScore (higher is better) across training checkpoints for 1 epoch. While LitTER uses inputspecific block lists to assess the literalness of translation outputs based on annotations of idiomatic
expressions in the input, MWEScore additionally relies on one or more gold translations of those idiomatic expressions and computes a score based on edit distance of output vs reference idiom tokens.
Similar to the WMT22 test set, we see that COMET scores improve until the first 1-3 checkpoints before stabilizing or starting to decrease. However, for idiomatic expressions even thed targeted metrics LitTER and MWEScore improve during fine-tuning or least remain stable. This indicates that even a large open-source model like LLaMA-65b still has room for improvement when it comes to idiomatic translations. Future work could investigate translation literalness and idiomaticity of LLM translations on stronger models such as GPT-3.5 during fine-tuning.6
6https://openai.com/blog/gpt-3-5-turbo-fine-tuning-andapi-updates

heading: 4 Fine-tuning on a mix of monolingual and parallel data
text: Having established that fine-tuning on parallel data leads to a decline in various advantages of LLMs for machine translation, this section delves into strategies to mitigate this degradation.
A potential approach to counteract degradation involves incorporating examples of desired behaviors during fine-tuning. For instance, the degradation of few-shot capabilities for domain adaptation can be partially mitigated by including few-shot examples during fine-tuning (Alves et al., 2023; Moslem et al., 2023b). However, our aim is to establish a more general solution that prevents degradation across a broader spectrum of behaviors, without the need to specifically include data for each behavior during fine-tuning. To achieve this, our experiments involve a blend of monolingual and parallel data in the fine-tuning phase. This strategy stems from the understanding that the pretraining on monolingual data contributed to these beneficial phenomena, and our goal is to retain these qualities during fine-tuning.

heading: 4.1 Experimental setup
text: To construct our monolingual dataset, we use the News-Commentary data from WMT22. This dataset includes document-level information, which we preserve by concatenating sentences within each paragraph to form a single data entry. The resulting processed data closely resembles the type of data used for LLM pre-training. We then merged this monolingual dataset with parallel data sourced from OPUS (we sample 89K parallel examples), maintaining a 1:1 ratio and resulting in a total of 178K examples. We refer to this arrangement as the FT 1:1 setup. We compare this setup to several baselines: 1) the base model prior to any fine-tuning; 2) the FT 1:0 setup, which in-
volves fine-tuning exclusively on parallel OPUS data (89K); and 3) the FT 2:0 setup, where finetuning is conducted on parallel OPUS data equal in size to our mixed monolingual and parallel dataset, totalling 178K examples. We use LLaMA-7B with a context window size of 2048 tokens for this experiment.

heading: 4.2 Results
text: General translation quality further improves The results, as detailed in Table 1, show the comparative performance of the baselines and the integration of monolingual data during the fine-tuning phase on general translation quality (WMT22) as measured by COMET. Including monolingual data
(FT 1:1) leads to translations that generally surpass those produced by parallel-only fine-tuning approaches (FT 1:0 and FT 2:0). Notably, the most significant improvement is observed in the en-zh direction, where the FT 1:1 setup yields an increase of 0.0131 COMET over the best baseline (FT 2:0). This can be attributed to the pre-training of LLaMA on an English-centric corpus, which contains only minimal amounts of (accidental) Chinese data. As suggested by Xu et al. (2023), out-of-English capabilities of the model can be substantially augmented through an additional monolingual fine-tuning step, a methodology akin to our approach.
While the enhancement of general translation quality is a beneficial outcome, our primary interest lies in evaluating the ability of our method to preserve and possibly enhance the qualitative behaviors inherent in Large Language Models. This aspect forms the next focus of our investigation. Figure 6 shows a comparison between our finetuning method on formality steering, documentlevel contextualization, technical translation, and idiom translation tasks. A consistent trend is observed: the integration of monolingual data with parallel data during fine-tuning generally results in more effective preservation of various translation capabilities.
Minimal formality steering degradation Baselines using only parallel data show accuracy drops as great as 0.198 for formal and 0.11 for informal steering. The inclusion of monolingual data mitigates this degradation, reducing it to just 0.025 for formal and 0.007 for informal steering, albeit some degradation persists when compared to the baseline.
Degradation of contextualization abilities are lessened The impact of combining parallel and monolingual data during fine-tuning is also evident here. For translations from German to English and Russian to English, the loss in accuracy is up to 0.073 and 0.075, respectively, when using only parallel data. Incorporating monolingual data diminishes this degradation for Russian to English translations (-0.035), and even shows a notable improvement of +0.021 over the base model.
Technical domain performance is enhanced The inclusion of monolingual data during finetuning enhances performance for English into Russian and Chinese, and vice versa. For instance, in the TICO English to Chinese translation task,
the blended approach of monolingual and parallel data in fine-tuning yields a +0.0089 COMET score improvement over the baseline. Conversely, relying solely on parallel data results in a 0.022 COMET score decrease (FT 2:0), indicating a substantial differential of 0.03. For English in and out of German in both the Law and Medical domain, the differences between fine-tuning with monolingual data plus parallel data and the base model are minimal. In contrast to using parallel data only, we observe no decline. Notably, we use parallel data up to 178K (FT 2:0), where degradation is relatively modest in the case of few-shot technical domain translation. When doing extended finetuning, this capability will further degrade, as we show in Section 3.2.
Performance on idioms improves Including monolingual data (1:1) improves the literalness of translations as measured by LitTER for ru-en and en-de. However, LLaMA-7b does not demonstrate good performance on idiomatic translations, making it an easy baseline to improve upon.
These findings underscore the nuanced benefits of incorporating monolingual data when finetuning English-centric LLMs for machine translation tasks, specifically in preserving task-relevant capabilities. However, for long-term progress, we advocate for the development of LLMs with multilingual data in mind. In this approach, parallel data would be combined with monolingual data during the pre-training phase (Anil et al., 2023; Wei et al., 2023). Nevertheless, even when beginning with a more robust multilingual LLM for translation purposes, the exploration of fine-tuning strategies that preserve the emerged capabilities of LLMs remains critical, especially when adapting these models for various use cases and objectives.

heading: 5 Conclusion
text: We investigated how fine-tuning on parallel data affects the qualitative advantages of LLMs for machine translation. While previous research predominantly focused on summary quality metrics like COMET, our findings reveal a more complex interplay between fine-tuning and LLM capabilities. Consistent with prior work, we find that fine-tuning enhances the general translation quality of LLMs. However, we show that fine-tuning adversely impacts several important qualitative advantages of LLMs. We observe declines in the abilities of LLMs to 1) perform formality steering, 2) perform
technical translation through few-shot examples, as well as 3) a decrease in their document-level translation capabilities. The ability to produce non-literal translations shows improvement post fine-tuning, likely because the publicly available LLMs we investigate do not perform strongly on this task to begin with. Furthermore, our results indicate that these degradations are more pronounced for larger fine-tuning datasets, even when generic translation quality continues to improve. These trends are consistent across different model scales (7b up to 65b), underscoring the generalizability of our findings. To prevent these degradations, we develop a fine-tuning method tailored for machine translation, that combines monolingual and parallel data. We show that this approach mitigates the degradation of LLMs’ qualitative advantages, thereby preserving their capabilities while improving general translation quality.

heading: Limitations
text: Because of the high cost of repeatedly fine-tuning LLMs of different sizes, we limited ourselves to 6 translation directions (German, Chinese, and Russian in and out of English). The impact of finetuning on emergent abilities of LLMs when translating in and out of low-resource languages is not studied in our work. It is therefore possible that our findings do not generalize to low-resource languages.

heading: Ethics statement
text: The IdiomsInCtx-MT dataset is annotated by professional translators and they were all paid a fair rate.

heading: Acknowledgments
text: We thank Zach Hille and Arda Keskiner for their support in our experimental work.
Christof Monz acknowledges funding by the Netherlands Organization for Scientific Research (NWO) under project number VI.C.192.080.

heading: A Details on experimental setup
text: We run our fine-tuning experiments with the Hugging Face transformers library (Wolf et al., 2020) and make use of DeepSpeed (Rasley et al., 2020). We store intermediate checkpoints during finetuning to track how abilities evaluate over time. We perform full fine-tuning on models up to 40B. We use the AdamW optimizer with a cosine learning rate scheduler and 3% warm-up percentage. We
empirically set the batch size to 128, learning rate to 2e − 5, and train for one epoch. During inference we use beam search with a beam size of 4. For LLaMA-65B, we fine-tune with QLoRA (Dettmers et al., 2023), using 8-bit quantization. Following Zhang et al. (2023) we set the low-rank approximation to 64 and the scaling factor for low-rank adaptation to 32.
Inference Depending on the evaluation set, we use a 0-shot or few-shot approach. The prompt used for fine-tuning and 0-shot is shown in Table 2. For our few-shot setup, we find the 5 most similar source sentences and their corresponding target sentences from a corresponding train set (if available) or validation set. The resulting prompt is displayed in Table 3. We use Sentence-BERT (Reimers and Gurevych, 2019) for encoding7 and FAISS (Johnson et al., 2019) for searching similar sentences.

heading: B Additional results

heading: B.1 General translation quality (WMT22)
text: Results on WMT22 for models trained on filtered web-crawled data are summarized in Figure 7. Note that the data size up until 89K has relatively small increments, and later data sizes are doubled compared to the data point before it. Generally, in contrast to training on human-written data, translation quality continues to increase when adding more data.
We compare the best checkpoints of models trained on human-written and OPUS data in Table 4. In 15 out of 18 cases, models fine-tuned on the larger OPUS dataset results in better scores on WMT22.

heading: B.2 Formality steering
text: Figure ?? shows COMET scores for models trained on WMT data. COMET scores stay relatively constant, in contrast to the accuracy of formality forms.
Figure 9 shows that the ability to generate correct formality markers also degrades when fine-tuning on OPUS data. Extended fine-tuning continues to show benefits in terms of general translation quality (see Figure 7), but the ability to generate correct formality markers continues to degrade.
7We use all-MiniLM-L6-v2 for English sentences and paraphrase-multilingual-MiniLM-L12-v2 for non-English

heading: B.3 Technical domains
text: Figure 10 shows the outcome of fine-tuning on human-written data across all evaluated domains and language directions. We observe a consistent trend: fine-tuning impairs the few-shot technical translation capabilities, and generally further finetuning results in more degradations. The COMET scores correlate negatively with datastore size (ρ = −0.27, p < 0.001), indicating that fine-tuning on more data results in larger degradations.
The effects of fine-tuning are also analyzed using filtered web-scraped data from OPUS, as shown in Figure 11. Similar to the previous findings, an increase in data volume for fine-tuning corresponds to performance degradations, evidenced by a negative correlation between COMET scores and datastore size (ρ = −0.33, p < 0.001).
However, OPUS data reveals that these degradations manifest more gradually compared to the WMT dataset. This discrepancy is likely due to OPUS’s broader domain coverage, in contrast to the specialized news content of the human-curated WMT dataset. Notably, while fine-tuning on OPUS data leads to deterioration in technical domain translation accuracy when leveraging few-shot examples, it concurrently continues to enhance overall translation quality (Figure 7), underscoring a nuanced impact of fine-tuning across different data types and translation tasks.
e

heading: B.4 Contextualization of document-level input
text: Figure 12 shows that the animacy contextualization accuracy of document-level input degrades for models fine-tuned on filtered web-crawled OPUS data. We observe a negative correlation between accuracy and fine-tuning dataset size (ρ = −0.49, p < 0.001).

heading: Future Work
text: Future work could investigate translation literalness and idiomaticity of LLM translations on stronger models such as GPT-3. 5 during fine-tuning. 6
6https://openai. com/blog/gpt-3-5-turbo-fine-tuning-andapi-updates