<figure><img src="image_1.png" /></figure>

HAL
open science

# Enhancing Translation Quality: A Comparative Study of Fine-Tuning and Prompt Engineering in Dialog-Oriented Machine Translation Systems Insights from the MULTITAN-GML Team

Lichao Zhu, Maria Zimina-Poirot, Behnoosh Namdarzadeh, Nicolas Ballier, Jean-Baptiste Yunès

<figure><img src="image_2.png" /></figure>

## To cite this version:

Lichao Zhu, Maria Zimina-Poirot, Behnoosh Namdarzadeh, Nicolas Ballier, Jean-Baptiste Yunès. Enhancing Translation Quality: A Comparative Study of Fine-Tuning and Prompt Engineering in Dialog-Oriented Machine Translation Systems Insights from the MULTITAN-GML Team. Ninth Conference on Machine Translation, Nov 2024, Miami (Floride), United States. 10.18653/v1/2024.wmt-1.103 . hal-04781595

HAL Id: hal-04781595

https://hal.science/hal-04781595v1

Submitted on 13 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Enhancing Translation Quality: A Comparative Study of Fine-Tuning and Prompt Engineering in Dialog-Oriented Machine Translation Systems

## Insights from the MULTITAN-GML Team

Lichao Zhu¹, Maria Zimina-Poirot¹, Behnoosh Namdarzadeh¹, Nicolas Ballier¹,³, Jean-Baptiste Yunès²

¹CLILLAC-ARP, ²IRIF, ³LLF
Université Paris Cité, F-75013 Paris, France
Contact: lichao.zhu@u-paris.fr

<figure><img src="image_4.png" /></figure>

## Introduction

→ This study focused on enhancing machine translation quality through:

• Fine-tuning strategies

• Prompt engineering techniques

• Analysis of translation errors

→ Research objectives:

• Assess fine-tuning feasibility with minimal unlabelled data

• Use LLMs for post-editing translations

• Analyse linguistic features in bilingual customer service interactions

## Methods

→ Fine-tuning and translation via NMT Engine:

• Used neural machine translation (NMT) engine

• Applied in-domain baseline model

• Utilised Chat Task 2022's train and valid datasets and 2024's valid dataset

→ Translation with baseline models:

• NLLB-200-3.3B baseline model

• Deep translator implementation

→ ChatGPT-4o for post-editing of all translations

## Translation Performance Comparison (BLEU)

→ Significant improvement over both baseline models in both translation directions

→ Importance and value of model fine-tuning in specialised domains

<table><thead><tr><th rowspan="2"></th><th colspan="2">Fine-tuned</th><th colspan="2">In-domain</th><th colspan="2">Generic</th></tr><tr><th>model</th><th>baseline</th><th>model</th><th>baseline</th><th>model</th><th>baseline</th></tr></thead><tbody><tr><td>en → fr</td><td>57.19</td><td>48.05</td><td>50.47</td><td>48.05</td><td>48.19</td><td>50.47</td></tr><tr><td>fr → en</td><td>55.02</td><td>48.28</td><td>48.19</td><td>48.28</td><td>48.19</td><td>50.47</td></tr></tbody></table>

## Primary and Contrastive Metrics Comparison

→ Deep translator and the fine-tuned Systran model: higher and more consistent scores

→ NLLB-200-3.3: noticeably lower scores

<table><thead><tr><th colspan="2">Primary (Systran® fine-tuned)</th><th colspan="2">Contrastive 1 (NLLB-200-3.3)</th><th colspan="2">Contrastive 2 (Deep translator)</th></tr><tr><th>sentenceBLEU</th><th>TER</th><th>sentenceBLEU</th><th>TER</th><th>sentenceBLEU</th><th>TER</th></tr></thead><tbody><tr><td>0.70</td><td>0.25</td><td>0.57</td><td>0.50</td><td>0.71</td><td>0.28</td></tr></tbody></table>

## Serendipity: Mistranslations Caused by Misalignment

<table><thead><tr><th></th><th>Source</th><th>Reference</th><th>Primary</th><th>sentenceBLEU</th><th>TER</th></tr></thead><tbody><tr><td>1</td><td>Is there anything else I can assist you with today?</td><td>Avez-vous besoin d'aide pour autre chose aujourd'hui ?</td><td>Y a-t-il autre chose que je puisse faire pour vous aider aujourd'hui ?</td><td>0.25</td><td>1.125</td></tr><tr><td>2</td><td>You are welcome!</td><td>Ce fut un plaisir de vous parler.</td><td>C'était agréable de parler avec vous.</td><td>0.00</td><td>1.0</td></tr><tr><td>3</td><td>I hope you have an excellent day.</td><td>J'espère que vous passez une excellente journée !</td><td>Merci pour l'information.</td><td>0.00</td><td>1.0</td></tr></tbody></table>

## Acknowledgements

We would like to acknowledge the support of the Pure Neural Server (PNS-UP) scientific platform, the 2024 MULTITAN-GML research equipment grant, and the 2021 PAPTAN research equipment grant from the Scientific Platforms and Equipment Committee, Université Paris Cité.

## Comparison of Specific Lexical Features

Many translation errors are over-represented in the NLLB-200-3.3B prediction, while the reference translation and the fine-tuned Systran prediction share common lexical features.

<figure><img src="image_5.png" /></figure>

## Correspondence Analysis of the Reference and Our Predicted Translations

→ Fine-tuned Systran model: close to the reference and similar to Deep translator

→ NLLB-200-3.3B model: displays a distinct profile

<figure><img src="image_6.png" /></figure>

## Observations and Conclusions

<figure><img src="image_7.png" /></figure>

Our primary translation has best BLEU and chrF scores but lower COMET score.

<figure><img src="image_8.png" /></figure>

Our fine-tuned model has higher precision than recall.

<figure><img src="image_9.png" /></figure>

Dataset quality is crucial, with preprocessing playing an important role.

<figure><img src="image_10.png" /></figure>

The RAG-based post-editing strategy for LLMs is expensive and often underperforms.