# Domain Adaptation for Arabic Machine Translation: Financial Texts as a Case Study
## 4 Results and Discussion
### Automatic Evaluation  
In Setup 1, OPUS and NLLB perform equally with inferior performances of around 14 and 42 for BLEU and chrF points, respectively. The TER score which is expressed as the ratio of the number of edits to the average number of words in the reference is high for the two models. Thus, it indicates that the translation is of poor quality. In terms of COMET score, both models have very poor results which means reference-based COMET may lose information from source, translation output, or reference embeddings, except for ChatGPT-3.5. But, BERTScores for all three models are high which means that they do not correlate with COMET score. In comparison, BERTScore and COMET have a significant difference in their scores. In contrast, ChatGPT-3.5 performs competitively better (BLEU 26.13) than OPUS and NLLB models. Indeed, we are not surprised by this fact which is in line with related research works [50; 62; 63]. However, these findings are not consistent with a previous finding [64] where the authors evaluated ChatGPT and GPT on 4000 Arabic-English pairs and found out that SoTA models like araT5 [44] outperforms ChatGPT by 19 BLEU Points. Similarly, ref. [65] found the English-to-Arabic translation of ChatGPT-3.5 was below average compared to 14 established MT systems. When, we analyze Setup 2, as expected, fine-tuning all models on authentic data has generally helped improve the BLEU scores and other metrics as well. This finding is also in-line with other previous research [7; 56]. However, ref. [62] noticed that for domain-specific translation (e.g., in the biomedical field), ChatGPT's performance degrades considerably. We attribute this behavior to the observation that ChatGPT is capable of translating our sentences better than terminologies in sentences from the biomedical domain, a very specific domain. Furthermore, we clearly, see that BLEU scores increase from 14.58 to 48.83, from 14.38 to 43.43 and from 26.13 to 51.15 for OPUS, NLLB and ChatGPT-3.5, respectively. In terms of COMET and BERTScore, both metrics had a high correlation which indicates acceptable translation outputs. Concerning ChatGPT, even though it only used 2000 pairs of AR-EN sentences for fine-tuning, it outperforms all other models which means the MT quality of ChatGPT can easily be improved with little additional data from the language pair, a fact that has not been previously confirmed for related approaches, since this is the first work that assesses the performance of ChatGPT fine-tuned models for AR-EN MT task. Nevertheless, for English, this work [66] has shown that ChatGPT has great robust translation capabilities over related SoTA MT models. Our experimental result confirms the latter finding and shows that with a carefully prepared certain amount of fine-tuning data, this model is capable of creating acceptable translations. As for the translation robustness, results from Setup 2 suggest that ChatGPT-3.5 performs competitively well on financial news. Regarding the \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Model** & **spBLEU \(\uparrow\)** & **chrF \(\uparrow\)** & **TER \(\downarrow\)** & **COMET \(\uparrow\)** & **BERTScore \(\uparrow\)** & **Human \(\uparrow\)** \\ \hline \multirow{4}{*}{3} & OPUS (bt-big) FT-BMT [5] & **47.56** & 64.53 & **54.30** & **57.21** & **0.95** & 2.94 \\  & OPUS (bt-big) FT-S [6] & 40.67 & 57.87 & 60.46 & 49.71 & 0.94 & 2.67 \\ \cline{1-1}  & NLLB 600M FT-BMT [7] & 43.38 & 60.92 & 54.63 & 52.77 & 0.94 & 2.67 \\  & NLLB 600M FT-S [8] & 40.77 & 58.26 & 57.48 & 49.44 & 0.94 & 2.85 \\ \cline{1-1}  & ChatGPT-3.5 FT-BMT & 45.07 & **67.64** & 55.07 & 33.55 & 0.93 & 2.93 \\  & ChatGPT-3.5 FT-S & 34.67 & 62.93 & 70.29 & 23.03 & 0.91 & 2.77 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: _Cont._human evaluation, all models in this setup reached possible and acceptable translations. We conclude that our experiment shows that providing in-domain examples to ChatGPT achieves comparable results to a SoTA model in terms of automatic and human evaluation.