# Domain Adaptation for Arabic Machine Translation: Financial Texts as a Case Study
## 4 Results and Discussion
### Automatic Evaluation
In Setup 3 we fine-tune the baseline models with the augmented data in two versions, one using back-translated data and the other using synthetic data. We observe that both lexical metrics (BLEU and chrF) show consistent degradation with all models. The same applies to the TER score. For instance, for ChatGPT, the BLEU score decreased dramatically from 51.15 to 34.67 when fine-tuned on synthetic data while maintaining an acceptable score (BLEU 45.38) when fine-tuned on back-translated data. We observe that the COMET score degraded massively for ChatGPT more than for OPUS and NLLB. One explanation could be that the synthetic data may have a lot of generated tokens that are grammatically correct, but they have nonsense meaning, as we know from the current state of the generative text. This could indicate that the translation results are not close in the embedding space with the source and reference. In contrast, BERScore maintained a good score over the two versions for all models. In this setup, OPUS (bt-big) FT (back MT) has made it the best model that provides reasonably good scores translations; however, it still lags behind the OPUS model fine-tuned on authentic data by at least 1.3 BLEU points. Generally, the drop in performance for all models in this setup is not consistent with others' research. For instance, the authors in [7] used synthetic data in the healthcare domain and achieved improvements on the in-domain test set. In comparison, with this work, the authors applied synthetic data generation using mGPT ([[https://huggingface.co/sberbank-ai/mGPT](https://huggingface.co/sberbank-ai/mGPT)]([https://huggingface.co/sberbank-ai/mGPT](https://huggingface.co/sberbank-ai/mGPT)), accessed on 12 March 2024) a multilingual language model. We argue that this model might have better perplexity in generated tokens compared to araGPT2. To the best of our knowledge, we did not find any research work investigating the performance of both models in regard to Arabic. We will further investigate this issue in future work. However, there are many general reasons explaining OPUS, NLLB and ChatGPT behavior in domain-specific MT, especially in the case of augmenting the dataset with synthetic data. One explanation is that the use of synthetic data may cause incorrect token choices, grammatical errors, or unnatural sentence structures to propagate into the translation outputs which make suboptimal translation outputs. Indeed, the results of this study demonstrate the models' robust translation capabilities for in-domain adaptation. They perform well when fine-tuned on authentic data. However, we observe a discrepancy between COMET and BERScore. For instance, ChatGPT-3.5 performs worse on augmented data yielding a lower COMET score (23.03) but still having high BERScore (0.91). This behavior seems uncommon. One possible explanation is that COMET with reference-based translation is failing to find closeness in all three resource embeddings, whereas BERScore is able the find closeness in the similarity between an MT output and a reference translation. This behavior encourages us to drive human evaluation a much-needed score for trustworthiness.