# Domain Adaptation for Arabic Machine Translation: Financial Texts as a Case Study
## 3 Methodology
### Experiment Setup
#### 3.2.2 Nmt Pre-Trained Models  
Our generic NMT pre-trained models use different Transformer architectures; however, we have implemented the fine-tuning objective using the huggingface NMT transformer (a sequence-to-sequence version in the Transformers library) procedure. We adopt a common Seq2 Seq architecture mainly composed of Encoder and Decoder network. The fine-tuning procedure is summarized as follows:  
1. Load a pre-trained model with the corresponding Tokenizer.
2. Tokenize the training, validation and test data into subwords (sentencePiece) so that training, validation and testing data will be truncated and tokenized.
3. Instantiate the loaded model using Seq2 seq Trainer ([[https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Seq2](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Seq2) SeqTrainer]([https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Seq2](https://huggingface.co/docs/transformers/en/main_classes/trainer#transformers.Seq2) SeqTrainer), accessed on 1 June 2024) from huggingface while setting the AdamW (Adam with Weight Decay) optimizer with appropriate hyperparameters.
4. Fine-tune the loaded model using backpropagation while minimizing cross-entropy.
5. Save the fine-tuned model on the huggingFace hub.
6. Load the fine-tuned model for inference, feed the tokenized test data and decode the outputs to obtain the translations.  
For Fine-tuning and inference, we use beam size 4 and batch size 16, on a GPU T4-15 GB (Google Colab). Further, we use ChatGPT as a baseline with zero-shot learning.  
OPUS (bt-big): We use OPUS [55] models from the Tatoeba-Challenge, specifically the models augmented with back-translated data of Wikimedia content and trained with Transformer-Big architecture. Here we picked the Helsinki-NLP/opus-mt-ar-en checkpoint. For tokenization, we instantiate our tokenizer which is based on SentencePiece [56] with the AutoTokenizer.from_pretrained method. This ensures that the tokenizer corresponds to the model architecture we want to use.  
NLLB: No-Language-Left-Behind (NLLB) [57] is a multilingual model which supports 200 languages with a massive size Transformer. Fine-tuning is carried out on NLLB using its distilled version facebook/nllb-200-distilled-600M checkpoint. For tokenization, we instantiate a multilingual model provided by NLLB for tokenization with the NllbTokenizerFast.from_pretrained method. This ensures that the tokenizer corresponds to the model architecture we are using.  
ChatGPT3.5: We use the ChatGPT-3.5-turbo model via its official API ([[https://chat.openai.com](https://chat.openai.com)]([https://chat.openai.com](https://chat.openai.com)), accessed on 20 August 2023) which powers ChatGPT. Here, we prepare our dataset in the format that is accepted by the API functions. In particular, we convert the AR-EN pairs into the Prompt template for sentence-level translation as recommended in the OpenAI playground for sentence-level translation tasks. In order to avoid errors, we truncate all the sentence pairs to a max size of 4290 characters before sending the request. Moreover, we set the size of the total tokens to about 378,460 tokens due to limit rate costs. For this model, we formatted the requests with the system message first 'You are a professional translator in the financial domain. Translate the following Arabic sentence: ar_en into English' followed by user content messages, where ar_en represents the AR-EN pairs.