## 5 Training an Arabic Transformer Model Using the AraFast Corpus  
In the present study, a large MSA corpus (AraFast) was developed to train an Arabic transformer model, and then we conducted a study to understand how well this corpus works. Four experiments were set up to identify the best version of the corpus and to determine how the size of the corpus would affect model training. Various forms of the AraFast corpus were tested, including segmented, unsegmented, web-scraped, and smaller versions.  
To construct the experiments and obtain the best version of the AraFast corpus, we proposed two Arabic transformer models: AraFastQA-mini, and AraFastQA-base, which were both examined. The AraFastQA-mini model was pretrained on a 10 GB segmented corpus (which is mentioned in Section 5.3); AraFastQA-base, in contrast, was pretrained on the 112 GB segmented AraFast corpus (which is mentioned in Section 5.2). Both models were developed with enhancements to the masking language modeling objective function, building on AraBERT [61] and other transformers based on BERT versions [19]. Departing from predictions of a single token in a sentence, the proposed models predict spans (continuous tokens) within a sentence. Additionally, the models employ a dynamic masking strategy inspired by RoBERTa [62], as opposed to the static masking strategy used in BERT. The use of this novel approach aims to improve the models' ability to capture context and dependencies within continuous segments of text.  
In BERT, token masking occurs before pretraining begins, and it remains constant throughout the training process. This static approach implies that the same set of tokens is consistently masked whenever a particular sentence appears in training instances. In contrast, RoBERTa employs a dynamic masking strategy, where tokens to be masked are randomly chosen anew in each training epoch. This means that each time a sentence is processed during training, a different set of tokens may be masked. This dynamic approach allows the model to learn from a diverse range of masked patterns, potentially leading  
Figure 4: Distribution of corpora sizes.  
to robust learning and generalization. All of the experiments were conducted using the proposed base version of AraFastQA model.  
Table 4 outlines the experimental setup with a maximum sequence length of 512, a training batch size of 256, 2.4 million training steps, and the AdamW optimizer. The optimizer was utilized for training, ensuring efficient and effective optimization of the model parameters. The experimental setup of the two models (AraFastQA mini and base) is identical. In addition, in Table 4, the AraFast mini and base configurations are provided, featuring 110 million parameters, a 100,000-token vocabulary size, 14.2 billion tokens, 534 million sentences, a 12-layer encoder architecture with 12 attention heads and 768 hidden sizes, a \(1\times 10^{-4}\) learning rate, span and dynamic masked language modeling tasks, and a WordPiece tokenizer.  
The findings of our experiments were used to address both RQ2, \"Does pretraining a transformer model on a large Arabic corpus that includes web-scraped data lead to worse performance than training without web-scraped data and noise?\", and RQ3, \"Does pretraining a transformer model on a segmented Arabic corpus lead to better performance than training the model on an unsegmented Arabic corpus?\". Further details about the experiments are provided below.