## 5 Training an Arabic Transformer Model Using the AraFast Corpus
### Results and Discussion: Training an Arabic Transformer Model  
In the following section, the key findings of the four experiments conducted using the AraFast corpus are discussed. To illustrate the general behavior of the transformer model, seven representative steps of the training process with extraction of their loss values were selected for all four versions of the AraFast corpus. Choosing these specific steps allowed for a focused analysis of the model's performance and the impact of different corpus configurations on the training outcomes. Figure 10: Training loss of the mini model with segmentation. Figure 9: Farasa sample for the Arabic corpus used. In Table 5 and Figure 11, the training loss values for the different versions of the proposed model are presented, namely, the mini model, the base model with segmentation (seg. ), the base model without segmentation, and the base model trained with noise, which included data obtained following web scraping. The mini version exhibited a consistent and steady decrease in loss as the number of steps increased, which suggests that effective learning and generalization took place. The lowest loss value was seen at 2 M steps, indicating that longer training leads to better performance with this model. The optimal value of loss should be between 0 and 1, and should be less than the value at the start of training. As the loss value decreases, the model's learning improves. For instance, the loss values of the mini and base models with segmentation, as shown in Table 5, dramatically decreased as the model was trained on a high-quality corpus excluding noisy data. The base model with segmentation also showed a decreasing trend in loss, with significant improvements as training progressed. Segmentation contributed positively to the learning process, as evidenced by the low loss values compared to the base model without segmentation. In contrast, the base model without segmentation initially paralleled the base model with segmentation in terms of loss reduction. However, there was a dramatic increase in loss after 500 K steps, peaking at 1.5 M steps. This finding indicates that the problem of exploding loss was due to the representation of noisy data. Finally, the loss values for the base model with noise were significantly higher than those of the other models, particularly in the initial stages. A decrease was observed after 100 K steps, followed by irregular increases and decreases. The high loss values may have been due to the model struggling to generalize the noisy data, although performance seemed to improve slightly over time before rising again after 1 M steps. The above findings indicate that the type of data used was not suitable for training, as it contained a considerable amount of noise and meaningless data. In sum, the training loss values of the various transformer models with Arabic text confirmed the significance of several NLP and DL factors. First, the results suggest that training a model on a large text corpus leads to better performance, as the mini and base  
Figure 11: Loss curves when training the model using the different versions of AraFast. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Step Number** & **Mini** & **Base with Seg.** & **Base without Seg.** & **Base with Noise** \\ \hline
10 K & 2.7 & 4.6 & 4.68 & 300 \\
50 K & 0.91 & 1.22 & 2.3 & 231 \\
100 K & 0.83 & 0.55 & 1.46 & 6.47 \\
500 K & 0.77 & 0.28 & 0.9 & 1.17 \\
1 M & 0.7 & 0.2 & 17.3 & 890 \\
1.5 M & 0.43 & 0.15 & 23.07 & 760 \\
2 M & 0.24 & 0.1 & 20.9 & 870 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Training loss values for the AraFast QA experimental models. models demonstrated continuous improvements in loss reduction with large corpus sizes. This finding further implies that a large corpus can provide diverse examples from which a model can learn, enhancing its ability to generalize and understand the nuances of the Arabic language.