## 5 Training an Arabic Transformer Model Using the AraFast Corpus
### Web-Scraped AraFast Corpus  
In the NLP domain, training language models on Arabic text, particularly data acquired through web scraping, involves significant challenges related to the nature and cleanness of data. Arabic, characterized by its rich morphology and complex script, presents unique challenges in NLP. Factors such as the use of dialects, the absence of diacritics, and variations in encoding can lead to inconsistencies in the text data, impacting training stability. The following subsection is a response to RQ2, \"Does pretraining a transformer model on a large Arabic corpus that includes web-scraped data lead to worse performance than training without web-scraped data and noise?\". Therefore, to investigate the efficiency of the Arabic web-scraping corpus, a proposed transformer model was trained on a dataset of Arabic text obtained through web scraping, specifically common crawl (CC) data [40]. The size of the Arabic corpus containing CC data was roughly 700 GB. Notably, when training the proposed model on a larger corpus of CC data, an exploding loss curve was observed, indicating that the model was unable to learn. Figure 5 shows the exploding gradients for the model, particularly the epochs where the training data included high levels of noise. This was evident in the sudden spikes in the loss curve. The loss value of training the model on Arabic text with noise was roughly 131.3 at step 700 K. This value indicates that the model could not correctly predict the masked tokens. \begin{table}
\begin{tabular}{l l} \hline \hline
**Argument** & **Value/Method** \\ \hline Maximum sequence length & 512 \\ \hline Training batch size & 256 \\ \hline Training steps & 2.4 M \\ \hline Optimizer & AdamW \\ \hline Parameters’ number & 110 M \\ \hline Vocabulary size & 100 K \\ \hline Number of tokens/sentences & 14.2 B/534 M \\ \hline Encoder architecture & 12 layers, 12 attention heads, and 768 hidden sizes \\ \hline Learning Rate & \(1\times 10^{-4}\) \\ \hline Pre-training task & Span and dynamic masked language modeling \\ \hline Tokenizer & WordPiece \\ \hline \hline \end{tabular}
\end{table} 
Table 4: AraFast mini and base configuration with their experimental setup. To investigate the reasons for the noise, the Arabic corpus was examined and cleaned several times. It was found that the main problem was in the method of scraping data from the web. The size of the CC data was about 457 GB, and many issues affecting this data could not be fixed. For example, many words were not meaningful due to repeated letters and concatenated words, for example, \", which means \"ha, aaaaa, kkkkk, wedesdaythursdaythursdaythur\" in English abbreviations such as #, which means \"peace be upon him\" in English and short letters. The high frequency of such words led to the created vocabulary list of 700 GB in size being unclean, featuring a lot of meaningless tokens (Figure 6). All of these tokens were found in the original files of text, even though they were preprocessed many times, indicating that there was a problem in the process of extracting the data from the web. In tokenization, the ## symbol indicates that a token is a continuation of a word that started with a previous token. An example of concatenating words is \", which equivalent to \"communityhelpEidyouobepprint\" in English.To overcome this and obtain a clean and meaningful corpus, the CC files were removed, as they were disqualified for the Arabic language models. The exploding gradient phenomenon during the training phase of the language model was a prominent issue encountered due to the noisy nature of web-scraped data [63]. The process of training language models on Arabic text sourced from the web entailed handling various forms of noise, such as inconsistent spellings, colloquial language variations, and a variety of dialects. Noise can worsen the problem of exploding gradients, a situation where the gradients of the loss value of a model become excessively large, impeding the training and learning process [64].