## 4 Methodology
### Preliminary Processing  
After the data collection and scraping processes were complete, the data for processing were then ready for preprocessing. This process involved filtering and tagging the data, which were primarily HTML pages with various Arabic encodings. For example, a Python verion 3.8.3 script was used to convert articles from the HTML format to plain text for the Watan and Khaleej news corpus [6; 21]. Meanwhile, certain files that contained unreadable symbols required decoding to UTF-8, which made the text accessible on web pages and to the Arabic model. It was important to decode all collected text to a unified Unicode standard format. Subsequently, the data were converted to follow a standardized encoding suitable for further analysis using Python scripts. In the case of Wikipedia text, the tags and titles were removed, leaving only the contents of the articles, which were then saved in plain text format. The WikiExtractor verion 3.3 tool was used for Arabic Wikipedia articles. Numerous corpora, especially Arabic news articles, were downloaded in CSV format. A Python script was then used to extract the articles and save them in plain text format. For large corpora such as common crawl (CC) [40], the BigTextFileSplitter verion 3.8.2 ([[https://www.withdata.com/big-text-file-splitter/](https://www.withdata.com/big-text-file-splitter/)]([https://www.withdata.com/big-text-file-splitter/](https://www.withdata.com/big-text-file-splitter/)) accessed on 2 April 2023) tool was used to split such large corpora. Furthermore, some texts, such as the Tashkeela corpora [22], contained emojis and diacritics, which were removed using Python scripts. Several corpora, such as the Saudi-NewsNeCorpus, were converted from JSON format to plain text via a Python script. The size of the collected corpora was 833 GB before scanning, filtering, and selecting the relevant data. After removing repeated articles and files, the size was reduced to 756 GB. Then, the process of extracting text and articles from various file types, such as converting EPUB to TXT format, was applied, resulting in about 720 GB of data. After applying preprocessing steps to clean the data, noisy data were identified and removed, reducing the corpus size to roughly 249 GB. Following this stage, the preprocessing step was conducted. For preprocessing in general, Figure 2 illustrates the main preprocessing steps for cleaning and organizing Arabic corpora. The process begins with the collection of corpora, followed by the inspection of files and articles to identify duplicates. If duplicate articles or files are found, they are removed. The next step involves checking for duplicated text within the files. If duplicated text is detected, this text is also removed. Once the data are free of duplicates, unwanted characters, symbols, and links are removed to ensure that the text is clean. Finally, the cleaned text is saved as a separate file in a directory named AraFast corpus, marking the end of the preprocessing stage. Figure 2: Flowchart of the preprocessing stage. In the preprocessing stage, the steps detailed in Algorithms 1 and 2 are followed to prepare and clean the overall corpus for training language models. Algorithm 1 enhances the quality and consistency of Arabic text data by removing redundancies and various forms of noise. It starts by identifying and excluding duplicate entries across different files to maintain dataset integrity. It then performs text cleaning operations, including removing URLs, links, emails, HTML line breaks, markup, repeated characters, extra spaces, and other unwanted elements such as Tatweet, punctuation, noisy characters, and diacritics. It also replaces slashes with dashes and ensures proper spacing around non-Arabic digits, English digits, brackets, and between words and numbers. This thorough cleaning and standardization process improve dataset quality, making it more suitable for training Arabic language models and other NLP tasks.