## 4 Building Arabic Word Embeddings
### Corpus Preprocessing  
Preprocessing is a critical step in preparing a corpus for training. Feeding a model with a corpus containing redundant vocabulary can result in unwanted output. To prevent the development of inadequate models, we implement several preprocessing steps on our corpora. Figure 3 illustrates the main steps of the corpus preprocessing. It includes tokenization, cleaning, normalization, repeated letters removal, and stop words removal. * **Tokenization**: Since the text is considered as a single input block for the preprocessing program, we utilized tokenization as the first step. Tokenization breaks the text into individual tokens (words), allowing for easier processing. To accomplish this, we employed the (tokenize) function, which is a part of the pyyarabic ([[https://pypi.org/project/PyArabic/](https://pypi.org/project/PyArabic/)]([https://pypi.org/project/PyArabic/](https://pypi.org/project/PyArabic/)) (accessed on 28 September 2024)) library. * **Cleaning**: Then, we cleaned the text to remove any unwanted data. This includes removing digits, non-Arabic text, punctuation marks, emojs, and symbols. * **Normalization**: Arabic is a highly complex language due to its richness, which includes representing the same word in various formats. To enhance the robustness of our models, we implemented several normalization steps to prevent word redundancy. * **Tashkeel Removal**: One of the unique features of Arabic is its use of specific marks for each letter in words. These marks represent the required sound for each letter in the word. Their existence affects the representation of the word, as changing one mark can lead the model to perceive the word as a new one. From the model's perspective, every unique input is treated as individual input. To prevent the model from establishing different vectors for each marked version \begin{table}
\begin{tabular}{c l l l} \hline \hline
**Corpus** & **Source** & **Number of Words** & **Number of Vocabulary** \\ \hline Watan & Watan newspaper & 6 M & 35,211 \\ Book & 32 Arabic books & 11 M & 181,432 \\ Wiki & Wikipedia & 111 M & 445,977 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Summery of the corpora. Figure 3: Illustration of the corpus preprocessing steps. of the word, we removed any tashkeel marks. To perform this, we used the (strip_tashkeel) function from the (pyarabic) library. * **Letters Normalization**: We conducted letter normalization to standardize the format of certain Arabic letters. This step is crucial in ensuring that the model treats each letter consistently, preventing redundancy and improving overall performance. * **Tatweel Removal**: Some Arabic writers use (Tatweel) to add a touch of elegance to the text by incorporating extra dashes within the words. We removed it from the text by utilizing the (strip_tatweel) function from the (pyarabic) library. * **Repeated Letters Removal**: One of the challenges that affects our training is the presence of unintentionally repeated letters within the same word. To address this issue, we implemented a step to remove repeated letters within the same word. * **Stop Words Removal**: In the context of the Arabic language, stop words are words that are frequently used but do not carry significant meaning. By removing these stop words, we can reduce the size of the corpus and focus on more meaningful words. We started by adopting the Arabic stop words list provided by NLTK ([[https://www.nltk.org/](https://www.nltk.org/)]([https://www.nltk.org/](https://www.nltk.org/)) (accessed on 28 September 2024)), which contains 754 words. However, we noticed that stop words in Arabic can appear in various forms, such as different tenses, formats, or with the conjunction letter attached directly to them. To address this, we expanded the NLTK Arabic stop words list by adding a conjunction letter to each word and applied normalization to words that can come in different formats. Additionally, we added present and future verb formats to the stop words list. The resulting stop words list contains around 936 words.