# The Detection of Fake News in Arabic Tweets Using Deep Learning
## 3 Methodology
### Tweets Preprocessing
#### 3.3.1 Textual Features Extraction  
Neural network models cannot deal with raw text directly. Instead, the feature extraction method is required to map words from their unstructured forms to fixed-length vectors of continuous real numbers [25]. Various methods of feature extraction exist in natural language processing (NLP). Pre-trained word embeddings have received considerable interest in recent years, displaying superior performance in different text classification domains [24, 26, 27, 28]. The Keras embedding layer and various pre-trained word embeddings are considered in this study. We explored pre-trained models extracting static textual representations, where embeddings are not allowed to update during the training process of the model. The benefit of utilizing this approach for a task-specific model is that it does not need parameter training, and therefore it consumes less memory and time. * Keras Embedding Layer Each word in the Keras embedding layer is represented by an embedding vector of a fixed length that is defined as part of the model, often 100, 200, or 300 dimensions. An embedding vector is learned during training the neural network on a specific NLP task. It requires each word to be represented using one-hot representation and padded with zeros for texts shorter than the maximum length. The embedding layer is initialized with small random numbers and learns the optimal values of the embeddings for all words using the backpropagation algorithm [29]. * Word2 vec Word2 vec is one of the most famous pre-trained word embeddings that is based on continuous bag-of-words (CBOW) or skip-gram (SG) learning models for obtaining vector representations for different words. These models use neural networks to  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Raw Tweet** & **Processed Tweet** \\ \hline
\begin{tabular}{c} \(a\) \\  
learn word representation but differ in their input and output variables. CBOW learns word vectors by training the model to predict a target word given its context, while in the SG model, the input word is used to predict the surrounding context words [30]. In [31], the word2 vec model is used to release pre-trained word vectors for the Arabic language called AraVec, which has been trained on two different Arabic content sources, Twitter and Wikipedia. In our experiment, we chose the version of AraVec built using Twitter data by applying CBOW for 100 and 300 dimensions with a window size of 3. * FastText This is essentially an extension of the word2 vec model. FastText treats each word as a bag of character n-grams instead of a word whole [29]. A vector representation is generated for each character n-gram, and the sum of these representations gives the final word vector. This feature enables it to find the vector representation for rare, misspelled, and out-of-vocabulary words [32]. FastText provides pre-trained word vectors for 157 different languages, including Arabic, which have been trained on datasets composed of a mixture of Common Crawl and Wikipedia [8]. These models were learned dense vectors using CBOW with a window size of 5 and character n-grams of length 5. In our experiment, we selected the 100-300-dimension Arabic version of pre-trained FastText embedding. * ARBERT and MARBERT BERT (Bidirectional Encoder Representations from Transformers) is a bidirectional transformer model that was pre-trained on a massive unlabelled English dataset. The bidirectional capability of BERT means that the model can read each word from both the right and left side of a text statement simultaneously; this helps the model to truly and deeply perceive the meaning of the word by all of its surrounding words [33]. BERT has many pre-trained model variations that are trained on the source dataset with a different domain or language. These models can be used either to extract numerical representations of text data or to fine-tune these models on a diverse range of NLP tasks that may not be relevant to the task the model was trained on to provide state-of-the-art predictions [32].