## 3 Methodology
### Word Embeddings  
The traditional techniques of words representation ignore the word context that degrades the accuracy of sentiment classification systems [51]. Several techniques have been proposed for word representation, such as N-gram, TF-IDF, and word embeddings [27]. Recently, pre-trained vectors (or embeddings) play a vital role in improving the accuracy of machine learning models in several problems related to the NLP [52].  
Word embeddings are used extensively in NLP to capture the semantic relations between sentence's words. Embedding techniques can be classified into static word embeddings (e.g., word2 vec, Glove) and contextual embeddings (e.g., BERT, ELMo). Static  
Figure 1: Preprocessing steps.  
Figure 2: Two Arabic tweets with different emojis.  
word embeddings encode the context of the word into the word vector (embedding), while contextual embedding considers the sequence of all words in the documents to learn sequence-level semantics. Therefore, contextual techniques learn different representations for polysemous words [53]. In this work, we utilized both embedding representations for Arabic sentiment analysis.  
Static word embedding learns a continuous representation for each word, such that each word is associated with exactly one dense vector. This word representation method was originally coined by [PERSON] et al. [54] to build a model that learns a distributed representation for each word along with the probability function for word sequences. The classic model of static word embedding consists of one embedding layer that feeds forward into a neural network that predicts the next word in a sequence. However, it was [PERSON] et al. [55] who brought word embedding to the forefront of deep learning models for NLP, through the creation of the _Word2 vec_ model. In this work, we used the _AraVec_ word embedding technique to represent the words in Arabic tweets [56]. This method provides powerful Arabic pre-trained word embedding models following the Word2 vec approach. AraVec has several word embedding models built from three Arabic content domains (web pages, Wikipedia articles, and Twitter tweets) with two methods, Skip-Gram and Common Bag Of Words (CBOW). We used in this work the Twitter-skip-gram model with 300 dimensions (Twt-SG-300) where each word in a tweet is represented as a 300 dimensional dense vector.  
We also employed contextual embedding for the sentiment analysis of COVID-19 tweets through pre-trained BERT-based transformers. The main advantage of BERT-based models is that they train word embedding based on a bidirectional transformer (or auto-encoder) rather than language model. Bidirectional transformer considers both the previous and next tokens when predicting the token, in contrast to the N-gram language model, which considers only the previous n words. Thus, the bidirectional transformers are able to incorporate contextual information from both directions at the same time. Two versions of BERT have been used in this work, AraBERT [44] and MARBERT [57]. These embeddings retrained the BERT transformer on Arabic texts. More details about these two models will be presented in the following Section.