## 2 Background
### Feature Extraction Techniques
#### 2.2.2 Innovative Techniques  
* **Word Embeddings**: word embeddings are useful for capturing the semantic connections between words [30]. The raw data used to train the network are fed into a low-dimensional dense vector. Following enough training, the lexicon's semantics are learned, and a map is created by grouping words with similar semantic connotations [18; 22]. Moreover, standalone representations that are independent of context are captured via static word embedding. Word and subword embeddings were applied using the word2 vec tool, which provides two models for representation: a continuous bag of words and a continuous skip-gram of words [31]. This application was for the purpose of detecting Arabic satire. Moreover, Arabic FastText was used to extract word embeddings [32]. The deep emojt technique was additionally utilized for the extraction of emotion features [23]. Even though the static word embedding method is effective, especially the embeddings trained on large datasets, it does not account for the meaning of a word in various contexts. Due to the possibility that a word in one domain could have a completely different meaning in another, it did not perform well on domain-specific datasets [33]. Contextual word embedding is, therefore, employed to represent the term in accordance with the context in which it appears [22; 34]. Technologies such as ElMo, for example, have successfully addressed this problem, although they need larger datasets, whereas domain-specific datasets do not. It is important to note that in non-contextual jobs, such as studying vector spaces, utilizing static word embedding is occasionally preferred. The computational cost of static word embedding is also significantly lower than that of contextual word embedding [35]. Contextual word embedding has been utilized in a variety of studies, including [22; 34], to identify Arabic sarcasm.  
Transformers take natural language text as input and generate a prediction for a classification problem. Its architecture uses an encoder-decoder structure, as shown in Figure 1. The encoder takes an input and converts then into a sequence of continuous representations that are fed to a decoder, which generates a prediction.  
According to [PERSON] et al. [36], \"The Transformer is the first transduction model relying purely on self-attention to calculate representations of its input and output without employing sequence-aligned RNNs or convolution\". Using layered self-attention and pointwise, completely linked layers, the transformer conforms with the encoder decoder's overall architectural design. By translating a query and a collection of key-value pairs to an output, the transformer's attention function is calculated. The result is then calculated as a weighted sum of the values, with each value's weight determined by the query's compatibility function with the connected key, where dimension \(dk\) is of the keys, and dimension \(dv\) is of the values. Moreover, the transformer performs the attention function in parallel, resulting in \(d\)_volimensional_ output values using \"multi-head attention\" as shown in Equation (1):  
\[MultiHead(Q,K,V)=Contact(head_{1},...,head_{h})W^{O}\] \[,where\,head_{i}=Attention((QW)_{i}^{Q},(KW)_{i}^{K},(VW)_{i}^{V}) \tag{1}\]  
Transformers ignore convolutional and recursive layers in favor of multiheaded self-attentional layers as their foundation. They take advantage of the attention mechanism in three different ways. First, it is employed in encoder-decoder attention layers, where the preceding decoder layer generates queries and the encoder output consists of memory keys and values. Second, there are six internal layers in every encoder and decoder. Each of them is made up of two sublayers: positionwise completely linked feedforward networks and multihead self-attention. Transformers are also less expensive in terms of time.