# Exploring the Role of Emotions in Arabic Rumor Detection in Social Media
## 2 Background
### Pre-Trained Language Models  
The field of NLP reached a milestone in language representation models with the emergence of pre-trained language models (PLMs). These language models are trained based on neural networks holding vast data in textual form and fine-tuned on downstream NLP tasks. The advancement in transformer architecture has been a key enabler for PLMs, including BERT [20], OpenAP GPT [21], RoBERTa [22], and XLNet [23]. Transformers are composed of multiple encoder and decoder components, as well as multi-head attention mechanisms, which contribute to their effectiveness in various NLP tasks. The attention mechanism allows for selectively focusing on pertinent information while disregarding irrelevant information. The fundamental principle underlying PLMs is using pre-trained models and then applying fine-tuning for tasks rather than training models from scratch. These models have been proposed to cope with the issue of polysemy in static word embeddings (i.e., word2 vec and GloVe) by considering the context in which the word occurs (referred to as contextualized word embeddings). BERT was released in 2018 by Google researchers [PERSON] and colleagues. It is now widely accepted as the preliminary step for research on NLP. The model was trained with two tasks. First, masked language modeling allows the representation to integrate the right and left contexts, thus enabling it to learn the context in contrast to previous language representation models that only captured context in one direction. Second, it was trained on next-sentence prediction, which simultaneously pre-train's text-pair representation. BERT performs remarkably well across a range of natural language processing tasks, including text categorization, thanks to this mix of features. It includes an encoder with 12 transformer blocks and a hidden size of 768 with 12 self-attention heads. BERT provides two predominant methods: feature-based and fine-tuning [20]. The feature-based approach extracts pre-training embedding vectors at various levels (word, phrase, or paragraph), which are subsequently supplied as extra features to a given model [24]. The second method, fine-tuning, entails training pre-trained parameters on the desired task. The fine-tuning method is simple in BERT since the self-attention feature enables it to represent several classification problems by switching appropriate inputs and outputs [20]. The most recent research demonstrates that fine-tuning outperforms the feature-based method on text classification tasks [25]. Models are initialized for diverse downstream tasks using the same pre-trained model parameters, and all parameters are adjusted during fine-tuning. Every single instance has the special symbol [CLS] before it and [SEP] as a separate separator token. Fortunately, a multilingual version of BERT was released for the top 104 languages, including Arabic [20]. In addition, the more recently developed XLM-RoBERTa model provides a framework containing languages, including Arabic, trained on a large corpus of text at 2.5 TB [26]. However, according to [27], the multilingual versions are not as effective compared to the English version due to the lack of language resources. Several pre-training monolingual models in Arabic have been introduced to address this issue of limited performance, including AraBERT and MARBERT. The outstanding performance of monolingual models may be determined by the amount of pre-training corpora that were utilized to train these models. AraBERT, for instance, was pre-trained with a corpus that comprises 2.7 billion tokens, in contrast to Multilingual BERT and XLM-RoBERTa, which had pre-training corpora with 110,000 tokens and 250,000 tokens, respectively. AraBERT and AraBERT-Twitter: The AraBERT model was trained specifically for the Arabic language on a sizable corpus in Modern Standard Arabic (MSA) news, which consisted of 70 million sentences and around 3 billion words. The aim was to achieve, in the Arabic language, what the BERT model had accomplished in the English language.