# Abusive Content Detection in Arabic Tweets Using Multi-Task Learning and Transformer-Based Models
## 3 Materials and Methods
### The Proposed Framework for Automatic Abusive Content Detection in the Arabic Language  
The proposed framework in Figure 3 for automatic abusive content detection in Arabic is designed by adopting the text classification pipeline [38]. It is represented as three main steps. The first involves the data collection and annotation. The second involves the feature selection and model training. The third step involves the model validation. In addition, the bottom part of the list refers to the implementation environments (see Section 4) that were used for data retrieval, processing, model development, and validation training. Figure 3: The framework for automatic abusive content detection in the Arabic language. Figure 2: The most frequent terms in Arabic tweets from the selected dataset: (**a**) a description of the word cloud; (**b**) the relative frequencies with document segments. In the first step, we selected the Arabic dataset that is stored in the GitHub repository [39]. More specifically, we selected all the attributes within the indication labels, where each attribute represented a multi-class classification task, except the directness attribute which used the binary classification task. The number of instances for each attribute and its indication labels are listed in Table 2.  
The second step involved the feature selection and model training level, whereby three main approaches were used (**ML**, **DL**, and **pretrained Arabic LMs**). Each approach with selected features and models is presented as follows. For the **ML** approach, two models were used (SVM and NB) as they were found to be efficient in previous studies [21; 40]. For the feature representation, we used term frequency-inverse document frequency (TF-IDF) [41] which performed well when combined with classification models [42; 43]. For the **DL** approach, two models were used (the LSTM and the CNN) for hate speech detection studies in the Arabic language [8; 22] as they showed the best performance compared to the other models. The LSTM is a special kind of RNN, capable of learning long-term dependencies. LSTMs are explicitly designed to avoid the long-term dependency problem that was encountered with the standard RNN. CNNs are known for their good performance in image analysis, but recent advancements show that CNNs perform well on text data as they extract important features from input feature vectors that help with any downstream task. For the representation of features, we used word embeddings, which are dense vector representations of words. More specifically, we utilized the Arabic word embedding model called AraVec [23]. For the **pretrained Arabic LMs**, we used the fine-tuning strategy for the text classification task [44]. The pretrained language models in Arabic were mainly based on a stacked BERT [10] model. BERT has two models: (1) the BERTBASE (12 encoders with 12 bidirectional self-attention heads) and (2) the BERT LARGE (24 encoders with 24 bidirectional self-attention heads). The original BERT was trained on 3.3 B words extracted from English Wikipedia and the book Corpus [45]. The BERT base configuration has 12 encoder blocks, 768 hidden dimensions, 12 attention heads, 512 maximum sequence lengths, and a total of \(\sim\)110 M parameters. Several researchers have released and benchmarked pretrained Arabic LMs; some of the main differences among these models are the genre and the amount of Arabic data that they are trained on. For example, AraBERT [15] was trained only on MSA, while AraBERTv0.2 [15] was trained on DA and MSA. MARBERT [11], ArabicBERT [12], and QARiB [14] used DA during training. CAMeLBERT-mix [13], used a combination of all types of Arabic text for training. We summarized the recent pretrained Arabic LMs in Table 3, and we focused on the models that trained on DA in order to fine-tune these models in our selected dataset. In addition, the low number of samples in the selected dataset led to a low resource scenario problem.