# Abusive Content Detection in Arabic Tweets Using Multi-Task Learning and Transformer-Based Models
## 2 Literature Review
### Automatic Abusive Content Detection in the Arabic Language  
Researchers' interest in automatic abusive content detection on social media has recently grown. Whether traditional ML or DL techniques are used, the majority of research in the literature has essentially modeled the issue as a supervised classification task [8; 19; 20; 21; 22]. Early attempts at tackling this problem involved term frequency weighting which was used to extract n-gram features and then fed to naive Bayes (NB) and support vector machine (SVM) classifiers, according to [PERSON] et al. [21]. The results indicate the outperformance of NB over SVM by achieving F1 scores of 89.6 for binary classification and 74.4 for ternary classification. [PERSON] et al. [19] examined the combined effect of the gated recurrent unit (GRU) and SVM with AraVec embeddings [23], and the best results achieved 79% accuracy. [PERSON] et al. [22] employed the SVM model as a baseline for comparing the LSTM, CNN + LSTM, GRU, and CNN + GRU deep learning models. They concluded that the performance of detection improved with 72% precision, 75% recall, and a 73% F1 score when the CNN was added as a layer to the LSTM. According to [PERSON] et al. (2018), CNN models outperformed the CNN-LSTM and achieved an accuracy score of 81%. When a binary classification task was used on the ArHS dataset, CNN-BiLSTM counterparts identified hate speech with greater accuracy. In recent years, there has been some competition among workshops and conferences that specialize in Arabic language processing. Among those competitions, there was a shared task for offensive language and hate speech detection. For instance, the shared task of offensive language identification was addressed in the 4 th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT4) (2018). There were two subtasks: Subtask A focused on detecting offensive language and Subtask B focused on detecting hate speech. Most teams used many pre-processing steps, usually via character normalization, punctuation removal, diacritics, the repetition of letters, and tokens that were not Arabic. The teams used deep neural network (DNN) algorithms (such as a CNN and an RNN, including a LSTM, a BiLSTM, and GRUs with and without attention), as well as classic ML techniques (such as the SVM and logistic regression (LR)) and fine-tuning contextual embeddings (such as BERT and AraBERT) (2018). The highest-ranking submissions incorporated a variety of learning techniques (including standard ML) and deep neural networks (DNN) to reach F1 scores of 90.5 ([PERSON] et al. (2018)) and 95.2 ([PERSON] (2018)) for Subtasks A and B. For the MTL approach, some studies employed this approach and achieved significantly better performance. For instance, the use of pretrained Arabic BERT was proposed by [PERSON] et al. (2018) for the appropriate classification of various tweets. They used MTL to enhance the AraBERT model, such that it can efficiently learn both tasks together, even in the presence of little per-task labeled data. With macro-F1 scores of 90.0 for Subtask A and 83.41 for Subtask B, their results ranked second in both Subtasks A and B of OSACT4. Another study conducted by [PERSON] and [PERSON] (2018) employed CNN-BiLSTM-based architecture and used MTL to detect hate speech and offensive language. The MTL model achieved a macro-F1 score of 0.76 for the hate speech task and 0.87 for the offensive language task. More recently, the multi-corpus-based learning strategy is explored by [PERSON] et al. (2018) and is built upon MARBERT (2018). For Arabic offensive and hate speech identification tasks, the created MTL model outperformed existing models in the literature on three of four datasets. [PERSON] and [PERSON] (2018) proposed an ensemble of models employing multitask learning and a self-consistency correction method. The proposed model (called AraHS) was found to outperform the baseline models. AraHS is an ensemble of [PERSON] (2018) model trained with different hyperparameters using MTL.