# Abusive Content Detection in Arabic Tweets Using Multi-Task Learning and Transformer-Based Models
###### Abstract
The developed framework used different models, including machine learning (ML), deep learning (DL), and Arabic pretrained language models (LMs), within a multi-aspect annotation dataset. Recently, there has been huge interest in adapting NLP techniques, together with ML and DL approaches, to address the problem of abusive content at large by developing automatic detection models. Due to the advancements in NLP techniques, for many natural language tasks, specifically deep learning models, transformer-based models have recently shown to be the most successful and extensively used method. Figure 1: A multi-aspect annotation schema within its indication labels. Transformer-based models analyze textual information in parallel and employ self-attention mechanisms to compute attention weights that quantify the influence of each word on another in place of the sequence word dependence architecture of recurrent neural network (RNN) models. Numerous pretrained models have been made available since 2018, including bidirectional encoder representations from transformers (BERTs) [10], which greatly aided the development of numerous NLP applications. More precisely, we intend to employ pretrained LMs in DA and MSA, such as MARBERT [11], ArabicBERT [12], CAMeLBERT [13], QARiB [14], and AraBERTv0.2 [15] (see Section 3.2). In addition, to investigate the impact of the other approaches, such as multi-task learning (MTL), we developed four MTL models built on top of a pretrained DA language model (called MARBERT) and trained on the multi-aspect annotation dataset (see Section 3.3). Our MTL models and pretrained Arabic LMs enhanced the performance compared to the existing DL model in the literature. Detecting abusive content is still a challenging task, not only in the Arabic language. Recently, many studies have been conducted for hate speech and abusive language detection in many languages. BERT models have recently demonstrated outstanding performance in abusive content identification. According to [PERSON] et al. [16] and [PERSON] et al. [17], the BERT model was found to achieve better performance in SemEval-2019 Task6 compared to ML and RNN models. [PERSON] et al. [18] built an offensive language detection system, combining MTL with BERT-based models, which achieved a 91.51% F1 score for English SemEval-2020 Task12. To the best of our knowledge, this study is the first to employ pretrained Arabic LMs in a multi-aspect annotated dataset using different fine-tuning strategies. In addition, no study has investigated the MTL approach with LMs in a multi-aspect annotated dataset. The main contributions in this study can be summarized as follows:  
* We present a framework for automatic abusive content detection in Arabic by using the multi-aspect annotated dataset and applying ML, DL, and pretrained LMs in DA and MSA. Then, we comprehensively evaluate the performance of each approach. * We propose a MTL model that is built upon pretrained LMs in DA (called MARBERT) to investigate its impact on automatic abusive Arabic content detection. * We apply four different neural network (NN) architectures to the MTL model and then comprehensively evaluate the performance of each experiment. The rest of the paper is organized as follows. Section 2 is the literature review which reviews the related work on automatic abusive content detection in Arabic and the available datasets. In Section 3, we present the proposed framework for automatic abusive content detection and the MTL model in detail. Section 4 provides the experimental setup and the performance measurements. The results of the performed experiments are then presented in Section 5. Section 6 presents the conclusion and future prospects.