# Abusive Content Detection in Arabic Tweets Using Multi-Task Learning and Transformer-Based Models
## 3 Materials and Methods
### The Proposed MTL Architecture  
Rather than learning each aspect individually, each with its own parameters, we intended to allow the model to share the hidden layer parameters among aspects in order to positively transfer knowledge and leverage information contained in the related aspects. Thus, we treated each aspect as a task in our dataset. MTL is employed in many ML applications to enhance a model's performance and generalizability [48; 49; 50]. MTL is a machine learning technique that involves simultaneously training a model on multiple related tasks in order to improve the performance of all tasks. By training on multiple tasks simultaneously, the model can better learn the shared information between the different tasks, leading to more efficient training and generalization. The advantage of MTL is that multiple tasks can be learned simultaneously, which allows for the transfer of information and the generalization of skills from one task to another. This kind of learning can result in a greater understanding of how tasks relate to each other, as well as the improved performance on each task. Since we are dealing with the hierarchical annotation dataset, we considered MTL to be a very useful strategy. Therefore, we adopted the MTL approach to improve the performance of automatic fine-grained abusive content detection in Arabic using a multi-aspect annotated dataset.  
The proposed MTL architecture is shown in Figure 4, where the bottom part is the MARBERT language model, which will be shared among tasks. We used MARBERT LM as it showed the best performance compared to other LMs. The upper parts were five modules which were used for each sub-task. We trained four MTL models in the five modules which were used for each sub-task. The models included MTL with MARBERT, MTL with LSTM + MARBERT, MTL with LSTM + CNN + MARBERT, and MTL with BiLSTM + CNN + MARBERT. Unlike previous work [5; 29] our paper focuses on improving fine-grained abusive Arabic content detection using MTL with a multi-aspect annotation dataset.  
Figure 4: The architecture of the proposed MTL model.  
The first model is based on MTL with MARBERT in its base feature extractor without any headers. The final prediction head with only one dense layer was used to map the feature vector into final classification targets. This design inspired the effect of the MARBERT-generalized feature extraction ability and was used in a previous study, but with a cross-corpora MTL model [29].  
The second model is based on MTL with MARBERT + LSTM heads; this architecture was used with BERT-based MTL for offensive language detection [18]. The feature extractor model architecture consists of RNNs with LSTM cells followed by a dropout layer to avoid over-fitting and a final classification dense layer.  
The third and fourth models are quite similar, apart from the fact that the LSTM was used in the third model and the BiLSTM was used in the fourth model. The third model is based on MTL with MARBERT + LSTM + CNN; this model's architecture is different from the above designs because the CNN network is applied in the architecture consisting of feature extraction, followed by the RNN head with the LSTM, the dropout layer to avoid model overfitting, the 1D convolution layer, batch normalization, ReLU activation, the dropout layer, followed by the final classification dense layer.  
Our experiment tested the four different MTL architectures to enhance different types of prediction heads, but the MARBERT-based feature extractor is common for each architecture. The input HS tweet is first fed into the shared MARBERT, and then each sub-task module takes the contextualized embeddings generated by MARBERT and produces a probability distribution for its own target labels. The overall loss \(L\) is calculated in Equation (1), where \(I\)= [T\({}_{1}\), T\({}_{2}\), T\({}_{3}\), T\({}_{4}\), T\({}_{5}\)] and \(w_{i}\) is the loss weight for each task-specific cross-entropy loss \(L_{i}\).  
\[L=\sum_{i}^{l}w_{i}L_{i} \tag{1}\]