# Detecting Hateful and Offensive Speech in Arabic Social Media Using Transfer Learning
## 4 Results and Discussion  
Based on the fine-tuning recommendations described in Table 2, we proceeded to train our candidate models on our dataset. Table 3 provides an overview on the performance of these models. \(\text{BERT}_{\text{EN}}\) outperformed all the other candidate models with an Accuracy of 98%, a Precision of 98%, a Recall of 98%, and an F1-Score of 98%. AraBERT was a close runner-up with an Accuracy of 96%, a Precision of 95%, a Recall of 96%, and an F1-Score of 95%, mBERT\({}_{\text{AR}}\) achieved an Accuracy of 83%, a Precision of 84%, a Recall, of 82%, and an F1-Score of 83%. mBERT\({}_{\text{EN}}\) achieved an Accuracy of 81%, a Precision of 82%, a Recall of 80% and an F1-Score of 81%. The results proved that the Precision and the Recall of the BERT-based models are almost alike. This means that these models are not as biased as the baseline models, such as LSTM [43] and LinearSVC [44], and perform equally well for both the positive and the negative comments. mBERT is versatile and generic at the cost of performance. Moreover, mBERT\({}_{\text{AR}}\) performs better than the mBERT\({}_{\text{EN}}\). AraBERT performs well in this case as the source data is quite similar to the target data. As BERT was trained on a relatively larger language corpus when compared to that of AraBERT, the dominance of \(\text{BERT}_{\text{EN}}\) validates that the volume of the language corpora for pretraining plays a vital role in a model's performance. As the \(\text{BERT}_{\text{EN}}\) and AraBERT were clear winners, with very close values in their confusion matrices as depicted in Figure 5, we conducted a more granular analysis on these two models to understand what really made the difference between these two models. A closer look at some of the comments that were wrongly predicted by AraBERT and correctly by \(\text{BERT}_{\text{EN}}\) made us realize that these comments had sarcasm in them. This assumption was verified after annotating all the comments of the dataset as to having sarcasm or not. Figure 5: The confusion matrices of \(\text{BERT}_{\text{EN}}\) (**a**) and AraBERT (**b**). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Algorithm** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline \(\text{BERT}_{\text{EN}}\) & 0.98 & 0.98 & 0.98 & 0.98 \\ AraBERT & 0.96 & 0.95 & 0.96 & 0.95 \\ LinearSVC [44] & 0.9 & 0.89 & 0.76 & 0.81 \\ mBERT\({}_{\text{AR}}\) & 0.83 & 0.84 & 0.82 & 0.83 \\ LSTM [43] & 0.82 & 0.92 & 0.74 & 0.82 \\ mBERT\({}_{\text{EN}}\) & 0.81 & 0.82 & 0.80 & 0.81 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Classification Performance Scores. Again, the difference in the size of the pre-training datasets of BERT and AraBERT has demonstrated their important roles in performance, especially in detecting the eventual presence of sarcasm. Taking a deeper look at why BERT-based models outperformed other language models such as LSTM [43] and LinearSVC [44], in the past, conventional language models could only interpret text input sequentially -- either from right to left or from left to right -- but not simultaneously. BERT is unique since it can simultaneously read in both directions. Bidirectionality is the name for this capacity, which the invention of Transformers [31] made possible. BERT is pre-trained on two distinct but related NLP tasks--Masked Language Modeling and Next Sentence Prediction--using this bidirectional capacity. The Masked Language Model (MLM) training's goal is to conceal a word in a phrase and then have the computer anticipate the hidden word based on the context of the concealed word. The goal of Next Sentence Prediction training is to make the algorithm determine if two provided phrases relate logically and sequentially or whether their relationship is just random [4]. This is the core reason why BERT-based models are very powerful when compared to other conventional unidirectional language models. Another reason behind the power of BERT based models when compared to other conventional models, is again, their huge pre-training corpus.