# A Deep Learning Approach for Automatic Hate Speech Detection in the Saudi Twittersphere
## 6 Experiments Results
### Results and Discussion
This was not expected, since the BERT model has been proven to be very powerful, achieving state-of-the-art results in many NLP tasks such as sentiment analysis, question-answering, and textual entailment [14]. The poor performance of BERT can be attributed to the fact, as stated by [PERSON] et al. [45], that BERT was trained on a different dataset genre (Wikipedia). In Wikipedia, pages are written in MSA (Modern Standard Arabic), whereas on Twitter, tweets are mostly written in dialectal Arabic. The linguistic variations between MSA and dialectal text might cause BERT to not perform very well on our tweet datasets. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{6}{c}{**GHDS (In-Domain)**} & \multicolumn{6}{c}{**RHSD (Out-Domain)**} \\  & **P** & **R** & **F1** & **A** & **HCR** & **AUROC** & **P** & **R** & **F1** & **A** & **HCR** & **AUROC** \\ \hline \multicolumn{12}{c}{**SVM**} \\ **(char** & 0.74 & 0.74 & 0.74 & 0.78 & 0.63 & 0.85 & 0.70 & 0.66 & 0.66 & 0.68 & 0.46 & 0.73 \\
**n-grams** & & & & & & & & & & & \\
**LR (char** & 0.75 & 0.74 & 0.75 & 0.79 & 0.63 & 0.84 & 0.68 & 0.66 & 0.65 & 0.68 & 0.47 & 0.72 \\
**n-grams** & & & & & & & & & & & & \\ \hline
**CNN** & **0.81** & **0.78** & **0.79** & **0.83** & **0.67** & **0.89** & 0.72 & **0.69** & **0.69** & **0.70** & **0.56** & **0.79** \\
**GRU** & 0.80 & 0.77 & 0.78 & 0.82 & 0.63 & 0.87 & 0.70 & 0.65 & 0.64 & 0.68 & 0.43 & 0.76 \\
**CNN +** & & & & & & & & & & & & \\
**GRU** & 0.80 & 0.76 & 0.77 & 0.82 & 0.62 & 0.88 & **0.74** & 0.67 & 0.67 & **0.70** & 0.44 & 0.77 \\ \hline
**BERT** & 0.76 & 0.76 & 0.76 & 0.80 & 0.65 & 0.76 & 0.63 & 0.60 & 0.59 & 0.63 & 0.38 & 0.6 \\ \hline \hline \end{tabular}  
* We denoted precision as P, recall as R, accuracy as A, and hate class recall as HCR.  
\end{table} 
Table 3: Evaluation results of the experimented models. The best result for each metric is boldfaced *. **Out-domain experiments:** The right part of Table 3 shows the results of the out-domain experiments. The main purpose of these experiments was to assess how well our models can generalize across different datasets. Not surprisingly, the performance of all models, including the baselines, was always lower on the out-domain dataset. While being always lower, results of the out-domain experiments were very consistent with the results obtained in the in-domain experiments. Specifically, CNN in both sets of experiments achieved the best results compared to the baselines and other models in almost all metrics. Moreover, GRU and CNN + GRU showed comparative results. However, in out-domain experiments, BERT failed to beat the baselines with respect to all metrics. There are many reasons that could cause the consistent drop in the classification performance of all experimented methods on the RHSD dataset. Firstly, RHSD focuses on a different type of hate speech, namely, religious hate speech. This type of hate speech was not directly considered while collecting the GHSD dataset (used for training), which focuses on a wider range of hate speech types including racist, ideological, and inter-religious hate speech. In fact, this could cause the two datasets to have different types of swear/abusive/offensive terms and also different target names. This can greatly impact the lexical distribution of the two datasets, hence impacting how/what the models learn as predictive features. To investigate this, we firstly generated two lexicons from our dataset (GHSD) following two corpus statistical-based approaches: chi-square (\(x^{2}\)) and pointwise mutual information (PMI), which leverages the labeled corpus in order to learn a domain-specific lexicon and measures the association strength between a term and a class (hate or non-hate classes, in our case). More details on how \(x^{2}\) and PMI values are calculated can be found in [46]. Then, we examined the lexicons generated from RHSD by [PERSON] et al. [12] and from GHSD and analyzed the terms with high positive scores (strongly associated with hate class).