# Social Media Hate Speech Detection Using Explainable Artificial Intelligence (XAI)
## 2 Materials and Methods
### Classification Methods and Explainable Techniques
#### 2.9.2 BERT (Bidirectional Encoder Representation from Transformers)  
The BERT model is a relatively new language model that was presented in a paper by Google in 2018 [31]. This model has presented state-of-the-art results in natural language processing. The key feature of BERT is the bidirectionality of the model. The BERT model makes use of the encoder component of the transformer to furnish the representation of words. BERT is used for the creation of language representation models that can serve various purposes.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer Type** & **Output Shape** & **\# of Parameters** \\ \hline Embedding & (None, none, 128) & 3,840,000 \\ \hline LSTM 1 & (None, none, 128) & 131,584 \\ \hline LSTM 2 & (None, 128) & 131,584 \\ \hline Dense & (None, 6) & 774 \\ \hline \hline \end{tabular}
\end{table}
Table 4: LSTM model on the Google Jigsaw dataset.  
BERT has a base layer of \"knowledge\" that is derived from its pretraining. From this base layer of \"knowledge\", BERT can further be trained to adapt to specifications provided. BERT's transformer processes any given word with respect to the word's relation to all other words in that particular sentence. This enables BERT to understand the context of the word after looking at all surrounding words, unlike other models that understand the meaning of a word in one dimension only. There is another BERT variant that was trained on specifically hate speech detection task called AngryBERT [32]. AngryBERT jointly learns hate speech detection with emotion classification. It can outperform standard BERT in some hate speech tasks. However, the objective of this research was to detect hate speech along with explainable AI to evaluate how explainable the current high-performing black-box algorithms can be. Therefore, standard BERT was applied rather than AngryBERT so as to not only learn the hate speech pattern using the standard BERT variant but also consider the cases where correctly identifying hate speech is difficult for machines (e.g., sarcasm), enabling the recipient of the explanations to make better decisions. BERT uses the following two semi-supervised models for pretraining [33]:  
1. Masked language model (MLM): In this task, BERT learns a featured representation for each of the words present in the vocabulary. About 85% of the words are used for training, and the remainder are used for evaluation. The selection of the training and evaluation sets is random and in iterations. Through this process, the model learns featured representation in a bidirectional way i.e., learns both the left and the right contexts of the words. In this task, some of the tokens from each sequence are replaced with the token [Mask]. The model is trained to predict these tokens using other tokens from sequence.
2. Next sentence prediction (NSP): In this task, [PERSON] learns the relationship between two different sentences. This task contributes to aspects such as question answering. The model is trained to predict the next sentence. It is similar to the textual entailment task where there are two sentences; it is a binary classification task to predict whether the second sentence succeeds the first sentence.