# Detecting Hateful and Offensive Speech in Arabic Social Media Using Transfer Learning
## 3 Materials and Methods
### BERT Based Models  
#### 3.1.1 BERT  
BERT, or Bidirectional Encoder Representations from Transformers, is pre-trained to provide word representations based on context learnt from both sides. It is based on the Transformer architecture [31], from which, it only takes the encoder side. According to [PERSON]'s description [32], BERT is presented in two versions:  
* BERTBASE: with 12 encoder layers, 768 hidden unit feedforward layers, and 12 attention heads;
* BERTLARGE: with 24 encoder layers, 1024 hidden unit feedforward layers, and 16 attention heads. Both of these versions are larger than the default version suggested in the original paper [31], which consists of 6 encoder layers, 512 hidden unit feedforward layers, and 8 attention heads. Figure 1 displays the encoder stacks in both BERTBASE and BERTLARGE. Each encoder can be broken down into two sublayers. The first layer is the self-attention layer whereas the second layer is the feed forward neural network layer. The self-attention layer allows the encoder to investigate other words from a sentence while encoding a specific word. This can be done in different steps. The first step is to multiply the embedding of the word \(x_{i}\) by three weight matrices \(W^{Q}\), \(W^{K}\), and \(W^{V}\), thus extracting three vectors: the query vector \(q_{i}\), the key vector \(k_{i}\), and the value vector \(v_{i}\), respectively. These vectors will be used, in the second step, to calculate the attention score of the processed word. This score evaluates the focus that needs to be placed on the other words of the sentence while processing the current word. This score is the dot product of the processed word's query vector \(q_{i}\) and the key vector \(k_{j}\) of each of the other words of the sentence. The third step is the division by the square root of the key vector's dimension \(\sqrt{d_{k}}\) to keep the gradients stable. The scores will be normalized in the fourth step using a softmax function. The fifth step is to multiply the softmax score by the value vector \(v_{i}\) so we can keep the needed words while discarding the irrelevant ones. These weighted value scores will be summarized in the sixth step to produce the output of the self-attention layer for the processed word. For better performance, the abovementioned steps are applied on matrices instead of individual vectors. For instance, the embedded sentences will be collected in a matrix \(X\) that will be multiplied by the weight matrices \(W^{Q}\), \(W^{K}\), and \(W^{V}\) to produce the query \(Q\), key \(K\), and value \(V\) matrices. The output of the self-attention layer \(Z\) will be calculated according to the following equation:  
\[Z=SOFTMAX(\frac{Q.K^{T}}{\sqrt{d_{k}}}).V \tag{1}\]  
Having multiple attention heads (12 for BERTBASE, and 24 for BERT LARGE) allows for the model to focus on different positions and have multiple representation subspaces with different \(Z\) matrices. As the feedforward layer expects only one \(Z\) matrix, the different matrices from the attention heads are concatenated into one, which will be multiplied by another weight matrix \(W^{O}\). Each encoder sublayer, as described in Figure 2, applies a Layer Norm operation [33] on its output before feeding it to the next sublayer (or layer). This output can be expressed as:  
\[Z=LayerNorm(X+Z) \tag{2}\]  
[PERSON]'s pre-training was based on two tasks going together. In the first task, \"Next Sentence Prediction\" (NSP), two sentences are fed to the model with an embedding \"A\" for the first sentence and \"B\" for the second sentence. The sentence \"B\" is provided as the real next sentence in half of the cases, whereas a random sentence is provided in the other half of the cases. In the second task, \"Masked Language Model\" (MLM), 15% of the tokens are replaced with a [MASK] token in 80% of the cases, and a random token for 10% of the cases, while they remain as they were for the last 10% of the cases. This task is applied right after the tokenization process. Figure 1: Encoder stacks in both BERTBASE and BERT LARGE [32].