## 5 Model Compilation Using Deep Learning Methods
### Experimental Setup  
For training the deep learning models, an Adam optimizer with a 0.01 learning rate, weight decay of 0.0005, and 128 batch size was used. A dropout value of 0.5 was used to avoid overfitting and to hasten the learning. The output layer used a softmax activation function. The experiments utilized Python programming Tensorflow and Keras libraries for the machine learning and deep learning models. For hardware, a Windows 10-based machine with a core i7 CPU and 16 GB RAM was used.  
Our experiments included two stages. First, we investigated the state-of-the-art deep learning models for Arabic text, namely CNN, LSTM, BiLSTM, and GRU, using different embedding techniques. The second stage included selecting the models that achieved the highest performance in order to build a hybrid model that added a CNN layer for feature extraction and to assess its impact on the models' performance. Below, we briefly explain the process of building each model.  
1. **CNN** The implemented architecture was, therefore, comprised of the following key pieces (as shown in Figure 5): * Word encoding layer: Each sentence consists of a sequence of words \(s_{i}=\{w_{i1},w_{i2},\dots,w_{ik},\dots,w_{in}\}\), where \(w_{ik}\) denotes the \(k\)th word in the \(i\)th sentence and \(n\) represents the length of the sentence. Each word in the sentence is represented as a vector, known as the word embedding. An embedding matrix is constructed to form the first layer of the model. * Convolutional layer: This layer learns to extract meaningful features and substructures from the text for the overall prediction task at hand. We implemented a convolutional neural network (CNN) with a linear ('relu') activation function, followed by a pooling layer to reduce the output of the convolutional layer by 20%. Next, the output was flattened to one long 2D vector to represent the features extracted by the CNN. Finally, we added dense layers, including scale and rotating by transforming the vector by multiplying the matrix and vector. A dropout of 0.2 was applied to reduce the model's overfitting. Finally, the output layer used a sigmoid activation function.  
2. **LSTM** The architecture is shown in Figure 6; the first layer was the word encoding layer. The next layer was the LSTM layer, with 128 memory units. The activation function was softmax, and categorical cross-entropy was used as the loss function.  
Figure 5: The CNN network architecture.  
3. **BiLSTM** The architecture is shown in Figure 7. The word encoding layer was the first layer of the model, then a bidirectional LSTM layer was added. Three dense layers were then added to scale, rotate and transform the vectors and the softmax activation function was used to generate the final classification.
4. **GRU** In GRU, the embedding layer was the first layer in the network fed into a GRU layer with 128 units, which had a dropout rate of 0.2 to prevent overfitting. A dense layer with a softmax activation function was added to output the classification probabilities for the three classes. The model was compiled with a categorical cross-entropy loss function Adam optimizer, as shown in Figure 8.  
Figure 8: The GRU network architecture.  
Figure 6: The LSTM network architecture.  
Figure 7: The BiLSTM network architecture.  
128 units, which had a dropout rate of 0.2 to prevent overfitting. A dense layer with a softmax activation function was used to generate the final classification, as shown in Figure 11.