# Text Classification of Patient Experience Comments in Saudi Dialect Using Deep Learning Techniques
## 3 Proposed Methodology
### Trained Models
#### 3.3.3 BERT-Based Model  
BERT is a contextual language representation model. It consists of the encoder part from the transformer architecture. The BERT implementation relies on two steps: the first is pre-training, which is the process of training the model on an unlabeled data set to grasp the context of the texts. The second part is fine tuning, which is the process of training the model with labeled data for a specific task [15]--in our case, PX multi-label classification.  
To capture language patterns, BERT must be pre-trained on a huge data set of the same language as the labeled data set that will be used to fine tune the model for a specific task. We utilized AraBERT [46], MarBERT [61], and Qarib [62], which are BERT models pre-trained on huge data sets of the Arabic language with both MSA and DA text. This signifies the ability of these contextual models to deal with Arabic text written in both forms. We fine-tuned these models on our pre-processed data sets to investigate the impact of using BERT-based models on the performance measures. Figure 7 depicts the fine-tuning process.  
* **Fine-tuned AraBERT:** We fine tuned the AraBERTv02 base model using 80% of the training data and the parameters listed in Table 17. We utilized the AraBERT tokenizer to transform the data into an appropriate format for the fine-tuning process.
* **Fine-tuned MarBERT:** We fine tuned the MarBERT pre-trained model using the parameters listed in Table 17.
* **Fine-tuned Qarib:** We fine tuned the pre-trained Qarib model for the task of multi-label text classification using the parameters listed in Table 17.
* **Fine-tuned PX-BERT:** We built a customized PX_BERT model by pre-training a BERT model using the PX unlabeled data set provided by the PX center at MOH _(PX_BERT _pretraining process described in Section 3.2.4)_. The pre-trained model was then fine tuned using BertForSequenceClassification from the transformers Python library, with the problem type set to multi-label classification, and the remaining parameters were set as detailed in Table 17.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Set of Values** \\ \hline Embedding layer & 300 \\ Bidirectional GRU (activation = tanh) & 32 \\ Dropout & 0.2 \\ Bidirectional GRU (activation = tanh) & 32 \\ Dropout & 0.2 \\ Bidirectional GRU (activation = linear) & 64 \\ Dropout & 0.2 \\ Dense (activation = sigmoid) & 64 \\ Recurrent dropout for all GRU & 0.4 \\ Optimizer & Adam \\ Loss & Binary Cross-entropy \\ Learning Rate & 0.001 \\ Epochs & 10 \\ Batch Size & 128 \\ \hline \hline \end{tabular}
\end{table}
Table 16: The structure of the best BiGRU model found through hyperparameter tuning.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Set of Values** \\ \hline Number of Bidirectional GRU layers & 2, 3, 4, 5, 6, 7, 8 \\ Number of units & 8, 16, 32, 64, 128 \\ Activation function & ReLU, tanh, sigmoid, \\ Recurrent dropout & 0.4 \\ Optimizer & Adam, SGD, RMSprop \\ Dropout & 0.2 \\ Loss & Binary Cross-entropy \\ \hline \hline \end{tabular}
\end{table}
Table 15: Hyperparameter values for the BiGRU model experiments.