# Sentiment Analysis on Algerian Dialect with Transformers
## 4 Classification of Algerian YouTube Comments Using Transformers
### Model Inference
LSTM obtained an F1-score of 0.7399, being fourth in the ranking. Even though the large BERT Arabic model is optimal in terms of F1-score and accuracy, it is significantly large and takes about 2 h to train with the high-performance GPU offered by Google Colab. The original BERT weights obviously underperformed because the model was pre-trained on English text instead of Arabic, and since 45,000 samples are not enough for language understanding, even using the large BERT did not increase the F1-score. The LSTM model has 2 LSTM layers of 128 units, a sequence length set to 180, and an embedding size set to 100. Bi-LSTM uses the same parameters but with bidirectional LSTM cells. The second experiment is tweaking the sequence length parameter explained in this section's previous subsection. Table 7 shows the training of large Arabic BERT using different sequence length values. As shown in Table 7, an sequence length of 180 is optimal. Increasing the parameter will slow down the training as expected, but what is not expected is that F1-score decreases after 180. This is because longer comments tend to have mixed sentiments or be neutral in our dataset, which can be quite hard for the model to determine. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model Version** & **No. of Parameters** & **Training Time** & **F1-Score** & **Accuracy** \\ \hline LSTM & \(\sim\)4 M & 3 min & 0.7399 & 0.7445 \\ Bi-LSTM & \(\sim\)4.3 M & 6 min 35 s & 0.7380 & 0.7437 \\ BERT Base & \(\sim\)109.5 M & 33 min 20 s & 0.6979 & 0.7500 \\ BERT Large & \(\sim\)335.1 M & 1 h 50 min & 0.6976 & 0.7484 \\ BERT Arabic Mini & \(\sim\)11.6 M & 2 min 40 s & 0.7057 & 0.7527 \\ BERT Arabic Medium & \(\sim\)42.1 M & 11 min 25 s & 0.7521 & 0.7860 \\ BERT Arabic Base & \(\sim\)110.6 M & 34 min 19 s & 0.7688 & 0.8002 \\ \hline
**BERT Arabic Large** & **\(\sim\)336.7 M** & **1 h 53 min** & **0.7838** & **0.8174** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Different model weights and architectures along with training time and evaluation metrics. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Sequence Length** & **Training Time** & **F1-Score** & **Accuracy** \\ \hline
140 & 1 h 40 min & 0.7744 & 0.8056 \\
160 & 1 h 46 min & 0.7809 & 0.8114 \\
**180** & **1 h 53 min** & **0.7838** & **0.8174** \\
200 & 2 h & 0.7829 & 0.8143 \\
220 & 2 h 8 min & 0.7799 & 0.8084 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Training time and evaluation metrics when adjusting the sequence length parameter. Our last experiment is trying to determine the best combination of preprocessing tasks to increase the model performance further. In Table 8, the first row shows the F1 and accuracy scores of the model in the testing set without any preprocessing. Each row after that corresponds to one of the following preprocessing tasks:

1. Removing elongation: This task consists of removing redundant letters; words like \"braaaavooooo\" are transformed to \"braavoo\", where 2 redundant characters is the maximum. 2. Replacing URLs, phone numbers, and emails: This task is straightforward; since URLs, phone numbers, and emails do not contribute to the sentiment of the sentence, they are simply replaced by a special token. 3. Removing HTML: This one removes HTML tags from all sentences. 4. Removing Emois: Removing all types of emojis from text. 5. Inserting spaces between emojis: A lot of comments come with multiple emojis that are attached to each other. This task inserts spaces between them to help the WordPiece tokenizer [35] learn these emojis as individual tokens. 6. Converting Latin to Arabic: As mentioned earlier, many Algerians use Latin characters to express their opinions on social media. As a result, we constructed a simple algorithm to convert Arabic words written in Latin characters into Arabic characters. Here are some examples: (a) Rbi y7 dhk \(\rightarrow\) <liag, <y> (May [PERSON] protect you) (b) Ro7 klih \(\rightarrow\) <liag, <z> (Let him)
7. Removing redundant punctuation: Some comments have multiple punctuation marks between two words.