# Sentiment Analysis on Algerian Dialect with Transformers
## 4 Classification of Algerian YouTube Comments Using Transformers
### Model Inference  
For the model inference, we made a Colab ( [[https://colab.research.google.com](https://colab.research.google.com)]([https://colab.research.google.com](https://colab.research.google.com)), accessed on 17 September 2023 ) notebook to use the inference API offered by Huggingface on our trained model. Now, let us start off with some examples of positive samples:  
``` \(\textit{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text







The comment's author is definitely upset that this is the last episode of their favorite show, yet the model classified it as positive with 53.3% confidence because the writer certainly likes the show. Here is another challenging sentence: (at last, it will happen to you my brother). Figure 7b shows the inference results. Even though the sentence does not contain any negative words, but does contain a positive word (Dear), the sentence is still seen as negative with 84.4% confidence. ## 5 Experimental Results and Discussions

Being able to set up an environment for training a deep learning model and conduct experimentation on parameters on custom data requires a lot of skills and tools to accomplish. In this section, we will show what tools were used for this work, as well as the working environment. Next, we discuss the settings and fixed parameters used during the experiments. After that, we share our model's performance results with various parameter tweaks and fine-tuning. ### Experiment Setting

During experimentation and hyperparameter fine-tuning, there are a lot of parameters to change and tweak to find the best possible combination to achieve the highest F1-score possible. In this section, we will discuss various training and preprocessing parameters that were fixed during the experiments, which we will reveal in the next section. * Sequence length: The maximum length of a sequence in the input text. We set it to 180 for our dataset, which means we truncate longer sentences into 180 tokens and pad shorter sentences with zeros on the left. More than 93% of the sentences in our dataset have at most 180 tokens. In the next section, we will experiment with this parameter too. * Learning rate: The configurable hyperparameter used to train neural networks during the backpropagation phase. For large models like Transformers, it is often between \(10^{-6}\) and \(10^{-4}\), and may increase or decrease depending on the model and data sizes. We ran the optuna library for more than 100 trials to obtain the best learning rate, and \(1.07\ \times\ 10^{-5}\) was chosen. * Training epochs: An epoch trains the whole training dataset for one cycle. We chose three as our number of training epochs. * Batch size: A hyperparameter that controls the number of samples to feed into the neural network in a single training iteration. Again, optuna chose 16. * Warmup steps: The number of steps at the beginning of the training, where we use a very low learning rate to avoid early overfitting and also to slowly start fine-tuning

Figure 7: Example inference on a challenging sentence:: (**a**) Prediction results of comment 1; (**b**) Prediction results of comment 2.

the weights in the transformers. We have over 5900 total training iterations, and 800 was set as the number of warmup steps. Since we are only fine-tuning BERT, there is not much we can change about the model architecture, as the model was pre-trained on a large amount of text, and we just fine-tuned it on our dataset. ### Results and Analysis

As a first experiment, we tried different weights and architectures, where we trained and evaluated an LSTM model, BERT base, BERT large [6], BERT Arabic mini, BERT Arabic medium, BERT Arabic base, and BERT Arabic large [34]. Table 6 shows the different model versions and their corresponding number of parameters, training time, and evaluation on F1 and accuracy scores on the test set. The large version of Arabic BERT outperforms all other models, with a 0.7838 F1-score. The base version comes in second place, and medium in third.