# BERT Models for Arabic Text Classification: A Systematic Review
## 4 Discussion
### How Effective Are They for Classifying Arabic Text? In the majority of the 66 comparisons that were performed in the included studies, Arabic BERT models showed superior performance for classifying Arabic text over the other machine-learning models. Even in the few comparisons in which the used BERT model did not perform particularly well, the reason does not seem to be the BERT technique itself, but rather the specific Arabic BERT model that was used, as in most of these comparisons the used model was Multilingual BERT. As was discussed earlier, Multilingual BERT was not designed for Arabic specifically, but rather supports a large number of languages, including Arabic. The superior performance of Arabic BERT models seems to be related to the unique features of the BERT technique, i.e., its ability to be pretrained from the unlabeled text that is enormously available on the web and its ability to predict words conditioned on both their left and right context [3; 5]. This finding is consistent with the many prior studies that found BERT models outperformed other machine learning models in a variety of natural language processing tasks including text classification [70; 71]. This indicates that researchers and practitioners should have BERT among their first options for Arabic text classification tasks. BERT models that were pretrained for the Arabic language specifically, e.g., AraBERT and MARBERT, showed much better performance than those that support multiple languages, i.e., Multilingual BERT and XLM-RoBERTa. An obvious explanation for this is the size of the pretraining corpora that were used to train these models. While Multilingual BERT and XLM-RoBERTa have 7292 and 2869 tokens in their pretraining corpus, respectively, AraBERT, for example, was pretrained with a corpus that has 2.7 billion tokens. This agrees with the results of studies on other languages. [PERSON], et al. [72] and [PERSON], et al. [73], for example, found monolingual BERT models that were designed specifically for Finnish and French outperformed Multilingual BERT on the natural language processing tasks, including text classification. Using Arabic BERT models as feature-extraction approaches seems to yield better results than finetuning these models. In three out of four comparisons between the two approaches (feature extraction vs. finetuning), the feature extraction gave a better performance. However, this finding should be taken with caution considering the very few studies that compared the two approaches. It is also different than the finding of the developer of the original BERT model [3] who compared the two approaches and reported that their performance is comparable, with feature extraction being 0.3 F1 behind finetuning. Arabic BERT models did not show significant differences in performance at multi-class classification compared to binary classification. The highest difference in performance was scored by Multilingual BERT. It had an average F1 score of 0.566 at multi-class classification and an average of 0.715 at binary classification. Most of the other models, however, scored slight differences in performance in the two classification tasks. For example, AraBERT had an average F1 score of 0.715 at multi-class classification compared to an average of 0.794 at binary classification. There is also the case of MARBERT, where the average F1 Score of the multi-class classification (0.724) was slightly higher than the one of binary classification (0.69). This indicates that the BERT technique is suitable for both Arabic multi-class and binary classification. Considering the performance of all nine models, they can be classified into low-performing models (the ones with an average F1 Score < 0.7) and high-performing models (the ones with an average F1 Score \(\geq\) 0.7). The low-performing models include Multilingual BERT, XLM-RoBERTa, Arabic ALBERT, and GigaBERT.