# BERT Models for Arabic Text Classification: A Systematic Review
## 5 Implications for Future Research  
The outcomes of this systematic review have opened up new research areas for further improvements and future work. First, the nine models that were identified in this review are general-purpose BERT models that were pretrained on data from Wikipedia, Common Crawl, and social media posts. Therefore, it is most likely that the high performance that they showed will not be achieved when they are applied in specific domains such as medical, financial law, and industry. [PERSON], et al. [80], for example, found that applying the original English BERT model to biomedical corpora did not yield satisfactory results due to a word-distribution shift from general domain corpora to biomedical corpora. They developed a new BERT model that they called BioBERT. The model was pretrained on large-scale biomedical corpora and therefore was able to outperform the original BERT model in a variety of biomedical natural language processing tasks. It would be interesting to investigate domain-specific Arabic BERT models, if any, evaluate them on text classification tasks, and explore the different domains in which more models are needed.  
It would be also interesting to investigate the impact of the quality of pretraining text on the performance of the pretrained model. As explained in the previous section, a large pretraining corpus does not always lead to a better performance. Previous studies also found that improving performance by relying on more pretraining data is very expensive because of the diminishing returns of such an approach [81; 82]. Pretraining BERT models costs substantial time and money. The Arabic language is a very rich language that contains millions of words [83]. Therefore, it seems that careful selection of the type and amount of pretraining data would have a substantial impact on the performance of Arabic BERT models. More studies in this area are needed.  
In addition, the study showed that using Arabic BERT models as a feature extraction approach mostly yields better results than finetuning. However, two issues should be noted regarding this finding. First, only few studies in this review compared the two approaches. Second, the finding does not agree with previous studies in other languages such as English and Chinese that compared the two approaches [3; 84]. Therefore, more studies in this regard should be conducted to confirm this finding and also to explore when and how each approach would perform the best.  
Furthermore, this review shows the need to develop more labeled datasets for Arabic natural language processing tasks. There is a relatively small number of Arabic datasets available for this type of task compared to English datasets. The majority of the available Arabic datasets, as found in this review, contain tweets only. According to [PERSON], [PERSON], [PERSON], [PERSON], [PERSON], [PERSON], [PERSON] and [PERSON] [71], the availability of labeled datasets for natural language processing tasks is a main driving force for the rapid advancement in this research field.  
Lastly, none of the reviewed studies provided insights into how the morphological, syntactic, and orthographic features of Arabic versus English would influence the performance of BERT models. As these features differ greatly between the two languages [85; 86], more research about their impact on the performance of BERT models is needed.