# BERT Models for Arabic Text Classification: A Systematic Review
## 1 Introduction  
Text classification is a machine-learning task in which a document is assigned to one or more predefined categories according to its content. It is a fundamental task in natural language processing with diverse applications such as sentiment analysis, email routing, offensive language detection, spam filtering, and language identification [1]. Of these applications, sentiment analysis has attracted the most attention. The objective of sentiment analysis is to identify the polarity of text content, which can take the form of a binary positive/negative classification, or a more granular set of categories, such as a five-point satisfaction scale [2]. Despite the progress that has been achieved in improving the performance of text classification, there is much room for improvement, especially for the Arabic language.  
Bidirectional Encoder Representations from Transformers (BERT) is a language representation model that was introduced in 2018 by [PERSON] and his colleagues from Google [3]. Since its introduction, it has become a ubiquitous baseline in natural language-processing research [4]. Unlike other language representation models that capture the context unidirectionally, BERT was designed as a bidirectional model that can predict words conditioned on both the left and right context [5]. BERT was also built as an unsupervised model that can be trained using the plain text corpus that is enormously available on the web in most languages. This combination of features allows BERT to demonstrate exceptional performance in various natural language-processing tasks, including text classification [6].  
There are two main approaches for using BERT: feature extraction and finetuning. In feature extraction, the architecture of the BERT model is preserved, i.e., the model's parameters are 'frozen'. Features are extracted from the pretrained BERT model and then fed into a classifier model to solve a given task. In finetuning, the model's parameters arefinetuned by adding extra layers to the original BERT architecture. These new layers are used to train the model on the downstream tasks [3].  
In the original paper of [PERSON], [PERSON], [PERSON] and [PERSON] [3], two BERT models were introduced: BERT-large and BERT-base. They are both in English. They were pretrained from extremely large corpora extracted from the internet. Therefore, they have heavy computing requirements and a high memory footprint. As BERT gained more interest from researchers and practitioners, more BERT models were introduced. The new models attempt to address some of the shortcomings of the original models, improving the performance [7] or improving the inference speed [8]. There were also models that were developed to support languages other than English [9; 10].  
Several BERT models were pretrained to support the Arabic language. For example, [PERSON] and his team developed a multilingual model that supports more than 100 languages, including Arabic. [PERSON], et al. [11] developed an Arabic model that they called Arabert. The model was pretrained on around 24 gigabytes of text. Similarly, [PERSON], et al. [12] used 1B tweets to train an Arabic BERT model that they named MARBERT. While these models have been used for Arabic text classification, it is not obvious which model is most suitable for the classification task. It is also not clear if one of them is more effective than the other and whether the process that has been used to pretrain each model has affected its performance. This systematic review provides a detailed examination of the different Arabic BERT models that have been used for text classification. The aim is to provide guidance for researchers and practitioners by critically appraising and summarizing existing research. To the best of our knowledge, this is the first systematic review study on this subject.