# BERT Models for Arabic Text Classification: A Systematic Review
## 4 Discussion
### What BERT Models Were Used for Arabic Text Classification, and How Do They Differ? As was shown in the previous section, nine different BERT models were used for Arabic text classification. Some of them were designed specifically for the Arabic language, while others support multiple languages, including Arabic. 1. Multilingual BERT:  
The model was developed by [PERSON] et al., the same developer of the original BERT model. It was not designed for Arabic specifically, but rather supports 104 languages, including Arabic [21]. It was pretrained on Modern Standard Arabic data from Wikipedia. The size of the Arabic pretraining corpus is less than 1.4 gigabytes and has only 7292 tokens [65; 66]. Regarding its architecture, Multilingual BERT has 12 layers of transformers blocks with 768 hidden units each. It also contains 12 self-attention heads and around 110 million trainable parameters [26]. 2. AraBERT:  
Unlike Multilingual BERT, AraBERT was built specifically for the Arabic language. The pretraining dataset contains Modern Standard Arabic news extracted from different Arabic media. Version 1 of the model has 77 million sentences and 2.7 billion tokens corresponding to around 23 gigabytes of text. This is 17 times the size of the Arabic pretraining dataset that was used to train the Multilingual BERT. The newest version of the model used 3.5 times more data for pretraining, i.e., 77 gigabytes of text. Similar to Multilingual BERT, AraBERT has 12 transformers blocks with 768 hidden units each. It also has 12 self-attention heads and a total of 110 million trainable parameters [11]. To support dialectical Arabic, the model was further pretrained on a corpus of 12,000 sentences written in different Arabic dialects. This customized version of AraBERT was called SalamBERT [59]. MARBERT:  
This model was also designed specifically for the Arabic language, but unlike AraBERT, it was pretrained on huge Twitter data which contains text in both Modern Standard Arabic and various Arabic dialects. The pretraining corpus has 1 billion tweets, which sum up to almost 128 gigabytes of text. The number of tokens in the corpora is around 15.6 billion, which is almost double the number of tokens of Version 2 of AraBERT. This makes it the largest pretraining corpora of the nine models. MARBERT has the same architecture as Multilingual BERT, but without the next-sentence prediction (NSP). The reason for omitting NSP, according to the model's developers, is that tweets are too short. The total number of trainable parameters in MARBERT is around 160 million [27]. 4 ArabicBERT:  
The model has several versions that were all trained specifically for Arabic. The base model has the same architecture as Multilingual BERT. It was pretrained on 8.2 billion tokens extracted from Wikipedia and other Arabic resources, which makes up around 95 gigabytes of text. The data are in Modern Standard Arabic. The multi dialect version, however, was further pretrained on 10 million tweets that were written in different Arabic dialects [58]. 5 ARBERT:  
This model was developed by the developer of MARABERT. The two differ in that ARBERT was pretrained on Modern Standard Arabic text only. The text was extracted from Wikipedia, news, and books sources. It has a total size of about 61 gigabytes and 6.2 billion tokens. ARBERT has the same architecture as the Multilingual BERT, with 12 layers of transformers blocks, 768 hidden units, and 12 self-attention heads, but a total of around 163 million trainable parameters [12]. 6 XLM-RoBERTa:  
This model was developed by researchers from Facebook. Similar to Multilingual BERT, XLM-RoBERTa supports multiple languages (100 languages), including Arabic. It was pretrained on Wikipedia and Common Crawl data. The Arabic pretraining corpus is in Modern Standard Arabic and it has 2869 tokens that sum up to around 28 gigabytes of data. The base version of the model has 12 layers of transformers blocks, 768 hidden units, 12 self-attention heads, and around 270 million trainable parameters.