# Location Analysis for Arabic COVID-19 Twitter Data Using Enhanced Dialect Identification Models
## 3 Arabic Dialect Identification Model
### The Classifier System
#### 3.2.3 BERT Model  
Finally, we investigated the recent Bidirectional Encoder Representations from Transformers (BERT) model [36], a transformer-based machine learning technique for NLP pre-training. The BERT process corrupts some tokens in the input by replacing them with MASK and then trains a model to recreate the original tokens (Figure 3). BERT uses unlabeled text to pre-train deep bidirectional representations, unlike other older language representation models, by jointly conditioning the right and left context in all layers. As a result, fine-tuning the pre-trained BERT model can be done using one extra output layer to provide state-of-the-art models for a big range of NLP tasks. In this work, we fine-tuned the BERT model to produce an Arabic dialect identification model.  
In our experiments, we utilized three pre-trained BERT models as the base models to be tuned. Pre-trained BERT models cannot be used directly for dialect identification. So given an Arabic BERT model, we used the AOC training data subset to fine-tune the model where the input is a sequence of comments associated with a specific dialect and the output is compared to the target dialect for minimizing the difference to fine-tune the  
Figure 2: An example of Long Short Term Memory (LSTM) neural network.  
weights to convert the model into a dialect identifier. Once the model is fine-tuned, it can be queried using an Arabic sentence and the model should propose a specific dialect for the input. A summary of pre-trained BERT models is the following.  
ArabicBERT  
ArabicBERT was the first pre-trained BERT model for Arabic [37] that used a corpus consists of an unshuffled version of OSCAR data [32] and a relatively recent data dump from Wikipedia, which sums up to 8.2 B words and a vocabulary set of 32,000 word pieces. The corpus and the vocabulary set were not restricted to Modern Standard Arabic (MSA) and contained some dialectical Arabic too, which boosted the model's performance in terms of data from social media platforms. The main specifications of the ArabicBERT model are:  
* Hidden size: 768;
* Num attention heads: 12;
* Num hidden layers: 12;
* Vocab size: 32,000.  
AraBERT  
AraBERT was pre-trained specifically for Arabic, hoping to replicate BERT's success with state-of-the-art results on the majority of tested Arabic natural language processing tasks [32]. It has been trained using crawling Arabic news websites, among other sources. The performance of AraBERT is better than multilingual BERT from Google (mBERT) in several downstream tasks. Specifications of the AraBERT model are summarized as follows:  
* Hidden size: 768;
* Num attention heads: 12;
* Num hidden layers: 12;
* Vocab size: 64,000.  
MARBERT  
The third model was MARBERT [38]. MARBERT is an Arabic large-scale masked language model that was proposed this year. It targets both MSA and dialectical Arabic. It has been trained using a large dataset of Arabic tweets. MARBERT is best suited for downstream tasks involving dialectal Arabic. It also has state-of-the-art results in many of the downstream tasks in Arabic. The following are the main characteristics of the model:  
* Hidden size: 768;  
Figure 3: Bidirectional Encoder Representations from Transformers (BERT) model.