# Fine-Tuning BERT-Based Pre-Trained Models for Arabic Dependency Parsing
## 6 Evaluation and Discussion
### Choosing the Best BERT Model
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Source\textbackslash{}Model** & **Camel-MSA** & **Camel-CA** & **Camel-mix** & **Multilingual Arabic** & **AraBERTv1** & **AraBERTv2** & **ARBERT** & **GigaBERT** \\ \hline AR\_Wiki & & & & & & & \\ El-Khair & & & & & & & \\ OSIAN & & & & & & & \\ OpenITI & & & & & & & \\ OSCAR & & & & & & & \\
2019 & & & & & & & \\ OSCAR & & & & & & & \\
2020 & & & & & & & \\ Gigaword & & & & & & & \\ Dialectal & & & & & & \\ Corpora & & & & & & \\ Multi\_Wiki & & & & & & \\ News & & & & & & \\ As-safir & & & & & & \\ Hindawi & & & & & & \\ Code- & & & & & & \\ switch & & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Data sources for the Arabic BERT-based models pre-training. To explore the effect of OOV tokens on the parsing accuracy of fine-tuned BERT-based models, we present the rate of OOV tokens generated by the tokenizers for each dataset in Table 11.  
In all corpora, the ratio of OOV produced by the CAMeL pre-trained models was lower. This was followed by AraBERTv2, which was the most accurate on dependency parsing. However, after checking the OOV tokens, we observed that punctuation is the most common OOV given by the CAMeL BERT models (Camel-MSA and Camel-CA Camel-mix) and AraBERT models (AraBERTv1 and AraBERTv2). Thus, these models did not record any OOV tokens in the unpunctuated freebank (ArPoT). In contrast, the rest of the models would identify very common words with \"/HAMZAH\" as OOV, such as \",\" and \",\", because of encoding issue (We used tokenizer.convert_tokens_to_ids method rather than tokenizer.encode). Such particles are essential for sentence comprehension and affect how the models understand a sentence. In the same context, Table 12 shows the average number of sub-words (pieces per word) given by the tokenizer of each model (We used pytorch-pretrained-bert, which gives different tokenization than the transformers), where the lower value means that most words are not divided into sub-words by the model tokenizer. Overall, the rates show that, on average, the tokenizers did not split the word into more than one word. A higher rate of pieces per word was recorded by the multilingual model on all treebanks. The ARBERT model records the lowest splitting rate on PADT (mostly, each word is one token), where it is trained on a 100 K vocabulary size. However, it was not as accurate as the AraBERT and CAMeL models. This can result from the factors mentioned earlier, such as the OOV rate and training batch size. \begin{table}
\begin{tabular}{l l l l l l l l l} \hline \hline \multirow{2}{*}{**Model**} & **(MLM)** & \multirow{2}{*}{**Tokenizer**} & **Batch** & \multirow{2}{*}{**MSL**} & **TD** & **Steps** & **Words** & **Data** & **Vocabulary** \\  & & **Objective** & & & & **(Days)** & **Size** & **Size** & **Size** \\ \hline Camel- & WWM + NSP & WP & 1024/256 & 128/512 & \(\sim\)4.5 & 1 M & 12.6 B & 107 GB & 30 K \\ Camel- & WWM + NSP & WP & 1024/256 & 128/512 & \(\sim\)4.5 & 1 M & 847 M & 6 GB & 30 K \\ Camel- & & WWM + NSP & WP & 1024/256 & 128/512 & \(\sim\)4.5 & 1 M & 17.3 B & 167 GB & 30 K \\ multilingual & SWM + NSP & WP & - & - & - & - & - & - & 120 K \\ Arabic & WWM + NSP & WP & 128 & - & - & 4 M & 8.2 B & 95 GB & 32 K \\ AraBERTv1 & WWM + NSP & SP & 512/128 & 128/512 & 4 & 1.25 M & 2.7 B & 23 GB & 60 K \\ AraBERTv2 & WWM + NSP & WP & 2560/384 & 128/512 & 36 & 3 M & 8.6 B & 77 GB & 60 K \\ ARBERT & WWM + NSP & WP & 256 & 128 & 16 & 8 M & 6.5 B & 61 GB & 100 K \\ GigaBERT & SWM + NSP & WP & 512 & 128/512 & - & 1.48 M & 10.4 B & - & 50 K \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Arabic BERT models configurations. MLM: masked language modeling. WWM: whole word masking. SWM: sub-word masking. NSP: next sentence prediction. WP: word piece. SP: sentence piece. MSL: maximum sentence length. TD: training duration.