# Fine-Tuning BERT-Based Pre-Trained Models for Arabic Dependency Parsing
## 6 Evaluation and Discussion
### Choosing the Best BERT Model  
The Arabic BERT-based models were evaluated on the corpora (PADT, CATiB, and ArPoT), and the UAS/LAS for the testing datasets are presented in Table 8. In our observations, the most accurate model among the nine fully tuned models was Ara-BERTv2. This corresponds to the results of [23], where AraBERTv2 was the best for six other NLP subtasks. Furthermore, the dependency parsing task is sensitive to the language variants of the pre-trained model. For instance, Camel-MSA and Camel-CA have the same setup, but they are trained on different Arabic variants, MSA and CA, respectively. Hence, the UAS/LAS of the fine-tuned Camel-MSA model on the MSA corpora (PADT and CATiB) is higher than that of the Camel-CA model. On the other hand, the fine-tuned Camel-CA model performed better on the Classical Arabic corpus (ArPoT). Similarly, the multilingual model (multilingual) reported worse scores for all corpora. Table 9 displays the Arabic training datasets used for pre-training the BERT-based models under study. \begin{table}
\begin{tabular}{c c c c} \hline  & **E1** & **E2** & **E3** \\ \hline PADT & 4513 & 2017 & 1793 \\ CATiB & 1923 & 563 & 297 \\ ArPoT & 305 & 164 & 173 \\ \hline \end{tabular}
\end{table} 
Table 6: Number of labels in each treebank using three head position encodings. \begin{table}
\begin{tabular}{c c c c c c} \hline \multirow{2}{*}{**Treebank (Model)**} & \multicolumn{3}{c}{**DEV**} & \multicolumn{3}{c}{**TEST**} \\ \cline{2-6}  & **E1** & **E2** & **E3** & **E1** & **E2** & **E3** \\ \hline PADT (Camel-MSA) & 64.25/60.65 & 81.79/77.46 & 83.12/79.00 & 85.48/55.40 & 82.18/77.77 & 83.06/79.17 \\ CATiB (Camel-MSA) & 46.88/45.89 & 83.89/82.40 & 86.35/85.23 & 44.26/43.07 & 83.75/82.20 & 86.52/85.33 \\ ArPoT (Camel-CA) & 64.63/57.77 & 78.74/70.77 & 75.47/70.28 & 64.39/58.38 & 78.91/72.39 & 76.70/72.06 \\ \hline \end{tabular}
\end{table} 
Table 7: Average UAS/LAS over 5 runs on development (DEV) and test datasets using three head position encodings. To investigate the effect of the pre-training configurations on fine-tuning the dependency parsing, we compared the pre-training settings of the models and the pre-training data size. Table 10 shows that training with larger data and vocabulary sizes does not necessarily lead to higher parsing accuracy. For instance, although ARBERT is trained on more than double the size of the pre-training data and vocabulary for AraBERTv1, it performed worse on all corpora. This confirms the [PERSON] et al. [23] suggestion for other NLP tasks where data size may not be an important factor in fine-tuning the performance. Looking at the pre-training objective and tokenizer in Table 10, we can see that they do not affect the model accuracy. They are the same in the case of the BERT-based models with a higher and lower UAS/LAS (AraBERTv2 and multilingual, respectively). Therefore, focusing on the most accurate BERT-based model for all corpora (AraBERTv2), we can see that the training batches sizes are larger, which verifies the [PERSON] et al. [40] proposed enhancement of using an increasing batch size schedule for pre-training to grow the accuracy of the BERT-based model. \begin{table}
\begin{tabular}{c c c c} \hline \hline  & & **TEST UAS/LAS** \\ \hline
**Model** & **PADT\({}_{\text{E3}}\)** & **CATiB\({}_{\text{E3}}\)** & **ArPoT\({}_{\text{E2}}\)** \\ \hline Camel-MSA & 83.10/79.17 & 86.47/85.29 & 78.72/71.83 \\ Camel-CA & 80.78/76.60 & 84.95/83.32 & 79.65/72.21 \\ Camel-mix & 82.37/78.37 & 85.64/84.35 & 78.48/71.95 \\ multilingual & 74.02/68.54 & 76.22/72.50 & 72.33/60.71 \\ Arabic & 80.02/76.52 & 82.65/80.59 & 73.58/64.30 \\ AraBERTv1 & 82.76/78.82 & 86.76/85.57 & 77.76/70.95 \\ AraBERTv2 & 84.03/80.26 & 87.54/86.41 & 79.79/74.13 \\ ARBERT & 80.37/76.11 & 78.31/75.95 & 75.06/66.19 \\ GigaBERT & 80.41/76.06 & 83.29/81.28 & 73.39/62.31 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Test UAS/LAS for all Arabic pre-trained models under study using the best encoding methods for each treebank.