# Fine-Tuning BERT-Based Pre-Trained Models for Arabic Dependency Parsing
## 6 Evaluation and Discussion
### Evaluation of the Tuning Techniques  
While the BERT models are well-regulated and relatively complex, several research papers have studied the modeling of their performance and investigated the need to tune all the pre-trained layers to accomplish NLP tasks [31]. [PERSON] et al. [30] have shown that dependency parsing requires tuning more layers to capture the syntactic information of the English dataset. To explore Arabic treebanks, we evaluated the following three fine-tuning techniques using the AraBERTv2 BERT model:  
1. Full Tuning (FT): Further train the entire architecture of the pre-trained model on the experimental datasets. 2. Full Freezing (FF): All layers of the pre-trained model are frozen and tested on the datasets under study. In this case, the weights of the 12 layers are not updated. 3. Partial Freezing (PF): The weights of some layers of the pre-trained model are kept frozen, while other layers are retrained. We also examined the attachment of two neural network (NN) layers to the FT and FF, and then trained the new model. Two types of neural networks have been explored, Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) [41; 42]. FT+LSTM and FF+LSTM are fully tuned and fully freeze models with two-layered BILSTM. In contrast, FT+GRU and FF+GRU are fully tuned and fully freeze models with two attached (GRU) layers. Table 13 shows the results of testing the six models (FT, FT+LSTM, FT+GRU, FF, FF+LSTM, and FF+GRU). It is noticeable that FT without attached NN layers recorded the best UAS/LAS for all three treebank test sets. On the other hand, GRU layers performed better than the LSTM layer, which is consistent with [PERSON] and [PERSON]'s results on the Arabic NER task [29]. However, the fully tuned BERT model with a two-layered BILSTM decoder achieved better results on the English treebank in [20]. We tuned the BERT layers partially and tracked the model accuracy using partial freezing of the bottom layers progressively. We froze one more layer in each run. Moreover, we gradually tuned one more bottom layer for each run and reported the LAS of the model. Figure 4a,b shows the results of both experiments, which correspond to the hypothesis of [30] that the hierarchical nature of analysis requires additional layers to understand syntactic dependencies. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **FT** & **FT+LSTM** & **FT+GRU** & **FF** & **FF+LSTM** & **FF+GRU** \\ \hline PADT\({}_{\text{E3}}\) & 84.03/80.26 & 82.90/78.92 & 83.64/79.74 & 65.30/60.25 & 72.54/67.72 & 75.44/70.98 \\ CATIB\({}_{\text{E3}}\) & 87.54/86.41 & 85.92/84.82 & 87.50/86.39 & 62.88/59.88 & 77.29/74.74 & 79.76/77.66 \\ ArPoT\({}_{\text{E2}}\) & 79.79/74.13 & 76.88/70.83 & 79.51/73.63 & 56.84/42.32 & 69.87/60.93 & 72.64/64.73 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: UAS/LAS on the test datasets for AraBERTv2 BERT model fully tuned (FT) and fully freezes (FF) with and without additional layers (GRU and LSTM). \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Camel-\({}_{\text{MSA,CA,mix}}\)** & **Multilingual** & **Arabic** & **AraBERTv1** & **AraBERTv2** & **ARBERT** & **GigaBERT** \\ \hline PADT\({}_{\text{E3}}\) & 1.1 & 1.6 & 1.1 & 1.1 & 1.1 & 1.0 & 1.1 \\ CATIB\({}_{\text{E3}}\) & 1.2 & 1.5 & 1.1 & 1.1 & 1.1 & 1.1 & 1.1 \\ ArPoT\({}_{\text{E2}}\) & 1.3 & 1.7 & 1.2 & 1.4 & 1.2 & 1.1 & 1.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 12: Average number of sub-words provided by each model’s tokenizer. More detailed scores for this experiment are listed in Table 14. Although fine-tuning of the higher layers is required in all cases, some datasets converge in earlier layers. For example, CATiB and ArPoT provided a higher UAS/LAS on the test dataset by tuning the first 10 and 11 layers, respectively. On the other hand, the best scores on the PADT required all 12 layers, which indicates full tuning.