# Fine-Tuning BERT-Based Pre-Trained Models for Arabic Dependency Parsing
## 3 Related Work  
This section discusses recent work on syntactic parsing using pre-trained language models, including BERT for sequence labeling and fine-tuning for various downstream tasks. The need for exploring BERT fine-tuning for syntactic dependency parsing is emphasized for its potential in improving Arabic syntactic parsers. Compared to transition- and graph-based methods, [PERSON] et al. [20] showed that it is possible to obtain fast and accurate parsers using a conventional BILSTM-based model for sequence labeling. However, the study was conducted only on the English language. Later, [PERSON] et al. [28] cast parsing as sequence labeling, and then with just pre-trained encoders, syntactic parsing models were presented. The work compared different pre-trained architectures, such as GloVe, word2 vec, FastText, ELMO, and BERT, on an English treebank only. The fully tuned BERT surpassed existing sequence tagging parsers on the same English dependency treebank. The paper gave a modest presentation of the scores for some of the dependency relations in the Treebank, such as obj, advmod, det, and nsubj, but did not analyze why recognizing some relations was more accurate than others, and there was no in-depth analysis of the dataset errors. Recent work [13] explored the use of BERT models jointly with a MALT parser for transition-based dependency parsing of Tamil, a morphologically rich, agglutinative language. The results show that appropriate vector representations from BERT trained on Indic languages directly improve the overall parser performance. An abundance of recent work has evaluated the fine-tuning of pre-trained language models for downstream tasks. Fine-tuning BERT-based Arabic pre-trained language models for sequence labeling problems has shown great success in named entity recognition, POS tagging, sentiment analysis, dialect identification, and poetry meter classification [23, 29]. Consequently, an investigation of BERT fine-tuning methods for syntactic dependency parsing is also needed. This will provide a foundation for improving the development of Arabic syntactic parsers. From an analytical perspective, several studies [30, 31, 32] have attempted to understand and explain what occurs during BERT fine-tuning. [PERSON] et al. [30] explained the extent to which the layers were affected during the fine-tuning process. The results show that, unlike natural language inference and reading comprehension, dependency parsing reconfigures most of the BERT models and involves deeper changes to the encoder (more top layers of BERT are affected). In this work, we intend to explore the Arabic dependency parsing as a sequence labeling problem. We aim to identify the optimal selections to perform syntactic dependency parsing by fine-tuning BERT-based pre-trained Arabic models. To implement this task, we aim to identify the best head position encoding method, pre-trained model, and fine-tuning strategy. Moreover, we present a detailed analysis of the errors given by the optimal parsing model to understand what happened during fine-tuning, and we shed light on the reasons for the weaknesses of the chosen model. \begin{table}
\begin{tabular}{l l c c} \hline \hline  & **Model Name \({}^{*}\)** & **Variants** & **Ref.** \\ \hline Camel-MSA & CAMeL-Lab/BERT-based-arabic-camelbert-msa & MSA & [23] \\ Camel-CA & CAMeL-Lab/BERT-based-arabic-camelbert-ca & CA & [23] \\ Camel-mix & CAMeL-Lab/BERT-based-arabic-camelbert-mix & MSA/DA/CA & [23] \\ multilingual & BERT-based-multilingual-uncased & MSA & [22] \\ Arabic & asaya/BERT-based-arabic & MSA & [24] \\ AraBERTv1 & aubmindlab/BERT-based-AraBERTv01 & MSA & [25] \\ AraBERTv2 & aubmindlab/BERT-based-AraBERTv02 & MSA & [25] \\ ARBERT & UBC-NLP/ARBERT & MSA & [26] \\ GigaBERT & lanwuwei/GigaBERT-v4-Arabic-and-English & MSA & [27] \\ \hline \hline \end{tabular} \({}^{*}\) Model name as in: [[https://huggingface.co](https://huggingface.co)]([https://huggingface.co](https://huggingface.co)) (accessed on 24 October 2021).  
\end{table} 
Table 2: Pre-trained language models.