# Fine-Tuning BERT-Based Pre-Trained Models for Arabic Dependency Parsing
## 8 Conclusions and Future Work  
In this study, we explored the formulation of dependency parsing for the Arabic language as a sequence labeling problem. We applied three encoding methods to the dependent head position and studied nine Arabic BERT models using different fine-tuning strategies. The effectiveness of the Arabic BERT models varied, but AraBERTv2 performed better on the three studied treebanks. Moreover, because the number of classes (possible labels) given by each dependent head position encoding method relies on the dataset sentence length and annotation schema, there is no preferred encoding method or tuning strategy. However, our experiments confirmed that tuning until the top layers of pre-trained BERT models is required to capture the syntactic information of Arabic sentences.  
We plan to work on future enhancements to address the faults stated in the results analysis and to address the issues with the Arabic BERT-based sequence labeling parser. For instance, we can conduct additional research to determine how improved tokenization contributes to better BERT-based parsing. In order to reduce the impact of the labels modifications in post-processing, we might also conduct a further search on how to effectively verify and correct the tree structure of the labeled sequence.  
Conceptualization, S.A.-G. and H.A.-K.; Data curation, S.A.-G.; Formal analysis, S.A.-G. and H.A.-K.; Funding acquisition, S.A.-G., H.A.-K. and A.-A.-S.; Investigation, S.A.-G. and H.A.-K.; Methodology, S.A.-G. and H.A.-K.; Project administration, S.A.-G. and H.A.-K.; Resources, S.A.-G., H.A.-K. and A.A.-S.; Software, S.A.-G.; Supervision, H.A.-K. and A.A.-S.; Validation, S.A.-G. and H.A.-K.; Visualization, S.A.-G. and H.A.-K.; Writing--original draft, S.A.-G.; Writing--review & editing, H.A.-K. and A.A.-S. All authors have read and agreed to the published version of the manuscript.  
This work was funded by the Deanship of Scientific Research at King Saud University through the initiative of DSR Graduate Students Research Support (GSR).  
Not applicable.  
**Data Availability Statement:** PADT datasets can be accessed at [[https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-3238](https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-3238)]([https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-3238](https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-3238)) (accessed on 4 April 2021). Restrictions apply to the availability of CATIB (the conversion of Penn Arabic Treebank (ATB) part2 v3.1) datasets, which are used under the license of the College of Computer and Information Sciences at King Saud University. The ArPoT treebank is provided by [19] at [[https://github.com/ArPoT-KSU/ArPoT_v1.0](https://github.com/ArPoT-KSU/ArPoT_v1.0)]([https://github.com/ArPoT-KSU/ArPoT_v1.0](https://github.com/ArPoT-KSU/ArPoT_v1.0)) (accessed on 21 June 2022).  
Figure 9: Example of syntactically wrong labeling caused by poor tokenization. The blue arrow represents an incorrect prediction. The word highlighted in red displays an incorrect tokenization.  
**Acknowledgments:** The authors would like to thank King Saud University and the College of Computer and Information Sciences. Additionally, the authors would like to thank the Deanship of Scientific Research at King Saud University for funding and supporting this research through the initiative of DSR Graduate Students Research Support (GSR).  
**Conflicts of Interest:** The authors declare no conflict of interest.