|
--- |
|
license: mit |
|
datasets: |
|
- tashkeela |
|
language: |
|
- ar |
|
metrics: |
|
- accuracy |
|
pipeline_tag: text2text-generation |
|
--- |
|
|
|
# Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization |
|
|
|
|
|
## Table of Contents |
|
- [Introduction](#introduction) |
|
- [Models](#models) |
|
- [ByT5](#model-name) |
|
- [Model Description](#model-description) |
|
- [Benchmarks](#benchmarks) |
|
- [Citation](#citation) |
|
- [Contact](#contact) |
|
|
|
## Introduction |
|
|
|
Most of previous work on learning diacritization of the Arabic language relied on training models from scratch. In this paper, we investigate how to leverage pre-trained language models to learn diacritization. We finetune token-free pre-trained multilingual models (ByT5) to learn to predict and insert missing diacritics in Arabic text, a complex task that requires understanding the sentence semantics and the morphological structure of the tokens. We show that we can achieve state-of-the-art on the diacritization task with minimal amount of training and no feature engineering, reducing WER by 40%. We release our finetuned models for the greater benefit of the researchers in the community. |
|
|
|
## Model Description |
|
|
|
The ByT5 model, distinguished by its innovative token-free architecture, directly processes raw text to adeptly navigate diverse languages and linguistic nuances. Pre-trained on a comprehensive text corpus mc4, ByT5 excels in understanding and generating text, making it versatile for various NLP tasks. We have further enhanced its capabilities by fine-tuning it on a Tashkeela data set for 13,000 steps, significantly refining its performance in restoring the diacritical marks for Arabic. |
|
|
|
## Benchmarks |
|
|
|
**Note: This model has been trained specifically for use with Classical Arabic.** |
|
|
|
Our model attained a Diacritics Error Rate (DER) of 0.95 and a Word Error Rate (WER) of 2.49. |
|
|
|
|
|
Code sample to use the model. |
|
|
|
|
|
```python |
|
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer |
|
import pandas as pd |
|
|
|
|
|
if __name__ == "__main__": |
|
|
|
text = "كيف الحال" |
|
|
|
model_name = "basharalrfooh/Fine-Tashkeel" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
model = AutoModelForSeq2SeqLM.from_pretrained(model_name) |
|
|
|
input_ids = tokenizer(text, return_tensors="pt").input_ids |
|
outputs = model.generate(input_ids, max_new_tokens=128) |
|
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=False) |
|
print("Generated output:", decoded_output) |
|
``` |
|
|
|
## Citation |
|
|
|
```bibtex |
|
@misc{alrfooh2023finetashkeel, |
|
title={Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization}, |
|
author={Bashar Al-Rfooh and Gheith Abandah and Rami Al-Rfou}, |
|
year={2023}, |
|
eprint={2303.14588}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
|
|
``` |
|
|
|
## Contact |
|
|
|
bashar@alrfou.com |
|
|
