|
--- |
|
license: cc-by-4.0 |
|
base_model: |
|
- facebook/nllb-200-3.3B |
|
pipeline_tag: translation |
|
--- |
|
# AraDiCE-msa-to-egy: An MSA to Egyptian Machine Translation model based on NLLB-3.3B |
|
|
|
This repository includes an MSA to Egyptian machine translation model that was finetuned based on nllb-3.3B. The model was used to curate benchmarks for the AraDiCE paper (citation below). The The human post-edited benchmarks can be found<a href="https://huggingface.co/datasets/QCRI/AraDiCE" target="_blank" style="margin-right: 15px; margin-left: 10px">here.</a> |
|
|
|
## Sample Usage |
|
|
|
```python |
|
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("QCRI/AraDiCE-msa-to-egy") |
|
model = AutoModelForSeq2SeqLM.from_pretrained("QCRI/AraDiCE-msa-to-egy") |
|
|
|
article = "من مصلحتك أن ترحل من كازابلانكا لفترة. هناك موقع لفرنسا الحرة بالقرب من برازفيل. قد أسهل لك العبور." |
|
inputs = tokenizer(article, return_tensors="pt") |
|
|
|
translated_tokens = model.generate( |
|
**inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("arz_Arab"), max_length=30 |
|
) |
|
translation = tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0] |
|
print(translation) |
|
``` |
|
## License |
|
|
|
The model is distributed under the **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0)**. The full license text can be found in the accompanying `licenses_by-nc-sa_4.0_legalcode.txt` file. |
|
|
|
## Citation |
|
Please find the paper<a href="https://arxiv.org/pdf/2409.11404" target="_blank" style="margin-right: 15px; margin-left: 10px">here.</a> |
|
|
|
``` |
|
@article{mousi2024aradicebenchmarksdialectalcultural, |
|
title={{AraDiCE}: Benchmarks for Dialectal and Cultural Capabilities in LLMs}, |
|
author={Basel Mousi and Nadir Durrani and Fatema Ahmad and Md. Arid Hasan and Maram Hasanain and Tameem Kabbani and Fahim Dalvi and Shammur Absar Chowdhury and Firoj Alam}, |
|
year={2024}, |
|
publisher={arXiv:2409.11404}, |
|
url={https://arxiv.org/abs/2409.11404}, |
|
} |
|
``` |
