|
--- |
|
language: ar |
|
datasets: |
|
- oscar |
|
- wikipedia |
|
--- |
|
|
|
# Arabic BERT Medium Model |
|
|
|
Pretrained BERT Medium language model for Arabic |
|
|
|
_If you use this model in your work, please cite this paper:_ |
|
|
|
``` |
|
@inproceedings{safaya-etal-2020-kuisail, |
|
title = "{KUISAIL} at {S}em{E}val-2020 Task 12: {BERT}-{CNN} for Offensive Speech Identification in Social Media", |
|
author = "Safaya, Ali and |
|
Abdullatif, Moutasem and |
|
Yuret, Deniz", |
|
booktitle = "Proceedings of the Fourteenth Workshop on Semantic Evaluation", |
|
month = dec, |
|
year = "2020", |
|
address = "Barcelona (online)", |
|
publisher = "International Committee for Computational Linguistics", |
|
url = "https://www.aclweb.org/anthology/2020.semeval-1.271", |
|
pages = "2054--2059", |
|
} |
|
``` |
|
|
|
## Pretraining Corpus |
|
|
|
`arabic-bert-medium` model was pretrained on ~8.2 Billion words: |
|
|
|
- Arabic version of [OSCAR](https://traces1.inria.fr/oscar/) - filtered from [Common Crawl](http://commoncrawl.org/) |
|
- Recent dump of Arabic [Wikipedia](https://dumps.wikimedia.org/backup-index.html) |
|
|
|
and other Arabic resources which sum up to ~95GB of text. |
|
|
|
__Notes on training data:__ |
|
|
|
- Our final version of corpus contains some non-Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER. |
|
- Although non-Arabic characters were lowered as a preprocessing step, since Arabic characters does not have upper or lower case, there is no cased and uncased version of the model. |
|
- The corpus and vocabulary set are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too. |
|
|
|
## Pretraining details |
|
|
|
- This model was trained using Google BERT's github [repository](https://github.com/google-research/bert) on a single TPU v3-8 provided for free from [TFRC](https://www.tensorflow.org/tfrc). |
|
- Our pretraining procedure follows training settings of bert with some changes: trained for 3M training steps with batchsize of 128, instead of 1M with batchsize of 256. |
|
|
|
## Load Pretrained Model |
|
|
|
You can use this model by installing `torch` or `tensorflow` and Huggingface library `transformers`. And you can use it directly by initializing it like this: |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModel |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("asafaya/bert-medium-arabic") |
|
model = AutoModelForMaskedLM.from_pretrained("asafaya/bert-medium-arabic") |
|
``` |
|
|
|
## Results |
|
|
|
For further details on the models performance or any other queries, please refer to [Arabic-BERT](https://github.com/alisafaya/Arabic-BERT) |
|
|
|
## Acknowledgement |
|
|
|
Thanks to Google for providing free TPU for the training process and for Huggingface for hosting this model on their servers 😊 |
|
|
|
|
|
|
