Hugging Face's logo

aubmindlab
/
araelectra-base-discriminator

Transformers
PyTorch
TensorFlow
TensorBoard
Safetensors
Arabic
electra
pretraining
Inference Endpoints
Model card Files Metrics Community
1
araelectra-base-discriminator / README.md
AUB MIND LAB
Update README.md
f6259b1
preview code
|
raw
history blame
5.58 kB
metadata
language: ar
datasets:
  - wikipedia
  - OSIAN
  - 1.5B Arabic Corpus
  - OSCAR Arabic Unshuffled

AraELECTRA

ELECTRA is a method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN. AraELECTRA achieves state-of-the-art results on Arabic QA dataset.

For a detailed description, please refer to the AraELECTRA paper AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding.

How to use the discriminator in transformers 

from transformers import ElectraForPreTraining, ElectraTokenizerFast
import torch

discriminator = ElectraForPreTraining.from_pretrained("aubmindlab/araelectra-base-discriminator")
tokenizer = ElectraTokenizerFast.from_pretrained("aubmindlab/araelectra-base-discriminator")

sentence = ""
fake_sentence = ""

fake_tokens = tokenizer.tokenize(fake_sentence)
fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt")
discriminator_outputs = discriminator(fake_inputs)
predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2)

[print("%7s" % token, end="") for token in fake_tokens]

[print("%7s" % int(prediction), end="") for prediction in predictions.tolist()]

Model

Model HuggingFace Model Name Size (MB/Params)
AraELECTRA-base-generator araelectra-base-generator 227MB/60M
AraELECTRA-base-discriminator araelectra-base-discriminator 516MB/135M

Compute

Model Hardware num of examples (seq len = 512) Batch Size Num of Steps Time (in days)
AraELECTRA-base TPUv3-8 - 256 2M 24

Dataset

The pretraining data used for the new AraELECTRA model is also used for AraGPT2 and AraBERTv2.

The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation)

For the new dataset we added the unshuffled OSCAR corpus, after we thoroughly filter it, to the previous dataset used in AraBERTv1 but with out the websites that we previously crawled:

Preprocessing

It is recommended to apply our preprocessing function before training/testing on any dataset. Install farasapy to segment text for AraBERT v1 & v2 pip install farasapy

from arabert.preprocess import ArabertPreprocessor

model_name="araelectra-base"
arabert_prep = ArabertPreprocessor(model_name=model_name, keep_emojis=True)

text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
arabert_prep.preprocess(text)

TensorFlow 1.x models

The TF1.x model are avaiable in the HuggingFace models repo. You can download them as follows:

  • via git-lfs: clone all the models in a repo
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install

!git clone https://huggingface.co/aubmindlab/MODEL_NAME
!tar -C ./MODEL_NAME -zxvf /content/MODEL_NAME/tf1_model.tar.gz

where MODEL_NAME is any model under the aubmindlab name

  • via wget:
    • Go to the tf1_model.tar.gz file on huggingface.co/models/aubmindlab/MODEL_NAME.
    • copy the oid sha256
    • then run wget https://cdn-lfs.huggingface.co/aubmindlab/aragpt2-base/INSERT_THE_SHA_HERE (ex: for aragpt2-base: wget https://cdn-lfs.huggingface.co/aubmindlab/aragpt2-base/3766fc03d7c2593ff2fb991d275e96b81b0ecb2098b71ff315611d052ce65248)

If you used this model please cite us as : 

@misc{antoun2020aragpt2,
      title={AraGPT2: Pre-Trained Transformer for Arabic Language Generation},
      author={Wissam Antoun and Fady Baly and Hazem Hajj},
      year={2020},
      eprint={2012.15520},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn't have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com