Package: arabert
AraBERTv2 / AraGPT2 / AraELECTRAThis repository now contains code and implementation for:AraBERT v0.1/v1: OriginalAraBERT v0.2/v2: Base and large versions with better vocabulary, more data, more trainingRead More...AraGPT2: base, medium, large and MEGA. Trained from scratch on ArabicRead More...AraELECTRA: Trained from scratch on ArabicRead More...If you want to clone the old repository:gitclonehttps://github.com/aub-mind/arabert/cdarabert&&gitcheckout6a58ca118911ef311cbe8cdcdcc1d03601123291Update17-jul-2022:You can now install arabert viapip install arabert8-Oct-2021:New AraBERT models that better supports tweets and emojies.13-Sep-2021:Arabic NLP Demo Space on HuggingFace02-Apr-2021:AraELECTRA powered Arabic Wikipedia QA systemInstallationInstall AraBERT from PyPI:pipinstallarabertthen use it as follows:fromarabertimportArabertPreprocessorfromarabert.aragpt2.grover.modeling_gpt2importGPT2LMHeadModelAraBERTv2What's New!AraBERTv0.2-Twitter-base/largeare two new models for Arabic dialects and tweets, trained by continuing the pre-training using the MLM task on ~60M Arabic tweets (filtered from a collection on 100M).The two new models have had emojies added to their vocabulary in addition to common words that weren't at first present.
The pre-training was done with a max sentence length of 64 only for 1 epoch.ModelsAraBERT comes in 6 variants:More Detail in the AraBERT folder and in theREADMEand in theAraBERT PaperModelHuggingFace Model NameSize (MB/Params)Pre-SegmentationDataSet (Sentences/Size/nWords)AraBERTv0.2-Twitter-basebert-base-arabertv02-twitter543MB / 136MNoSame as v02 + 60M Multi-Dialect TweetsAraBERTv0.2-Twitter-largebert-large-arabertv02-twitter1.38G / 371MNoSame as v02 + 60M Multi-Dialect TweetsAraBERTv0.2-basebert-base-arabertv02543MB / 136MNo200M / 77GB / 8.6BAraBERTv0.2-largebert-large-arabertv021.38G / 371MNo200M / 77GB / 8.6BAraBERTv2-basebert-base-arabertv2543MB / 136MYes200M / 77GB / 8.6BAraBERTv2-largebert-large-arabertv21.38G / 371MYes200M / 77GB / 8.6BAraBERTv0.1-basebert-base-arabertv01543MB / 136MNo77M / 23GB / 2.7BAraBERTv1-basebert-base-arabert543MB / 136MYes77M / 23GB / 2.7BAll models are available in theHuggingFacemodel page under theaubmindlabname. Checkpoints are available in PyTorch, TF2 and TF1 formats.Better Pre-Processing and New VocabWe identified an issue with AraBERTv1's wordpiece vocabulary. The issue came from punctuations and numbers that were still attached to words when we trained the wordpiece vocab. We now insert a space between numbers and characters and around punctuation characters.The new vocabulary was learnt using theBertWordpieceTokenizerfrom thetokenizerslibrary, and now supports the Fast tokenizer implementation from thetransformerslibrary.P.S.: All the old BERT codes should work with the new BERT, just change the model name and check the new preprocessing functionPlease read the section on how to use thehttps://github.com/aub-mind/arabert#preprocessingBigger Dataset and More ComputeWe used ~3.5 times more data, and trained for longer.
For Dataset Sources see thehttps://github.com/aub-mind/arabert#DatasetModelHardwarenum of examples with seq len (128 / 512)128 (Batch Size/ Num of Steps)512 (Batch Size/ Num of Steps)Total StepsTotal Time (in Days)AraBERTv0.2-baseTPUv3-8420M / 207M2560 / 1M384/ 2M3M36AraBERTv0.2-largeTPUv3-128420M / 207M13440 / 250K2056 / 300K550K7AraBERTv2-baseTPUv3-8420M / 207M2560 / 1M384/ 2M3M36AraBERTv2-largeTPUv3-128520M / 245M13440 / 250K2056 / 300K550K7AraBERT-base (v1/v0.1)TPUv2-8-512 / 900K128 / 300K1.2M4AraGPT2More details and code are available in the AraGPT2 folder andREADMEModelModelHuggingFace Model NameSize / ParamsAraGPT2-basearagpt2-base527MB/135MAraGPT2-mediumaragpt2-medium1.38G/370MAraGPT2-largearagpt2-large2.98GB/792MAraGPT2-megaaragpt2-mega5.5GB/1.46BAraGPT2-mega-detector-longaragpt2-mega-detector-long516MB/135MAll models are available in theHuggingFacemodel page under theaubmindlabname. Checkpoints are available in PyTorch, TF2 and TF1 formats.Dataset and ComputeFor Dataset Source see thehttps://github.com/aub-mind/arabert#DatasetModelHardwarenum of examples (seq len = 1024)Batch SizeNum of StepsTime (in days)AraGPT2-baseTPUv3-1289.7M1792125K1.5AraGPT2-mediumTPUv3-89.7M801M15AraGPT2-largeTPUv3-1289.7M256220k3AraGPT2-megaTPUv3-1289.7M256800K9AraELECTRAMore details and code are available in the AraELECTRA folder andREADMEModelModelHuggingFace Model NameSize (MB/Params)AraELECTRA-base-generatoraraelectra-base-generator227MB/60MAraELECTRA-base-discriminatoraraelectra-base-discriminator516MB/135MDataset and ComputeModelHardwarenum of examples (seq len = 512)Batch SizeNum of StepsTime (in days)ELECTRA-baseTPUv3-8-2562M24DatasetThe pretraining data used for the newAraBERTmodel is also used forAraGPT2 and AraELECTRA.The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation)For the new dataset we added the unshuffled OSCAR corpus, after we thoroughly filter it, to the previous dataset used in AraBERTv1 but with out the websites that we previously crawled:OSCAR unshuffled and filtered.Arabic Wikipedia dumpfrom 2020/09/01The 1.5B words Arabic CorpusThe OSIAN CorpusAssafir news articles. Huge thank you for Assafir for the dataPreprocessingIt is recommended to apply our preprocessing function before training/testing on any dataset.Install farasapy to segment text for AraBERT v1 & v2pip install farasapyfromarabert.preprocessimportArabertPreprocessormodel_name="aubmindlab/bert-base-arabertv2"arabert_prep=ArabertPreprocessor(model_name=model_name)text="ولن نبالغ إذا قلنا: إن 'هاتف' أو 'كمبيوتر المكتب' في زمننا هذا ضروري"arabert_prep.preprocess(text)>>>"و+ لن نبالغ إذا قل +نا : إن ' هاتف ' أو ' كمبيوتر ال+ مكتب ' في زمن +نا هذا ضروري"You can also use theunpreprocess()function to reverse the preprocessing changes, by fixing the spacing around non alphabetical characters, and also de-segmenting if the model selected need pre-segmentation. We highly recommend unprocessing generated content ofAraGPT2model, to make it look more natural.output_text="و+ لن نبالغ إذا قل +نا : إن ' هاتف ' أو ' كمبيوتر ال+ مكتب ' في زمن +نا هذا ضروري"arabert_prep.unpreprocess(output_text)>>>"ولن نبالغ إذا قلنا: إن 'هاتف' أو 'كمبيوتر المكتب' في زمننا هذا ضروري"TheArabertPreprocessorclass:ArabertPreprocessor(model_name="",keep_emojis=False,remove_html_markup=True,replace_urls_emails_mentions=True,strip_tashkeel=True,strip_tatweel=True,insert_white_spaces=True,remove_non_digit_repetition=True,replace_slash_with_dash=None,map_hindi_numbers_to_arabic=None,apply_farasa_segmentation=None)model_name(str): model name from the HuggingFace Models page without the aubmindlab tag. Will default to a base Arabic preprocessor if model name was not found.keep_emojis(bool,optional, defaults toFalse): don't remove emojis while preprocessing.remove_html_markup(bool,optional, defaults toTrue): Whether to remove html artfacts,
should be set to False when preprocessing TyDi QA.replace_urls_emails_mentions(bool,optional, defaults toTrue): Whether to replace email urls
and mentions by special tokens.strip_tashkeel(bool,optional, defaults toTrue): remove diacritics (FATHATAN, DAMMATAN, KASRATAN, FATHA, DAMMA,  KASRA, SUKUN, SHADDA).strip_tatweel(bool,optional, defaults toTrue): remove tatweel '\u0640'.insert_white_spaces(bool,optional, defaults toTrue): insert whitespace before and after all non Arabic digits or English digits or Arabic and English Alphabet or the 2 brackets, then inserts whitespace between words and numbers or numbers and words.remove_non_digit_repetition(bool,optional, defaults toTrue): replace repetition of more than 2 non-digit character with 2 of this character.replace_slash_with_dash(bool,optional, defaults toNone): Will be automatically set to True in AraBERTv02,  AraELECTRA and AraGPT2.Set to False to force disable, and True to force enable. Replaces the "/"  with "-", since "/" is missing from AraBERTv2, AraELECTRA and ARAGPT2 vocabulary.map_hindi_numbers_to_arabic(bool,optional, defaults toNone): Will be automatically set to True in AraBERTv02, AraELECTRA and AraGPT2.Set to False to force disable, and True to force enable.Replaces hindi numbers with the corresponding Arabic one. ex: "١٩٩٥" --> "1995". This is behavior is present by default in AraBERTv1 and v2 (with pre-segmentation), and fixes the issue of caused by a bug when inserting white spaces.apply_farasa_segmentation(bool,optional, defaults toNone): Will be automatically set to True in
AraBERTv2, and AraBERTv1. Set to False to force disable, and True to force enable.Examples NotebooksYou can find the old examples that work with AraBERTv1 in theexamples/oldfolderCheck theReadme.mdfile in the examples folder for new links to colab notebooksTensorFlow 1.x modelsYou can find the PyTorch, TF2 and TF1 models in HuggingFace's Transformer Library under theaubmindlabusernamewget https://huggingface.co/aubmindlab/MODEL_NAME/resolve/main/tf1_model.tar.gzwhereMODEL_NAMEis any model under theaubmindlabnameIf you used this model please cite us as :AraBERTGoogle Scholar has our Bibtex wrong (missing name), use this instead@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}AraGPT2@inproceedings{antoun-etal-2021-aragpt2,
    title = "{A}ra{GPT}2: Pre-Trained Transformer for {A}rabic Language Generation",
    author = "Antoun, Wissam  and
      Baly, Fady  and
      Hajj, Hazem",
    booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
    month = apr,
    year = "2021",
    address = "Kyiv, Ukraine (Virtual)",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.wanlp-1.21",
    pages = "196--207",
}AraELECTRA@inproceedings{antoun-etal-2021-araelectra,
    title = "{A}ra{ELECTRA}: Pre-Training Text Discriminators for {A}rabic Language Understanding",
    author = "Antoun, Wissam  and
      Baly, Fady  and
      Hajj, Hazem",
    booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
    month = apr,
    year = "2021",
    address = "Kyiv, Ukraine (Virtual)",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.wanlp-1.20",
    pages = "191--195",
}AcknowledgmentsThanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn't have done it without this program, and to theAUB MIND LabMembers for the continous support. Also thanks toYakshofand Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.ContactsWissam Antoun:Linkedin|Twitter|Github| wfa07 (AT) mail (DOT) aub (DOT) edu | wissam.antoun (AT) gmail (DOT) comFady Baly:Linkedin|Twitter|Github| fgb06 (AT) mail (DOT) aub (DOT) edu | baly.fady (AT) gmail (DOT) com