aubmindlab commited on
Commit
ed115b9
1 Parent(s): d1046f0

Added model files

Browse files
README.md ADDED
@@ -0,0 +1,134 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: ar
3
+ datasets:
4
+ - wikipedia
5
+ - OSIAN
6
+ - 1.5B Arabic Corpus
7
+ - OSCAR Arabic Unshuffled
8
+ widget:
9
+ - text: " عاصمة لبنان هي [MASK] ."
10
+ ---
11
+
12
+ # AraBERT v1 & v2 : Pre-training BERT for Arabic Language Understanding
13
+
14
+ <img src="https://raw.githubusercontent.com/aub-mind/arabert/master/arabert_logo.png" width="100" align="left"/>
15
+
16
+ **AraBERT** is an Arabic pretrained lanaguage model based on [Google's BERT architechture](https://github.com/google-research/bert). AraBERT uses the same BERT-Base config. More details are available in the [AraBERT Paper](https://arxiv.org/abs/2003.00104) and in the [AraBERT Meetup](https://github.com/WissamAntoun/pydata_khobar_meetup)
17
+
18
+ There are two versions of the model, AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were splitted using the [Farasa Segmenter](http://alt.qcri.org/farasa/segmenter.html).
19
+
20
+
21
+ We evalaute AraBERT models on different downstream tasks and compare them to [mBERT]((https://github.com/google-research/bert/blob/master/multilingual.md)), and other state of the art models (*To the extent of our knowledge*). The Tasks were Sentiment Analysis on 6 different datasets ([HARD](https://github.com/elnagara/HARD-Arabic-Dataset), [ASTD-Balanced](https://www.aclweb.org/anthology/D15-1299), [ArsenTD-Lev](https://staff.aub.edu.lb/~we07/Publications/ArSentD-LEV_Sentiment_Corpus.pdf), [LABR](https://github.com/mohamedadaly/LABR)), Named Entity Recognition with the [ANERcorp](http://curtis.ml.cmu.edu/w/courses/index.php/ANERcorp), and Arabic Question Answering on [Arabic-SQuAD and ARCD](https://github.com/husseinmozannar/SOQAL)
22
+
23
+ # AraBERTv2
24
+
25
+ ## What's New!
26
+
27
+ AraBERT now comes in 4 new variants to replace the old v1 versions:
28
+
29
+ More Detail in the AraBERT folder and in the [README](https://github.com/aub-mind/arabert/blob/master/AraBERT/README.md) and in the [AraBERT Paper](https://arxiv.org/abs/2003.00104v2)
30
+
31
+ Model | HuggingFace Model Name | Size (MB/Params)| Pre-Segmentation | DataSet (Sentences/Size/nWords) |
32
+ ---|:---:|:---:|:---:|:---:
33
+ AraBERTv0.2-base | [bert-base-arabertv02](https://huggingface.co/aubmindlab/bert-base-arabertv02) | 543MB / 136M | No | 200M / 77GB / 8.6B |
34
+ AraBERTv0.2-large| [bert-large-arabertv02](https://huggingface.co/aubmindlab/bert-large-arabertv02) | 1.38G 371M | No | 200M / 77GB / 8.6B |
35
+ AraBERTv2-base| [bert-base-arabertv2](https://huggingface.co/aubmindlab/bert-base-arabertv2) | 543MB 136M | Yes | 200M / 77GB / 8.6B |
36
+ AraBERTv2-large| [bert-large-arabertv2](https://huggingface.co/aubmindlab/bert-large-arabertv2) | 1.38G 371M | Yes | 200M / 77GB / 8.6B |
37
+ AraBERTv0.1-base| [bert-base-arabertv01](https://huggingface.co/aubmindlab/bert-base-arabertv01) | 543MB 136M | No | 77M / 23GB / 2.7B |
38
+ AraBERTv1-base| [bert-base-arabert](https://huggingface.co/aubmindlab/bert-base-arabert) | 543MB 136M | Yes | 77M / 23GB / 2.7B |
39
+
40
+ All models are available in the `HuggingFace` model page under the [aubmindlab](https://huggingface.co/aubmindlab/) name. Checkpoints are available in PyTorch, TF2 and TF1 formats.
41
+
42
+ ## Better Pre-Processing and New Vocab
43
+
44
+ We identified an issue with AraBERTv1's wordpiece vocabulary. The issue came from punctuations and numbers that were still attached to words when learned the wordpiece vocab. We now insert a space between numbers and characters and around punctuation characters.
45
+
46
+ The new vocabulary was learnt using the `BertWordpieceTokenizer` from the `tokenizers` library, and should now support the Fast tokenizer implementation from the `transformers` library.
47
+
48
+ **P.S.**: All the old BERT codes should work with the new BERT, just change the model name and check the new preprocessing dunction
49
+ **Please read the section on how to use the [preprocessing function](#Preprocessing)**
50
+
51
+ ## Bigger Dataset and More Compute
52
+
53
+ We used ~3.5 times more data, and trained for longer.
54
+ For Dataset Sources see the [Dataset Section](#Dataset)
55
+
56
+ Model | Hardware | num of examples with seq len (128 / 512) |128 (Batch Size/ Num of Steps) | 512 (Batch Size/ Num of Steps) | Total Steps | Total Time (in Days) |
57
+ ---|:---:|:---:|:---:|:---:|:---:|:---:
58
+ AraBERTv0.2-base | TPUv3-8 | 420M / 207M | 2560 / 1M | 384/ 2M | 3M | -
59
+ AraBERTv0.2-large | TPUv3-128 | 420M / 207M | 13440 / 250K | 2056 / 300K | 550K | 7
60
+ AraBERTv2-base | TPUv3-8 | 420M / 207M | 2560 / 1M | 384/ 2M | 3M | -
61
+ AraBERTv2-large | TPUv3-128 | 520M / 245M | 13440 / 250K | 2056 / 300K | 550K | 7
62
+ AraBERT-base (v1/v0.1) | TPUv2-8 | - |512 / 900K | 128 / 300K| 1.2M | 4
63
+
64
+ # Dataset
65
+
66
+ The pretraining data used for the new AraBERT model is also used for Arabic **GPT2 and ELECTRA**.
67
+
68
+ The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation)
69
+
70
+ For the new dataset we added the unshuffled OSCAR corpus, after we thoroughly filter it, to the previous dataset used in AraBERTv1 but with out the websites that we previously crawled:
71
+ - OSCAR unshuffled and filtered.
72
+ - [Arabic Wikipedia dump](https://archive.org/details/arwiki-20190201) from 2020/09/01
73
+ - [The 1.5B words Arabic Corpus](https://www.semanticscholar.org/paper/1.5-billion-words-Arabic-Corpus-El-Khair/f3eeef4afb81223df96575adadf808fe7fe440b4)
74
+ - [The OSIAN Corpus](https://www.aclweb.org/anthology/W19-4619)
75
+ - Assafir news articles. Huge thank you for Assafir for giving us the data
76
+
77
+ # Preprocessing
78
+
79
+ It is recommended to apply our preprocessing function before training/testing on any dataset.
80
+ **Install farasapy to segment text for AraBERT v1 & v2 `pip install farasapy`**
81
+
82
+ ```python
83
+ from arabert.preprocess import ArabertPreprocessor
84
+
85
+ model_name="bert-base-arabertv02"
86
+ arabert_prep = ArabertPreprocessor(model_name=model_name, keep_emojis=True)
87
+
88
+ text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
89
+ arabert_prep.preprocess(text)
90
+ ```
91
+
92
+ ## Accepted_models
93
+ ```
94
+ bert-base-arabertv01
95
+ bert-base-arabert
96
+ bert-base-arabertv02
97
+ bert-base-arabertv2
98
+ bert-large-arabertv02
99
+ bert-large-arabertv2
100
+ araelectra-base
101
+ aragpt2-base
102
+ aragpt2-medium
103
+ aragpt2-large
104
+ aragpt2-mega
105
+ ```
106
+
107
+ # TensorFlow 1.x models
108
+
109
+ The TF1.x model are avaiable in the HuggingFace models repo.
110
+ To download them as follows:
111
+ ```bash
112
+ wget https://s3.amazonaws.com/models.huggingface.co/bert/aubmindlab/MODEL_NAME/tf1_model.tar.gz
113
+ ```
114
+ where `MODEL_NAME` is any model under the `aubmindlab` name
115
+
116
+
117
+ # If you used this model please cite us as :
118
+ Google Scholar has our Bibtex wrong (missing name), use this instead
119
+ ```
120
+ @inproceedings{antoun2020arabert,
121
+ title={AraBERT: Transformer-based Model for Arabic Language Understanding},
122
+ author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
123
+ booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
124
+ pages={9}
125
+ }
126
+ ```
127
+ # Acknowledgments
128
+ Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn't have done it without this program, and to the [AUB MIND Lab](https://sites.aub.edu.lb/mindlab/) Members for the continous support. Also thanks to [Yakshof](https://www.yakshof.com/#/) and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.
129
+
130
+ # Contacts
131
+ **Wissam Antoun**: [Linkedin](https://www.linkedin.com/in/wissam-antoun-622142b4/) | [Twitter](https://twitter.com/wissam_antoun) | [Github](https://github.com/WissamAntoun) | <wfa07@mail.aub.edu> | <wissam.antoun@gmail.com>
132
+
133
+ **Fady Baly**: [Linkedin](https://www.linkedin.com/in/fadybaly/) | [Twitter](https://twitter.com/fadybaly) | [Github](https://github.com/fadybaly) | <fgb06@mail.aub.edu> | <baly.fady@gmail.com>
134
+
config.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "BertForMaskedLM"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "hidden_act": "gelu",
7
+ "hidden_dropout_prob": 0.1,
8
+ "hidden_size": 768,
9
+ "initializer_range": 0.02,
10
+ "intermediate_size": 3072,
11
+ "max_position_embeddings": 512,
12
+ "model_type": "bert",
13
+ "num_attention_heads": 12,
14
+ "num_hidden_layers": 12,
15
+ "type_vocab_size": 2,
16
+ "vocab_size": 64000
17
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eac91bc3c317f3624abb5d562f5a49180a8aa991fdb21c57ce6a5e1266968be1
3
+ size 543490667
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
tf1_model.tar.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:38d3200f86bb1934c7d2ebc844f3743f573482eebfc952e1da6a7c29f54f2426
3
+ size 504967460
tf_model.h5 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7f1bc598fbc8c01a2d24e3a0d4a4f4e4ce79e479e011bffc6e5d91bb1590bce2
3
+ size 741883760
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"do_lower_case": false, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "do_basic_tokenize": true, "never_split": ["[بريد]", "[مستخدم]", "[رابط]"], "special_tokens_map_file": null, "name_or_path": "aubmindlab/bert-base-arabertv02"}
vocab.txt ADDED
The diff for this file is too large to render. See raw diff