heBERT oscar based v0.1
Browse files- README.md +1 -1
- config.json +5 -5
- pytorch_model.bin +2 -2
- training_args.bin +1 -1
- vocab.txt +0 -0
README.md
CHANGED
@@ -6,7 +6,7 @@ HeBERT is a Hebrew pretrained language model. It is based on Google's BERT archi
|
|
6 |
|
7 |
### HeBert was trained on three dataset:
|
8 |
1. A Hebrew version of OSCAR [(Ortiz, 2019)](https://oscar-corpus.com/): ~9.8 GB of data, including 1 billion words and over 20.8 millions sentences.
|
9 |
-
2. A Hebrew dump of Wikipedia: ~650 MB of data, including over 63 millions words and 3.8 millions sentences
|
10 |
3. Emotion UGC data that was collected for the purpose of this study. (described below)
|
11 |
We evaluated the model on emotion recognition and sentiment analysis, for a downstream tasks.
|
12 |
|
|
|
6 |
|
7 |
### HeBert was trained on three dataset:
|
8 |
1. A Hebrew version of OSCAR [(Ortiz, 2019)](https://oscar-corpus.com/): ~9.8 GB of data, including 1 billion words and over 20.8 millions sentences.
|
9 |
+
2. A Hebrew dump of [Wikipedia](https://dumps.wikimedia.org/hewiki/latest/): ~650 MB of data, including over 63 millions words and 3.8 millions sentences
|
10 |
3. Emotion UGC data that was collected for the purpose of this study. (described below)
|
11 |
We evaluated the model on emotion recognition and sentiment analysis, for a downstream tasks.
|
12 |
|
config.json
CHANGED
@@ -10,12 +10,12 @@
|
|
10 |
"initializer_range": 0.02,
|
11 |
"intermediate_size": 3072,
|
12 |
"layer_norm_eps": 1e-12,
|
13 |
-
"max_position_embeddings":
|
14 |
"model_type": "bert",
|
15 |
"num_attention_heads": 12,
|
16 |
-
"num_hidden_layers":
|
17 |
"pad_token_id": 0,
|
18 |
-
"total_flos":
|
19 |
-
"type_vocab_size":
|
20 |
-
"vocab_size":
|
21 |
}
|
|
|
10 |
"initializer_range": 0.02,
|
11 |
"intermediate_size": 3072,
|
12 |
"layer_norm_eps": 1e-12,
|
13 |
+
"max_position_embeddings": 512,
|
14 |
"model_type": "bert",
|
15 |
"num_attention_heads": 12,
|
16 |
+
"num_hidden_layers": 12,
|
17 |
"pad_token_id": 0,
|
18 |
+
"total_flos": 6997313242916978688,
|
19 |
+
"type_vocab_size": 2,
|
20 |
+
"vocab_size": 30522
|
21 |
}
|
pytorch_model.bin
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
-
size
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:b219b9d76997d1933f01c362ae4fdc838600a5dcc5323869af1466959b74e6ed
|
3 |
+
size 438146887
|
training_args.bin
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
size 1775
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:d028009bf2029df744e09caee6ef5c5fe830318d5e02b381011099c51f69c0d2
|
3 |
size 1775
|
vocab.txt
CHANGED
The diff for this file is too large to render.
See raw diff
|
|