Update README.md
Browse files
README.md
CHANGED
@@ -8,7 +8,7 @@ license: bigscience-openrail-m
|
|
8 |
---
|
9 |
|
10 |
|
11 |
-
[GeoV](https://huggingface.co/docs/transformers/model_doc/geov)-9B is a 9 billion parameter
|
12 |
|
13 |
The GeoV model was designed by Georges Harik and uses
|
14 |
[Rotary Positional Embeddings with Relative distances (RoPER)](http://research.labml.ai/RoPER.html)
|
@@ -36,9 +36,12 @@ RoPER has given better performance in some algorithmic tasks, and seems comparab
|
|
36 |
| n<sub>heads</sub> | 40 |
|
37 |
| d<sub>head</sub> | 128 |
|
38 |
| n<sub>vocab</sub> | 65500 |
|
39 |
-
| Sequence Length |
|
40 |
</figure>
|
41 |
|
|
|
|
|
|
|
42 |
|
43 |
## Generation
|
44 |
|
|
|
8 |
---
|
9 |
|
10 |
|
11 |
+
[GeoV](https://huggingface.co/docs/transformers/model_doc/geov)-9B is a 9 billion parameter causal language model.
|
12 |
|
13 |
The GeoV model was designed by Georges Harik and uses
|
14 |
[Rotary Positional Embeddings with Relative distances (RoPER)](http://research.labml.ai/RoPER.html)
|
|
|
36 |
| n<sub>heads</sub> | 40 |
|
37 |
| d<sub>head</sub> | 128 |
|
38 |
| n<sub>vocab</sub> | 65500 |
|
39 |
+
| Sequence Length | 2048 |
|
40 |
</figure>
|
41 |
|
42 |
+
The released weights were trained on ~70 billion tokens.
|
43 |
+
We plan to continue training up to 300 billion tokens and update the weights at every 20b tokens.
|
44 |
+
This training run is monolingual and uses c4en and english wikipedia datasets.
|
45 |
|
46 |
## Generation
|
47 |
|