Update README.md
Browse files
README.md
CHANGED
|
@@ -14,6 +14,8 @@ tags:
|
|
| 14 |
- llama-2-ko
|
| 15 |
---
|
| 16 |
|
|
|
|
|
|
|
| 17 |
# **Llama-2-Ko** 🦙🇰🇷
|
| 18 |
|
| 19 |
Llama-2-Ko serves as an advanced iteration of Llama 2, benefiting from an expanded vocabulary and the inclusion of a Korean corpus in its further pretraining. Just like its predecessor, Llama-2-Ko operates within the broad range of generative text models that stretch from 7 billion to 70 billion parameters. This repository focuses on the 7B pretrained version, which is tailored to fit the Hugging Face Transformers format. For access to the other models, feel free to consult the index provided below.
|
|
@@ -34,7 +36,8 @@ Llama-2-Ko is an auto-regressive language model that uses an optimized transform
|
|
| 34 |
|
| 35 |
||Training Data|Params|Content Length|GQA|Tokens|LR|
|
| 36 |
|---|---|---|---|---|---|---|
|
| 37 |
-
|Llama 2|*A new mix of Korean online data*|7B|4k|✗|>40B
|
|
|
|
| 38 |
|
| 39 |
**Vocab Expansion**
|
| 40 |
|
|
|
|
| 14 |
- llama-2-ko
|
| 15 |
---
|
| 16 |
|
| 17 |
+
> 🚧 Note: this repo is under construction 🚧
|
| 18 |
+
|
| 19 |
# **Llama-2-Ko** 🦙🇰🇷
|
| 20 |
|
| 21 |
Llama-2-Ko serves as an advanced iteration of Llama 2, benefiting from an expanded vocabulary and the inclusion of a Korean corpus in its further pretraining. Just like its predecessor, Llama-2-Ko operates within the broad range of generative text models that stretch from 7 billion to 70 billion parameters. This repository focuses on the 7B pretrained version, which is tailored to fit the Hugging Face Transformers format. For access to the other models, feel free to consult the index provided below.
|
|
|
|
| 36 |
|
| 37 |
||Training Data|Params|Content Length|GQA|Tokens|LR|
|
| 38 |
|---|---|---|---|---|---|---|
|
| 39 |
+
|Llama 2|*A new mix of Korean online data*|7B|4k|✗|>40B*|1e<sup>-5</sup>|
|
| 40 |
+
*Plan to train upto 200B tokens
|
| 41 |
|
| 42 |
**Vocab Expansion**
|
| 43 |
|