|
--- |
|
language: bg,cs,da,de,el,en,es,et,fi,fr,ga,hr,hu,it,lt,lv,mt,nl,pl,pt,ro,sk,sl,sv,uk |
|
license: mit |
|
--- |
|
|
|
# EuroGPT2 |
|
|
|
**NOTE: THIS IS THE ORIGINAL MEGATRON-DEEPSPEED CHECKPOINT INCLUDING OPTIMIZER STATES** |
|
|
|
A GPT2 language model for European languages (EU-24 + Ukrainian). |
|
The model follows the original architecture as [OpenAI's GPT2](https://huggingface.co/gpt2/) apart from using [rotary](https://arxiv.org/abs/2104.09864) instead of learned positional embeddigs. |
|
|
|
## Model settings |
|
|
|
- parameters: 124M |
|
- number of layers: 12 |
|
- hidden size: 768 |
|
- number of heads: 12 |
|
- sequence length: 1024 |
|
- batch size: 168 |
|
- test PPL after training: 23.6 (steps: 436,940) |
|
|
|
## Training data |
|
|
|
- [Wikimedia dumps](https://dumps.wikimedia.org/) (Wikipedia, Wikinews, Wikibooks, Wikisource, Wikivoyage; 20230301) |
|
- [EUR-Lex](https://huggingface.co/datasets/joelito/eurlex_resources) |
|
- [OSCAR 2023.01](https://huggingface.co/datasets/oscar-corpus/OSCAR-2301) |
|
- Tokens: 75,167,662,080 |
|
|
|
## Languages |
|
|
|
Included languages: Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Irish, Croatian, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovenian, Swedish, and Ukrainian. |
|
|
|
| Language | Ratio | |
|
| -------- | ------ | |
|
| bg | 5,92% | |
|
| cs | 4,77% | |
|
| da | 2,19% | |
|
| de | 7,36% | |
|
| el | 8,60% | |
|
| en | 10,11% | |
|
| es | 6,57% | |
|
| et | 1,67% | |
|
| fi | 2,70% | |
|
| fr | 7,18% | |
|
| ga | 0,25% | |
|
| hr | 1,09% | |
|
| hu | 6,38% | |
|
| it | 5,80% | |
|
| lt | 2,01% | |
|
| lv | 1,76% | |
|
| mt | 1,49% | |
|
| nl | 5,20% | |
|
| pl | 4,82% | |
|
| pt | 4,64% | |
|
| ro | 2,93% | |
|
| sk | 2,03% | |
|
| sl | 1,54% | |
|
| sv | 3,00% | |
|
|
|
## License |
|
|
|
MIT |
|
|
|
|