| language: | |
| - bg | |
| - cs | |
| - da | |
| - de | |
| - el | |
| - en | |
| - es | |
| - et | |
| - fi | |
| - fr | |
| - ga | |
| - hr | |
| - hu | |
| - it | |
| - lt | |
| - lv | |
| - mt | |
| - nl | |
| - pl | |
| - pt | |
| - ro | |
| - sk | |
| - sl | |
| - sv | |
| - uk | |
| - multilingual | |
| license: mit | |
| # EuroGPT2 | |
| **NOTE: THIS IS THE ORIGINAL MEGATRON-DEEPSPEED CHECKPOINT INCLUDING OPTIMIZER STATES** | |
| A GPT2 language model for European languages (EU-24 + Ukrainian). | |
| The model follows the original architecture as [OpenAI's GPT2](https://huggingface.co/gpt2/) apart from using [rotary](https://arxiv.org/abs/2104.09864) instead of learned positional embeddigs. | |
| ## Model settings | |
| - parameters: 124M | |
| - number of layers: 12 | |
| - hidden size: 768 | |
| - number of heads: 12 | |
| - sequence length: 1024 | |
| - batch size: 168 | |
| - test PPL after training: 23.6 (steps: 436,940) | |
| ## Training data | |
| - [Wikimedia dumps](https://dumps.wikimedia.org/) (Wikipedia, Wikinews, Wikibooks, Wikisource, Wikivoyage; 20230301) | |
| - [EUR-Lex](https://huggingface.co/datasets/joelito/eurlex_resources) | |
| - [OSCAR 2023.01](https://huggingface.co/datasets/oscar-corpus/OSCAR-2301) | |
| - Tokens: 75,167,662,080 | |
| ## Languages | |
| Included languages: Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Irish, Croatian, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovenian, Swedish, and Ukrainian. | |
| | Language | Ratio | | |
| | -------- | ------ | | |
| | bg | 5,92% | | |
| | cs | 4,77% | | |
| | da | 2,19% | | |
| | de | 7,36% | | |
| | el | 8,60% | | |
| | en | 10,11% | | |
| | es | 6,57% | | |
| | et | 1,67% | | |
| | fi | 2,70% | | |
| | fr | 7,18% | | |
| | ga | 0,25% | | |
| | hr | 1,09% | | |
| | hu | 6,38% | | |
| | it | 5,80% | | |
| | lt | 2,01% | | |
| | lv | 1,76% | | |
| | mt | 1,49% | | |
| | nl | 5,20% | | |
| | pl | 4,82% | | |
| | pt | 4,64% | | |
| | ro | 2,93% | | |
| | sk | 2,03% | | |
| | sl | 1,54% | | |
| | sv | 3,00% | | |
| ## License | |
| MIT | |