|
--- |
|
language: |
|
- fi |
|
pipeline_tag: text-generation |
|
--- |
|
Generative Pretrained Transformer with 8B parameteres for Finnish. |
|
|
|
TurkuNLP Finnish GPT-3-models are a model family of pretrained monolingual GPT-style language models that are based on BLOOM-architecture. |
|
Note that the models are pure language models, meaning that they are not [instruction finetuned](https://arxiv.org/abs/2203.02155) for dialogue |
|
or answering questions. |
|
|
|
These models are intended to be used as foundational models that can be e.g. instruction finetuned to serve as modern chat-models. |
|
|
|
|
|
|
|
**Parameters** |
|
| Model | Layers | Dim | Heads | Params | |
|
|--------|--------|------|-------|--------| |
|
| Small | 12 | 768 | 12 | 186M | |
|
| Medium | 24 | 1024 | 16 | 437M | |
|
| Large | 24 | 1536 | 16 | 881M | |
|
| XL | 24 | 2064 | 24 | 1.5B | |
|
| ”2.7B” | 32 | 2560 | 32 | 2.8B | |
|
| ”6.7B” | 32 | 4096 | 32 | 7.5B | |
|
| "13B" | 40 | 5120 | 40 | 13.3B | |
|
|
|
|
|
|
|
More documentation coming soon! |
|
|
