metadata
language:
- fi
pipeline_tag: text-generation
Generative Pretrained Transformer with 8B parameteres for Finnish.
TurkuNLP Finnish GPT-3-models are a model family of pretrained monolingual GPT-style language models that are based on BLOOM-architecture. Models are pure language models meaning that they are not instruction finetuned for dialogue or answering questions.
These models are intended to be used as foundational models that can be e.g. instruction finetuned to serve as modern chat-models.
Parameters
| Model | Layers | Dim | Heads | Params |
|---|---|---|---|---|
| Small | 12 | 768 | 12 | 186M |
| Medium | 24 | 1024 | 16 | 437M |
| Large | 24 | 1536 | 16 | 881M |
| XL | 24 | 2064 | 24 | 1.5B |
| ”2.7B” | 32 | 2560 | 32 | 2.8B |
| ”6.7B” | 32 | 4096 | 32 | 7.5B |
| "13B" | 40 | 5120 | 40 | 13.3B |
More documentation coming soon!