Update README.md
Browse files
README.md
CHANGED
@@ -3,5 +3,26 @@ language:
|
|
3 |
- fi
|
4 |
pipeline_tag: text-generation
|
5 |
---
|
|
|
6 |
|
7 |
-
GPT-3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
- fi
|
4 |
pipeline_tag: text-generation
|
5 |
---
|
6 |
+
Generative Pretrained Transformer with 8B parameteres for Finnish.
|
7 |
|
8 |
+
TurkuNLP Finnish GPT-3-models are a model family of pretrained monolingual GPT-style language models that are based on BLOOM-architecture.
|
9 |
+
Models are pure language models meaning that they are not [instruction finetuned](https://arxiv.org/abs/2203.02155) for dialogue or answering questions.
|
10 |
+
|
11 |
+
These models are intended to be used as foundational models that can be e.g. instruction finetuned to serve as modern chat-models.
|
12 |
+
|
13 |
+
|
14 |
+
|
15 |
+
**Parameters**
|
16 |
+
| Model | Layers | Dim | Heads | Params |
|
17 |
+
|--------|--------|------|-------|--------|
|
18 |
+
| Small | 12 | 768 | 12 | 186M |
|
19 |
+
| Medium | 24 | 1024 | 16 | 437M |
|
20 |
+
| Large | 24 | 1536 | 16 | 881M |
|
21 |
+
| XL | 24 | 2064 | 24 | 1.5B |
|
22 |
+
| ”2.7B” | 32 | 2560 | 32 | 2.8B |
|
23 |
+
| ”6.7B” | 32 | 4096 | 32 | 7.5B |
|
24 |
+
| "13B" | 40 | 5120 | 40 | 13.3B |
|
25 |
+
|
26 |
+
|
27 |
+
|
28 |
+
More documentation coming soon!
|