rluukkon commited on
Commit
6038656
1 Parent(s): b490fd9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +22 -1
README.md CHANGED
@@ -3,5 +3,26 @@ language:
3
  - fi
4
  pipeline_tag: text-generation
5
  ---
 
6
 
7
- GPT-3 8B for Finnish. Documentation coming soon!
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  - fi
4
  pipeline_tag: text-generation
5
  ---
6
+ Generative Pretrained Transformer with 8B parameteres for Finnish.
7
 
8
+ TurkuNLP Finnish GPT-3-models are a model family of pretrained monolingual GPT-style language models that are based on BLOOM-architecture.
9
+ Models are pure language models meaning that they are not [instruction finetuned](https://arxiv.org/abs/2203.02155) for dialogue or answering questions.
10
+
11
+ These models are intended to be used as foundational models that can be e.g. instruction finetuned to serve as modern chat-models.
12
+
13
+
14
+
15
+ **Parameters**
16
+ | Model | Layers | Dim | Heads | Params |
17
+ |--------|--------|------|-------|--------|
18
+ | Small | 12 | 768 | 12 | 186M |
19
+ | Medium | 24 | 1024 | 16 | 437M |
20
+ | Large | 24 | 1536 | 16 | 881M |
21
+ | XL | 24 | 2064 | 24 | 1.5B |
22
+ | ”2.7B” | 32 | 2560 | 32 | 2.8B |
23
+ | ”6.7B” | 32 | 4096 | 32 | 7.5B |
24
+ | "13B" | 40 | 5120 | 40 | 13.3B |
25
+
26
+
27
+
28
+ More documentation coming soon!