File size: 17,374 Bytes
f8bd133 8d544d9 f8bd133 8d544d9 f8bd133 8d544d9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 |
---
language:
- fi
license: apache-2.0
tags:
- finnish
- llama
datasets:
- Finnish-NLP/mc4_3.1.0_fi_cleaned
- Finnish-NLP/oscar_2301_fi_cleaned
- Finnish-NLP/Reddit_fi_2006_2022
- Finnish-NLP/wikipedia_20230501_fi_cleaned
- intfloat/multilingual_cc_news
inference: false
pipeline_tag: text-generation
---
# Llama-7b for Finnish
Pretrained Llama model on Finnish language using a causal language modeling (CLM) objective. Llama model was introduced in
[this paper](https://arxiv.org/abs/2302.13971)
and first released at [this page](https://github.com/facebookresearch/llama).
There are two different sized Llama models, all pretrained from scratch for 131B tokens:
| Model | Context length | Layers | Dim | Heads | Params |
|---------------------------------------------------------------------------------|----------------|--------|------|-------|--------|
| [llama-3b-finnish](https://huggingface.co/Finnish-NLP/llama-3b-finnish) | 2048 | 26 | 3200 | 32 | 3.6B |
| [llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish) | 2048 | 32 | 4096 | 32 | 7.0B |
## Model description
Finnish Llama is a GPT-style model pretrained on a very large corpus of Finnish data in a self-supervised fashion. This
means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots
of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely,
it was trained to predict the next word in sentences.
More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence,
shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the
predictions for the token `i` only uses the inputs from `1` to `i` but not the future tokens.
This way, the model learns an inner representation of the Finnish language that can then be used to extract features
useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a
prompt.
## Intended uses & limitations
This model was only pretrained in a self-supervised way excluding any supervised training. You can use the this model for text generation or fine-tune it to a downstream task, like instruction following. Fine-tuning is recommended to get better results.
### How to use
Here is an example of using this model in PyTorch with some generation arguments you can modify:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Finnish-NLP/llama-7b-finnish")
model = AutoModelForCausalLM.from_pretrained("Finnish-NLP/llama-7b-finnish")
inputs = tokenizer("Olipa kerran tekoälylaama", return_tensors="pt")
generated_ids = model.generate(inputs.input_ids, **{
"temperature": 0.8,
"penalty_alpha": 0.6,
"top_k": 4,
"do_sample": True,
"repetition_penalty": 1.2,
"min_length": 20,
"max_length": 2048
})
generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
```
### Limitations and bias
The training data used for this model contains a lot of content from the internet, which is far from neutral. Therefore, the model can have biased predictions. This bias will also affect all fine-tuned versions of this model.
To reduce toxic content, training data was filtered with a toxicity classifier but it cannot truly eliminate all toxic text.
## Training data
This Finnish Llama model was pretrained on the combination of 12 datasets:
- [mc4_3.1.0_fi_cleaned](https://huggingface.co/datasets/Finnish-NLP/mc4_3.1.0_fi_cleaned), mC4 is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the mC4 version 3.1.0 dataset and further cleaned it with our own text data cleaning codes (check the dataset repo).
- [oscar_2301_fi_cleaned](https://huggingface.co/datasets/Finnish-NLP/oscar_2301_fi_cleaned), Oscar is a multilingual, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the Oscar version 23.01 dataset and further cleaned it with our own text data cleaning codes.
- [wikipedia_20230501_fi_cleaned](https://huggingface.co/datasets/Finnish-NLP/wikipedia_20230501_fi_cleaned), we used the Finnish subset of the wikipedia (May 2023) dataset
- [Reddit_fi_2006_2022](https://huggingface.co/datasets/Finnish-NLP/Reddit_fi_2006_2022), filtered and post-processed dataset of Finnish Reddit
- [Yle Finnish News Archive 2011-2018](http://urn.fi/urn:nbn:fi:lb-2017070501)
- [Yle Finnish News Archive 2019-2020](http://urn.fi/urn:nbn:fi:lb-2021050401)
- [Finnish News Agency Archive (STT)](http://urn.fi/urn:nbn:fi:lb-2018121001)
- [The Suomi24 Sentences Corpus](http://urn.fi/urn:nbn:fi:lb-2020021803)
- [Project Lönnrot](http://www.lonnrot.net/)
- [Finnish parliament speeches](https://avoindata.eduskunta.fi)
- [multilingual_cc_news](https://huggingface.co/datasets/intfloat/multilingual_cc_news), we used the Finnish subset of the multilingual CC-News dataset
- [fi-news-corpus](https://github.com/nkrusch/fi-news-corpus)
Raw datasets were automatically cleaned to filter out bad quality and non-Finnish examples. Also, a [perplexity](https://huggingface.co/course/chapter7/3#perplexity-for-language-models) score was calculated for all texts with a KenLM model which was trained with very clean Finnish texts only. This perplexity score can then be used to determine how "clean" Finnish language the text contains. To reduce toxic text, we used Finnish toxicity classifier [TurkuNLP/bert-large-finnish-cased-toxicity](https://huggingface.co/TurkuNLP/bert-large-finnish-cased-toxicity) released by TurkuNLP to classify all text examples. Classified toxicity label scores can then be used to determine how toxic the text is.
All datasets were concatenated and the whole dataset deduplicated. Top 95% perplexity score was used as a filtering threshold to filter out the worst quality 5% of texts. To reduce amount of toxic content, the dataset was filtered to include text examples having lower than 80% score for the toxicity labels "label_identity_attack", "label_insult", "label_threat" and "label_severe_toxicity".
Finally, 20,000 text examples from each of the Wikipedia, Yle, STT, Suomi24, and Reddit datasets were randomly selected for evaluation dataset in addition to the existing validation split of the [mc4_3.1.0_fi_cleaned](https://huggingface.co/datasets/Finnish-NLP/mc4_3.1.0_fi_cleaned) dataset.
The final training dataset had 19 billion words and the evaluation dataset had 25 million words. After tokenization, the training dataset had 28 billion tokens and the evaluation dataset had 38 million tokens. Training dataset is divided between different datasets as follows:
|Dataset | Words | Ratio |
|------------------------------|-------------|-------------|
|mc4_3.1.0_fi_cleaned | 11.462B | 60.7\% |
|oscar_2301_fi_cleaned | 3.295B | 17.4\% |
|Suomi24 | 3.045B | 16.1\% |
|multilingual_cc_news | 0.295B | 1.6\% |
|STT | 0.249B | 1.3\% |
|Yle | 0.201B | 1.1\% |
|Reddit_fi_2006_2022 | 0.138B | 0.7\% |
|wikipedia_20230501_fi_cleaned | 0.096B | 0.5\% |
|Project Lönnrot | 0.078B | 0.4\% |
|Finnish parliament speeches | 0.021B | 0.1\% |
|fi-news-corpus | 0.004B | 0.1\% |
|**TOTAL** | **18.884B** | **100.0\%** |
## Training procedure
### Preprocessing
Texts are tokenized using Byte Pair Encoding (BPE) based on sentencepiece with a vocabulary size of 64,256. Inputs are sequences of 2048 consecutive tokens. Texts are not lower cased so this model is case-sensitive: it makes a difference between finnish and Finnish.
### Pretraining
The model was trained on TPUv4-32 VM, sponsored by the [Google TPU Research Cloud](https://sites.research.google/trc/about/). The model was trained for 131B tokens which is about 4.7 epochs of data, inspired by the findings of [this paper](https://arxiv.org/abs/2305.16264). The optimizer used was a [Lion](https://arxiv.org/abs/2302.06675), linear learning rate warmup for 2K steps to a peak learning rate of 3e-5, and then cosine decay to 3e-6.
Training was conducted with a slightly modified Jax/Flax based [EasyLM](https://github.com/young-geng/EasyLM) framework, and inspired by the [OpenLLaMA](https://github.com/openlm-research/open_llama) project.
## Evaluation results
This model was evaluated using [FIN-bench by TurkuNLP](https://github.com/TurkuNLP/FIN-bench) with zero-shot setting, but the evaluation script had some problems running succesfully, so the results reported below should perhaps be viewed with some caution.
[llama-3b-finnish](https://huggingface.co/Finnish-NLP/llama-3b-finnish):
| Task |Version| Metric |Value | |Stderr|
|------------------------------------------------|------:|---------------------|-----:|---|-----:|
|bigbench_analogies | 0|multiple_choice_grade|0.3308|± |0.0414|
|bigbench_arithmetic_1_digit_addition | 0|multiple_choice_grade|0.2000|± |0.0402|
|bigbench_arithmetic_1_digit_division | 0|multiple_choice_grade|0.3478|± |0.1015|
|bigbench_arithmetic_1_digit_multiplication | 0|multiple_choice_grade|0.2500|± |0.0435|
|bigbench_arithmetic_1_digit_subtraction | 0|multiple_choice_grade|0.1800|± |0.0386|
|bigbench_arithmetic_2_digit_addition | 0|multiple_choice_grade|0.0900|± |0.0288|
|bigbench_arithmetic_2_digit_division | 0|multiple_choice_grade|0.2200|± |0.0416|
|bigbench_arithmetic_2_digit_multiplication | 0|multiple_choice_grade|0.0100|± |0.0100|
|bigbench_arithmetic_2_digit_subtraction | 0|multiple_choice_grade|0.0900|± |0.0288|
|bigbench_arithmetic_3_digit_addition | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_3_digit_division | 0|multiple_choice_grade|0.1500|± |0.0359|
|bigbench_arithmetic_3_digit_multiplication | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_3_digit_subtraction | 0|multiple_choice_grade|0.0100|± |0.0100|
|bigbench_arithmetic_4_digit_addition | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_4_digit_division | 0|multiple_choice_grade|0.1500|± |0.0359|
|bigbench_arithmetic_4_digit_multiplication | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_4_digit_subtraction | 0|multiple_choice_grade|0.0100|± |0.0100|
|bigbench_arithmetic_5_digit_addition | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_5_digit_division | 0|multiple_choice_grade|0.0900|± |0.0288|
|bigbench_arithmetic_5_digit_multiplication | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_5_digit_subtraction | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_cause_and_effect_one_sentence | 0|multiple_choice_grade|0.4510|± |0.0704|
|bigbench_cause_and_effect_one_sentence_no_prompt| 0|multiple_choice_grade|0.7255|± |0.0631|
|bigbench_cause_and_effect_two_sentences | 0|multiple_choice_grade|0.5294|± |0.0706|
|bigbench_emotions | 0|multiple_choice_grade|0.2437|± |0.0340|
|bigbench_empirical_judgments | 0|multiple_choice_grade|0.3232|± |0.0472|
|bigbench_general_knowledge | 0|multiple_choice_grade|0.1286|± |0.0403|
|bigbench_hhh_alignment_harmless | 0|multiple_choice_grade|0.3966|± |0.0648|
|bigbench_hhh_alignment_helpful | 0|multiple_choice_grade|0.3051|± |0.0605|
|bigbench_hhh_alignment_honest | 0|multiple_choice_grade|0.4068|± |0.0645|
|bigbench_hhh_alignment_other | 0|multiple_choice_grade|0.5814|± |0.0761|
|bigbench_intent_recognition | 0|multiple_choice_grade|0.0838|± |0.0105|
|bigbench_misconceptions | 0|multiple_choice_grade|0.5373|± |0.0432|
|bigbench_paraphrase | 0|multiple_choice_grade|0.4900|± |0.0354|
|bigbench_sentence_ambiguity | 0|multiple_choice_grade|0.5500|± |0.0648|
|bigbench_similarities_abstraction | 0|multiple_choice_grade|0.3553|± |0.0553|
[llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish):
| Task |Version| Metric |Value | |Stderr|
|------------------------------------------------|------:|---------------------|-----:|---|-----:|
|bigbench_analogies | 0|multiple_choice_grade|0.2692|± |0.0391|
|bigbench_arithmetic_1_digit_addition | 0|multiple_choice_grade|0.2600|± |0.0441|
|bigbench_arithmetic_1_digit_division | 0|multiple_choice_grade|0.3043|± |0.0981|
|bigbench_arithmetic_1_digit_multiplication | 0|multiple_choice_grade|0.2100|± |0.0409|
|bigbench_arithmetic_1_digit_subtraction | 0|multiple_choice_grade|0.1300|± |0.0338|
|bigbench_arithmetic_2_digit_addition | 0|multiple_choice_grade|0.0800|± |0.0273|
|bigbench_arithmetic_2_digit_division | 0|multiple_choice_grade|0.2400|± |0.0429|
|bigbench_arithmetic_2_digit_multiplication | 0|multiple_choice_grade|0.0300|± |0.0171|
|bigbench_arithmetic_2_digit_subtraction | 0|multiple_choice_grade|0.1100|± |0.0314|
|bigbench_arithmetic_3_digit_addition | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_3_digit_division | 0|multiple_choice_grade|0.1300|± |0.0338|
|bigbench_arithmetic_3_digit_multiplication | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_3_digit_subtraction | 0|multiple_choice_grade|0.0300|± |0.0171|
|bigbench_arithmetic_4_digit_addition | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_4_digit_division | 0|multiple_choice_grade|0.1300|± |0.0338|
|bigbench_arithmetic_4_digit_multiplication | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_4_digit_subtraction | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_5_digit_addition | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_5_digit_division | 0|multiple_choice_grade|0.0800|± |0.0273|
|bigbench_arithmetic_5_digit_multiplication | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_arithmetic_5_digit_subtraction | 0|multiple_choice_grade|0.0000|± |0.0000|
|bigbench_cause_and_effect_one_sentence | 0|multiple_choice_grade|0.4510|± |0.0704|
|bigbench_cause_and_effect_one_sentence_no_prompt| 0|multiple_choice_grade|0.7647|± |0.0600|
|bigbench_cause_and_effect_two_sentences | 0|multiple_choice_grade|0.5098|± |0.0707|
|bigbench_emotions | 0|multiple_choice_grade|0.3688|± |0.0383|
|bigbench_empirical_judgments | 0|multiple_choice_grade|0.3434|± |0.0480|
|bigbench_general_knowledge | 0|multiple_choice_grade|0.1429|± |0.0421|
|bigbench_hhh_alignment_harmless | 0|multiple_choice_grade|0.3793|± |0.0643|
|bigbench_hhh_alignment_helpful | 0|multiple_choice_grade|0.3051|± |0.0605|
|bigbench_hhh_alignment_honest | 0|multiple_choice_grade|0.4068|± |0.0645|
|bigbench_hhh_alignment_other | 0|multiple_choice_grade|0.5814|± |0.0761|
|bigbench_intent_recognition | 0|multiple_choice_grade|0.1850|± |0.0148|
|bigbench_misconceptions | 0|multiple_choice_grade|0.5373|± |0.0432|
|bigbench_paraphrase | 0|multiple_choice_grade|0.5000|± |0.0354|
|bigbench_sentence_ambiguity | 0|multiple_choice_grade|0.5333|± |0.0649|
|bigbench_similarities_abstraction | 0|multiple_choice_grade|0.4474|± |0.0574|
## Acknowledgements
This project would not have been possible without compute generously provided by Google through the
[TPU Research Cloud](https://sites.research.google/trc/).
## Team Members
- Aapo Tanskanen, [Hugging Face profile](https://huggingface.co/aapot), [LinkedIn profile](https://www.linkedin.com/in/aapotanskanen/)
- Rasmus Toivanen, [Hugging Face profile](https://huggingface.co/RASMUS), [LinkedIn profile](https://www.linkedin.com/in/rasmustoivanen/)
Feel free to contact us for more details 🤗 |