Llama-3b for Finnish

Pretrained Llama model on Finnish language using a causal language modeling (CLM) objective. Llama model was introduced in this paper and first released at this page.

There are two different sized Llama models, all pretrained from scratch for 131B tokens:

Model	Context length	Layers	Dim	Heads	Params
llama-3b-finnish	2048	26	3200	32	3.6B
llama-7b-finnish	2048	32	4096	32	7.0B

Note: this model is pretrained, not fine-tuned for instruction following or chat use. For instruction tuned models, you can check models listed in this collection: https://huggingface.co/collections/Finnish-NLP/instruction-tuned-models-65c08db79fa020161be2e942

Model description

Finnish Llama is a GPT-style model pretrained on a very large corpus of Finnish data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it was trained to predict the next word in sentences.

More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence, shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the predictions for the token i only uses the inputs from 1 to i but not the future tokens.

This way, the model learns an inner representation of the Finnish language that can then be used to extract features useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a prompt.

Intended uses & limitations

This model was only pretrained in a self-supervised way excluding any supervised training. You can use the this model for text generation or fine-tune it to a downstream task, like instruction following. Fine-tuning is recommended to get better results.

How to use

Here is an example of using this model in PyTorch with some generation arguments you can modify:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Finnish-NLP/llama-3b-finnish")
model = AutoModelForCausalLM.from_pretrained("Finnish-NLP/llama-3b-finnish")

inputs = tokenizer("Olipa kerran tekoälylaama", return_tensors="pt")
generated_ids = model.generate(inputs.input_ids, **{
      "temperature": 0.8,
      "penalty_alpha": 0.6,
      "top_k": 4,
      "do_sample": True,
      "repetition_penalty": 1.2,
      "min_length": 20,
      "max_length": 2048
    })
generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

Limitations and bias

The training data used for this model contains a lot of content from the internet, which is far from neutral. Therefore, the model can have biased predictions. This bias will also affect all fine-tuned versions of this model.

To reduce toxic content, training data was filtered with a toxicity classifier but it cannot truly eliminate all toxic text.

Training data

This Finnish Llama model was pretrained on the combination of 12 datasets:

mc4_3.1.0_fi_cleaned, mC4 is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the mC4 version 3.1.0 dataset and further cleaned it with our own text data cleaning codes (check the dataset repo).
oscar_2301_fi_cleaned, Oscar is a multilingual, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the Oscar version 23.01 dataset and further cleaned it with our own text data cleaning codes.
wikipedia_20230501_fi_cleaned, we used the Finnish subset of the wikipedia (May 2023) dataset
Reddit_fi_2006_2022, filtered and post-processed dataset of Finnish Reddit
Yle Finnish News Archive 2011-2018
Yle Finnish News Archive 2019-2020
Finnish News Agency Archive (STT)
The Suomi24 Sentences Corpus
Project Lönnrot
Finnish parliament speeches
multilingual_cc_news, we used the Finnish subset of the multilingual CC-News dataset
fi-news-corpus

Raw datasets were automatically cleaned to filter out bad quality and non-Finnish examples. Also, a perplexity score was calculated for all texts with a KenLM model which was trained with very clean Finnish texts only. This perplexity score can then be used to determine how "clean" Finnish language the text contains. To reduce toxic text, we used Finnish toxicity classifier TurkuNLP/bert-large-finnish-cased-toxicity released by TurkuNLP to classify all text examples. Classified toxicity label scores can then be used to determine how toxic the text is.

All datasets were concatenated and the whole dataset deduplicated. Top 95% perplexity score was used as a filtering threshold to filter out the worst quality 5% of texts. To reduce amount of toxic content, the dataset was filtered to include text examples having lower than 80% score for the toxicity labels "label_identity_attack", "label_insult", "label_threat" and "label_severe_toxicity".

Finally, 20,000 text examples from each of the Wikipedia, Yle, STT, Suomi24, and Reddit datasets were randomly selected for evaluation dataset in addition to the existing validation split of the mc4_3.1.0_fi_cleaned dataset.

The final training dataset had 19 billion words and the evaluation dataset had 25 million words. After tokenization, the training dataset had 28 billion tokens and the evaluation dataset had 38 million tokens. Training dataset is divided between different datasets as follows:

Dataset	Words	Ratio
mc4_3.1.0_fi_cleaned	11.462B	60.7%
oscar_2301_fi_cleaned	3.295B	17.4%
Suomi24	3.045B	16.1%
multilingual_cc_news	0.295B	1.6%
STT	0.249B	1.3%
Yle	0.201B	1.1%
Reddit_fi_2006_2022	0.138B	0.7%
wikipedia_20230501_fi_cleaned	0.096B	0.5%
Project Lönnrot	0.078B	0.4%
Finnish parliament speeches	0.021B	0.1%
fi-news-corpus	0.004B	0.1%
TOTAL	18.884B	100.0%

Training procedure

Preprocessing

Texts are tokenized using Byte Pair Encoding (BPE) based on sentencepiece with a vocabulary size of 64,256. Inputs are sequences of 2048 consecutive tokens. Texts are not lower cased so this model is case-sensitive: it makes a difference between finnish and Finnish.

Pretraining

The model was trained on TPUv4-32 VM, sponsored by the Google TPU Research Cloud. The model was trained for 131B tokens which is about 4.7 epochs of data, inspired by the findings of this paper. The optimizer used was a Lion, linear learning rate warmup for 2K steps to a peak learning rate of 3e-5, and then cosine decay to 3e-6.

Training was conducted with a slightly modified Jax/Flax based EasyLM framework, and inspired by the OpenLLaMA project.

Evaluation results

This model was evaluated using FIN-bench by TurkuNLP with zero-shot setting, but the evaluation script had some problems running succesfully, so the results reported below should perhaps be viewed with some caution.

llama-3b-finnish:

Task	Metric	Value		Stderr
bigbench_analogies	multiple_choice_grade	0.3308	±	0.0414
bigbench_arithmetic_1_digit_addition	multiple_choice_grade	0.2000	±	0.0402
bigbench_arithmetic_1_digit_division	multiple_choice_grade	0.3478	±	0.1015
bigbench_arithmetic_1_digit_multiplication	multiple_choice_grade	0.2500	±	0.0435
bigbench_arithmetic_1_digit_subtraction	multiple_choice_grade	0.1800	±	0.0386
bigbench_arithmetic_2_digit_addition	multiple_choice_grade	0.0900	±	0.0288
bigbench_arithmetic_2_digit_division	multiple_choice_grade	0.2200	±	0.0416
bigbench_arithmetic_2_digit_multiplication	multiple_choice_grade	0.0100	±	0.0100
bigbench_arithmetic_2_digit_subtraction	multiple_choice_grade	0.0900	±	0.0288
bigbench_arithmetic_3_digit_addition	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_3_digit_division	multiple_choice_grade	0.1500	±	0.0359
bigbench_arithmetic_3_digit_multiplication	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_3_digit_subtraction	multiple_choice_grade	0.0100	±	0.0100
bigbench_arithmetic_4_digit_addition	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_4_digit_division	multiple_choice_grade	0.1500	±	0.0359
bigbench_arithmetic_4_digit_multiplication	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_4_digit_subtraction	multiple_choice_grade	0.0100	±	0.0100
bigbench_arithmetic_5_digit_addition	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_5_digit_division	multiple_choice_grade	0.0900	±	0.0288
bigbench_arithmetic_5_digit_multiplication	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_5_digit_subtraction	multiple_choice_grade	0.0000	±	0.0000
bigbench_cause_and_effect_one_sentence	multiple_choice_grade	0.4510	±	0.0704
bigbench_cause_and_effect_one_sentence_no_prompt	multiple_choice_grade	0.7255	±	0.0631
bigbench_cause_and_effect_two_sentences	multiple_choice_grade	0.5294	±	0.0706
bigbench_emotions	multiple_choice_grade	0.2437	±	0.0340
bigbench_empirical_judgments	multiple_choice_grade	0.3232	±	0.0472
bigbench_general_knowledge	multiple_choice_grade	0.1286	±	0.0403
bigbench_hhh_alignment_harmless	multiple_choice_grade	0.3966	±	0.0648
bigbench_hhh_alignment_helpful	multiple_choice_grade	0.3051	±	0.0605
bigbench_hhh_alignment_honest	multiple_choice_grade	0.4068	±	0.0645
bigbench_hhh_alignment_other	multiple_choice_grade	0.5814	±	0.0761
bigbench_intent_recognition	multiple_choice_grade	0.0838	±	0.0105
bigbench_misconceptions	multiple_choice_grade	0.5373	±	0.0432
bigbench_paraphrase	multiple_choice_grade	0.4900	±	0.0354
bigbench_sentence_ambiguity	multiple_choice_grade	0.5500	±	0.0648
bigbench_similarities_abstraction	multiple_choice_grade	0.3553	±	0.0553

llama-7b-finnish:

Task	Metric	Value		Stderr
bigbench_analogies	multiple_choice_grade	0.2692	±	0.0391
bigbench_arithmetic_1_digit_addition	multiple_choice_grade	0.2600	±	0.0441
bigbench_arithmetic_1_digit_division	multiple_choice_grade	0.3043	±	0.0981
bigbench_arithmetic_1_digit_multiplication	multiple_choice_grade	0.2100	±	0.0409
bigbench_arithmetic_1_digit_subtraction	multiple_choice_grade	0.1300	±	0.0338
bigbench_arithmetic_2_digit_addition	multiple_choice_grade	0.0800	±	0.0273
bigbench_arithmetic_2_digit_division	multiple_choice_grade	0.2400	±	0.0429
bigbench_arithmetic_2_digit_multiplication	multiple_choice_grade	0.0300	±	0.0171
bigbench_arithmetic_2_digit_subtraction	multiple_choice_grade	0.1100	±	0.0314
bigbench_arithmetic_3_digit_addition	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_3_digit_division	multiple_choice_grade	0.1300	±	0.0338
bigbench_arithmetic_3_digit_multiplication	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_3_digit_subtraction	multiple_choice_grade	0.0300	±	0.0171
bigbench_arithmetic_4_digit_addition	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_4_digit_division	multiple_choice_grade	0.1300	±	0.0338
bigbench_arithmetic_4_digit_multiplication	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_4_digit_subtraction	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_5_digit_addition	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_5_digit_division	multiple_choice_grade	0.0800	±	0.0273
bigbench_arithmetic_5_digit_multiplication	multiple_choice_grade	0.0000	±	0.0000
bigbench_arithmetic_5_digit_subtraction	multiple_choice_grade	0.0000	±	0.0000
bigbench_cause_and_effect_one_sentence	multiple_choice_grade	0.4510	±	0.0704
bigbench_cause_and_effect_one_sentence_no_prompt	multiple_choice_grade	0.7647	±	0.0600
bigbench_cause_and_effect_two_sentences	multiple_choice_grade	0.5098	±	0.0707
bigbench_emotions	multiple_choice_grade	0.3688	±	0.0383
bigbench_empirical_judgments	multiple_choice_grade	0.3434	±	0.0480
bigbench_general_knowledge	multiple_choice_grade	0.1429	±	0.0421
bigbench_hhh_alignment_harmless	multiple_choice_grade	0.3793	±	0.0643
bigbench_hhh_alignment_helpful	multiple_choice_grade	0.3051	±	0.0605
bigbench_hhh_alignment_honest	multiple_choice_grade	0.4068	±	0.0645
bigbench_hhh_alignment_other	multiple_choice_grade	0.5814	±	0.0761
bigbench_intent_recognition	multiple_choice_grade	0.1850	±	0.0148
bigbench_misconceptions	multiple_choice_grade	0.5373	±	0.0432
bigbench_paraphrase	multiple_choice_grade	0.5000	±	0.0354
bigbench_sentence_ambiguity	multiple_choice_grade	0.5333	±	0.0649
bigbench_similarities_abstraction	multiple_choice_grade	0.4474	±	0.0574

Acknowledgements

This project would not have been possible without compute generously provided by Google through the TPU Research Cloud.

Team Members

Aapo Tanskanen, Hugging Face profile, LinkedIn profile
Rasmus Toivanen, Hugging Face profile, LinkedIn profile

Feel free to contact us for more details 🤗

Finnish-NLP
/

llama-3b-finnish