Hugging Face's logo

Finnish-NLP
/
llama-7b-finnish

Text Generation
Transformers
PyTorch
Safetensors
Finnish
llama
finnish
text-generation-inference
Model card Files Community
1
llama-7b-finnish / README.md
aapot's picture
aapot
Update README.md
80a4ab9 verified
preview code
|
raw
history blame
17.6 kB
metadata
language:
  - fi
license: apache-2.0
tags:
  - finnish
  - llama
datasets:
  - Finnish-NLP/mc4_3.1.0_fi_cleaned
  - Finnish-NLP/oscar_2301_fi_cleaned
  - Finnish-NLP/Reddit_fi_2006_2022
  - Finnish-NLP/wikipedia_20230501_fi_cleaned
  - intfloat/multilingual_cc_news
inference: false
pipeline_tag: text-generation

Llama-7b for Finnish

Pretrained Llama model on Finnish language using a causal language modeling (CLM) objective. Llama model was introduced in this paper and first released at this page.

There are two different sized Llama models, all pretrained from scratch for 131B tokens:

Model Context length Layers Dim Heads Params
llama-3b-finnish 2048 26 3200 32 3.6B
llama-7b-finnish 2048 32 4096 32 7.0B

Note: this model is pretrained, not fine-tuned for instruction following or chat use. For instruction tuned models, you can check models listed in this collection: https://huggingface.co/collections/Finnish-NLP/instruction-tuned-models-65c08db79fa020161be2e942

Model description

Finnish Llama is a GPT-style model pretrained on a very large corpus of Finnish data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it was trained to predict the next word in sentences.

More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence, shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the predictions for the token i only uses the inputs from 1 to i but not the future tokens.

This way, the model learns an inner representation of the Finnish language that can then be used to extract features useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a prompt.

Intended uses & limitations

This model was only pretrained in a self-supervised way excluding any supervised training. You can use the this model for text generation or fine-tune it to a downstream task, like instruction following. Fine-tuning is recommended to get better results.

How to use

Here is an example of using this model in PyTorch with some generation arguments you can modify:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Finnish-NLP/llama-7b-finnish")
model = AutoModelForCausalLM.from_pretrained("Finnish-NLP/llama-7b-finnish")

inputs = tokenizer("Olipa kerran tekoälylaama", return_tensors="pt")
generated_ids = model.generate(inputs.input_ids, **{
      "temperature": 0.8,
      "penalty_alpha": 0.6,
      "top_k": 4,
      "do_sample": True,
      "repetition_penalty": 1.2,
      "min_length": 20,
      "max_length": 2048
    })
generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

Limitations and bias

The training data used for this model contains a lot of content from the internet, which is far from neutral. Therefore, the model can have biased predictions. This bias will also affect all fine-tuned versions of this model.

To reduce toxic content, training data was filtered with a toxicity classifier but it cannot truly eliminate all toxic text.

Training data

This Finnish Llama model was pretrained on the combination of 12 datasets:

Raw datasets were automatically cleaned to filter out bad quality and non-Finnish examples. Also, a perplexity score was calculated for all texts with a KenLM model which was trained with very clean Finnish texts only. This perplexity score can then be used to determine how "clean" Finnish language the text contains. To reduce toxic text, we used Finnish toxicity classifier TurkuNLP/bert-large-finnish-cased-toxicity released by TurkuNLP to classify all text examples. Classified toxicity label scores can then be used to determine how toxic the text is.

All datasets were concatenated and the whole dataset deduplicated. Top 95% perplexity score was used as a filtering threshold to filter out the worst quality 5% of texts. To reduce amount of toxic content, the dataset was filtered to include text examples having lower than 80% score for the toxicity labels "label_identity_attack", "label_insult", "label_threat" and "label_severe_toxicity".

Finally, 20,000 text examples from each of the Wikipedia, Yle, STT, Suomi24, and Reddit datasets were randomly selected for evaluation dataset in addition to the existing validation split of the mc4_3.1.0_fi_cleaned dataset.

The final training dataset had 19 billion words and the evaluation dataset had 25 million words. After tokenization, the training dataset had 28 billion tokens and the evaluation dataset had 38 million tokens. Training dataset is divided between different datasets as follows:

Dataset Words Ratio
mc4_3.1.0_fi_cleaned 11.462B 60.7%
oscar_2301_fi_cleaned 3.295B 17.4%
Suomi24 3.045B 16.1%
multilingual_cc_news 0.295B 1.6%
STT 0.249B 1.3%
Yle 0.201B 1.1%
Reddit_fi_2006_2022 0.138B 0.7%
wikipedia_20230501_fi_cleaned 0.096B 0.5%
Project Lönnrot 0.078B 0.4%
Finnish parliament speeches 0.021B 0.1%
fi-news-corpus 0.004B 0.1%
TOTAL 18.884B 100.0%

Training procedure

Preprocessing

Texts are tokenized using Byte Pair Encoding (BPE) based on sentencepiece with a vocabulary size of 64,256. Inputs are sequences of 2048 consecutive tokens. Texts are not lower cased so this model is case-sensitive: it makes a difference between finnish and Finnish.

Pretraining

The model was trained on TPUv4-32 VM, sponsored by the Google TPU Research Cloud. The model was trained for 131B tokens which is about 4.7 epochs of data, inspired by the findings of this paper. The optimizer used was a Lion, linear learning rate warmup for 2K steps to a peak learning rate of 3e-5, and then cosine decay to 3e-6.

Training was conducted with a slightly modified Jax/Flax based EasyLM framework, and inspired by the OpenLLaMA project.

Evaluation results

This model was evaluated using FIN-bench by TurkuNLP with zero-shot setting, but the evaluation script had some problems running succesfully, so the results reported below should perhaps be viewed with some caution.

llama-3b-finnish:

Task Version Metric Value Stderr
bigbench_analogies 0 multiple_choice_grade 0.3308 ± 0.0414
bigbench_arithmetic_1_digit_addition 0 multiple_choice_grade 0.2000 ± 0.0402
bigbench_arithmetic_1_digit_division 0 multiple_choice_grade 0.3478 ± 0.1015
bigbench_arithmetic_1_digit_multiplication 0 multiple_choice_grade 0.2500 ± 0.0435
bigbench_arithmetic_1_digit_subtraction 0 multiple_choice_grade 0.1800 ± 0.0386
bigbench_arithmetic_2_digit_addition 0 multiple_choice_grade 0.0900 ± 0.0288
bigbench_arithmetic_2_digit_division 0 multiple_choice_grade 0.2200 ± 0.0416
bigbench_arithmetic_2_digit_multiplication 0 multiple_choice_grade 0.0100 ± 0.0100
bigbench_arithmetic_2_digit_subtraction 0 multiple_choice_grade 0.0900 ± 0.0288
bigbench_arithmetic_3_digit_addition 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_3_digit_division 0 multiple_choice_grade 0.1500 ± 0.0359
bigbench_arithmetic_3_digit_multiplication 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_3_digit_subtraction 0 multiple_choice_grade 0.0100 ± 0.0100
bigbench_arithmetic_4_digit_addition 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_4_digit_division 0 multiple_choice_grade 0.1500 ± 0.0359
bigbench_arithmetic_4_digit_multiplication 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_4_digit_subtraction 0 multiple_choice_grade 0.0100 ± 0.0100
bigbench_arithmetic_5_digit_addition 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_5_digit_division 0 multiple_choice_grade 0.0900 ± 0.0288
bigbench_arithmetic_5_digit_multiplication 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_5_digit_subtraction 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_cause_and_effect_one_sentence 0 multiple_choice_grade 0.4510 ± 0.0704
bigbench_cause_and_effect_one_sentence_no_prompt 0 multiple_choice_grade 0.7255 ± 0.0631
bigbench_cause_and_effect_two_sentences 0 multiple_choice_grade 0.5294 ± 0.0706
bigbench_emotions 0 multiple_choice_grade 0.2437 ± 0.0340
bigbench_empirical_judgments 0 multiple_choice_grade 0.3232 ± 0.0472
bigbench_general_knowledge 0 multiple_choice_grade 0.1286 ± 0.0403
bigbench_hhh_alignment_harmless 0 multiple_choice_grade 0.3966 ± 0.0648
bigbench_hhh_alignment_helpful 0 multiple_choice_grade 0.3051 ± 0.0605
bigbench_hhh_alignment_honest 0 multiple_choice_grade 0.4068 ± 0.0645
bigbench_hhh_alignment_other 0 multiple_choice_grade 0.5814 ± 0.0761
bigbench_intent_recognition 0 multiple_choice_grade 0.0838 ± 0.0105
bigbench_misconceptions 0 multiple_choice_grade 0.5373 ± 0.0432
bigbench_paraphrase 0 multiple_choice_grade 0.4900 ± 0.0354
bigbench_sentence_ambiguity 0 multiple_choice_grade 0.5500 ± 0.0648
bigbench_similarities_abstraction 0 multiple_choice_grade 0.3553 ± 0.0553

llama-7b-finnish:

Task Version Metric Value Stderr
bigbench_analogies 0 multiple_choice_grade 0.2692 ± 0.0391
bigbench_arithmetic_1_digit_addition 0 multiple_choice_grade 0.2600 ± 0.0441
bigbench_arithmetic_1_digit_division 0 multiple_choice_grade 0.3043 ± 0.0981
bigbench_arithmetic_1_digit_multiplication 0 multiple_choice_grade 0.2100 ± 0.0409
bigbench_arithmetic_1_digit_subtraction 0 multiple_choice_grade 0.1300 ± 0.0338
bigbench_arithmetic_2_digit_addition 0 multiple_choice_grade 0.0800 ± 0.0273
bigbench_arithmetic_2_digit_division 0 multiple_choice_grade 0.2400 ± 0.0429
bigbench_arithmetic_2_digit_multiplication 0 multiple_choice_grade 0.0300 ± 0.0171
bigbench_arithmetic_2_digit_subtraction 0 multiple_choice_grade 0.1100 ± 0.0314
bigbench_arithmetic_3_digit_addition 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_3_digit_division 0 multiple_choice_grade 0.1300 ± 0.0338
bigbench_arithmetic_3_digit_multiplication 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_3_digit_subtraction 0 multiple_choice_grade 0.0300 ± 0.0171
bigbench_arithmetic_4_digit_addition 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_4_digit_division 0 multiple_choice_grade 0.1300 ± 0.0338
bigbench_arithmetic_4_digit_multiplication 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_4_digit_subtraction 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_5_digit_addition 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_5_digit_division 0 multiple_choice_grade 0.0800 ± 0.0273
bigbench_arithmetic_5_digit_multiplication 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_arithmetic_5_digit_subtraction 0 multiple_choice_grade 0.0000 ± 0.0000
bigbench_cause_and_effect_one_sentence 0 multiple_choice_grade 0.4510 ± 0.0704
bigbench_cause_and_effect_one_sentence_no_prompt 0 multiple_choice_grade 0.7647 ± 0.0600
bigbench_cause_and_effect_two_sentences 0 multiple_choice_grade 0.5098 ± 0.0707
bigbench_emotions 0 multiple_choice_grade 0.3688 ± 0.0383
bigbench_empirical_judgments 0 multiple_choice_grade 0.3434 ± 0.0480
bigbench_general_knowledge 0 multiple_choice_grade 0.1429 ± 0.0421
bigbench_hhh_alignment_harmless 0 multiple_choice_grade 0.3793 ± 0.0643
bigbench_hhh_alignment_helpful 0 multiple_choice_grade 0.3051 ± 0.0605
bigbench_hhh_alignment_honest 0 multiple_choice_grade 0.4068 ± 0.0645
bigbench_hhh_alignment_other 0 multiple_choice_grade 0.5814 ± 0.0761
bigbench_intent_recognition 0 multiple_choice_grade 0.1850 ± 0.0148
bigbench_misconceptions 0 multiple_choice_grade 0.5373 ± 0.0432
bigbench_paraphrase 0 multiple_choice_grade 0.5000 ± 0.0354
bigbench_sentence_ambiguity 0 multiple_choice_grade 0.5333 ± 0.0649
bigbench_similarities_abstraction 0 multiple_choice_grade 0.4474 ± 0.0574

Acknowledgements

This project would not have been possible without compute generously provided by Google through the TPU Research Cloud.

Team Members

Feel free to contact us for more details 🤗