datasets:
- oscar
language:
- da
widget:
- text: Der var engang
What is this?
A GPT-2 model (small version, 124 M parameters) for Danish text generation. The model was not pre-trained from scratch but adapted from the English version.
How to use
Test the model using the pipeline from the 🤗 Transformers library:
from transformers import pipeline
generator = pipeline("text-generation", model = "KennethTM/gpt2-small-danish")
text = generator("Manden arbejdede som")
print(text[0]["generated_text"])
Or load it using the Auto* classes:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("KennethTM/gpt2-small-danish")
model = AutoModelForCausalLM.from_pretrained("KennethTM/gpt2-small-danish")
Model training
The model is trained using the Danish part of the oscar dataset ('unshuffled_deduplicated_da') and a context length of 1024 tokens.
The model is initialized from the English GPT-2 small model with new word token embeddings created for Danish using WECHSEL.
Initially, only the word token embeddings are trained using 50.000 samples. Finally, the whole model is trained using 1.000.000 samples.
Model training is carried out on an 8 GB GPU.
Notes
This is a pre-trained model, for optimal performance it should be finetuned for new tasks.