rugpt3xl / README.md
mrrano's picture
Update README.md
e24bc52
|
raw
history blame
1.15 kB
metadata
language:
  - ru
tags:
  - PyTorch
  - Transformers
thumbnail: https://github.com/sberbank-ai/ru-gpts

rugpt3xl

Model was trained with 512 sequence length using Deepspeed and Megatron code by SberDevices team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.
Note! Model has sparse attention blocks.

Total training time was around 10 days on 256 GPUs.
Final perplexity on test set is 12.05. Model parameters: 1.3B. from transformers import GPT2LMHeadModel, GPT2Tokenizer model_name_or_path = "sberbank-ai/rugpt3large_based_on_gpt2" (можно использовать sberbank-ai/rugpt3xl) tokenizer = GPT2Tokenizer.from_pretrained(model_name_or_path) model = GPT2LMHeadModel.from_pretrained(model_name_or_path).cpu() text = "Иисус Христос родился в " input_ids = tokenizer.encode(text, return_tensors="pt").cpu() out = model.generate(input_ids.cpu()) print(generated_text) generated_text = list(map(tokenizer.decode, out))[0] print(generated_text)