language:
- ru
tags:
- PyTorch
- Transformers
thumbnail: https://github.com/sberbank-ai/ru-gpts
rugpt3xl
Model was trained with 512 sequence length using Deepspeed and Megatron code by SberDevices team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.
Note! Model has sparse attention blocks.
Total training time was around 10 days on 256 GPUs.
Final perplexity on test set is 12.05.
Model parameters: 1.3B.
from transformers import GPT2LMHeadModel, GPT2Tokenizer
model_name_or_path = "sberbank-ai/rugpt3large_based_on_gpt2" (можно использовать sberbank-ai/rugpt3xl)
tokenizer = GPT2Tokenizer.from_pretrained(model_name_or_path)
model = GPT2LMHeadModel.from_pretrained(model_name_or_path).cpu()
text = "Иисус Христос родился в "
input_ids = tokenizer.encode(text, return_tensors="pt").cpu()
out = model.generate(input_ids.cpu())
print(generated_text)
generated_text = list(map(tokenizer.decode, out))[0]
print(generated_text)