File size: 593 Bytes
c3c2888 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
---
language:
- ru
tags:
- PyTorch
- Transformers
thumbnail: "https://github.com/sberbank-ai/ru-gpts"
---
# rugpt3xl
Model was trained with 512 sequence length using [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron](https://github.com/NVIDIA/Megatron-LM) code by [SberDevices](https://sberdevices.ru/) team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.
*Note! Model has sparse attention blocks.*
Total training time was around 10 days on 256 GPUs.
Final perplexity on test set is `12.05`.
Model parameters: 1.3B.
|