|
--- |
|
language: |
|
- ru |
|
tags: |
|
- PyTorch |
|
- Transformers |
|
thumbnail: "https://github.com/sberbank-ai/ru-gpts" |
|
--- |
|
|
|
# rugpt3xl |
|
Model was trained with 512 sequence length using [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron](https://github.com/NVIDIA/Megatron-LM) code by [SberDevices](https://sberdevices.ru/) team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048. |
|
*Note! Model has sparse attention blocks.* |
|
|
|
Total training time was around 10 days on 256 GPUs. |
|
Final perplexity on test set is `12.05`. |
|
Model parameters: 1.3B. |
|
|
