add model card

README.md

@@ -1 +1,16 @@
-ruGPT3xl language model with sparse attention
+---
+language:
+- ru
+tags:
+- PyTorch
+- Transformers
+thumbnail: "https://github.com/sberbank-ai/ru-gpts"
+---
+
+# rugpt3xl
+Model was trained with 512 sequence length using [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron](https://github.com/NVIDIA/Megatron-LM) code by [SberDevices](https://sberdevices.ru/) team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.  
+*Note! Model has sparse attention blocks.*
+
+Total training time was around 10 days on 256 GPUs.  
+Final perplexity on test set is `12.05`.
+Model parameters: 1.3B.