File size: 1,425 Bytes
2ba65e8 f397ab4 2bb66cb f397ab4 2ba65e8 f397ab4 2bb66cb f397ab4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
---
license: apache-2.0
language:
- ru, en
pipeline_tag: text-generation
tags:
- PyTorch
- Transformers
- gpt3
- gpt2
- Deepspeed
- Megatron
thumbnail: "https://github.com/sberbank-ai/mgpt"
---
# Multilingual GPT model
We introduce family of autoregressive GPT-like models with 1.3 billion parameters trained on 60 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus.
We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism, [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron]() frameworks allows us to effectively parallelize the training and inference steps. Resulting models show performance on par with the recently released [XGLM](https://arxiv.org/pdf/2112.10668.pdf) models at the same time covering more languages and enhance NLP possibilities for low resource languages.
## Code
The source code for the mGPT XL model is available on [Github](https://github.com/sberbank-ai/mgpt)
## Paper
[Arxiv preprint](https://arxiv.org/user)
Cite us:
```{
bibtex
}
```
## Languages
## Training Data Statistics
## Details
Model was trained with sequence length 1024 using transformers lib by [SberDevices](https://sberdevices.ru/) team on 80B tokens for 3 epochs. After that model was finetuned 1 epoch with sequence length 2048.
Total training time was around n days on n GPUs for n context and few days on n GPUs for n context.
|