| # mGPT | |
| mGPT is pre-trained on the [mC4 dataset](https://huggingface.co/datasets/mc4) using a causal language modeling objective. It was introduced in this [paper](https://arxiv.org/abs/2110.06609) and first released on this page. | |
| ## Model description | |
| mGPT is a Transformer-based model which pre-trained on massive multilingual data covering over 101 languages. Similar to GPT-2, It was pre-trained on the raw texts only, with no human labeling. We use the same tokenization and vocabulary as the [mT5 model](https://huggingface.co/google/mt5-base). | |
| ## Intended uses | |
| You can use the raw model for text generation or using prompts for adapting it to a downstream task. | |
| ## How to use | |
| You can use this model directly with a pipeline for text generation. Here is how to use this model to get the features of a given text in PyTorch: | |
| ```python | |
| from transformers import MT5Tokenizer, GPT2LMHeadModel, TextGenerationPipeline | |
| tokenizer = MT5Tokenizer.from_pretrained("THUMT/mGPT") | |
| model = GPT2LMHeadModel.from_pretrained("THUMT/mGPT") | |
| pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer) | |
| text = "Replace me by any text you'd like." | |
| text = pipeline(text, do_sample=True, max_length=1024)[0]["generated_text"] | |
| ``` | |
| ## Preprocessing | |
| The texts are tokenized using `sentencepiece` and a vocabulary size of 250,100. The inputs are sequences of 1,024 consecutive tokens. We use `<extra_id_0>` to separate lines in a document. | |
| ## BibTeX entry and citation info | |
| ```bibtex | |
| @misc{tan2021msp, | |
| title={MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators}, | |
| author={Zhixing Tan and Xiangwen Zhang and Shuo Wang and Yang Liu}, | |
| year={2021}, | |
| eprint={2110.06609}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CL} | |
| } | |
| ``` | |