Instruction for Using XLM Text  Generations
Author: yusufani
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I've reviewed every file in the document. I couldn't find an instruction to use **XLM** for text generation. What I really want to do is use a pre-trained **XLM** model for **English** text generation and examine the results. Then I would like to train a model or use pre-trained model for text generation in my language , which is **Turkish** ,   to examine the results .

How do I perform these operations step-by-step?


Comments:
I am working on similar situation. If anyone solves this problem please help me

It is really hard issue in my project. I can’t find anything as helpful about it and I really need this. I spent on this problem hours and hours. I found several resources but they didn’t have enough information. Please help us that title.

Hello! Thanks for opening this issue. As XLM indeed works slightly differently than other models, I have added it to the `run_generation.py` script.

Here's the difference from the other models: as a multilingual model, you can specify which language should be used when generating text. In order to do so, you should specify a language embedding during generation. Here's the way to do that:

Let's say we're using the pre-trained checkpoint `xlm-clm-1024-enfr`, which has two languages: English and French.
```py
import torch
from transformers import XLMTokenizer, XLMWithLMHeadModel

tokenizer = XLMTokenizer.from_pretrained("xlm-clm-1024-enfr")
```
You can see the different languages this tokenizer handles, as well as the ids of these languages using the `lang2id` attribute:
```py
print(tokenizer.lang2id)  # {'en': 0, 'fr': 1}
```
These ids should be used when passing a language parameter during a model pass. Let's define our inputs:
```py
input_ids = torch.tensor([tokenizer.encode("Wikipedia was used to")]) # batch size of 1
```
We should now define the language embedding by using the previously defined language id. We want to create a tensor filled with the appropriate language ids, of the same size as `input_ids`. For english, the id is `0`:
```py
language_id = tokenizer.lang2id['en']  # 0
langs = torch.tensor([language_id] * input_ids.shape[1])  # torch.tensor([0, 0, 0, ..., 0])

# We reshape it to be of size (batch_size, sequence_length)
langs = langs.view(1, -1) # is now of shape [1, sequence_length] (we have a batch size of 1) 
```
You can then feed it all as input to your model:
```py
outputs = model(input_ids, langs=langs)
```

You can see all of this implemented in the [`run_generation.py`](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py) script, and how to decode the results. I hope this clears things up

Please also note that for accurate generation, only the `clm` models should be used. I believe the `mlm` could also be used, but would output worse text generation.

Furthermore, the `langs` value I explained works for the models that have `use_lang_emb` set to `True`. This is not the case for the 17 languages and 100 languages models.

I can't tell you how grateful I am for your answer and for updating the `run_generations` file. But I have one little problem.

I ran the model in the Colab environment with the following entry:
```
!python run_generation.py \
    --model_type=xlm \
    --model_name_or_path=xlm-clm-enfr-1024
```
Code gave me this error:
```
10/05/2019 20:17:55 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-clm-enfr-1024-vocab.json from cache at /root/.cache/torch/transformers/e6f5fa1cd0da83c700ab5b38483774463b599ee8f73d995e6779dcd5f2777e84.892e5b45d85e254d5a121ca6986484acd0cf78f26b2d377b89be3771422779b6
10/05/2019 20:17:55 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-enfr-1024-merges.txt from cache at /root/.cache/torch/transformers/6fcd506cac607ea4adeb88dddc38fef209ebeb4b2355132d43dc63b76863b81e.9da5d5f88a7619d42b4a6cc26c9bfd7c2186d3f0c3a1563b9d8176c58b44a745
10/05/2019 20:17:56 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-clm-enfr-1024-config.json from cache at /root/.cache/torch/transformers/fbf61a111106c863e3566853bb101241339254ea07d761d4ba9d19642bcf471f.ba7ab938fe4de8fa5f7d97ad12ed8d5dfe6dc702abc18c55e9bd29db21fc7b8c
10/05/2019 20:17:56 - INFO - transformers.configuration_utils -   Model config {
  "asm": false,
  "attention_dropout": 0.1,
  "bos_index": 0,
  "causal": false,
  "dropout": 0.1,
  "emb_dim": 1024,
  "embed_init_std": 0.02209708691207961,
  "end_n_top": 5,
  "eos_index": 1,
  "finetuning_task": null,
  "gelu_activation": true,
  "id2lang": {
    "0": "en",
    "1": "fr"
  },
  "init_std": 0.02,
  "is_encoder": true,
  "lang2id": {
    "en": 0,
    "fr": 1
  },
  "layer_norm_eps": 1e-12,
  "mask_index": 5,
  "max_position_embeddings": 512,
  "max_vocab": -1,
  "min_count": 0,
  "n_heads": 8,
  "n_langs": 2,
  "n_layers": 6,
  "n_words": 64139,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "pad_index": 2,
  "pruned_heads": {},
  "same_enc_dec": true,
  "share_inout_emb": true,
  "sinusoidal_embeddings": false,
  "start_n_top": 5,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "first",
  "summary_use_proj": true,
  "torchscript": false,
  "unk_index": 3,
  "use_bfloat16": false,
  "use_lang_emb": true
}

10/05/2019 20:17:56 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-clm-enfr-1024-pytorch_model.bin from cache at /root/.cache/torch/transformers/bb34c23dd1c8c4a03862aa4347291a7bd0a405511ab9e6ac05c53ede177c2d09.ddfff42a040dae9a73f7b93c30f1b0a72bad65fa82637f63ab38ac9ed1bc425c
Namespace(device=device(type='cuda'), length=20, model_name_or_path='xlm-clm-enfr-1024', model_type='xlm', n_gpu=1, no_cuda=False, padding_text='', prompt='', seed=42, stop_token=None, temperature=1.0, top_k=0, top_p=0.9, xlm_lang='')
Using XLM. Select language in ['en', 'fr'] >>> en
Model prompt >>> Today is a nice day
  0% 0/20 [00:00<?, ?it/s]Printing Inputs  {'input_ids': tensor([[ 497,   29,   17, 3370,  206]], device='cuda:0'), 'langs': tensor([[0, 0, 0, 0, 0]])}

Traceback (most recent call last):
  File "run_generation.py", line 220, in <module>
    main()
  File "run_generation.py", line 206, in main
    device=args.device,
  File "run_generation.py", line 130, in sample_sequence
    outputs = model(**inputs)  # Note: we could also use 'past' with GPT-2/Transfo-XL/XLNet (cached hidden-states)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlm.py", line 637, in forward
    head_mask=head_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_xlm.py", line 485, in forward
    tensor = tensor + self.lang_embeddings(langs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1467, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
```
Where could i make mistake when runnig the code ? 

Indeed, sorry about that I didn't assign the correct device to the new tensor. It should be fixed now.

Thank you for quick fix. I ran [run_generations](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py) file in [all pre-trained  XLM models](https://huggingface.co/transformers/pretrained_models.html) except for `xlm-mlm-en-2048` model ( I think the problem is that there is only one language in the model. So the model does have some parameters ) . 

As you said, although it is not very successful in CLM models, I can get some results (I will continue to try) but I can't get meaningful results in MLM models.For example, for `xlm-mlm-ende-1024` model I wrote the outputs I received for 10 different inputs : 

Outputs:
- ] ] ] ] ] ] ] ] ] " " " " " " " " " " "
- ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ]
- ] ] ] ] ] ) ) ) ) ) ) ) ) " ) ) ) ) ) )
- ] ] ] ] ] ] ] ] ] ] ] ) ) ) ) " " " " "
- ] ] ] ] ] • • • • • • • • • • • • • • •
- ] ] ] ] ] ] " " " " " " " " " " " " " "
- ] ] ] ] " " " " " " " " " " " " " " " "
- ] ] ] ] ] " " " " " " " " " " " " " " "
- stlike like like like like like like like like like like like like like like like like like like
- est blast stab at....docdocdocdocdocdocdoctooo"..

How can i generate meaningful outputs in mlm models ?

Unfortunately, MLM models won´t be of use useful for text generation. By nature, MLM models require left and right context to predict masked tokens, and in the case of text generation they only have access to the left context.

XLM is the only model in our library which was trained with both MLM and CLM; all other models in the `run_generation` script are CLM-only.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@LysandreJik 

> Indeed, sorry about that I didn't assign the correct device to the new tensor. It should be fixed now.

This is still a problem, even with updated torch. Please see #2360