Generating from mT5
Author: tomhosking
Pull request: no

## Environment info
     
- `transformers` version: #9c0afdaf7b091c341072b432ad6ee17ba7a5016b
- Platform: Google colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0
No GPU

### Who can help
mT5: @patrickvonplaten
 

## Information

Generating from `mT5-small` gives (nearly) empty output:
```
from transformers import MT5ForConditionalGeneration, T5Tokenizer
model = MT5ForConditionalGeneration.from_pretrained("google/mt5-small")
tokenizer = T5Tokenizer.from_pretrained("google/mt5-small")
article = "translate to french: The capital of France is Paris."
batch = tokenizer.prepare_seq2seq_batch(src_texts=[article], return_tensors="pt")
output_ids = model.generate(input_ids=batch.input_ids, num_return_sequences=1, num_beams=8, length_penalty=0.1)
tokenizer.decode(output_ids[0])
```
`>>> <pad> <extra_id_0>
`

Using the same input for T5 gives reasonable output:
```
from transformers import T5ForConditionalGeneration, T5Tokenizer
model = T5ForConditionalGeneration.from_pretrained("t5-small")
tokenizer = T5Tokenizer.from_pretrained("t5-small")
article = "translate to french: The capital of France is Paris."
batch = tokenizer.prepare_seq2seq_batch(src_texts=[article], return_tensors="pt")
output_ids = model.generate(input_ids=batch.input_ids, num_return_sequences=1, num_beams=8, length_penalty=0.1)
tokenizer.decode(output_ids[0])
```
`>>> <pad> La capitale de la France est Paris.
`

My understanding is that mT5 is trained in the same way as T5, and should work in a very similar way?


Comments:
mT5 is not pretrained on downstream tasks like T5 was - see: https://huggingface.co/transformers/master/model_summary.html#mt5 

So it not surprising that mT5 won't work well out-of-the-box without fine-tuning.

Ah, I hadn't realised that. But in that case, wouldn't the expected output be a reconstruction of the input?

> Ah, I hadn't realised that. But in that case, wouldn't the expected output be a reconstruction of the input?

Hard to say if the model does not include any sentinel tokens (`<extra_id_1>`) and if one uses `generate()` instead of just the forward pass.... . Wolud be interesting to play around with the two pre-trained model variants though and see what differences they show...

I agree that I would only get reconstruction if the decoding setup matched training :) Can you point me at any documentation that describes what special tokens are expected? I dug around in your implementation and the official repo but couldn't see anything. The output of `tokenizer.prepare_seq2seq_batch()` is the same for src and tgt as well (presumably because it uses the T5 tokenizer - does it not need its own?)

Edit: Looking again, it seems like the sentinel tokens are just the equivalent of `[MASK]`? In which case the model should be able to reconstruct the input if it has access to the full (un-noised) sequence.

Maybe these pointers help:
- https://github.com/huggingface/transformers/issues/7451
- https://github.com/huggingface/transformers/issues/7910
- https://github.com/huggingface/transformers/issues/3985

mT5 is pretrained exactly like T5 only without the downstream supersived training mixin. I think the T5 paper should explain in detail how this in done.

Does anybody have some more pointers on how to use (train) the mT5 model that has been added to master for text generation? Anything explaining how the finetuning is done in practice using Huggingface Transformers would be greatly appreciated!

Hey @Rijgersberg, what exactly do you mean by text generation ? GPT2-like open-end text generation?

Well not open-end text generation in the sense of "writing", but using text-to-text generation to perform all types of different NLP tasks with little to no training. Basically what the GPT-3-paper calls "few shot learning".

Specifically, I would be interested in replicating the [WT5?! Training Text-to-Text Models to Explain their Predictions](https://arxiv.org/abs/2004.14546) results in languages other than English. But I'm having some trouble understanding what the differences between the T5 and mT5 models in Transformers mean for accomplishing that task.

Hey @tomhosking how did you use  MT5ForConditionalGeneration, T5Tokenizer
I used 
```
pip install transformers
```
But it is showing 
```
ImportError: cannot import name 'MT5ForConditionalGeneration'
```
How can we install it?🤔


@parthplc You can specify version of package You would like to install. For me it was experimental: `transformers==4.0.0rc1` and it works fine.

For training mT5 model for generating summary You can check out [this](https://towardsdatascience.com/fine-tuning-a-t5-transformer-for-any-summarization-task-82334c64c81) post. It worked for me.

[edit]
I forgot to mention, the only modification You have to make is to replace `T5ForConditionalGeneration` with `MT5ForConditionalGeneration`.

> Well not open-end text generation in the sense of "writing", but using text-to-text generation to perform all types of different NLP tasks with little to no training. Basically what the GPT-3-paper calls "few shot learning".
> 
> Specifically, I would be interested in replicating the [WT5?! Training Text-to-Text Models to Explain their Predictions](https://arxiv.org/abs/2004.14546) results in languages other than English. But I'm having some trouble understanding what the differences between the T5 and mT5 models in Transformers mean for accomplishing that task.

In this case, I would just fine-tune mT5 with the normal causal language modeling objective meaning:

```python
from transformers import MT5ForConditionalGeneration, T5Tokenizer 
mt5 = MT5ForConditionalGeneration.from_pretrained("google/mt5-base")
mt5_tok = T5Tokenizer.from_pretrained("google/mt5-base")

input_ids = mt5_tok("explain sentiment: I went to see this movie with my husband, and we both thought the acting was terrible!", return_tensors="pt").input_ids  # in the language of your choice
labels = mt5_tok("negative explanation: the acting was terrible.", return_tensors="pt").input_ids  # in the language of your choice

loss = mt5(input_ids=input_ids, labels=labels).loss
```

I took one of the visual examples of the paper you mentioned.

In short, there is no difference in how mt5 and t5 should be fine-tuned.

Also, @mrm8488 already successfully fine-tuned an mT5 model: https://twitter.com/mrm8488/status/1329478063768350723
sorry to ping you here @mrm8488 - but maybe you have some tips/tricks for mt5 fine-tuning? 

Also pinging our T5 fine-tuning expert @patil-suraj 

> Well not open-end text generation in the sense of "writing", but using text-to-text generation to perform all types of different NLP tasks with little to no training. Basically what the GPT-3-paper calls "few shot learning".

I'm not sure if you can use mT5 with no training (fine-tuning), since it was not pre-trained with any supervised objective like `T5`. 

One experiment to try is to fine-tune `mT5` on the english data and see if it works for your language without any language specific fine-tuning (In my experiments, `T5` trained on English SQuAD for que gen gave interesting results for French and German without any language specific fine-tuning).

But for better results you should fine-tune `mT5` on the language specific dataset.

And also as Patrick said, you can fine-tune `mT5` and `T5` the same way. 
The major differences between `mT5` and `T5` are
- `mT5` is based on [`T51.1`](https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511)
- pre-trained on 101 languages
- no supervised pre-training

Hi, I slightly modified the script provided by @patil-suraj  to fine-tune [`T5` on SQUAD] (https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb) and after many epochs (I think I am missing anything/doing something wrong) I got 'decent' results fine-tuning mT5-small on tydiQA for multilingual QA https://huggingface.co/mrm8488/mT5-small-finetuned-tydiqa-for-xqa. The [PR with the model card](https://github.com/huggingface/transformers/pull/8729) for more details is not approved yet.

> Hi, I slightly modified the script provided by @ patil-suraj to fine-tune [`T5` on SQUAD] (https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb) and after many epochs (I think I am missing anything/doing something wrong) I got 'decent' results fine-tuning mT5-small on tydiQA for multilingual QA https://huggingface.co/mrm8488/mT5-small-finetuned-tydiqa-for-xqa. The [PR with the model card](https://github.com/huggingface/transformers/pull/8729) for more details is not approved yet.

just merged it :-) BTW, you can now directly create the model cards online - no need for PRs anymore ;-)

> > Well not open-end text generation in the sense of "writing", but using text-to-text generation to perform all types of different NLP tasks with little to no training. Basically what the GPT-3-paper calls "few shot learning".
> 
> I'm not sure if you can use mT5 with no training (fine-tuning), since it was not pre-trained with any supervised objective like `T5`.
> 
> One experiment to try is to fine-tune `mT5` on the english data and see if it works for your language without any language specific fine-tuning (In my experiments, `T5` trained on English SQuAD for que gen gave interesting results for French and German without any language specific fine-tuning).
> 
> But for better results you should fine-tune `mT5` on the language specific dataset.
> 
> And also as Patrick said, you can fine-tune `mT5` and `T5` the same way.
> The major differences between `mT5` and `T5` are
> 
> * `mT5` is based on [`T51.1`](https://github.com/google-research/text-to-text-transfer-transformer/blob/master/released_checkpoints.md#t511)
> * pre-trained on 101 languages
> * no supervised pre-training

hey @patil-suraj @mrm8488 how can we finetune mT5 for other languages. Let's suppose we have language translation problem for any language other than English and if we finetune using T5 tokenizer we would be replacing each word with unk tokens. how will it be fine-tuned? eg.
```
print(tokenizer.decode(data['source_ids']))
print(tokenizer.decode(data['target_ids']))
```
```
English to Hindi: Tell me the name of the ninth month.
 <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad>
<unk> <unk> <unk> <unk> <unk> <unk> 
 <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad>
```


@parthplc - I don't really understand your question. Since mT5 was trained on 101 languages it's tokenizer can obviously handle all those languages, *e.g.*:

```python 
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("google/mt5-small")
tok.decode(tok("Der Satz wird auch definiert als sprachliche Einheit, die aus Subjekt und Prädikat besteht. Dies soll auf Aristoteles zurückgehen. Entsprechend definiert die traditionelle Grammatik den Satz als bestehend aus: Satzaussage (Prädikat), Satzergänzung (Objekt) und Satzgegenstand (Subjekt).").input_ids)  # gives no <unk> symbols
```

Hopefully, this makes more sense now

> ## Environment info
> * `transformers` version: #9c0afdaf7b091c341072b432ad6ee17ba7a5016b
> * Platform: Google colab
> * Python version: 3.6.9
> * PyTorch version (GPU?): 1.7.0
>   No GPU
> 
> ### Who can help
> mT5: @patrickvonplaten
> 
> ## Information
> Generating from `mT5-small` gives (nearly) empty output:
> 
> ```
> from transformers import MT5ForConditionalGeneration, T5Tokenizer
> model = MT5ForConditionalGeneration.from_pretrained("google/mt5-small")
> tokenizer = T5Tokenizer.from_pretrained("google/mt5-small")
> article = "translate to french: The capital of France is Paris."
> batch = tokenizer.prepare_seq2seq_batch(src_texts=[article], return_tensors="pt")
> output_ids = model.generate(input_ids=batch.input_ids, num_return_sequences=1, num_beams=8, length_penalty=0.1)
> tokenizer.decode(output_ids[0])
> ```
> 
> `>>> <pad> <extra_id_0>
`
> 
> Using the same input for T5 gives reasonable output:
> 
> ```
> from transformers import T5ForConditionalGeneration, T5Tokenizer
> model = T5ForConditionalGeneration.from_pretrained("t5-small")
> tokenizer = T5Tokenizer.from_pretrained("t5-small")
> article = "translate to french: The capital of France is Paris."
> batch = tokenizer.prepare_seq2seq_batch(src_texts=[article], return_tensors="pt")
> output_ids = model.generate(input_ids=batch.input_ids, num_return_sequences=1, num_beams=8, length_penalty=0.1)
> tokenizer.decode(output_ids[0])
> ```
> 
> `>>> <pad> La capitale de la France est Paris.
`
> 
> My understanding is that mT5 is trained in the same way as T5, and should work in a very similar way?

Hi, I met the same problem when fine-tuning mt5 to a Chinese QG environment. I'm wondering if you have solved this issue?

hi @nomoreoneday 

`mT5` is a pre-trained model and it's not finetuned on any downstream task, whereas T5 was already trained on translation task as part of its supervised pre-training mixture, which could explain the empty output of mT5.

You should fine-tune the model on your task, to use for generation. 

> I met the same problem when fine-tuning mt5 to a Chinese QG environment

And if you are having trouble with fine-tuning then please post a shot code snippet so we can reproduce your issue.

> hi @nomoreoneday
> 
> `mT5` is a pre-trained model and it's not finetuned on any downstream task, whereas T5 was already trained on translation task as part of its supervised pre-training mixture, which could explain the empty output of mT5.
> 
> You should fine-tune the model on your task, to use for generation.
> 
> > I met the same problem when fine-tuning mt5 to a Chinese QG environment
> 
> And if you are having trouble with fine-tuning then please post a shot code snippet so we can reproduce your issue.

hi @patil-suraj 

thanks for replying. I'm trying to replicate your project(https://github.com/patil-suraj/question_generation) on a Chinese QG task. I got decent results when I run 
```
export CUDA_VISIBLE_DEVICES=0
python3 eval.py \
    --model_name_or_path  mt5-small-ncp-qg-hl-base_epoch30  \
    --valid_file_path data/valid_data_qa_hl_mt5_ncp_all_task.pt \
    --model_type mt5 \
    --num_beams 4 \
    --max_decoding_length 32 \
    --output_path hypothesis_mt5-small-ncp-qg-hl-base_epoch30_ncp_all_task.txt
```
But when I trying to construct the pipeline and run:

`
def _extract_answers(self,context):

        sents,inputs = self._prepare_inputs_for_ans_extraction(context)
        inputs = self._tokenize(inputs,padding = True,truncation = True) #encoding
        print("inputs after encoding:",inputs)

        outs = self.ans_model.generate(
            input_ids = inputs['input_ids'].to(self.device),
            attention_mask = inputs['attention_mask'].to(self.device),
            max_length = 32,
        )

        dec = [self.ans_tokenizer.decode(ids,skip_special_tokens=False) for ids in outs] #decoding
        print("dec:", dec)
        answers = [item.split('<sep>') for item in dec]
        print("answers1:",answers)
        answers = [i[:-1] for i in answers]
        print("answers2:",answers)

        return sents, answers
`

I got the empty answers. like this

`dec: ['<pad> <extra_id_0>
']
answers1: [['<pad> <extra_id_0>
']]
answers2: [[]]`

I wondering if there is any difference in data preprocessing between t5 and mt5. 

> > hi @nomoreoneday
> > `mT5` is a pre-trained model and it's not finetuned on any downstream task, whereas T5 was already trained on translation task as part of its supervised pre-training mixture, which could explain the empty output of mT5.
> > You should fine-tune the model on your task, to use for generation.
> > > I met the same problem when fine-tuning mt5 to a Chinese QG environment
> > 
> > 
> > And if you are having trouble with fine-tuning then please post a shot code snippet so we can reproduce your issue.
> 
> hi @patil-suraj
> 
> thanks for replying. I'm trying to replicate your project(https://github.com/patil-suraj/question_generation) on a Chinese QG task. I got decent results when I run
> 
> ```
> export CUDA_VISIBLE_DEVICES=0
> python3 eval.py \
>     --model_name_or_path  mt5-small-ncp-qg-hl-base_epoch30  \
>     --valid_file_path data/valid_data_qa_hl_mt5_ncp_all_task.pt \
>     --model_type mt5 \
>     --num_beams 4 \
>     --max_decoding_length 32 \
>     --output_path hypothesis_mt5-small-ncp-qg-hl-base_epoch30_ncp_all_task.txt
> ```
> 
> But when I trying to construct the pipeline and run:
> 
> `
> def _extract_answers(self,context):
> 
> ```
>     sents,inputs = self._prepare_inputs_for_ans_extraction(context)
>     inputs = self._tokenize(inputs,padding = True,truncation = True) #encoding
>     print("inputs after encoding:",inputs)
> 
>     outs = self.ans_model.generate(
>         input_ids = inputs['input_ids'].to(self.device),
>         attention_mask = inputs['attention_mask'].to(self.device),
>         max_length = 32,
>     )
> 
>     dec = [self.ans_tokenizer.decode(ids,skip_special_tokens=False) for ids in outs] #decoding
>     print("dec:", dec)
>     answers = [item.split('<sep>') for item in dec]
>     print("answers1:",answers)
>     answers = [i[:-1] for i in answers]
>     print("answers2:",answers)
> 
>     return sents, answers
> ```
> 
> `
> 
> I got the empty answers. like this
> 
> `dec: ['<pad> <extra_id_0>
'] answers1: [['<pad> <extra_id_0>
']] answers2: [[]]`

having the same issue here

I have finally overcome the ['<pad> <extra_id_0>
'] issue and obtained decent post-training predictions with MT5, just had to 
1. lower the lr (set to 0.001, as indicated in mt5 paper) 
and 
2. train for a lot more epochs in comparison with T5 for the same task (60 epochs for MT5 vs 10 for T5 for a simple text style transfer task fine-tuning). 

@nomoreoneday I have no touched anything but the model name when switching between t5 and mt5 in my training pipeline, wonder if I should?

> Hi, I slightly modified the script provided by @patil-suraj to fine-tune [`T5` on SQUAD] (https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb) and after many epochs (I think I am missing anything/doing something wrong) I got 'decent' results fine-tuning mT5-small on tydiQA for multilingual QA https://huggingface.co/mrm8488/mT5-small-finetuned-tydiqa-for-xqa. The [PR with the model card](https://github.com/huggingface/transformers/pull/8729) for more details is not approved yet.

@mrm8488 Hi, for this model mrm8488/mT5-small-finetuned-tydiqa-for-xqa , I tried to run your demo script, but failed with error loading the tokenizer.  And the Hosted inference API on this page doesn't work as well.
 hope to see your feedback.

> > hi @nomoreoneday
> > `mT5` is a pre-trained model and it's not finetuned on any downstream task, whereas T5 was already trained on translation task as part of its supervised pre-training mixture, which could explain the empty output of mT5.
> > You should fine-tune the model on your task, to use for generation.
> > > I met the same problem when fine-tuning mt5 to a Chinese QG environment
> > 
> > 
> > And if you are having trouble with fine-tuning then please post a shot code snippet so we can reproduce your issue.
> 
> hi @patil-suraj
> 
> thanks for replying. I'm trying to replicate your project(https://github.com/patil-suraj/question_generation) on a Chinese QG task. I got decent results when I run
> 
> ```
> export CUDA_VISIBLE_DEVICES=0
> python3 eval.py \
>     --model_name_or_path  mt5-small-ncp-qg-hl-base_epoch30  \
>     --valid_file_path data/valid_data_qa_hl_mt5_ncp_all_task.pt \
>     --model_type mt5 \
>     --num_beams 4 \
>     --max_decoding_length 32 \
>     --output_path hypothesis_mt5-small-ncp-qg-hl-base_epoch30_ncp_all_task.txt
> ```
> 
> But when I trying to construct the pipeline and run:
> 
> `
> def _extract_answers(self,context):
> 
> ```
>     sents,inputs = self._prepare_inputs_for_ans_extraction(context)
>     inputs = self._tokenize(inputs,padding = True,truncation = True) #encoding
>     print("inputs after encoding:",inputs)
> 
>     outs = self.ans_model.generate(
>         input_ids = inputs['input_ids'].to(self.device),
>         attention_mask = inputs['attention_mask'].to(self.device),
>         max_length = 32,
>     )
> 
>     dec = [self.ans_tokenizer.decode(ids,skip_special_tokens=False) for ids in outs] #decoding
>     print("dec:", dec)
>     answers = [item.split('<sep>') for item in dec]
>     print("answers1:",answers)
>     answers = [i[:-1] for i in answers]
>     print("answers2:",answers)
> 
>     return sents, answers
> ```
> 
> `
> 
> I got the empty answers. like this
> 
> `dec: ['<pad> <extra_id_0>
'] answers1: [['<pad> <extra_id_0>
']] answers2: [[]]`

Same too 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> I have finally overcome the [' <extra_id_0>'] issue and obtained decent post-training predictions with MT5, just had to
> 
> 1. lower the lr (set to 0.001, as indicated in mt5 paper)
>    and
> 2. train for a lot more epochs in comparison with T5 for the same task (60 epochs for MT5 vs 10 for T5 for a simple text style transfer task fine-tuning).
> 
> @nomoreoneday I have no touched anything but the model name when switching between t5 and mt5 in my training pipeline, wonder if I should?

Thankyou，I have same problem， I am trying train more epochs, to see if it can be correct

> > I have finally overcome the [' <extra_id_0>'] issue and obtained decent post-training predictions with MT5, just had to
> > 
> > 1. lower the lr (set to 0.001, as indicated in mt5 paper)
> >    and
> > 2. train for a lot more epochs in comparison with T5 for the same task (60 epochs for MT5 vs 10 for T5 for a simple text style transfer task fine-tuning).
> > 
> > @nomoreoneday I have no touched anything but the model name when switching between t5 and mt5 in my training pipeline, wonder if I should?
> 
> Thankyou，I have same problem， I am trying train more epochs, to see if it can be correct

Do you have any newer ideas about this problem?

> > > hi @nomoreoneday
> > > `mT5` is a pre-trained model and it's not finetuned on any downstream task, whereas T5 was already trained on translation task as part of its supervised pre-training mixture, which could explain the empty output of mT5.
> > > You should fine-tune the model on your task, to use for generation.
> > > > I met the same problem when fine-tuning mt5 to a Chinese QG environment
> > > 
> > > 
> > > And if you are having trouble with fine-tuning then please post a shot code snippet so we can reproduce your issue.
> > 
> > 
> > hi @patil-suraj
> > thanks for replying. I'm trying to replicate your project(https://github.com/patil-suraj/question_generation) on a Chinese QG task. I got decent results when I run
> > ```
> > export CUDA_VISIBLE_DEVICES=0
> > python3 eval.py \
> >     --model_name_or_path  mt5-small-ncp-qg-hl-base_epoch30  \
> >     --valid_file_path data/valid_data_qa_hl_mt5_ncp_all_task.pt \
> >     --model_type mt5 \
> >     --num_beams 4 \
> >     --max_decoding_length 32 \
> >     --output_path hypothesis_mt5-small-ncp-qg-hl-base_epoch30_ncp_all_task.txt
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > But when I trying to construct the pipeline and run:
> > `
> > def _extract_answers(self,context):
> > ```
> >     sents,inputs = self._prepare_inputs_for_ans_extraction(context)
> >     inputs = self._tokenize(inputs,padding = True,truncation = True) #encoding
> >     print("inputs after encoding:",inputs)
> > 
> >     outs = self.ans_model.generate(
> >         input_ids = inputs['input_ids'].to(self.device),
> >         attention_mask = inputs['attention_mask'].to(self.device),
> >         max_length = 32,
> >     )
> > 
> >     dec = [self.ans_tokenizer.decode(ids,skip_special_tokens=False) for ids in outs] #decoding
> >     print("dec:", dec)
> >     answers = [item.split('<sep>') for item in dec]
> >     print("answers1:",answers)
> >     answers = [i[:-1] for i in answers]
> >     print("answers2:",answers)
> > 
> >     return sents, answers
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > `
> > I got the empty answers. like this
> > `dec: ['<pad> <extra_id_0>
'] answers1: [['<pad> <extra_id_0>
']] answers2: [[]]`
> 
> having the same issue here

I am also having the same issue

Sorry I'm loosing a bit track of what the problem is here. Note that `mt5` cannot generate coherent sentences out-of-the-box because it's only be pretrained on the span-mask filling task and not on any down-stream tasks.

But how can hugging face maek an nlp course and train the ưhole model from mT5 model as this link:  https://huggingface.co/learn/nlp-course/chapter7/5#summarization
Is it because they use the older package lib?