Performance of mbart-large-50-many-to-many-mmt on de/fr/it
Author: lvwerra
Pull request: no

Hi everybody

I am using ` mbart-large-50-many-to-many-mmt` and I am running into the following problem.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0 (installed from source)
- Platform: Linux
- Python version: 3.8.5
- PyTorch version (CPU): 1.7.1

### Who can help
@patrickvonplaten, @patil-suraj

## Information

I am using the `mbart-large-50-many-to-many-mmt` model for translation and it works as expected when translating German to English but when translating to other languages such as French or Italian it seems broken. I am using the same code as highlighted in the model card.

```Python
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

input_text = "Der Mars hat einen neuen Besucher, wenn auch einen robotischen: \
Nach einer mehr als 472 Millionen Kilometer langen Reise setzte am Donnerstagabend \
das amerikanische Roboterfahrzeug Perseverance sanft im Marsstaub auf. "

tokenizer.src_lang = "de_DE"
encoded = tokenizer(input_text, return_tensors="pt")
generated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)[0]

#--> '
en_XX Mars has a new visitor, but also a robotic one: After a journey of more than 472 million kilometers, the American robotic vehicle Perseverance gently set off in Mars dust on Thursday evening.
'

tokenizer.src_lang = "de_DE"
encoded = tokenizer(input_text, return_tensors="pt")
generated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)[0]

#--> '
fr_XX On Mars, on Mars, on Mars, on Mars, on Mars, on Mars, on Mars, on Mars, on Mars.
'

tokenizer.src_lang = "de_DE"
encoded = tokenizer(input_text, return_tensors="pt")
generated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.lang_code_to_id["it_IT"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)[0]

#--> '
it_IT Mars has a new visitor, anche robotico: After a journey di più di 472 milioni di chilometri, Thursday evening, the American robot vehicle Perseverance si è calmato in the dust of Mars.
'
```

Am I doing something wrong with the translation or is the performance on these languages expected to be worse?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Unstale

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.