MBartForConditionalGeneration doesn't seem to be able to complete the task of filling mask.
Author: 5i-wanna-be-the-666
Pull request: no

### System Info

transformers version: 4.29.2
Platform: Linux ubt-4090 5.15.0-75-generic 
Python version: 3.9.5
PyTorch version (GPU?): 1.12.1+cu113 (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @younesbelkada @patrickvonplaten

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When I used the official document on huggingface for mask filling, I got the expected output.
```python
from transformers import AutoTokenizer, MBartForConditionalGeneration

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")

# de_DE is the language symbol id <LID> for German
TXT = "
 Meine Freunde sind <mask> nett aber sie essen zu viel Kuchen. 
 de_DE"

input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt")["input_ids"]
logits = model(input_ids).logits

masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
probs = logits[0, masked_index].softmax(dim=0)
values, predictions = probs.topk(5)

tokenizer.decode(predictions).split()
['nett', 'sehr', 'ganz', 'nicht', 'so']
```

But when I changed the characters that need to be filled into Chinese, there was an accident.
```python
from transformers import AutoTokenizer, MBartForConditionalGeneration

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")

# de_DE is the language symbol id <LID> for German
TXT = "
 今天<mask>真好,我准备去公园打羽毛球. 
 zh_ZH"

input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt")["input_ids"]
logits = model(input_ids).logits

masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
probs = logits[0, masked_index].softmax(dim=0)
values, predictions = probs.topk(5)

tokenizer.decode(predictions).split()
[',·:.']
```
![image](https://github.com/huggingface/transformers/assets/58644245/7d710492-b3cd-4024-bf85-4ed926308eec)
After that, I tried to get mBART to restore a sentence with multiple masks for me, and the effect was even worse.
```python
from transformers import MBartTokenizer,DataCollatorForLanguageModeling,MBartForConditionalGeneration
tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-cc25")
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)
model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")

TXT_input = "
The weather is so nice today, I am going to play badminton in the park
en_xx"

inputs = tokenizer([TXT_input], add_special_tokens=False, return_tensors="pt",max_length=32,  padding='max_length')

masked_inputs_and_labels = data_collator([inputs]) 

input_ids = masked_inputs_and_labels['input_ids'][0]
attention_mask = masked_inputs_and_labels['attention_mask'][0]
labels = masked_inputs_and_labels['labels'][0]

masked_inputs={key:value[0] for key,value in masked_inputs_and_labels.items()}
outputs = model(input_ids = input_ids,attention_mask = attention_mask,labels = labels)
logits = outputs.logits

print(f'after mask: {tokenizer.decode(masked_inputs["input_ids"][0])}')

predictions = outputs.logits.argmax(dim=-1)

print(f'Predicted sentence: {tokenizer.decode(predictions[0])}')
after mask: 
 The weather is so nice today, I am going tosähkö badminton in the park
 en_xx<pad><pad><pad><pad><pad><pad><pad><mask><pad><pad><pad>
Predicted sentence: 
นยยยยยนนนนนน badmintonนนนap














```
**Excuse me, is there something wrong with my usage?In that case, how can I correctly use mBART to fill the mask?**



### Expected behavior

I think mBART has at least one Chinese token with five highest probabilities.Or restore the masked sentence for me.
such as:['天气','心情',.....]
or:Predicted sentence: "The weather is so nice today, I am going to play badminton in the park en_xx"

Comments:
Hey! It seems that the model you are trying to use was not trained on `zh_ZH` but `zh_CN`. Could you try to use this instead? (It might just be the token that need to be updated). 

For the second script, I don't think you changed the `src_lang` of the `tokenizer` which is not Chinese by default.
I got `[',我是早上去']` as an output with:
```python 
from transformers import AutoTokenizer, MBartForConditionalGeneration

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")

# de_DE is the language symbol id <LID> for German
TXT = "
 今天<mask>真好,我准备去公园打羽毛球. 
 zh_CN"

input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt")["input_ids"]
logits = model(input_ids).logits

masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
probs = logits[0, masked_index].softmax(dim=0)
values, predictions = probs.topk(5)

tokenizer.decode(predictions).split()
```
which is already a lot better 😉 

Thank you for your reply!I did get good results after changing zh_ZH to zh_CN. The reason why I think it is zh_ZH is that I accidentally read the document wrong.
![0611921C](https://github.com/huggingface/transformers/assets/58644245/229e15c7-10f8-46d2-973d-371a28d8a29f)

**But how can I solve the problem as in the last script? Even if there are multiple mask marks, I also want to know the MLM loss of this model.**

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Sorry for the late reply, recommend you to have a look at #10222  and search on our [forum](https://discuss.huggingface.co/) were this has been answered! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I also have problem with this.
I want to use 【facebook/mbart-large-50-many-to-many-mmt】 to do mask filling task. But the output is always strange.
I modify the input format as the Model Card from https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt suggested.
My code is as follows:
```
from transformers import (
AutoTokenizer,
BertForMaskedLM,
MBart50TokenizerFast,
MBartForConditionalGeneration,
DataCollatorForLanguageModeling
)
model_name_or_path = 'my_path/mbart-large-50-many-to-many-mmt'
model = MBartForConditionalGeneration.from_pretrained(model_name_or_path)
tokenizer = MBart50TokenizerFast.from_pretrained(model_name_or_path)

tokenizer.src_lang = 'en_XX'

src = "So that such a thing won’t happen <mask>."
encoded_src = tokenizer([src], return_tensors="pt")
input_ids = encoded_src["input_ids"]
src_tokens = tokenizer.convert_ids_to_tokens(input_ids[0])

model_outputs = model(**encoded_src)
logits = model_outputs.logits

masked_index = torch.nonzero((input_ids[0] == tokenizer.mask_token_id)).item()
probs = logits[0, masked_index].softmax(dim=0)
values, predictions = probs.topk(5)

print(tokenizer.convert_ids_to_tokens(predictions))
```

The output is:
['.', '☎', '↔', '∏', '∴']

When I change my input, it always output strange symbols. I think this is wrong.

I am confused whether this model is not suitable for this task. How should I modify to get proper outputs? Thank you so much!