Problem about using mBART50 for Russian to Chinese translation
Author: ALUKErnel
Pull request: no
Labels: name: WIP

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.1
- Platform: ubuntu 18.04 
- Python version: 3.6.9. 
- PyTorch version (GPU?): 1.8.0 
- Tensorflow version (GPU?): 2.3.0 
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no 

### Who can help

mbart-large-50-many-to-many-mmt：@LysandreJik @patrickvonplaten  

## Information

Model I am using: mbart-large-50-many-to-many-mmt

The problem arises when using:  
  
* my own modified scripts: (give details below)  

We originally wanted to do a Russian-Chinese translation task, but our translation results showed a lot of English. We used a script to test.

## To reproduce

Steps to reproduce the behavior:

1.The code is as follow:  
```python
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

text_list = ['Это позволит облегчить транспортировку грузов для Китая и Германии.',
             'Россия останется одним из лидеров, возможности для наращивания экспорта есть.',
             'Это позволит оптимизировать торговые отношения.']

src_lang = 'ru_RU'
tgt_lang = 'zh_CN'
model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")


for text in text_list:
    tokenizer.src_lang = src_lang
    encoded_hi = tokenizer(text, return_tensors="pt")
    generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.lang_code_to_id[tgt_lang])
    translated = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
    print(translated)
```  
when the src_lang was ‘ru_RU’ and the tgt_lang was ‘zh_CN’, the results were: 
```
['This will facilitate the transport of goods for China and Germany.']
['Russia will remain one of the leaders, there are opportunities to increase export.']
['This will allow to optimize trade relations.']
```


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
We wanted to obtain a set of Chinese translations. Here are the Chinese translations for reference.
```
['这将使中国和德国更容易运输货物。']
['俄罗斯仍将是一个领导者，有机会增加出口。']
['这将有助于改善贸易关系。']
```



Comments:
Same problem, please have a look~ @patil-suraj

Yes I've seen similar issues with mBART50 returning random sentences as output. Related issues are #12104 and #12958 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I don't think this was fixed

@patil-suraj - pinging again here. Would be great to take some action here soon

Sorry about being super slow here.

Going to take look at it this week. First step is to do the same generation with the original model [here](https://github.com/pytorch/fairseq/tree/main/examples/multilingual), the setup is very complicated. Will do it and post the instructions here as well. If the generations match then the issue is with the model itself.


is there a translation from English to Chinese ?
Or From Chinese to English ? 

It also happens from Spanish to English, using `mbart50_m2en`.
```
Original text: "un perro rojo con sombrero de colores"
Translation to English: "The Committee recommends that the State party take all necessary measures to ensure the full implementation of the present recommendations, inter alia, by transmitting them to the members of the Council of Ministers, the Parliament, the Parliamentary Assembly and the Senate..."
```
<details>
  <summary> Packages installed </summary>

```
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile
#
aiohttp==3.8.4
    # via runpod
aiosignal==1.3.1
    # via aiohttp
anyio==3.6.2
    # via
    #   httpcore
    #   starlette
    #   watchfiles
async-timeout==4.0.2
    # via aiohttp
attrs==23.1.0
    # via aiohttp
backoff==2.2.1
    # via runpod
boto3==1.26.133
    # via runpod
botocore==1.29.133
    # via
    #   boto3
    #   s3transfer
certifi==2023.5.7
    # via
    #   httpcore
    #   httpx
    #   requests
charset-normalizer==3.1.0
    # via
    #   aiohttp
    #   requests
click==8.1.3
    # via
    #   nltk
    #   sacremoses
    #   uvicorn
cmake==3.27.2
    # via triton
dnspython==2.3.0
    # via email-validator
easynmt==2.0.2
    # via -r requirements.in
email-validator==2.0.0.post2
    # via fastapi
exceptiongroup==1.1.3
    # via pytest
fastapi[all]==0.95.1
    # via runpod
fasttext==0.9.2
    # via easynmt
filelock==3.12.0
    # via
    #   huggingface-hub
    #   torch
    #   transformers
    #   triton
frozenlist==1.3.3
    # via
    #   aiohttp
    #   aiosignal
fsspec==2023.5.0
    # via huggingface-hub
h11==0.14.0
    # via
    #   httpcore
    #   uvicorn
httpcore==0.17.0
    # via httpx
httptools==0.5.0
    # via uvicorn
httpx==0.24.0
    # via fastapi
huggingface-hub==0.14.1
    # via transformers
idna==3.4
    # via
    #   anyio
    #   email-validator
    #   httpx
    #   requests
    #   yarl
iniconfig==2.0.0
    # via pytest
itsdangerous==2.1.2
    # via fastapi
jinja2==3.1.2
    # via
    #   fastapi
    #   torch
jmespath==1.0.1
    # via
    #   boto3
    #   botocore
joblib==1.2.0
    # via
    #   nltk
    #   sacremoses
lit==16.0.6
    # via triton
markupsafe==2.1.2
    # via jinja2
mpmath==1.3.0
    # via sympy
multidict==6.0.4
    # via
    #   aiohttp
    #   yarl
networkx==3.1
    # via torch
nltk==3.8.1
    # via easynmt
numpy==1.24.3
    # via
    #   easynmt
    #   fasttext
    #   transformers
nvidia-cublas-cu11==11.10.3.66
    # via
    #   nvidia-cudnn-cu11
    #   nvidia-cusolver-cu11
    #   torch
nvidia-cuda-cupti-cu11==11.7.101
    # via torch
nvidia-cuda-nvrtc-cu11==11.7.99
    # via torch
nvidia-cuda-runtime-cu11==11.7.99
    # via torch
nvidia-cudnn-cu11==8.5.0.96
    # via torch
nvidia-cufft-cu11==10.9.0.58
    # via torch
nvidia-curand-cu11==10.2.10.91
    # via torch
nvidia-cusolver-cu11==11.4.0.1
    # via torch
nvidia-cusparse-cu11==11.7.4.91
    # via torch
nvidia-nccl-cu11==2.14.3
    # via torch
nvidia-nvtx-cu11==11.7.91
    # via torch
orjson==3.8.12
    # via fastapi
packaging==23.1
    # via
    #   huggingface-hub
    #   pytest
    #   transformers
pillow==9.5.0
    # via runpod
pluggy==1.0.0
    # via pytest
protobuf==3.20.3
    # via
    #   -r requirements.in
    #   easynmt
pybind11==2.10.4
    # via fasttext
pydantic==1.10.7
    # via fastapi
pytest==7.3.1
    # via -r requirements.in
python-dateutil==2.8.2
    # via botocore
python-dotenv==1.0.0
    # via
    #   runpod
    #   uvicorn
python-multipart==0.0.6
    # via fastapi
pyyaml==6.0
    # via
    #   fastapi
    #   huggingface-hub
    #   transformers
    #   uvicorn
regex==2023.5.5
    # via
    #   nltk
    #   sacremoses
    #   transformers
requests==2.30.0
    # via
    #   huggingface-hub
    #   runpod
    #   transformers
runpod==0.9.7
    # via -r requirements.in
s3transfer==0.6.1
    # via boto3
sacremoses==0.0.53
    # via -r requirements.in
sentencepiece==0.1.99
    # via easynmt
six==1.16.0
    # via
    #   python-dateutil
    #   sacremoses
sniffio==1.3.0
    # via
    #   anyio
    #   httpcore
    #   httpx
starlette==0.26.1
    # via fastapi
sympy==1.12
    # via torch
tokenizers==0.13.3
    # via transformers
tomli==2.0.1
    # via pytest
torch==2.0.1
    # via
    #   easynmt
    #   triton
tqdm==4.65.0
    # via
    #   easynmt
    #   huggingface-hub
    #   nltk
    #   sacremoses
    #   tqdm-loggable
    #   transformers
tqdm-loggable==0.1.3
    # via runpod
transformers==4.29.1
    # via easynmt
triton==2.0.0
    # via torch
typing-extensions==4.5.0
    # via
    #   huggingface-hub
    #   pydantic
    #   torch
ujson==5.7.0
    # via fastapi
urllib3==1.26.15
    # via
    #   botocore
    #   requests
uvicorn[standard]==0.22.0
    # via fastapi
uvloop==0.17.0
    # via uvicorn
watchfiles==0.19.0
    # via uvicorn
websockets==11.0.3
    # via uvicorn
wheel==0.41.2
    # via
    #   nvidia-cublas-cu11
    #   nvidia-cuda-cupti-cu11
    #   nvidia-cuda-runtime-cu11
    #   nvidia-curand-cu11
    #   nvidia-cusparse-cu11
    #   nvidia-nvtx-cu11
yarl==1.9.2
    # via aiohttp

# The following packages are considered to be unsafe in a requirements file:
# setuptools

```

</details>

Python 3.9.17

See #25425, with an example.
@leandroalbero You would need to give a full reproducer 

I also got half English output when asking the model to ranslate from Spanish to Chinese. Is this problem fixed in any other issue?