seamlessm4tv2 performance on HF is significantly lower than facebook's implementation
Author: aliencaocao
Pull request: no

### System Info

not environment dependent. Reproducible on CPU, CUDA on windows and linux
Also reproducible on collab T4

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Translating the same string from Chinese to english, the result is very poor on HF's implementation, but good on FB's HF space. See https://github.com/facebookresearch/seamless_communication/issues/370
I can eliminate the possibility of using mintox as I turned it off myself.

All code used are official examples.
```
from transformers import AutoProcessor, SeamlessM4Tv2ForTextToText
processor = AutoProcessor.from_pretrained("models/seamless-m4t-v2-large")
model = SeamlessM4Tv2ForTextToText.from_pretrained("models/seamless-m4t-v2-large").half().to("cuda")
src_lang, tgt_lang = "cmn", "eng"
text_inputs = processor(text='11:160 31.7K/s· VPN? 4GI 5G 55 1 实验李校长 这三个电话号码打市局电话举 报说初七开学, 查查是哪班学 生 被背刺的三个学生: X', src_lang=src_lang, return_tensors="pt").to("cuda")
decoder_input_ids = model.generate(**text_inputs, tgt_lang=tgt_lang, num_beams=5, do_sample=True)[0].tolist()
translated_text = processor.decode(decoder_input_ids, skip_special_tokens=True)
print(f"{tgt_lang}: {translated_text}")
```

### Expected behavior

Correct result is `11:1060 31.7K/s·VPN? 4GI 5G 55 1 Experimental Li Principal These three phone numbers call the city office to report that the first seven start school, check which class of students were stabbed in the back by three students: X`

![image](https://github.com/huggingface/transformers/assets/20109683/d70cbbf3-b18d-4157-b573-359666097d46)


Comments:
cc @ylacombe 

Hi @aliencaocao, thanks for opening this issue, I've quickly check the model and at first sight, the model did seem to work. 

Firstly, a quick note on model initialization. You should pass `dtype=torch.float16` to the model init arguments instead of using `.half`. I'm actually not sure if it could change something (cc @ArthurZucker who might know), but it's the proper way of doing it in `transformers`.

Secondly, the model does hallucinate with your current code, but it also gets the translation right half of the time.

Thirdly, just switching to `cmn_Hant`, I get generations that are almost always the same, and also more correct:
 
`eng: 11:160 31.7K/s·VPN? 4GI 5G 55 1 Experimental Principal Lee These three phone numbers call the city office to report that the first seven schools have opened, check which class the students are in, the three students who were stabbed in the back: X`

Fourthly, the original model uses greedy search with batch size 5, you did sampling, that's why you get different results each time.

However, it's still really weird that the greedy sample with batch size 5 and using `cmn` is that bad:
`eng: 11:36 p.m. - 4G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 5G, 6G, 6G, 6G, 6G, 7G, 7G, 8G, 8G, 8G, 9G, 9G, 9G, 9G, 9G, 9G, 9G, 9G, 10G, 10G, 10G, 10G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G, 11G,`


Thanks again for opening this issue, I'll dig in more when I have time

Thanks for the details. May I know where did you get `cmn_Hant`? it is not listed in the list of languages here https://huggingface.co/spaces/facebook/seamless-m4t-v2-large/blob/main/lang_list.py

I also tried num_beams=5 and sample=False, it also give me completely wrong translation, and it is very consistent for me and also my peer running on different OS, different generations of GPU. I dont know how did you get it to be right half of the time. I used the [official colab notebook ](https://colab.research.google.com/github/ylacombe/scripts_and_notebooks/blob/main/v2_seamless_m4t_hugging_face.ipynb)

Another thing that might be related is, if I save the model using `SeamlessM4Tv2ForTextToText`, it breaks inference on Meta's `seamless_communications` implementation too and the result is very similar (repetition).


You can find the list of supported languages [here](https://github.com/facebookresearch/seamless_communication/blob/75ed7ef2db2bcc66deb794e19626127ca10cfb4a/src/seamless_communication/cards/unity_nllb-100.yaml#L28).

> I also tried num_beams=5 and sample=False, it also give me completely wrong translation

As said, I've the same issue when doing `cmn`, this is what I'll dig into latter. I've been particularly attentive to get a one-to-one correspondence with the original implementation, but I might have missed something here for sure.
Here is my script that got right half of the time:
```python
from transformers import AutoProcessor, SeamlessM4Tv2ForTextToText
processor = AutoProcessor.from_pretrained("facebook/seamless-m4t-v2-large")
model = SeamlessM4Tv2ForTextToText.from_pretrained("facebook/seamless-m4t-v2-large", torch_dtype=torch.float16).to("cuda")
src_lang, tgt_lang = "cmn", "eng"
text_inputs = processor(text='11:160 31.7K/s· VPN? 4GI 5G 55 1 实验李校长 这三个电话号码打市局电话举 报说初七开学, 查查是哪班学 生 被背刺的三个学生: X', src_lang=src_lang, return_tensors="pt").to("cuda")

print("------------------")
print("cmn")
for i in range(10):
    decoder_input_ids = model.generate(**text_inputs, tgt_lang=tgt_lang, num_beams=5, do_sample=True)[0].tolist()
    translated_text = processor.decode(decoder_input_ids, skip_special_tokens=True)
    print(f"{tgt_lang}: {translated_text}")
print("greedy ---")

decoder_input_ids = model.generate(**text_inputs, tgt_lang=tgt_lang, num_beams=5, do_sample=False)[0].tolist()
translated_text = processor.decode(decoder_input_ids, skip_special_tokens=True)
print(f"{tgt_lang}: {translated_text}")
```

> Another thing that might be related is, if I save the model using SeamlessM4Tv2ForTextToText, it breaks inference on Meta's seamless_communications implementation too and the result is very similar (repetition).

This is really weird as well, do you mind sharing a script to reproduce ? Many thanks!


About the last point, I think the issue was weight names were different. HF does give warning when loading from a weight saved by seamless_communications, but not the other way.

I am able to repro your findings on using 'cmn_Hant', but its weird as the text arent traditional Chinese...

This is what we saved from the TextToText: https://huggingface.co/panoyo9829/seamless-m4t-v2-large-fp16

```
src_lang, tgt_lang = "cmn_Hant", "eng"
text_inputs = processor(text='11:160 31.7K/s· VPN? 4GI 5G 55 1 你这个人 真的是 闲的没事干: X', src_lang=src_lang, return_tensors="pt").to("cuda")
decoder_input_ids = model.generate(**text_inputs, tgt_lang=tgt_lang, num_beams=5)[0].tolist()
translated_text = processor.decode(decoder_input_ids, skip_special_tokens=True)
print(f"{tgt_lang}: {translated_text}")
```

Try this example. Both `cmn` and `cmn_Hant` doesnt work. But I am able to repro the same wrong output in HF space so at least this is now matching the official impl.

i get
```
eng: 1 Corinthians 11:37-40 (KJV) 1 Corinthians 11:37-40 (KJV) 1 Corinthians 11:37-40 (KJV) 2 Corinthians 11:37-40 (KJV) 2 Corinthians 11:37-40 (KJV) 2 Corinthians 11:37-40 (KJV) 2 Corinthians 11:38-40 (KJV) 2 Corinthians 11:38-40 (KJV) 2 Corinthians 11:38-40 (KJV) 2 Corinthians 11:38-40 (KJV) 2 Corinthians 11:38-40 (KJW) 2 Corinthians 11:38-40 (KJW) 2 Corinthians 11:38-40 (KJW) 2 Corinthians 11:38-40 (KJW) 2 Corinthians
```

>  HF does give warning when loading from a weight saved by seamless_communications, but not the other way.

You shouldn't load the weight directly saved by seamless_communication, instead you should use the converting script provided [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/seamless_m4t_v2/convert_fairseq2_to_hf.py).

> This is what we saved from the TextToText: https://huggingface.co/panoyo9829/seamless-m4t-v2-large-fp16

Make sure to first convert the seamless_communication model to HF implementation before saving it.

> I am able to repro your findings on using 'cmn_Hant', but its weird as the text arent traditional Chinese...

I don't speak Chinese so I can't help you on that, sorry !


Finally, note that the seamless communication models are known to face some hallucination issues. There are very useful when dealing with a lot of languages, but if you want fixed translation from Chinese to English, you should probably look at dedicated translation models that are focused on this translation direction!
I hope it helps




I tried to use 'zsm' which is in the list of langs you linked but HF says unknown token. Instead, there is a zlm but they produce different results.

> Make sure to first convert the seamless_communication model to HF implementation before saving it.

This was loaded using HF and then saved using HF, so it should already been in HF format.

Re: `zsm` and `zlm`.

Looks like they specifically changed `zlm` to `zsm` a few months ago ([source](https://github.com/facebookresearch/seamless_communication/commit/d31144406acf1eff2505ccfca907d48a535954c6)). I've had reused the M4T-v1 vocabulary so the change went unnoticed. I invit you to use `zlm` while I fix this !


I think this can be safely closed ! 🤗 Thanks @ylacombe for explaining. TLDR greedy vs beam sampling for speed 👍🏻 

The sampling output for cmn is still very bad though, like the chain of 5G 5G 5G 5G etc.