Remove unnecessary tied weights for Seamless M4T?
Author: ebezzam
Pull request: yes
Labels: name: bug

name: Audio

# What does this PR do?

The tied weights refactoring from https://github.com/huggingface/transformers/pull/41580 may have introduced an error for the Seamless M4T model? Opening this PR to not lose track.

Perhaps https://github.com/huggingface/transformers/pull/42362 will address?

Modeling tests before fix (11 errors)
```
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithSpeechInputTest::test_model_from_pretrained - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_model_from_pretrained - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_resize_embeddings_untied_with_deepspeed - ImportError: libmpi.so.40: cannot open shared object file: No such file or directory
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_resize_tokens_embeddings_with_deepspeed - ImportError: libmpi.so.40: cannot open shared object file: No such file or directory
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_speech_to_speech_model - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_speech_to_text_model - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_text_to_speech_model - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_text_to_text_model - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_eng_text - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_rus_speech - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_swh_text - ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_mo... 
========== 11 failed, 79 passed, 193 skipped, 5 warnings in 116.67s (0:01:56) ====
```

Modeling tests after fix (7 errors)
```
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_resize_embeddings_untied_with_deepspeed - ImportError: libmpi.so.40: cannot open shared object file: No such file or directory
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_resize_tokens_embeddings_with_deepspeed - ImportError: libmpi.so.40: cannot open shared object file: No such file or directory
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_speech_to_speech_model - AssertionError: 86400 != 86080
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_text_to_speech_model - AssertionError: 90560 != 86720
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_eng_text - AssertionError: Lists differ: [2, 10051, 8980, 8212, 949, 1270, 4311, 1123, 5918, 2333] != [2, 10051, 9253, 9253, 9253, 9253, 3765, 3765, 3765, 3765]
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_rus_speech - AssertionError: Lists differ: [2, 10067, 5729, 4798, 9631, 8378, 4446, 2393, 6901, 5983] != [2, 10067, 5560, 5560, 5560, 5560, 5560, 5560, 5560, 5560]
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelIntegrationTest::test_to_swh_text - AssertionError: Lists differ: [2, 10071, 5729, 9995, 3089, 7546, 1204, 1721, 2532, 4340] != [2, 10071, 2679, 2679, 2679, 2679, 2679, 2679, 9178, 9178]
=============== 7 failed, 89 passed, 187 skipped, 5 warnings in 171.26s (0:02:51) =========
```

Essentially such errors are resolved:
```
ValueError: This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie t2u_model.model.decoder.embed_tokens.weight (which should be present and is not), to t2u_model.lm_head.weight (which is present).
```
But integration tests are failing which is probably better for another PR?

cc: @vasqu, @ArthurZucker



Comments:
run-slow: seamless_m4t

This comment contains `run-slow`, running the specified jobs: 

models: ["models/seamless_m4t"]
quantizations: []

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42377). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19661588575)

✅ No failing test specific to this PR 🎉 !


Integration tests are actually failing: https://github.com/huggingface/transformers/actions/runs/19661588575/job/56309175315#step:14:21132

but as mentioned above, this is the case before the above mentioned fix

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: seamless_m4t

Hi! likely linked to https://github.com/huggingface/transformers/pull/42385 as well

Converting to draft for now and essentially closing it when we have the reverse mapping PR landing. Seems like more models are affected so it's not worth to create exceptions everywhere!

Should be fixed with #42507