🚨  Generalize `get_decoder()` for multimodal and delete redundant code 🔪 
Author: zucchini-nlp
Pull request: yes
Labels: name: for_v5?

# What does this PR do?

As per title, blocked by https://github.com/huggingface/transformers/pull/41589 for VLMs! We should be able to use `get_decoder()` to get the LM part of any model after this and have much less duplicate code. Same foes for the `get_encoder()` to get the encoder if the model has a separate encoding module. In comparison to decoder, we can have specific encoder per modality so the helper will accept `modality` as arg

Universal helper first reduces duplicate code, nudges us to use standardized names for major modules and can be used by 3rd party libraries. Right now we have 5 ways to name a vision encoder!

####  🚨 Breaking changes, ig we can break helpers for v5:
- VLMs now will not have a property to get `self.language_model` directly from task-model and users will need to call `self.get_decoder()`
- Deleted `get_text_encoder` and `get_audio_encoder` in some audio models because functionality is covered now by the `get_encoder()`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42156). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merge conflicts after a big refactor 😢 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, autoformer, aya_vision, bart, bigbird_pegasus, blenderbot, blenderbot_small, blip_2, cohere2_vision, colqwen2, conditional_detr, d_fine, dab_detr, deformable_detr, detr, dia

hey @jackzhxng, I remember you requested this feature for `torch.export`. Now you can

```python
multimodal_model.get_decoder() -> returns the decoding LM
multimodal_model.get_encoder() -> returns the encoding LM if any
multimodal_model.get_encoder(modality="image") -> returns the encoding vision tower if any
multimodal_model.get_encoder(modality="audio") -> returns the encoding audio tower if any
```

also cc @hmellor, we also discussed it re vLLM

Hi @zucchini-nlp this PR causes an issue with PEFT as (at least some) decoder models now have `get_encoder`. As an example:

```python
from transformers import AutoModelForCausalLM

model_id = "facebook/opt-125m"
model = AutoModelForCausalLM.from_pretrained(model_id)
assert not hasattr(model, "get_encoder")
# after this PR, model.get_encoder() returns model.model
```

This works with the previous commit (`a5c903f877fda21e739027eed133e03162eb7712`) but fails after this PR (`e2fb8d6062a05f69f976cf6e39618df6c31a3bfd`). Is this change intended?

@zucchini-nlp this is amazing thank you!

@BenjaminBossan ideally it should return `self` and not the base model. I see where it comes from and will fix it. Then you can check with smth like - `self.get_encoder != self: encoder = self.get_encoder()`

All models will have `get_encoder()` implemented, but if a model has no actual encoder it should return `self`

> I see where it comes from and will fix it.

Great, please let me know when the PR is there.

> All models will have `get_encoder()` implemented, but if a model has no actual encoder it should return `self`

We can modify PEFT to take this into account. But at least to me, this API feels a bit strange to be honest.

@BenjaminBossan yeah, it is because we have `get_encoder()` in PreTrainedModel so all models will have it as an attribute. Might feel weird though it is same as it was with `get_decoder()` and I think it saves us from duplicating the same code in all models

@zucchini-nlp I believe Voxtral actually still has top-level `language_model` - https://github.com/huggingface/transformers/blob/main/src/transformers/models/voxtral/modeling_voxtral.py#L379

Additionally woult it be possible for `get_decoder` to return the causal variant of the model which includes the `lm_head`?

Ah, Voxtral has a causal model as backbone. We need to explicitly overwrite `get_decoder` for the model in that case, will do

Oh, which includes the lm_head, I misread it!

Nah, the base model is supposed to be the model without any task head on top so in Voxtral, it is going to be the language model without a head