Make `VisionEncoderDecoderModel.from_pretrained` take different arguments for encoder and decoder 
Author: MohamedAliRashad
Pull request: no
Labels: name: Usage

name: Feature request

### Feature request

```python
model = VisionEncoderDecoderModel.from_pretrained("nougat", encoder_kwargs={...}, decoder_kwargs={...})
```

### Motivation

I want to load VisionEncoderDecoderModel with flash_attention_2 for the decoder only (the encoder is not supporting this to my knowledge) but i don't find an easy way to do so

### Your contribution

I can help if its simple

Comments:
You can already pass `encoder_config` and `decoder_config` 🤗 

@MohamedAliRashad The simple way to specify attention implementation for sub-models will be added with  https://github.com/huggingface/transformers/pull/32238 when it's merged.

@qubvel Any idea when will this PR merged ?


@ArthurZucker The decoder config doesn't have something about loading the model in flash attention 2

Yeah it's actually not supported yet, we are cooking something to make it simpler, in the mean time you can overwrite the properly directly if you want / use the linked PR! 
Hope to get this merged soon! 
🤗 

@MohamedAliRashad the PR is merged and you should be able to load models with smth like this:
`Model.from_pretrained(model_id, attn_implementation={"decoder_config": "flash_attention_2", "encoder_config": "eager"})`



@zucchini-nlp I tried it and it's working
Thanks a lot guys ^^

@zucchini-nlp 
False Alarm, the loaded model is in `sdba` and not `flash-attn-2`
```
model = VisionEncoderDecoderModel.from_pretrained(
    "facebook/nougat-base", torch_dtype=torch.bfloat16, attn_implementation={"decoder_config": "flash_attention_2", "encoder_config": "eager"}
)
print(model)
```

![Screenshot from 2024-10-30 15-36-19](https://github.com/user-attachments/assets/a2df82e6-8b59-4b32-ab7d-ef11355e9691)



@MohamedAliRashad my bad, for the model you are using the keys are `decoder` and `encoder`. The main thing is that the keys in attn implementation should match 100% to the keys in `model.config`. So if you have `model.config.decoder` then it should be `decoder`. The below code should work

```python
model = VisionEncoderDecoderModel.from_pretrained(
    "facebook/nougat-base", torch_dtype="bfloat16", attn_implementation={"decoder": "flash_attention_2", "encoder": "eager"}
)
```

@zucchini-nlp It Worked
![Screenshot from 2024-10-30 17-31-53](https://github.com/user-attachments/assets/74a79597-1675-4fae-aa62-94031f9b0fa4)

Thanks a lot, i will close the issue for good this time ^^


Very cool to see it merged! Amazing work @zucchini-nlp! 🚀 
Regarding the issue above, can we check that there are no keys left that do not match config attributes and raise an error? wdyt?

@qubvel maybe we can try to add a "logger.info" in case if user passes a dict and the dict has no keys associated with some of sub-configs. I just don't want to make more warnings, afaik ppl are trying to reduce number of unnecessary warnings. I was thinking lately about better docs for VLMs and other composite models, so I'd better document that clearly. Just need to figure out what content goes in it

Feel free to share if you had ideas or things to be added in docs :)

Agreed about the warning, it can be skipped or disabled, so it's not actionable enough. IMO this situation should raise an _error_ because passed keys are wrong, so anyone can immediately reflect and fix it