Mistral: Add support for interleaved attention
Author: manueldeprada
Pull request: yes

Adds support for interleaved attention masks to the Mistral model.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39799). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: mistral

This comment contains run-slow, running the specified jobs: 

models: ['models/mistral']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: minimax, mistral, mixtral, phi3, phi4_multimodal, starcoder2

run-slow: mistral

This comment contains run-slow, running the specified jobs: 

models: ['models/mistral']
quantizations: [] ...

> Mistral model was never interleaved, so my first review is to go towards no. 
We can maybe define this in the config but hardcode it to be sliding. Our philosophy is to not add abstraction when it is not relevant.

[mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) is interleaved, as reported by @hmellor, quoting model's readme _"Trained with a 128k context window with interleaved sliding-window attention"_

I had understood that we wanted to support this 😅 But if Ministral isn’t relevant enough to justify the modeling change, sure just close the PR 🤗 

> We can maybe define this in the config but hardcode it to be sliding. 

The PR already does this: defaults are set to sliding or full attention unless the model's config says otherwise:
https://github.com/huggingface/transformers/blob/2166534a2d9660aadeeb67983925aaad1e4f39f7/src/transformers/models/mistral/configuration_mistral.py#L164-L167
(several Mistral models use full attention, [example here](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2))

> Same for mixtral: modeling code should be unchanged.

This PR doesn’t change Mixtral’s modeling, it only moves `Attention`’s `sliding_window` to `__init__` so inheriting models can modify it.

I don't mind the config, but the modeling code is not longer `mistral`, but `qwen2`. We need to check because that architecture already exists -> does not make sense for me to do an exception here ! So as much as possible let's check which model is thus the closest 

> I don't mind the config, but the modeling code is not longer `mistral`, but `qwen2`. We need to check because that architecture already exists -> does not make sense for me to do an exception here ! So as much as possible let's check which model is thus the closest

🤯 wow, you have every model on your head! I made a quick Ministral modular from Qwen2 and it matches exactly this PR's outputs with just the bias removal:
```
class MinistralAttention(Qwen2Attention):
    def __init__(self, config, layer_idx: int):
        super().__init__(config, layer_idx)
        self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
```

So I understand now! Whats the preferred approach?:
1. Give up on Ministral and only do the config change for external libraries (and maybe warn that Ministral may be incorrect on long context windows?).
2. Add a new Ministral model with the slim modular from Qwen2.


https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/blob/main/params.json (Mistral format config) shows that Ministral was supposed to be interleaved.

From what I understand about the history, when this model was originally contributed to Transformers, interleaved sliding attention was not supported, so the model was capped to use all sliding attention.

2. Add a new Ministral model with the slim modular from Qwen2. would be the best IMO! We can open a PR to support or just do a hardcode case for mistral + interleaved -> load Ministral

> 2. Add a new Ministral model with the slim modular from Qwen2. would be the best IMO! We can open a PR to support or just do a hardcode case for mistral + interleaved -> load Ministral

I am interested in learning about model bring-up, so I’ll open the PR! Closing this