ViTLayer.forward() needs to be in "eager" mode when `output_attentions=True`
Author: alexsherstinsky
Pull request: no

### System Info

Transformers version 4.41.1 (Python 3.10).
@amyeroberts 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hello:

With `output_attentions=True` in the latest transformers, _attn_implementation still stays at "sdpa" when one uses `transformer = ViTModel.from_pretrained(pretrained_model)`.  Unfortunately, this leaves the `output_attentions` as all `None`.  In version `4.40.0` for example, the element of the `output` at index one had the correct values of the self-attention tensor, because there was no eager/sdpa mode then.  In order to fix this situation, the user has to obtain the configuration, update `_attn_implementation`, and the re-instantiate the `ViTModel` (e.g., `transformer = ViTModel(config)`).  Could this please instead be handled in `https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/modeling_vit.py#L382` instead by doing something like this:

1. Save the config in the constructor --
```
self._config = config
```

and then in `forward()`:
```
if output_attentions:
            attention = VIT_ATTENTION_CLASSES["eager"](self._config)
            self_attention_outputs = attention(
                self.layernorm_before(hidden_states),  # in ViT, layernorm is applied before self-attention
                head_mask,
                output_attentions=output_attentions,
            )
else:
            self_attention_outputs = self.attention(
                self.layernorm_before(hidden_states),  # in ViT, layernorm is applied before self-attention
                head_mask,
                output_attentions=output_attentions,
            )
```

If you think that this would work, then I would be happy to issue a pull request (and so, please, kindly, point me to the right module, where I would add a unit test).

Thank you very much for your time and consideration.

### Expected behavior

One does not get `None` values for self_attentions when `output_attentions=True` -- one gets actual tensors, like it was in earlier versions.

Comments:
Hi @alexsherstinsky, thanks for raising this issue! 

So, there's two things here. 

1) Defaulting to SDPA like this is indeed a breaking change - we should default to the eager option and allow the user to opt-in to using SDPA or flash attention. The way the default attention is selected is a bit fiddly/annoying. Changing this might also be inconsistent with other model implementations, which assume defaults to e.g. FA2 if possible. I'm going look into this and see if there's a nice way to revert. 

2) You don't need to save a config or reload a model. You can specify the attention mechanism upon instantiation: 

```py
In [1]: from transformers import ViTConfig, ViTForImageClassification
   ...:
   ...: # Randomly initialized weights
   ...: config = ViTConfig(attn_implementation="eager")
   ...: model = ViTForImageClassification(config)
   ...: print(model.config._attn_implementation)
   ...:
   ...: # From a checkpoint
   ...: model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224", attn_implementation="eager")
   ...: print(model.config._attn_implementation)
eager
eager
```

@amyeroberts Thank you very much for your response with the explanation as well as your valuable suggestion for how to simplify my work-around (I implemented it, and it worked).  One quick request: if you end up changing the default implementation of the attention layer method, please, kindly, include it in the release notes, so that we can update our code accordingly.  Thanks again!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.