🐛 `attn_implementation="sdpa"` slower than `BetterTransformer.transform`?
Author: vibhas-singh
Pull request: no

### System Info

- `transformers` version: 4.41.1
- Platform: Linux-5.10.215-203.850.amzn2.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.14
- Huggingface_hub version: 0.23.2
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @younesbelkada


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to optimise a fine-tuned BERT model for sequence classification using lower precision and `SDPA`.
I am observing different behaviour while opting for `SDPA` using native transformers as compared to using `BetterTransformers`.

I have a local dataset and I am using that for recording the inference time for different settings - any dummy dataset or any dummy model can be used to reproduce the beviour.
Every experiment done is using same dataset. 
`batch_size` is 128 and `max_length` is 128 for all the runs.
Model performance is unchanged for all the runs.
GPU: A10G


### Experiment 1
```python
model = AutoModelForSequenceClassification.from_pretrained(model_path)
model.eval().to("cuda")
### Inference time: 133s
```
### Experiment 2
```python
model = AutoModelForSequenceClassification.from_pretrained(model_path)
model.eval().to("cuda")
model = model.half()
### Inference time: 55s
```

### Experiment 3
```python
model = AutoModelForSequenceClassification.from_pretrained(model_path, attn_implementation="sdpa")
model.eval().to("cuda")
model = model.half()
### Inference time: 55s
```

### Experiment 4
```python
model = AutoModelForSequenceClassification.from_pretrained(model_path, torch_dtype=torch.float16, attn_implementation="sdpa")
model.eval().to("cuda")
### Inference time: 55s
```

### Experiment 5
```python
model = AutoModelForSequenceClassification.from_pretrained(model_path)
model.eval().to("cuda")
model = model.half()
model = BetterTransformer.transform(model)
### RuntimeError: shape '[128, 128]' is invalid for input of size 2097152
```

### Experiment 6
```python
### When downgrading to transformers==4.36.1
model = AutoModelForSequenceClassification.from_pretrained(model_path)
model.eval().to("cuda")
model = model.half()
model = BetterTransformer.transform(model)
### Inference time: 30s
```





### Expected behavior

1. Ideally, going from Exp 2 to Exp 3/4 should give some time reduction in inference which is not the case here.
2. Is there any difference between `SDPA` implementations in `Transformers` vs `BetterTransformers`? Because I am able to achieve much better performance in terms of inference time using `BetterTransformers` as compared to `Transformers` (compare Exp 6 with Exp 3/4) - which isn't intuitive. Ideally, both should be same.

Comments:
1. "sdpa" is the default attention implementation even if you don't specify explicitly
2. BetterTransformer will do more optimizations than just replace the model's attention implementation
3. different input settings might influence the exact backend sdpa uses, you might want to set the backend of sdpa explicitly and use a profiler to see which part exactly is running faster/slower

> 2. BetterTransformer will do more optimizations than just replace the model's attention implementation

Why, in that case, is the recommendation in the [documentation](https://huggingface.co/docs/optimum/bettertransformer/overview) for `optimum` to deprecate BetterTransformer where SDPA is available?

I'm not exactly sure what BetterTransformer is doing but I have observed that it is able to significantly speed up my models (typically encoder models) on Windows despite flash attention not being available. Trying to use SDPA on Windows has, from my memory, not worked.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Encoder models might not alll have sdpa available in transformers directly! 

We checked the performance between sdpa and BetterTransformers on my companies project, we observed no difference in performance.

yep that is what we expect!

@ArthurZucker @Rithin-Draup personally, I have observed a *significant* difference, both in company projects and in my personal projects, both when it comes to training and when it comes to inference.

This is a super simple example that illustrates just how valuable BetterTransformer is:
```python
import torch

from transformers import RobertaModel, RobertaTokenizerFast

# BEGIN CONFIG #
MODEL_NAME = 'umarbutler/emubert'
EXAMPLE_INPUT = "\
The Parliament shall, subject to this Constitution,\
have power to make laws for the peace, order, and good\
government of the Commonwealth with respect to:\
    (i) trade and commerce with other countries, and among\
        the States;\
    (ii) taxation; but so as not to discriminate between"""
# END CONFIG #

sdpa_model = RobertaModel.from_pretrained(MODEL_NAME, attn_implementation = 'sdpa').to(torch.bfloat16).to('cuda').eval()
bettertransformer_model = RobertaModel.from_pretrained(MODEL_NAME).to(torch.bfloat16).to_bettertransformer().to('cuda').eval()
tokenizer = RobertaTokenizerFast.from_pretrained(MODEL_NAME)
input_tokens = tokenizer(EXAMPLE_INPUT, return_tensors='pt').to('cuda')

with torch.inference_mode():
    # Do unbenched forward passes to control for potential caching effects.
    for _ in range(10):
        bettertransformer_model(**input_tokens)
        sdpa_model(**input_tokens)
    
    # Benchmark the models.
    %timeit bettertransformer_model(**input_tokens)
    %timeit sdpa_model(**input_tokens)
```

On my 4090, BetterTransformer achieves `1.93 ms ± 104 μs` and SDPA achieves `3.64 ms ± 259 μs`. BetterTransformer is almost 2x faster (1.88x)...

See also [optimum#2083](https://github.com/huggingface/optimum/issues/2083), which I have created to advocate for the retention of BetterTransformer until Hugging Face `transformers` is able, if ever, to achieve parity.

RoBERTa base is downloaded 20 million times a month on its own. A 1.88x speed is not only meaningful but immensely valuable to users of BERT-like models.

See also [transformers#34488](https://github.com/huggingface/transformers/issues/34488).

Hey! thanks for your feedback. You are right in the fact that it's important to make sure `transformers` has great performances, and I am devoted to make sure you don't have to use better transformers for that! 🤗 

1. Are you sure better transformers does not use flash attention by default
2. You can compile the transformers models, which should also boost the performances
3. we'd need to spend some time investigating a bit more, and the people who implemented this are both gone, so would take a bit more time ! 

As you have used both heavily, if you have a better idea of where the slowdown comes from, I can review a PR ! 🤗