Performance regression due to Bidirectional masks
Author: jiqing-feng
Pull request: no
Labels: name: bug

### System Info

torch                 2.10.0.dev20251008+cpu

### Who can help?

@vasqu @zucchini-nlp 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`numactl -C 0-7 --membind 0 python test_bert.py`

```python
import torch
import time
from transformers import pipeline

model_id = "sentence-transformers/all-MiniLM-L6-v2"
sentences = ["This is an example sentence", "Each sentence is converted"]
extractor = pipeline("feature-extraction", model=model_id, dtype=torch.float16)
encoded_input = extractor.tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")
# warm up
for _ in range(10):
    model_output = extractor(sentences, return_tensors=True, batch_size=2)

for _ in range(10):
    start = time.time()
    model_output = extractor(sentences, return_tensors=True, batch_size=2)
    end = time.time()
    print(f"latency: {(end-start)*1000} ms")
```

### Expected behavior

The performance has 40% regression after PR [41265](https://github.com/huggingface/transformers/pull/41265)

Comments:
The problem seems to be in sdpa_mask_recent_torch from masking_utils . For simple encoders it seems to be overkill . So we can probably override it with a conditional .
Before the fix:
latency: 17.93813705444336 ms
latency: 18.224716186523438 ms
latency: 18.910884857177734 ms
latency: 20.073652267456055 ms
latency: 12.088537216186523 ms
latency: 16.454696655273438 ms
latency: 14.774322509765625 ms
latency: 18.1427001953125 ms
latency: 17.52448081970215 ms
latency: 22.339582443237305 ms

After applying it 
latency: 11.677742004394531 ms
latency: 12.84337043762207 ms
latency: 10.970354080200195 ms
latency: 11.156558990478516 ms
latency: 9.819269180297852 ms
latency: 13.221979141235352 ms
latency: 9.453058242797852 ms
latency: 10.178327560424805 ms
latency: 10.057449340820312 ms
latency: 20.432472229003906 ms


If this is the correct solution I would love to submit a PR

Hmm, yea seems like the current vmapping is too much. 

@i3hz you can gladly submit a (draft) PR and I take a look! We would likely need to check if things are still in place for torch export and the other attention flavors.

cc @Cyrilvallez for masks as well, might affect more downstream perf as previously assumed

Can confirm the regression locally, e.g. something around 7ms vs 5ms. Also this regression definitely seems to come from the vmapping as indicated by @i3hz as when switching to the no vmapping version from executorch at https://github.com/huggingface/transformers/blob/a99b1be3c73173509f71c261834d93d3d5e30ef1/src/transformers/integrations/executorch.py#L1301-L1346
I also get numbers around 5ms again

https://github.com/huggingface/transformers/pull/41586
@vasqu  yeah I'm getting around ~5ms with this implementation too . Lmk if there's anything that I missed

<img width="1771" height="429" alt="Image" src="https://github.com/user-attachments/assets/b82dee24-3860-4acb-8a6a-95368256ef8d" />

@jiqing-feng can you check perf again. Checked locally and we are somewhat back to normal perf (max -5%) thanks to @i3hz 

Closing this PR but ping me if you notice something is off!