ModernBERT fails to work without FlashAttention !
Author: benhachy
Pull request: no
Labels: name: bug

name: flex attention

### System Info

# Issue: ModernBERT Without Flash Attention on Tesla V100

## Problem Description

I am trying to use **ModernBERT** without **Flash Attention**, as it requires an **Ampere GPU or newer** (see error below), while I only have a **Tesla V100 16GB**.

![Error Screenshot](https://cdn-uploads.huggingface.co/production/uploads/63ad8d4120176b2d2162112b/5XxXxd0fXzCkWaqg1EMKv.png)
*Figure 1: Error message indicating that Flash Attention requires an Ampere or newer GPU.*

To bypass **Flash Attention**, I explored the **transformers** source code and found that ModernBERT allows alternative attention implementations like **SDPA** or **Eager**. This can be set using:

```python
model.config._attn_implementation = ['flash_attention_2', 'eager', 'sdpa'][2]
```

However, this leads to the following unexpected error:

```python
TypeError: ModernBertUnpaddedRotaryEmbedding.forward() got an unexpected keyword argument 'position_ids'
```

The error originates from the **ModernBertAttention** module, specifically when applying the **rotary embeddings function** on the `qkv` tensor.

![Stack Trace Screenshot](https://cdn-uploads.huggingface.co/production/uploads/63ad8d4120176b2d2162112b/gX9eaF5DeHpnCW9ErF9t4.png)
*Figure 2: Stack trace showing the unexpected keyword argument error in ModernBertUnpaddedRotaryEmbedding.*

---

## Reproduction Code

Here is the full code to reproduce the issue:

```python
# import flash_attn_2_cuda as flash_attn_cud
from transformers import AutoTokenizer, AutoModel

# Model name
model_name = "answerdotai/ModernBERT-base"  

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name).to('cuda')

# Force SDPA Attention instead of Flash Attention 2
model.config._attn_implementation = ['flash_attention_2', 'eager', 'sdpa'][2]

# Test the model
text = "The capital of France is [MASK]."
inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model(**inputs)
```

---

## Error Location

After investigating the source code, the error occurs at **line 180** in the following file:  
🔗 [Transformers GitHub - ModernBERT](https://github.com/huggingface/transformers/blob/main/src/transformers/models/modernbert/modeling_modernbert.py#L180)

![Source Code Screenshot](https://cdn-uploads.huggingface.co/production/uploads/63ad8d4120176b2d2162112b/v3L77EW6s98T3qCmpmFrw.png)
*Figure 3: Source code snippet highlighting the line where the error occurs in ModernBertAttention.*

---

## Thoughts and Next Steps

This issue restricts ModernBERT's usability for those without Ampere GPUs, limiting **accessibility** to **Open Source models**. It would be helpful if:

- The model could gracefully fall back to **SDPA** or **Eager Attention** without errors.
- The **position_ids** issue in `ModernBertUnpaddedRotaryEmbedding` was addressed.

I hope this post helps locate the issue and contributes to making **Open Source AI more accessible**! 🚀


### Who can help?

@tomaarsen @ArthurZucker 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I have provided the full reproduction code in the description 

### Expected behavior

The expected behavior was to get the logits and not an Error :)

Comments:
Thanks to Tom Aarsen, the error has been resolved. Check out the discussion here if you're looking to run ModernBert on GPUs older than the Ampere series:

https://huggingface.co/answerdotai/ModernBERT-base/discussions/56