Impossible to change attention implementation
Author: ghost
Pull request: no
Labels: name: bug

### System Info

```
transformers==4.46.3
torch==2.5.1
flash-attn==2.7.1
```

### Who can help?

@amyeroberts @LysandreJik

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The `attn_implementation` argument is not respected in `from_pretrained`.

This means it is impossible to change the attention implementation for certain models, e.g. `google/gemma-2-27b`.


```python
from transformers import AutoModelForCausalLM
from transformers.utils import is_flash_attn_2_available

print(is_flash_attn_2_available())
# True

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-27b",
    torch_dtype="bfloat16",
    device_map="auto",
    attn_implementation="flash_attention_2",
)
print(model.config._attn_implementation)
# 'eager'
```

I think this is related the following changes:
- @amyeroberts: https://github.com/huggingface/transformers/pull/32383
- @LysandreJik: https://huggingface.co/google/gemma-2-27b/discussions/12

### Expected behavior

It should be possible to specify the `attn_implementation`. Especially as Flash Attention 2 is officially supported for Gemma 2.

Comments:
Can you update to 4.47? There was a refactor for gemma2 recently so it might be fixed already. 

cc @ArthurZucker 

Thanks for the suggestion @vasqu! Unfortunately the issue still exists after updating to v4.47.

Just to clarify, this issue is not specific to Gemma 2. It affects any model with `_attn_implementation` set in the `config.json`.

Gemma 2 is probably the most notable example on the Hugging Face Hub that uses this property.

maybe you can try this, this works for me
```python
AutoModelForCausalLM.from_config(
                             config,
                             trust_remote_code=True,
                             attn_implementation="flash_attention_2"
```

Hey @stoical07 

I think this should help if you are trying to directly change the `_attn_implementation`.

```diff
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-27b",
    torch_dtype="bfloat16",
    device_map="auto",
-  attn_implementation="flash_attention_2",
+ _attn_implementation="flash_attention_2",
)
```

Hey! Not sure I agree with @ariG23498, in config utils you have: `self._attn_implementation_internal = kwargs.pop("attn_implementation", None)` .
Are you sure you did not get any warnings? 
<img width="541" alt="image" src="https://github.com/user-attachments/assets/7196bd68-42ee-47d7-b4a8-2827c25dd104">
works alright for sdpa

Hi @ArthurZucker, I notice you changed the model from Gemma 2 27B to 2B. 

`_attn_implementation` is only set in the `config.json` for the 27B version.

@ArthurZucker I can reproduce the issue with SDPA also:

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-27b",
    torch_dtype="bfloat16",
    device_map="auto",
    attn_implementation="sdpa",
)
print(model.config._attn_implementation)
# 'eager'
```

Mmm, the `_attn_implementation` should not be serialized, but for gemma, we needed to force eager at some point. 
Now, the arg should take priority I think. If you update transformers it should be fixed!

Same outcome with v4.47.0 unfortunately @ArthurZucker 😢

I think we should remove the default to `eager` on the hub. This might break for `transformers<4.47`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.