LlamaSdpaAttention vs output_attentions=True
Author: iBibek
Pull request: no

I am trying to get the attention matrix from LLAMA-3-70B-Instruct by doing:

```
outputs = model.generate(
            input_ids,
            output_attentions=True,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            return_dict_in_generate=True,
        )
```
And I received this warning.

Warning:

LlamaModel is using LlamaSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.

**Is the attention matrix received this way equivalent to the original LLAMA attention implementation?**

Comments:
Facing the same issue in Llama-2 as well. Any resolution @iBibek ?

The warning indicates that when `output_attentions=True` is set, the `LlamaSdpaAttention` mechanism reverts to a manual implementation because `torch.nn.functional.scaled_dot_product_attention` does not support this feature. From Transformers version v5.0.0 onwards, specifying the manual implementation explicitly will be necessary.

### Attention Matrix Equivalence
The attention matrix obtained through the manual implementation should be equivalent to the original LLAMA attention implementation. The fallback mechanism ensures consistency in functionality.

### Solution
To resolve this warning and ensure the correct attention mechanism is used, you should load the model with the `attn_implementation="eager"` argument. Here’s how you can update your code:

```python
from transformers import LlamaForCausalLM, LlamaTokenizer

# Load model with eager attention implementation
model = LlamaForCausalLM.from_pretrained("facebook/llama-3b", attn_implementation="eager")
tokenizer = LlamaTokenizer.from_pretrained("facebook/llama-3b")

# Prepare inputs
input_ids = tokenizer.encode("Your input text here", return_tensors="pt")

# Generate outputs with attention
outputs = model.generate(
    input_ids,
    output_attentions=True,
    max_new_tokens=50,
    do_sample=False,
    return_dict_in_generate=True,
)

# Access the attention matrices
attention_matrices = outputs.attentions
print(attention_matrices)
```

### Summary
- **Warning**: This is due to the `LlamaSdpaAttention` mechanism and the lack of support for `output_attentions=True` in PyTorch's scaled dot product attention.
- **Solution**: Use `attn_implementation="eager"` when loading the model.
- **Equivalence**: The attention matrix obtained this way is equivalent to the original LLAMA attention implementation.

@ashishpatel26 Please refrain from using LLM generated reply to issues. They are generally overly verbose and most often do not answer the question, resulting in a worse experience for our users. 

Hi @iBibek, thanks for raising an issue! 

> Is the attention matrix received this way equivalent to the original LLAMA attention implementation?

Could you clarify a bit what you mean? When you say "the attention matrix received this way" - what operation are you asking about i.e. what is receiving the attention matrix from what? 

The flag `output_attentions` controls whether you will get the attention weights returned in the model output. The attention calculation should essentially be the same (with some minor numerical differences). 

You can see the [eager implementation here](https://github.com/huggingface/transformers/blob/cb23d1b20b4450195564f107ac873e5ecd616def/src/transformers/models/llama/modeling_llama.py#L359); and its [SDPA equivalent here](https://github.com/huggingface/transformers/blob/cb23d1b20b4450195564f107ac873e5ecd616def/src/transformers/models/llama/modeling_llama.py#L556)

My apologies on the bad question.

I was trying to get the attention using `output_attentions=True` from LLAMA3, when I got the above warning message. 

> The attention calculation should essentially be the same (with some minor numerical differences.

This is what I was trying to verify.

Thank you for the shared resources. 

Hi @amyeroberts , I am trying to evaluate Meta-Llama-3-8B performance on PiQA dataset using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). When I was using the default config `attn_implementation='sdpa'`, I got the result below:
```
hf (pretrained=/cpfs01/user/landisen/models/Meta-Llama-3-8B), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 256
|Tasks|Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-----|------:|------|-----:|--------|---|-----:|---|-----:|
|piqa |      1|none  |     0|acc     |↑  |0.7949|±  |0.0094|
|     |       |none  |     0|acc_norm|↑  |0.8047|±  |0.0092|
```

But if I used `attn_implementation='eager'`, the model performance droped dramatically:

```
hf (pretrained=/cpfs01/user/landisen/models/Meta-Llama-3-8B), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 256
|Tasks|Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-----|------:|------|-----:|--------|---|-----:|---|-----:|
|piqa |      1|none  |     0|acc     |↑  |0.5218|±  |0.0117|
|     |       |none  |     0|acc_norm|↑  |0.5109|±  |0.0117|
```

So I guess there will be obvious differences between `LlamaSdpaAttention` and `LlamaAttention`.