GPT2Model model output inconsistency between different transformers versions
Author: wenzhong2005
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.41.0 or 4.42.0
- Platform: macOS-15.1-arm64-arm-64bit
- Python version: 3.11.7
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

The python code to reproduce the results:

import torch
import tokenizers
import transformers
from transformers import GPT2Model, GPT2Tokenizer

#Sample input
tokenizer = GPT2Tokenizer.from_pretrained(“distilgpt2”)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = “left”

text = ‘Model output changed’
model_inputs = tokenizer(text, padding=‘max_length’, max_length=12,
truncation=True, return_tensors=“pt”)
input_ids, attention_mask = model_inputs[“input_ids”], model_inputs[“attention_mask”]
print(‘input_ids:’, input_ids)
print(‘mask:’, attention_mask)

#Load GPT-2 Model
model = GPT2Model.from_pretrained(“distilgpt2”)
model.eval()

#Run model
with torch.no_grad():
outputs = model(input_ids=input_ids, attention_mask=attention_mask)

last_hidden_state = outputs.last_hidden_state
print(last_hidden_state)


### Expected behavior

We fine-tuned the GPT2Model (distilgpt2) some time ago. Due to tool vulnerability issues, we have to upgrade transformers 4.48.0 or above. However, the exact same GPT2 model produces different outputs for the exact same input after the upgrading. Therefore, after applying a classification head (linear layer) on top of GPT-2 output, we got different scores for the same input. After digging into it a little deeper, I found that the model output inconsistency was introduced between transformers v4.41.0 and v4.42.0. Also, the model output inconsistency lies in the masked portion only, but not in the unmasked portion. Can anyone help to take a look at it?

Here are the 2 requirements.txt files with the 2 versions of transformers and their corresponding model outputs:
First run:
torch==2.4.0
transformers==4.41.0
huggingface_hub==0.27.1

input_ids: tensor([[50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 17633, 5072, 3421]])
mask: tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]])
Model output:
tensor([[[-0.1352, 0.0991, -0.2160, …, -0.1755, -0.0512, -0.0338],
[-0.5171, -0.0978, -0.3561, …, -0.3091, 0.1552, -0.1503],
[-0.4233, -0.1778, -0.1415, …, -0.0925, 0.1203, -0.1014],
…,
[-0.3410, 0.2196, -0.1369, …, -0.4246, 0.3772, -0.4357],
[-0.6979, 0.1779, -1.0862, …, -0.5422, 0.1065, -0.2090],
[-0.5766, 0.1015, -0.2526, …, -1.4290, -0.1708, 0.1124]]])

Second run:
torch==2.4.0
transformers==4.42.0
huggingface_hub==0.27.1

input_ids: tensor([[50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 17633, 5072, 3421]])
mask: tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]])
Model output:
tensor([[[-5.1260e-02, 1.1421e-01, -6.7051e-02, …, -8.8936e-02,
-7.6510e-02, 8.6264e-03],
[-1.5280e-01, -5.6395e-02, 2.1665e-01, …, 1.1190e-01,
2.2004e-02, -9.5938e-02],
[-1.1987e-01, -5.4886e-02, 2.0053e-01, …, 1.3524e-01,
-4.1297e-04, -8.2952e-02],
…,
[-3.4099e-01, 2.1960e-01, -1.3687e-01, …, -4.2462e-01,
3.7722e-01, -4.3574e-01],
[-6.9789e-01, 1.7786e-01, -1.0862e+00, …, -5.4218e-01,
1.0647e-01, -2.0897e-01],
[-5.7657e-01, 1.0148e-01, -2.5263e-01, …, -1.4290e+00,
-1.7080e-01, 1.1240e-01]]])

Comments:
It seems that the default attention changed to "sdpa" for torch > ="2.1.1". If I set attn_implementation=“eager”, it resolves the model output consistency issue. Therefore, the problem probably lies in the "sdpa" implementation. 

Confirmed the bug! Outputs are equivalent on GPU and CPU with `eager` attn_implementation, but different when `attn_implementation=="sdpa"`. The CPU and GPU outputs for SDPA are also different from each other, in addition to both differing from the `eager` outputs. The size of the error in the final states is > 0.1, which is very large for a model as small as `distilgpt2`.

cc @arthurzucker for attention but let me know if you're overloaded and I can reassign this one!

@wenzhong2005 Have you tried newer versions e.g. `4.43<=`? Will probably have time on the weekend to look into this.

@vasqu I tried almost all the available versions, including 4.48, 4.49, and I got the same results. So I don't think it's been fixed yet.

Yea edited my comment because I was too fast and only saw it afterwards - sry about that. Like I said I'll probably take a look on the weekend.

On another thought, it's only affecting padding tokens iiuc? Are you applying some pooling as well? I'd assume that if you only apply a linear layer then every token independently works correctly (for the non-padded tokens). 

Edit: Imo it might be a non-issue if it's only affecting padding tokens as subsequent usage should ignore padding tokens as well. Otherwise, flash attention 2 would also be considered as incorrect implementation as padding is handled differently there as well.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

might be juste the default going from `eager` to `sdpa` WDYT? 

Yup, it's the reason for different padded values but that's also likely due to hacks to enable sdpa efficient backend (when full masking a full 0 row for example iirc).

But since it only happens on padded values and non-padded values are not affected, I think it's a non-issue. @ArthurZucker