Backwards incompatible change in returned hidden states
Author: BenjaminBossan
Pull request: no
Labels: name: bug

### System Info

Independent of system, requires latest transformers from main.

### Who can help?

As discussed internally @ArthurZucker @ydshieh @LysandreJik 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

The model outputs returned by passing `return_hidden_states=True` have changed recently. In particular, the order of the hidden states is affected by that change. This is a potentially severe issue, because on the surface, everything looks fine (same number of outputs, same shapes of outputs) but the indices are off by 1. As a user, it would be very hard to figure out what went wrong here, as there can be so many possible sources of why the output could have changed and because there was no deprecation period.

To reproduce the issue, run the following script, once with the current main of transformers and once with the last release, v4.53.2:

```python
import torch
import transformers
from transformers import AutoModelForCausalLM

print(f"{transformers.__version__=}")
model_id = "facebook/opt-125m"        # OPT works
model_id = "meta-llama/Llama-3.2-1B"  # Llama is broken
model = AutoModelForCausalLM.from_pretrained(model_id)
inputs = torch.arange(10).view(1, -1)
with torch.inference_mode():
    output = model(inputs, output_hidden_states=True).hidden_states
print(f"Number of returned hidden states: {len(output)}")

for i, hs in enumerate(output):
    print(f"hidden state {i:>2} values (slice): {hs[0, 0, :3]}")
```

The results are for main:

```
transformers.__version__='4.54.0.dev0'
Number of returned hidden states: 17
hidden state  0 values (slice): tensor([-0.0010,  0.0339,  0.0018])
hidden state  1 values (slice): tensor([ 0.2473, -0.5012,  1.5765])
hidden state  2 values (slice): tensor([ 0.2444, -0.5238,  1.5824])
hidden state  3 values (slice): tensor([ 0.2463, -0.5339,  1.5775])
hidden state  4 values (slice): tensor([ 0.2797, -0.5208,  1.6037])
hidden state  5 values (slice): tensor([ 0.2424, -0.4786,  1.6114])
hidden state  6 values (slice): tensor([ 0.2291, -0.4361,  1.5910])
hidden state  7 values (slice): tensor([ 0.2474, -0.3626,  1.5842])
hidden state  8 values (slice): tensor([ 0.2487, -0.3277,  1.4880])
hidden state  9 values (slice): tensor([ 0.1552, -0.2746,  1.4607])
hidden state 10 values (slice): tensor([ 0.1235, -0.2442,  1.3794])
hidden state 11 values (slice): tensor([ 0.1107, -0.2346,  1.3300])
hidden state 12 values (slice): tensor([ 0.0789, -0.2590,  1.3076])
hidden state 13 values (slice): tensor([ 0.3292, -0.3711,  1.1052])
hidden state 14 values (slice): tensor([ 0.4628, -0.6839,  1.1164])
hidden state 15 values (slice): tensor([ 0.4699, -0.7124,  4.8454])
hidden state 16 values (slice): tensor([ 0.2919, -0.4089,  1.8382])
```

v4.53.2

```
transformers.__version__='4.53.2'
Number of returned hidden states: 17
hidden state  0 values (slice): tensor([0.0045, 0.0166, 0.0210])
hidden state  1 values (slice): tensor([-0.0010,  0.0339,  0.0018])
hidden state  2 values (slice): tensor([ 0.2473, -0.5012,  1.5765])
hidden state  3 values (slice): tensor([ 0.2444, -0.5238,  1.5824])
hidden state  4 values (slice): tensor([ 0.2463, -0.5339,  1.5775])
hidden state  5 values (slice): tensor([ 0.2797, -0.5208,  1.6037])
hidden state  6 values (slice): tensor([ 0.2424, -0.4786,  1.6114])
hidden state  7 values (slice): tensor([ 0.2291, -0.4361,  1.5910])
hidden state  8 values (slice): tensor([ 0.2474, -0.3626,  1.5842])
hidden state  9 values (slice): tensor([ 0.2487, -0.3277,  1.4880])
hidden state 10 values (slice): tensor([ 0.1552, -0.2746,  1.4607])
hidden state 11 values (slice): tensor([ 0.1235, -0.2442,  1.3794])
hidden state 12 values (slice): tensor([ 0.1107, -0.2346,  1.3300])
hidden state 13 values (slice): tensor([ 0.0789, -0.2590,  1.3076])
hidden state 14 values (slice): tensor([ 0.3292, -0.3711,  1.1052])
hidden state 15 values (slice): tensor([ 0.4628, -0.6839,  1.1164])
hidden state 16 values (slice): tensor([ 0.2919, -0.4089,  1.8382])
```

As can be seen, the results are _mostly_ off by one, i.e. index `i` for main corresponds to index `i+1` for v4.53.2. However, the last index is identical for both, the 0th index of v4.53.2 is not found in main, and the second to last index of main is not found in v4.53.2.

The exact PR causing this change is #39120, which can be verified by checking out its commit (`ca7e1a3756c022bf31429c452b2f313f043f32de`) and comparing the results to the previous commit (`e6a8063ef1af16df964b644b07e1d17e96555d23`).

When replacing Llama with OPT, the error goes away, the hidden states remain the same between transformers versions. This is probably because the embedding is part of the decoder block in OPT but outside the decoder block of Llama.

### Expected behavior

The returned hidden states should not change between transformers versions.

Comments: