past functionality broken in release 2.9.0 and 2.9.1
Author: Damiox
Pull request: no

# 🐛 Bug

## Information

Model: gpt2

## To reproduce

Steps to reproduce the behavior:

```
from transformers.tokenization_gpt2 import GPT2Tokenizer
from transformers.modeling_gpt2 import GPT2LMHeadModel
import torch

# Remember to run transformers with latest master (not release 2.5.1)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='
')
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Complete phrases are: "I like to drink soda without sugar" and "Go watch TV alone, I am not going"
doc = "I like to"
# note: comment the above line and uncomment the following line to make it work with 1 document
docs_tensors = tokenizer.batch_encode_plus([doc], pad_to_max_length=True, return_tensors='pt')

docs_next = [" soda and ", " with this"]
# note: comment the above line and uncomment the following line to make it work with 1 document
docs_next_tensors = tokenizer.batch_encode_plus(
    [d for d in docs_next], pad_to_max_length=True, return_tensors='pt')

# predicting the first part of each phrase
_, past = model(docs_tensors['input_ids'], attention_mask=docs_tensors['attention_mask'])

# manipulating the past
past_expanded = [torch.repeat_interleave(layer, torch.LongTensor([2]), dim=1) for layer in past]
past_attention_mask = torch.ones(docs_next_tensors['attention_mask'].shape[0], len(docs_tensors['input_ids'][0]), dtype=torch.int64)
attn_mask = torch.cat([past_attention_mask, docs_next_tensors['attention_mask']], dim=-1)

# predicting the rest of the phrase with past
logits, _ = model(docs_next_tensors['input_ids'], attention_mask=attn_mask, past=past_expanded)
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Predictions for:", [doc + n for n in docs_next])
print("Results with past:", words)

#####################
docs_full_tensors = tokenizer.batch_encode_plus(
    [doc + n for n in docs_next], pad_to_max_length=True, return_tensors='pt')
logits, _ = model(docs_full_tensors['input_ids'], attention_mask=docs_full_tensors['attention_mask'])
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Predictions for:", [doc + n for n in docs_next])
print("Results without past:", words)

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I am expecting the past functionality to work. But I'm getting the following error:
```
RuntimeError: The size of tensor a (4) must match the size of tensor b (5) at non-singleton dimension 3
```
This was working correctly up to release 2.8.0

## Environment info
    
- `transformers` version:
- Platform: Macos
- Python version: 1.4.0 / 1.5.0
- PyTorch version (GPU?): 2.9.0 cpu / 2.9.1 cpu
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
Hi @Damiox,

I think I see the problem. That's quite an edge case here. Since `2.8.0` (here the PR: https://github.com/huggingface/transformers/pull/3734), the `input_ids` will always be cut to just the last token when using `past` to have a consistent API with other models and allow to keep putting in the whole `input_ids` when speeding up decoding. Also `past` was never supposed to be used together with `input_ids` longer than 1, so I did not think of this edge case :-/. I'm quite surprised that this even worked before, but I'm also not sure whether the `causal mask` in GPT2 was correct for this case before.

I think a simple fix in your case would be to generate the output incrementally and only use `input_ids` of length 1 or not use the `past` variable at all here.

@LysandreJik, looking at this script, I think I did introduce a breaking change in the PR: https://github.com/huggingface/transformers/pull/3734 for this edge case when:
`past` is given to the model **and** `input_ids` is longer than 1. It's quite an edge case here and I'm not sure whether it's worth reversing the PR. What do you think? 




Hi @patrickvonplaten - I'm a bit worried about this... I cannot apply the simple fix because I'm unable to get rid of `past` variable or run more inferences at this point because the performance would be affected for my use case which is over a realtime bigdata dataset. The current results are accurate and are based on discussions in https://github.com/huggingface/transformers/issues/3095 - this is already deployed.

Just to be clear... Will this not be fixed?

That's an interesting use-case indeed. I don't believe we ever intended the `past` to be used with `input_ids` larger than one, and I'm quite surprised as well that it did work.

@thomwolf, do you have any opinion on this?

I'm really looking forward to knowing if this can be fixed so to continue working as it was working up to v2.8.0. Any new thoughts on this? Thanks

Hi @Damiox,

Sorry to answer so late. We had some discussion internally about it. And you are correct, we should revert the PR I merged earlier. This is done in #4581 and should be included in the next version. @thomwolf @LysandreJik