Higher memory allocation for GPT-2 forward pass when not using past_key_values.
Author: LFruth
Pull request: no

I have following problem: 
I have a context text as input and then GPT-2 generates a text based on the input.
Currently I do this token for token the following way:

```
model_outputs = model(input_ids=last_token, past_key_values=past)
logits = model_outputs['logits']
past = model_outputs['past_key_values']
# do stuff with the logits to determine the next token...
```

This does work and is pretty fast, but the problem is the high gpu memory consumption by always saving the past_key_values.
AMP is enabled too!

To avoid this I had a trade-off in mind: Only save and use the past_key_values from the context text and put in the generated text so far as input_ids, instead of just the last token! 
```
model_outputs = model(input_ids=generated_tokens, past_key_values=past, use_cache=False)
logits = model_outputs['logits']
# do stuff with the logits to determine the next token...
```

But now the memory allocation per generated token is even higher. Even though I didn't change anything else in the code.
Can anyone help me with this, or explain what exactly happens here?

## Environment info
- `transformers` version: 4.15.0
- Platform: Windows-10-10.0.19041-SP0 (also tested on Ubuntu)
- Python version: Python 3.7.3
- PyTorch version (GPU?):  1.10.1+cu102 (True)
- Tensorflow version (GPU?): not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten, @narsil



Comments:
Hi @LFruth ,

I think `past_key_values` should never incure more memory usage than without (since it's really a cache of matrices that have to be caculated anyway during inference.).


What could be at play here, is that you are using custom code, keeping references to old past_key_values, which means an ever growing amount of memory. Can you provide your full script ?

Do you mind trying to use `model.generate(....)` and see if it also has ever growing memory usage (more than expected) ? And if you want to play with the logits, you can add a custom `LogitsProcessor` in order to use some rule which isn't already integrated into `transformers`.

I can help providing an example on how to do so if that's the case.

Hi @Narsil,

Thank you for the quick answer.
What you say about past_key_values makes sense to me! 
I don't keep any references to past_key_values anywhere afaik, but I copy my code here later!

I already have the same thing implemented with model.generate! Which is quite interesting, as it is a little slower but has almost no extra memory allocation. I tried to play with past_key_values and use_cache here too, but didn't notice a difference in speed and or memory allocation at all.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I encountered this same issue, and in case it helps anyone, my mistake is that I forgot to add the `torch.no_grad()`.