GPT2 past usage
Author: cronoik
Pull request: no

Hello everyone,

I tried to answer this [stackoverflow question](https://stackoverflow.com/questions/62703391/estimate-token-probability-logits-given-a-sentence-without-computing-the-entire) and stumbled about a strange beheaviour I can't explain. 

The following code will calculate the loss for a sentence with different single words injected:
```
from  transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

model = GPT2LMHeadModel.from_pretrained("gpt2")
model.eval()
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

def score(sentence):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    loss = model(tensor_input, labels=tensor_input)
    return -loss[0].item()

candidates = ["watch", "run", "think", "apple", "light"]
sent_template = "I like sitting in my new chair and {} about life"
print({candidate: score(sent_template.format(candidate)) for candidate in candidates})
```
Output:
```
{'watch': -5.406847953796387, 'run': -5.533411502838135, 'think': -4.525279521942139, 'apple': -6.158637046813965, 'light': -5.835141658782959}
```
Now I wanted to use the past parameter according to [documentation](https://huggingface.co/transformers/quickstart.html#using-the-past) and expected the same result:
```
from  transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

model = GPT2LMHeadModel.from_pretrained("gpt2")
model.eval()
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

past = "I like sitting in my new chair and"
past_tokenize_input = tokenizer.tokenize(past)
past_tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(past_tokenize_input)])

_, _, past = model(past_tensor_input, labels=past_tensor_input)

def score(sentence, past):
    tokenize_input = tokenizer.tokenize(sentence, )
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])

    loss = model(tensor_input, labels=tensor_input, past=past)
    return -loss[0].item()


candidates = ["watch", "run", "think", "apple", "light"]
sent_template = " {} about life"
print({candidate: score(sent_template.format(candidate), past) for candidate in candidates})
```
but the loss is different:
```
{'watch': -7.811002731323242, 'run': -6.370519638061523, 'think': -3.460831642150879, 'apple': -9.08120346069336, 'light': -8.28120231628418}
```
Is this the intended behaviour or am I doing something wrong?

Comments:
Hey @cronoik, 

Thanks for your issue. It does not really surprise me that the loss is different. 
In the first case the following loss is calculated:

loss = CrossEntropy(`input_ids`: "I like sitting in my new chair and {} about" vs. `labels`: "like sitting in my new chair and {} about life").

where as in the second case the following loss is calculated:

loss = CrossEntropy(`input_ids`: "{} about" vs. `labels`: "about life").

This is simplied - in reality the loss between the tokens of those words are calculated. 
The important part to note here is that 1) `past` should not be used for training. It should be used to speed up inference.
2) When using `past` only the output embeddings of the `input_ids` (in your case for "{} about life") are calculated and not also for the "cached" past input_ids.

Hope this answers your question

@patrickvonplaten Thank you a lot for your answer.

Hello! I have a question for gpt-2 lmhead model's input 'past_key_values'
I want to use this option for model.generate module but there is an error if I use this option by specifying **model_specific_kwargs={'past':past} in model.generate's inputs... dimension error...
what should I do for using this option for generation...?

Hey @chaeyoon-jang,

Could you open a new issue for this?