confusion about past_key_values in GPT2 
Author: VulnDetector
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:4.15.0



Models:


- GPT-2: @patrickvonplaten, @LysandreJik


## Information

In fact,there is no error here,or I'm not sure if this is a bug or I still have some misunderstanding about GPT2,but I sincerely wish you can read this,cause I turned to a lot of people with no one gave me right answer.

So,I wanna to question some implementation tricks in GPT2,that is,the design of past_key_values.
I debugged the code, and I suppose I have understood the idea of this amazing trick,I mean, for the auto-regressive model,when generating the new token,the trick is to save the key and value of the previous sequence of tokens,so maybe we don't need to compute the same thing.

But I found something confusing here.At first,the past_key_values should be none,when predicting the next token,the model returns the (key,value) tuple with each block(in gpt2,the number is 12).Here is the code of transformers:
```python
            query = self.q_attn(hidden_states)
            key, value = self.c_attn(encoder_hidden_states).split(self.split_size, dim=2)
            attention_mask = encoder_attention_mask
        else:
            query, key, value = self.c_attn(hidden_states).split(self.split_size, dim=2)

        query = self._split_heads(query, self.num_heads, self.head_dim)
        key = self._split_heads(key, self.num_heads, self.head_dim)
        value = self._split_heads(value, self.num_heads, self.head_dim)

        if layer_past is not None:
            past_key, past_value = layer_past
            key = torch.cat((past_key, key), dim=-2)
            value = torch.cat((past_value, value), dim=-2)
```
With the new token as `hidden_states`,we generate q,k,v for this new token,then we simply concat to the past_key,past_value,which works fine for the first block,but I suppose it not right for the second and the following blocks,cause the output of the first block,which is also the input of the second block,should be changed here,for the left tokens should also attend the new token's key,value,so the softmax of the atten_weights should be changed,which leads to the change of the output of the first block,so in the second and the following block,the (key,value) history is not useable.

I hope I have make my confusion clear,and I'm really confused with this.

I really thank you for your patience to reading whole words above.If this is not wrong but something wrong in my understanding towards GPT2,I really appreciate you for correct things for me.





Comments:
I know the next token is only related to the output of the last token , but the previous token's values are supposed to change(I mean),which we just use the past ones,so will cause the wrong value of the output of the last token.

Hey @VulnDetector,

I'm having a hard time following here exactly what you mean.
If you think the problem is that the past key values of GPT2's first block are incorrectly re-used by GPT2's second block - this is not the case. 
You can easily verify this when looking into the `past_key_values`. There you should notice that you have past key values for **each** transformer block.

Thank you very much @patrickvonplaten 
I'm sorry that I didn't express quite clear so that you had such a hard time,but in fact I know what you told me,I debugged the code,and I know,say,if there is 12 blocks in GPT2, the first return of the model contains the tuple of the (key,value) of the 12 blocks each.

Ok, I will take the code below to explain more specifically:
```python
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')



generated = tokenizer.encode("The Manhattan bridge")
context = torch.tensor([generated])

past_key_values = None

for i in range(30):

    output = model(context, past_key_values=past_key_values)
    past_key_values = output.past_key_values
    token = torch.argmax(output.logits[..., -1, :])

    context = token.unsqueeze(0)
   
    generated += [token.tolist()]


sequence = tokenizer.decode(generated)
sequence = sequence.split(".")[:-1]
print(sequence)
```
At first,cause `"The Manhattan bridge"` is the begin of the sentence,so my past_key_values here should be None.When I run model for the first time(in the first cycle),I got past_key_values which is (key,value) of 12 blocks,I guess you mean this above.Then I obtain the next word the model predict, put it as context alone,with the tuple of (key,value) returned as the past_key_values,I run the model a second time.

In this time,in the first block,I suppose that(maybe I'm wrong) the first three tokens should use their query(obtained from `past_key_values`) to attend the key of the new generated token,which is different from last time,because last time we don't have the new token,we only have 3 tokens,but now , we have 4 tokens ,so the output of the first three tokens in the first block has changed,so the input of the second block changes ,if we do not use `past_key_values` in the second block,we should  recalculate the q,k,v for the first three tokens with the changed first block output ,if so,which is absolutely not equal to the `(key,value)` of the second block generated in the first cycle.

I hope you can forgive me for my ambiguous words,and I wish you can understand what I mean this time.

I know there is something I ignored.
I thought the GPT2 attend any given tokens at any position.I ignored that the token only attend to those which is on the left side.
I ignored there is still the code:
```python
attn_weights = torch.where(causal_mask, attn_weights, self.masked_bias.to(attn_weights.dtype))
```
Thanks,anyway. @patrickvonplaten