ERROR about using layer_past and use_cache in Attention Layer of GPT2
Author: ouwenjie03
Pull request: no

Hi,
I am trying to use "use_cache" and "past_key_values" to speed up the decode steps.
But I have some questions about the Attention Layer, here are some codes in forward function:
[layer_past code](https://github.com/huggingface/transformers/blob/master/src/transformers/models/gpt2/modeling_gpt2.py#L230)
``` python
query = self.split_heads(query)
key = self.split_heads(key, k=True)
value = self.split_heads(value)
if layer_past is not None:
    past_key, past_value = layer_past[0].transpose(-2, -1), layer_past[1]  # transpose back cf below
    key = torch.cat((past_key, key), dim=-1)
    value = torch.cat((past_value, value), dim=-2)
```

If I send the layer_past value, it raise a size unmatched ERROR. 
It shows that the shapes of "key" and "value" are not match with the attention mask.
Maybe the "key" before torch.cat has the same shape as the attention mask, but after torch.cat with past_key, the shape of "key" change.

Here is an example:
``` python
import torch
from transformers import GPT2Model, GPT2Config

config = GPT2Config()
config.use_cache = True
model = GPT2Model(config=config)

input_ids = torch.randint(0, 100, (2, 6))
attention_mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]], dtype=torch.bool)
past_key_values = None
outputs = model(input_ids=input_ids, attention_mask=attention_mask, past_key_values=None)
logits = outputs[0]
past_key_values = outputs[1]
print(logits.size())
print(len(past_key_values))
print([_kv.size() for _kv in past_key_values])

# we get the past_key_values and add the next step decoder input.
added_input_ids = torch.randint(0, 100, (2, 1))
added_attention_mask = torch.tensor([[1], [1]], dtype=torch.bool)
input_ids = torch.cat([input_ids, added_input_ids], dim=1)
attention_mask = torch.cat([attention_mask, added_attention_mask], dim=1)
print(input_ids.size(), attention_mask.size())
outputs = model(input_ids=input_ids, attention_mask=attention_mask, past_key_values=past_key_values)
# here occur the ERROR

logits = outputs[0]
past_key_values = outputs[1]
print(logits.size())
print(len(past_key_values))
print([_kv.size() for _kv in past_key_values])
```

``` 
/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py in _attn(self, q, k, v, attention_mask, head_mask, output_attentions)
    175         if attention_mask is not None:
    176             # Apply the attention mask
--> 177             w = w + attention_mask
    178 
    179         w = nn.Softmax(dim=-1)(w)

RuntimeError: The size of tensor a (13) must match the size of tensor b (7) at non-singleton dimension 3
```

Comments:
Hey @ouwenjie03,

you should not do this steps:

```python
input_ids = torch.cat([input_ids, added_input_ids], dim=1)
```

It should just be 

```pyhton 
input_ids = added_input_ids
```

When passing `past_key_values` the input_ids should correspond **only** to the last tokens. I think if you take a look at this test: https://github.com/huggingface/transformers/blob/3f290e6c8403c6a2cf80dce068869793bde49540/tests/test_modeling_gpt2.py#L446 you'll understand a bit better.


Oh, get it! Thank you so much!