GPT-2 past behaves incorrectly when attention_mask is used
Author: s-jse
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT-2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import GPT2LMHeadModel
import torch

model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

# w/ `attention_mask` w/ `past`
input_ids      = torch.tensor([[0, 0, 1, 2]])
position_ids   = torch.tensor([[0, 0, 1, 2]])
attention_mask = torch.tensor([[0, 1, 1, 1]])
output = model(input_ids=input_ids, position_ids=position_ids, attention_mask=attention_mask)
input_ids      = torch.tensor([[3]])
position_ids   = torch.tensor([[3]])
attention_mask = torch.tensor([[1]])
output = model(input_ids=input_ids, position_ids=position_ids, attention_mask=attention_mask, past=output[1])
print(output[0][0, -1, :].detach().numpy())

# w/ `attention_mask` w/o `past`
input_ids      = torch.tensor([[0, 0, 1, 2, 3]])
position_ids   = torch.tensor([[0, 0, 1, 2, 3]])
attention_mask = torch.tensor([[0, 1, 1, 1, 1]])
output = model(input_ids=input_ids, position_ids=position_ids, attention_mask=attention_mask)
print(output[0][0, -1, :].detach().numpy())

# w/o `attention_mask` w/ `past`
input_ids      = torch.tensor([[0, 1, 2]])
position_ids   = torch.tensor([[0, 1, 2]])
output = model(input_ids=input_ids, position_ids=position_ids)
input_ids      = torch.tensor([[3]])
position_ids   = torch.tensor([[3]])
output = model(input_ids=input_ids, position_ids=position_ids, past=output[1])
print(output[0][0, -1, :].detach().numpy())

# w/o `attention_mask` w/o `past`
input_ids      = torch.tensor([[0, 1, 2, 3]])
position_ids   = torch.tensor([[0, 1, 2, 3]])
output = model(input_ids=input_ids, position_ids=position_ids)
print(output[0][0, -1, :].detach().numpy())
```

## Expected behavior

All printed outputs should be equal. But this is what I see:

```
[-63.085373 -62.34121  -61.08647  ... -76.70221  -73.235085 -65.88774 ]
[-67.13892  -66.132126 -64.81706  ... -80.93089  -77.780525 -70.23946 ]
[-67.1389   -66.13211  -64.81704  ... -80.930855 -77.780495 -70.239426]
[-67.13892  -66.132126 -64.81706  ... -80.93089  -77.780525 -70.23946 ]
```
Specifically, it seems like `past` does not work correctly when used with `attention_mask`.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: Linux-5.4.19-100.fc30.x86_64-x86_64-with-fedora-30-Thirty
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Solved. `attention_mask ` should be the full tensor (not just the last step), even when `past` is used.