GPT2 text generation produces different results w and w/o `past`
Author: mksenzov
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Here is two snippets that seem to expose the problem (or perhaps I just use the model incorrectly):

V1: no `past`

```python
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

generated = tokenizer.encode("This apple is")

for i in tqdm(range(100)):
    context = torch.tensor([generated])
    outputs = model(context)
    predictions = outputs[0]

    token = torch.argmax(predictions[0, -1, :]).item()
    generated.append(token)

tokenizer.decode(generated)
```

This produces:

```python
"This apple is a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very"
```
----

V2: conceptually the same but this time using `past`: 

```python
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

generated = tokenizer.encode("This apple is")
context = torch.tensor([generated])
past = None

for i in tqdm(range(100)):
    output, past = model(context, past=past)
    token = torch.argmax(output[0, :])

    generated += [token.item()]
    context = token.unsqueeze(0)

sequence = tokenizer.decode(generated)
print(sequence)
```

This produces:

```python
'This apple is is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a very good apple. It is a'
```

## Expected behavior


I expected the outputs to be the same... am I doing this wrong?

* OS: this is a docker image built on top of NVIDIA's `nvcr.io/nvidia/pytorch:19.11-py3`
* Python version: Python 3.6.9 :: Anaconda, Inc.
* PyTorch version: '1.4.0a0+649135b'
* PyTorch Transformers version (or branch): '2.3.0'
* Using GPU ? Seem to be reproducible on both CPU and GPU
* Distributed or parallel setup ? no
* Any other relevant information:



Comments:
Turns lout this is a non-issue; there is a subtle nuance that I have overlooked: on the very first iteration of V2 (using `past`) we do not have any `past` and therefore the code should be modified to take argmax differently fixed code produces the correct output:

V2 (fixed):

```python
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

generated = tokenizer.encode("This apple is")
print(generated)
context = torch.tensor([generated])
past = None

for i in tqdm(range(100)):
    output, past = model(context, past=past)
    if i == 0:
        token = output[0, -1, :].argmax()
    else:
        token = output[0, :].argmax()

    generated += [token.item()]
    context = token.unsqueeze(0)

tokenizer.decode(generated)
```

produces 

```python
"This apple is a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very good apple. It's a very"
```