gpt2-large and gpt2-xl behave strangely with pad tokens
Author: boblus
Pull request: no

### System Info

- `transformers` version: 4.18.0
- Platform: Linux-4.18.0-425.13.1.el8_7.x86_64-x86_64-with-centos-8.7-Green_Obsidian
- Python version: 3.6.8
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.10.2+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
pretrained = 'gpt2-large'
model = GPT2LMHeadModel.from_pretrained(pretrained).to('cuda')
tokenizer = AutoTokenizer.from_pretrained(pretrained, padding_side='left')
tokenizer.add_special_tokens({'pad_token': 
})

torch.cuda.manual_seed_all(2266)

input0 = '



































Austin knew Quinn intimately and they slept together many times. Why did Austin do this? (0) Hated Quinn, (1) Found QUinn attractive, (2) Ask Quinn on a date\nANSWER:'
input_ids = tokenizer(input0, return_tensors='pt').input_ids.to(device)
with torch.no_grad():
    output0 = model.generate(input_ids, max_new_tokens=16, top_k=20, pad_token_id=50256, eos_token_id=50256, do_sample=True, temperature=0.01)

input1 = 'Austin knew Quinn intimately and they slept together many times. Why did Austin do this? (0) Hated Quinn, (1) Found QUinn attractive, (2) Ask Quinn on a date\nANSWER:'
input_ids = tokenizer(input1, return_tensors='pt').input_ids.to(device)
with torch.no_grad():
    output1 = model.generate(input_ids, max_new_tokens=8, top_k=20, pad_token_id=50256, eos_token_id=50256, do_sample=True, temperature=0.01)
```

### Expected behavior

```
output0 = '\nThe The The The The'
output1 = ' Austin was jealous of Quinn's relationship'
```

pad_token works just fine with other gpt models such as gpt2, and gpt2-small.

Comments:
Hi @boblus 
In order for your script to work, you need to properly set the attention mask by masking out the padding tokens and call generate with the attention mask

Do you mean `tokenizer.add_special_tokens({'pad_token': pad_token})`?
I have done this. I forgot to put that in my initial comment. Just added it.

I think that attention masks are created only if you pass multiple sentences to the tokenizer, in your case I think you may need to create it manually unless I am wrong and there is a simpler solution.

The below script worked for me:
```python
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

pretrained = 'gpt2-large'
device = 'cuda'
model = GPT2LMHeadModel.from_pretrained(pretrained).to('cuda')
tokenizer = GPT2Tokenizer.from_pretrained(pretrained)

torch.cuda.manual_seed_all(2266)

input0 = '



































Austin knew Quinn intimately and they slept together many times. Why did Austin do this? (0) Hated Quinn, (1) Found QUinn attractive, (2) Ask Quinn on a date\nANSWER:'

tokenizer.pad_token = tokenizer.eos_token
tokenizer.pad_token_id = tokenizer.eos_token_id

inputs = tokenizer(input0, return_tensors='pt').to(device)
inputs["attention_mask"] = inputs["input_ids"].ne(tokenizer.pad_token_id).long().to(device)
with torch.no_grad():
    output0 = model.generate(**inputs, max_new_tokens=16, top_k=20, pad_token_id=50256, eos_token_id=50256, do_sample=True, temperature=0.01)
    print(tokenizer.decode(output0[0]))

input1 = 'Austin knew Quinn intimately and they slept together many times. Why did Austin do this? (0) Hated Quinn, (1) Found QUinn attractive, (2) Ask Quinn on a date\nANSWER:'
input_ids = tokenizer(input1, return_tensors='pt').input_ids.to(device)
with torch.no_grad():
    output1 = model.generate(input_ids, max_new_tokens=8, top_k=20, pad_token_id=50256, eos_token_id=50256, do_sample=True, temperature=0.01)
    print(tokenizer.decode(output1[0]))
```

Thanks @younesbelkada, it works now. I thought setting `pad_token_id` in `model.generate()` is equivalent to setting the attention mask. And my codes worked well with other models, that's why it confused me.

linked to #22155  and #21080. GPT2 is an old model and does not necessarly create everything by default like our recent models