`resize_token_embeddings` breaks `gpt2` generation
Author: vwxyzjn
Pull request: no

### System Info

```
- `transformers` version: 4.30.1
- Platform: Linux-5.15.0-1023-aws-x86_64-with-glibc2.2.5
- Python version: 3.8.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
```

### Who can help?

Maybe @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM, AutoTokenizer

pretrained_model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
device = "cpu"
input_ids = tokenizer.encode('I enjoy walking with my cute dog', return_tensors='pt')
greedy_output = pretrained_model.generate(
    input_ids=input_ids.to(device),
    max_new_tokens=50,
    temperature=0.7,
    pad_token_id=tokenizer.pad_token_id,
)
print("Output:\n" + 100 * '-')
print(tokenizer.decode(greedy_output[0]))

###
pretrained_model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
device = "cpu"
input_ids = tokenizer.encode('I enjoy walking with my cute dog', return_tensors='pt')
greedy_output = pretrained_model.generate(
    input_ids=input_ids.to(device),
    max_new_tokens=50,
    temperature=0.7,
    pad_token_id=tokenizer.pad_token_id,
)
print("Output2:\n" + 100 * '-')
print(tokenizer.decode(greedy_output[0]))

###
pretrained_model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
device = "cpu"
pretrained_model.resize_token_embeddings(len(tokenizer))
# encode context the generation is conditioned on
input_ids = tokenizer.encode('I enjoy walking with my cute dog', return_tensors='pt')
greedy_output = pretrained_model.generate(
    input_ids=input_ids.to(device),
    max_new_tokens=50,
    temperature=0.7,
)

print("Output3:\n" + 100 * '-')
print(tokenizer.decode(greedy_output[0]))
```
```
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Output:
----------------------------------------------------------------------------------------------------
I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with my dog. I'm not sure if I'll ever be able to walk with my dog.

I'm not sure if I'll ever be able to walk with my
Output2:
----------------------------------------------------------------------------------------------------
I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with my dog. I'm not sure if I'll ever be able to walk with my dog.

I'm not sure if I'll ever be able to walk with my
Output3:
----------------------------------------------------------------------------------------------------
I enjoy walking with my cute dog[PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD]
```

### Expected behavior

According to https://stackoverflow.com/a/69194717/6611317, 

```
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
pretrained_model.resize_token_embeddings(len(tokenizer))
```

Should work as expected and does not completely break generation.

Comments:
You are adding a random line in the model. Without fine-tuning it there is no reason for it to continue working.