_keys_to_ignore_on_load_unexpected not working with GPT2 model
Author: alex-dima
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.11.4
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@younesbelkada  @ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I saved a PyTorch lightning model which contained a RoGPT2-large model from version 4.27.4. When trying to load the model in version 4.30.2, I get the following error:

Traceback (most recent call last):
  File "/Users/alexandrudima/home/Research/Outlook Add-ins/Optimize/backend.py", line 23, in <module>
    generation_model = GenerationModel.load_from_checkpoint(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml/lib/python3.11/site-packages/pytorch_lightning/core/module.py", line 1520, in load_from_checkpoint
    loaded = _load_from_checkpoint(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml/lib/python3.11/site-packages/pytorch_lightning/core/saving.py", line 89, in _load_from_checkpoint
    storage = _load_state(cls, checkpoint, strict=strict, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml/lib/python3.11/site-packages/pytorch_lightning/core/saving.py", line 154, in _load_state
    keys = obj.load_state_dict(checkpoint["state_dict"], strict=strict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for GenerationModel:
        Unexpected key(s) in state_dict: "model.transformer.h.0.attn.bias", "model.transformer.h.0.attn.masked_bias", "model.transformer.h.1.attn.bias", "model.transformer.h.1.attn.masked_bias", "model.transformer.h.2.attn.bias", "model.transformer.h.2.attn.masked_bias", "model.transformer.h.3.attn.bias", "model.transformer.h.3.attn.masked_bias", "model.transformer.h.4.attn.bias", "model.transformer.h.4.attn.masked_bias", "model.transformer.h.5.attn.bias", "model.transformer.h.5.attn.masked_bias", "model.transformer.h.6.attn.bias", "model.transformer.h.6.attn.masked_bias", "model.transformer.h.7.attn.bias", "model.transformer.h.7.attn.masked_bias", "model.transformer.h.8.attn.bias", "model.transformer.h.8.attn.masked_bias", "model.transformer.h.9.attn.bias", "model.transformer.h.9.attn.masked_bias", "model.transformer.h.10.attn.bias", "model.transformer.h.10.attn.masked_bias", "model.transformer.h.11.attn.bias", "model.transformer.h.11.attn.masked_bias", "model.transformer.h.12.attn.bias", "model.transformer.h.12.attn.masked_bias", "model.transformer.h.13.attn.bias", "model.transformer.h.13.attn.masked_bias", "model.transformer.h.14.attn.bias", "model.transformer.h.14.attn.masked_bias", "model.transformer.h.15.attn.bias", "model.transformer.h.15.attn.masked_bias", "model.transformer.h.16.attn.bias", "model.transformer.h.16.attn.masked_bias", "model.transformer.h.17.attn.bias", "model.transformer.h.17.attn.masked_bias", "model.transformer.h.18.attn.bias", "model.transformer.h.18.attn.masked_bias", "model.transformer.h.19.attn.bias", "model.transformer.h.19.attn.masked_bias", "model.transformer.h.20.attn.bias", "model.transformer.h.20.attn.masked_bias", "model.transformer.h.21.attn.bias", "model.transformer.h.21.attn.masked_bias", "model.transformer.h.22.attn.bias", "model.transformer.h.22.attn.masked_bias", "model.transformer.h.23.attn.bias", "model.transformer.h.23.attn.masked_bias", "model.transformer.h.24.attn.bias", "model.transformer.h.24.attn.masked_bias", "model.transformer.h.25.attn.bias", "model.transformer.h.25.attn.masked_bias", "model.transformer.h.26.attn.bias", "model.transformer.h.26.attn.masked_bias", "model.transformer.h.27.attn.bias", "model.transformer.h.27.attn.masked_bias", "model.transformer.h.28.attn.bias", "model.transformer.h.28.attn.masked_bias", "model.transformer.h.29.attn.bias", "model.transformer.h.29.attn.masked_bias", "model.transformer.h.30.attn.bias", "model.transformer.h.30.attn.masked_bias", "model.transformer.h.31.attn.bias", "model.transformer.h.31.attn.masked_bias", "model.transformer.h.32.attn.bias", "model.transformer.h.32.attn.masked_bias", "model.transformer.h.33.attn.bias", "model.transformer.h.33.attn.masked_bias", "model.transformer.h.34.attn.bias", "model.transformer.h.34.attn.masked_bias", "model.transformer.h.35.attn.bias", "model.transformer.h.35.attn.masked_bias". 

I suspect that something is not ok with the _keys_to_ignore_on_load_unexpected parameter defined in GPT2LMHeadModel, but I have no idea what could be the problem.
P.S. The model can be loaded without problems when using transformers==4.27.4. 

### Expected behavior

The model should also be loadeed without problems in 4.30.2.

Comments:
The problem stems from PyTorch lightning trying to load the model as far as I can see. Models should be loaded via the `from_pretrained` method we provide. If you are using `.load_state_dict()` it's up to you to clean the state dict you have and make sure it only contains keys the model accepts/

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.