Tortoise's GPT2PreTrainedModel regression from 4.29.2 to newer versions
Author: rsxdalv
Pull request: no

### System Info

Separated from previous issue:
https://github.com/huggingface/transformers/issues/24657

- `transformers` version: 4.31.0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Tortoise stops working with the following error in newer versions:

RuntimeError: Error(s) in loading state_dict for UnifiedVoice:
	Unexpected key(s) in state_dict: "gpt.h.0.attn.bias", "gpt.h.0.attn.masked_bias", "gpt.h.1.attn.bias", "gpt.h.1.attn.masked_bias", "gpt.h.2.attn.bias", "gpt.h.2.attn.masked_bias", "gpt.h.3.attn.bias", "gpt.h.3.attn.masked_bias", "gpt.h.4.attn.bias", "gpt.h.4.attn.masked_bias", "gpt.h.5.attn.bias", "gpt.h.5.attn.masked_bias", "gpt.h.6.attn.bias", "gpt.h.6.attn.masked_bias", "gpt.h.7.attn.bias", "gpt.h.7.attn.masked_bias", "gpt.h.8.attn.bias", "gpt.h.8.attn.masked_bias", "gpt.h.9.attn.bias", "gpt.h.9.attn.masked_bias", "gpt.h.10.attn.bias", "gpt.h.10.attn.masked_bias", "gpt.h.11.attn.bias", "gpt.h.11.attn.masked_bias", "gpt.h.12.attn.bias", "gpt.h.12.attn.masked_bias", "gpt.h.13.attn.bias", "gpt.h.13.attn.masked_bias", "gpt.h.14.attn.bias", "gpt.h.14.attn.masked_bias", "gpt.h.15.attn.bias", "gpt.h.15.attn.masked_bias", "gpt.h.16.attn.bias", "gpt.h.16.attn.masked_bias", "gpt.h.17.attn.bias", "gpt.h.17.attn.masked_bias", "gpt.h.18.attn.bias", "gpt.h.18.attn.masked_bias", "gpt.h.19.attn.bias", "gpt.h.19.attn.masked_bias", "gpt.h.20.attn.bias", "gpt.h.20.attn.masked_bias", "gpt.h.21.attn.bias", "gpt.h.21.attn.masked_bias", "gpt.h.22.attn.bias", "gpt.h.22.attn.masked_bias", "gpt.h.23.attn.bias", "gpt.h.23.attn.masked_bias", "gpt.h.24.attn.bias", "gpt.h.24.attn.masked_bias", "gpt.h.25.attn.bias", "gpt.h.25.attn.masked_bias", "gpt.h.26.attn.bias", "gpt.h.26.attn.masked_bias", "gpt.h.27.attn.bias", "gpt.h.27.attn.masked_bias", "gpt.h.28.attn.bias", "gpt.h.28.attn.masked_bias", "gpt.h.29.attn.bias", "gpt.h.29.attn.masked_bias".

File "C:\Users\Jonathan\Documents\one-click-installers-tts-6.0\tts-generation-webui\src\tortoise\gen_tortoise.py", line 77, in get_tts
MODEL = TextToSpeech(
File "C:\Users\Jonathan\Documents\one-click-installers-tts-6.0\installer_files\env\lib\site-packages\tortoise\api.py", line 231, in init
self.autoregressive.load_state_dict(torch.load(get_model_path('autoregressive.pth', models_dir)))
File "C:\Users\Jonathan\Documents\one-click-installers-tts-6.0\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(

The error happens when UnifiedVoice which depends on GPT2PreTrainedModel tries to load state dict.

Notably some transformers versions work, like 4.19 and 4.29.2, while others do not.

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Reproduction is still WIP, here's the current state:

```python
self.autoregressive = UnifiedVoice(max_mel_tokens=604, max_text_tokens=402, max_conditioning_inputs=2, layers=30,
  model_dim=1024,
  heads=16, number_text_tokens=255, start_text_token=255, checkpointing=False,
  train_solo_embeddings=False).cpu().eval()
self.autoregressive.load_state_dict(torch.load(get_model_path('autoregressive.pth', models_dir)))
```

where UnifiedVoice has:
```python
self.inference_model = GPT2InferenceModel(
    gpt_config,
    self.gpt,
    self.mel_pos_embedding,
    self.mel_embedding,
    self.final_norm,
    self.mel_head,
    kv_cache=kv_cache,
)
```
and 
```python
from transformers import GPT2Config, GPT2PreTrainedModel, LogitsProcessorList
...
class GPT2InferenceModel(GPT2PreTrainedModel):
```


### Expected behavior

load_state_dict works without dictionary keyerrors.

Comments:
Hi @rsxdalv, thanks for raising this issue! 

In the issue information, it's written that 'reproduction still WIP'. Once this is finalised and updated here, could you ping @sanchit-gandhi again to make sure the issue isn't lost? The repro is necessary for us to be able to help debug. For UnifiedVoice, could you link to the relevant pieces of code? 

Note, the officially supported way to load a transformers model checkpoint is through the `from_pretrained` method. 

> The error happens when UnifiedVoice which depends on GPT2PreTrainedModel tries to load state dict.

Let's try and isolate this particular line and extract the `transformers` related code! The aim is to run the reproducible codesnippet with just the `transformers` library, which sounds like is possible based on the above statement

> Hi @rsxdalv, thanks for raising this issue!
> 
> In the issue information, it's written that 'reproduction still WIP'. Once this is finalised and updated here, could you ping @sanchit-gandhi again to make sure the issue isn't lost? The repro is necessary for us to be able to help debug. For UnifiedVoice, could you link to the relevant pieces of code?
> 
> Note, the officially supported way to load a transformers model checkpoint is through the `from_pretrained` method.

From what I'm seeing so far, using save_pretrained makes what used to be one 1.6gb file into one 1.47gb file for GPT and another 1.5gb file for GPT Inference model.

Perhaps it will be possible to refactor and switch to from_pretrained. Or if I'm lucky, 4.32.0 will magically work again.

With @manmay-nakhashi's help I was able to test removing these keys and it still works the same. According to him GPT2 model was changed which causes this issue, so for now I will use torch.load(..., strict=False) before migrating to a more accurate approach.
While I can't guarantee that no work needs to be done on transformers, it is highly likely that it doesn't.
I will close this issue for now as I do not have the resources to completely reproduce this with from_pretrained, and it _probably_ works.

For those users who might find this later - removing those keys might be sufficient. In my case they all contained this:
```
('h.0.attn.bias',
tensor([[[[ True, False, False,  ..., False, False, False],
        [ True,  True, False,  ..., False, False, False],
        [ True,  True,  True,  ..., False, False, False],
        ...,
        [ True,  True,  True,  ...,  True, False, False],
        [ True,  True,  True,  ...,  True,  True, False],
        [ True,  True,  True,  ...,  True,  True,  True]]]])),
('h.0.attn.masked_bias', tensor(-10000.)),
```


Thanks for digging deeper into this @rsxdalv! We try extremely hard in the `transformers` repo to minimise breaking changes / avoid them entirely, so I'm pretty intrigued by this issue since it could be affecting others! I hope it's not been too arduous to patch it on your side without a clear reason for the break.

Perhaps what you can do is save the model weights using `transformers==4.30.0`:

```python
self.inference_model.save_pretrained("./")
```

And then load it from pre-trained using `transformers` on main:
```python
self.inference_model.from_pretrained("./")
```

=> this should check whether there has been any change in the `transformers` weight structure (you'll get a warning if there's a key mis-match)

If there hasn't, it seems unlikely that the issue has come from `transformers` modelling code