[Bug] Failiure to generate Diffusion images / AI language responses when upgrading past 4.19.2
Author: BlackWyvern
Pull request: no

### System Info

I've posted this to both Auto1x4 and Opinionated, but I don't think it's an issue on their end. So here I am.

For some reason, I am completely unable to generate images or use ai language models if I upgrade my transformers past 4.19.2.

This ticket details my enite [install/diagnostic workflow](https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7351) and other examples of the issue.

In a nut-shell, I can git-pull any of my AI based generators, and they will all exhibit this issue until I change requirements.txt to transformers = 4.19.2

My Auto1x4 & Opinionated Environ and prompts are as follow:

1.5 ema-only safetensor from [runwayml](https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main)
On vae-ft-mse-840000-ema-pruned.ckpt from [stabilityai](https://huggingface.co/stabilityai/sd-vae-ft-mse-original/tree/main)
ETA weight: 31337, CFG: 7

30 Step Euler A
"3d rendering of a small metal cube sitting on a glass table"

4.19.2
![00183-2512420455](https://user-images.githubusercontent.com/7834910/233856974-771adabf-fdb3-47f6-8ee8-206191ef1466.png)

4.25.1
![00182-2512420455](https://user-images.githubusercontent.com/7834910/233856981-c73ef918-857c-45c4-8412-acec871f5675.png)

4.26.1
![00001-2512420455-3d rendering of a small metal cube sitting on a glass table](https://user-images.githubusercontent.com/7834910/233857561-95454cef-85c5-49d1-a92f-7dbbdcb9890c.jpg)

4.28.1
![00012-2512420455-3d rendering of a small metal cube sitting on a glass table](https://user-images.githubusercontent.com/7834910/236566215-77c1caa9-656a-42ae-80b1-3555ec71f35c.jpg)

The strangest thing is that after 4.19.2 all versions are wrong, but they're all CONSISTENTLY wrong. I don't really know where else to turn.

```
- `transformers` version: 4.29.0.dev0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.6
- Huggingface_hub version: 0.13.3
- Safetensors version: 0.3.0
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Presumably
- Using distributed or parallel set-up in script?: No idea
```

Launch details from Opinionated
```
13:48:03-071493 INFO     Starting SD.Next
13:48:03-092427 INFO     Python 3.10.6 on Windows
13:48:03-350737 INFO     Version: f6898c9a Fri May 5 13:40:53 2023 -0400
13:48:03-741693 INFO     Setting environment tuning
13:48:03-745682 INFO     nVidia CUDA toolkit detected
13:48:05-379334 INFO     Torch 2.0.0+cu118
13:48:05-397286 INFO     Torch backend: nVidia CUDA 11.8 cuDNN 8700
13:48:05-400261 INFO     Torch detected GPU: NVIDIA GeForce RTX 3080 VRAM 10240 Arch (8, 6) Cores 68
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Personally reproducible by changing requirements.txt in any project to any version of transformers higher than 4.19.2

I have not gotten confirmation of anyone else having or being able to reproduce this issue.

### Expected behavior

Parity or near parity of model generation.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.