load_in_8bit=True broken with new transformers
Author: pseudotensor
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.20.3
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: 0
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Script based upon: https://huggingface.co/Salesforce/blip2-flan-t5-xl
```python
import requests
from PIL import Image
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
from transformers import Blip2Processor, Blip2ForConditionalGeneration
gpu_id = 0
device_map = {"": gpu_id}
blip_model = blip_processor = 'Salesforce/blip2-flan-t5-xl'
prompt = 'an image of'

device = 'cuda'
load_half = False
import torch
with torch.no_grad():
    context_class_cast = torch.autocast
    with context_class_cast(device):
        processor = Blip2Processor.from_pretrained(blip_processor,
                                                   load_in_8bit=True,
                                                        device_map=device_map)
        model = Blip2ForConditionalGeneration.from_pretrained(blip_model,
                                                              load_in_8bit=True,
                                                                   device_map=device_map)

        inputs = processor(image, prompt, return_tensors="pt")
        output = model.generate(**inputs)

        caption: str = processor.decode(output[0], skip_special_tokens=True)
        print(caption)
```
output:
```
/home/jon/miniconda3/envs/h2ogpt/bin/python3.10 /home/jon/h2ogpt/checkblip2_mine.py 
Console output is saving to: /home/jon/h2ogpt/pycharm.log

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so...
/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/jon/miniconda3/envs/h2ogpt did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/clang+llvm-4.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'), PosixPath('/opt/rstudio-1.0.136/bin'), PosixPath('/usr/lib/jvm/default-java/jre/lib/amd64/server'), PosixPath('/home/jon/lib'), PosixPath('/usr/local/cuda/extras/CUPTI/lib64')}
  warn(msg)
Loading checkpoint shards: 100%|██████████| 2/2 [00:07<00:00,  3.88s/it]
/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/generation/utils.py:1369: UserWarning: Using `max_length`'s default (20) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.
  warnings.warn(
```

Gives empty output for caption in transformers==4.31.0 but gives correct output for all older transformers from 4.30.2 and backwards.

### Expected behavior

All dependencies fixed, just do:
```
pip install transformers==4.30.2
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting transformers==4.30.2
  Downloading transformers-4.30.2-py3-none-any.whl (7.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 55.8 MB/s eta 0:00:00
Requirement already satisfied: filelock in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (3.12.2)
Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (0.16.4)
Requirement already satisfied: numpy>=1.17 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (1.23.5)
Requirement already satisfied: packaging>=20.0 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (23.1)
Requirement already satisfied: pyyaml>=5.1 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (6.0)
Requirement already satisfied: regex!=2019.12.17 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (2023.6.3)
Requirement already satisfied: requests in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (2.31.0)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (0.13.3)
Requirement already satisfied: safetensors>=0.3.1 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (0.3.1)
Requirement already satisfied: tqdm>=4.27 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from transformers==4.30.2) (4.65.0)
Requirement already satisfied: fsspec in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2) (2023.6.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2) (4.7.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from requests->transformers==4.30.2) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from requests->transformers==4.30.2) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from requests->transformers==4.30.2) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages (from requests->transformers==4.30.2) (2023.5.7)
Installing collected packages: transformers
  Attempting uninstall: transformers
    Found existing installation: transformers 4.31.0
    Uninstalling transformers-4.31.0:
      Successfully uninstalled transformers-4.31.0
Successfully installed transformers-4.30.2
(h2ogpt) jon@pseudotensor:~/h2ogpt$ 
```

Then re-run script and get:

```
/home/jon/miniconda3/envs/h2ogpt/bin/python3.10 /home/jon/h2ogpt/checkblip2_mine.py 
Console output is saving to: /home/jon/h2ogpt/pycharm.log

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so...
/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/jon/miniconda3/envs/h2ogpt did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/lib/jvm/default-java/jre/lib/amd64/server'), PosixPath('/opt/rstudio-1.0.136/bin'), PosixPath('/opt/clang+llvm-4.0.0-x86_64-linux-gnu-ubuntu-16.04/lib'), PosixPath('/home/jon/lib'), PosixPath('/usr/local/cuda/extras/CUPTI/lib64')}
  warn(msg)
Loading checkpoint shards: 100%|██████████| 2/2 [00:08<00:00,  4.04s/it]
/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:318: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
  warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/generation/utils.py:1353: UserWarning: Using `max_length`'s default (20) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.
  warnings.warn(
a woman and her dog on the beach
```

i.e. returns the caption: `a woman and her dog on the beach`

I also tried the latest bitsandbytes==0.41.0  and same effect as with older 0.39.0.

I also tried newer accelerate==0.21.0 and has same effect as with older 0.20.3

I also tried latest transformers 4.32.0.dev0 and same effect.

Comments:
cc @younesbelkada 

Hi @pseudotensor 
Thanks for the issue and the clean reproducer, I can confirm the issue persists on the current main brach of transformers and does not occur with #25047 - once that PR merged your issue will be fixed

Also, there is no need to specify `load_in_8bit=True` and `device_map="auto"` in the `Blip2Processor.from_pretrained` method
```python
import requests
from PIL import Image
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
from transformers import Blip2Processor, Blip2ForConditionalGeneration
gpu_id = 0
device_map = {"": gpu_id}
blip_model = blip_processor = 'Salesforce/blip2-flan-t5-xl'
prompt = 'an image of'

device = 'cuda'
load_half = False
import torch
with torch.no_grad():
    context_class_cast = torch.autocast
    with context_class_cast(device):
        processor = Blip2Processor.from_pretrained(blip_processor)
        model = Blip2ForConditionalGeneration.from_pretrained(blip_model,
                                                              load_in_8bit=True,
                                                                   device_map=device_map)

        inputs = processor(image, prompt, return_tensors="pt")
        output = model.generate(**inputs)

        caption: str = processor.decode(output[0], skip_special_tokens=True)
        print(caption)
```