Transformers 4.31.0 Runtime error trying to load model saved as 8bit on HF fails
Author: mediocreatmybest
Pull request: no

### System Info

Transformers 4.31.0
Python 3.10.6
Linux and Windows (Same issue on both)
Bitsandbytes 0.39.1 to 0.40x


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi! 

I've just found I'm now getting the following error when trying to load a model I've saved as 8bit on the Huggingface.co website:

The error is:

```
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /home/user/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...
Traceback (most recent call last):
  File "/home/user/testing/blip2_testing.py", line 12, in <module>
    model = Blip2ForConditionalGeneration.from_pretrained(
  File "/home/user/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/user/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3260, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/home/user/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 725, in _load_state_dict_into_meta_model
    set_module_quantized_tensor_to_device(
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/bitsandbytes.py", line 116, in set_module_quantized_tensor_to_device
    new_value = nn.Parameter(new_value, requires_grad=old_value.requires_grad)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/parameter.py", line 36, in __new__
    return torch.Tensor._make_subclass(cls, data, requires_grad)
RuntimeError: Only Tensors of floating point and complex dtype can require gradients
```

A minimal python script that I was testing:

```
from PIL import Image
import requests
from transformers import AutoProcessor, Blip2ForConditionalGeneration
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(
    "Mediocreatmybest/blip2-opt-2.7b_8bit",
    load_in_8bit=True,
)
model = Blip2ForConditionalGeneration.from_pretrained(
    "Mediocreatmybest/blip2-opt-2.7b_8bit",
    load_in_8bit=True,

)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
url1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
image1 = Image.open(requests.get(url, stream=True).raw)
batch = [image, image1]

inputs = processor(images=batch, return_tensors="pt").to(device, torch.float16)

generated_ids = model.generate(**inputs, min_new_tokens=8, max_new_tokens=30)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)


print(generated_text)
```

If I try loading a model that hasn't been saved in 8bit, for example the original Salesforce/blip2-opt-2.7b this then loads without issue and runs fine, from what I've been able to test it seems to be the 8bit saved model only.

Dropping the Transformers version back to 4.30.2 and the script runs fine without error. 



### Expected behavior

Using Transformers version 4.30.2 the above example script run normally and outputs the described text.

Updating to Transformers 4.31.0 the above example script fails when trying to use an 8bit saved model such as Mediocreatmybest/blip2-opt-2.7b_8bit.

Using Transformers 4.31.0 on the original model works correctly when passing load_in_8bit=True. 

Comments:
cc @younesbelkada 

Hi @mediocreatmybest 
Thanks for the very clean reproducer, I managed to repro the issue and propose a fix in https://github.com/huggingface/transformers/pull/25047
The fix should be now live on the main branch of transformers and you should be able to use that by installing transformers from source. 

Champion :) thanks! 👍