Having trouble loading Blip2 4bit 
Author: joaopedrosdmm
Pull request: no

### System Info

It throws and import error although I already restarted the notebook, tried to import bitsandbytes and acceleraet different ways. I'm not even loading 8bit but in the error thrown, it seems to believe I am. Help would be appreciated.

Code:
```python
!pip install -q -U git+https://github.com/TimDettmers/bitsandbytes
!pip install -q -U git+https://github.com/huggingface/transformers.git 
!pip install -q -U git+https://github.com/huggingface/peft.git
!pip install -q -U git+https://github.com/huggingface/accelerate.git


import torch
from transformers import Blip2ForConditionalGeneration, AutoProcessor, Blip2Processor, AutoModelForCausalLM, BitsAndBytesConfig
from peft import prepare_model_for_kbit_training

nf4_config = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_quant_type="nf4",
  bnb_4bit_use_double_quant=True,
  bnb_4bit_compute_dtype=torch.bfloat16
)

processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b-coco")
model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b-coco", device_map='auto', quantization_config=nf4_config)"
```
Output:
```python
ImportError                               Traceback (most recent call last)
[<ipython-input-8-00160637c6d2>](https://localhost:8080/#) in <cell line: 13>()
     11 
     12 processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b-coco")
---> 13 model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b-coco", device_map='auto', quantization_config=nf4_config)

[/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
   2478         if load_in_8bit or load_in_4bit:
   2479             if not (is_accelerate_available() and is_bitsandbytes_available()):
-> 2480                 raise ImportError(
   2481                     "Using `load_in_8bit=True` requires Accelerate: `pip install accelerate` and the latest version of"
   2482                     " bitsandbytes `pip install -i https://test.pypi.org/simple/ bitsandbytes` or"

ImportError: Using `load_in_8bit=True` requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes `pip install -i https://test.pypi.org/simple/ bitsandbytes` or pip install bitsandbytes` 

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
```
### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Step 1: run the code provided in colab

### Expected behavior

Load normally in 4 bit. I actually used the exact same code a few weeks back and it worked perfectly.

Comments:
Did you try after installing the packages? Your environnement should be restarted to make sure you are using the installed packages

Hey. Thanks for the quick reply. I seem to have fixed it by importing a different version of transformers: 
"!pip install -U -q git+https://github.com/huggingface/transformers@de9255de27abfcae4a1f816b904915f0b1e23cd9"

Btw since you are already here, do you know of a way to remove the LLM head from Blip and use another LLM?

No idea no,[ the forum ](https://discuss.huggingface.co/)is a good place to ask this 😉 

Thanks for the reply I guess this issue is more or less closed, but I have a follow up issue regarding BNB and quantization of Blip. Maybe I should make another issue, but I will try here anyway. The model is Blip2ForConditionalGeneration 4bit.

Code:
```
prompt= "What is thos image?"
raw_image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=raw_image, text=prompt, return_tensors="pt").to(device, torch.float16)
generated_ids = model.generate(**inputs).to(torch.float16)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
```

Error:

```
/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1349: UserWarning: Using `max_length`'s default (20) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.
  warnings.warn(
FP4 quantization state not initialized. Please call .cuda() or .to(device) on the LinearFP4 layer first.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-37-fe415740fe90>](https://localhost:8080/#) in <cell line: 4>()
      2 model
      3 inputs = processor(images=raw_image, text=prompt, return_tensors="pt").to(torch.float16)
----> 4 generated_ids = model.generate(**inputs).to(torch.float16)
      5 generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()

10 frames
[/usr/local/lib/python3.10/dist-packages/bitsandbytes/nn/modules.py](https://localhost:8080/#) in forward(self, x)
    246 
    247         bias = None if self.bias is None else self.bias.to(self.compute_dtype)
--> 248         out = bnb.matmul_4bit(x, self.weight.t(), bias=bias, quant_state=self.weight.quant_state)
    249 
    250         out = out.to(inp_dtype)

AttributeError: 'Parameter' object has no attribute 'quant_state'
```

I tried the suggested solution, to use .to(device) on the model which contains  LinearFP4:  (language_projection): Linear4bit(in_features=768, out_features=2560, bias=True)

I got the following error:

```
WARNING:accelerate.big_modeling:You shouldn't move a model when it is dispatched on multiple devices.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-40-85b3397abcc2>](https://localhost:8080/#) in <cell line: 3>()
      1 prompt= "What is thos image?"
      2 image = Image.open(requests.get(url, stream=True).raw)
----> 3 model.to(device, torch.float16)
      4 
      5 inputs = processor(images=raw_image, text=prompt, return_tensors="pt").to(device, torch.float16)

1 frames
[/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py](https://localhost:8080/#) in to(self, *args, **kwargs)
   1884         # Checks if the model has been loaded in 8-bit
   1885         if getattr(self, "is_quantized", False):
-> 1886             raise ValueError(
   1887                 "`.to` is not supported for `4-bit` or `8-bit` models. Please use the model as it is, since the"
   1888                 " model has already been set to the correct devices and casted to the correct `dtype`.


ValueError: `.to` is not supported for `4-bit` or `8-bit` models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
```

Should I try another float?

Edit:

I tried not change another dtype and removing .to(), I got the following error:
"
RuntimeError                              Traceback (most recent call last)
[<ipython-input-50-b68255e99642>](https://localhost:8080/#) in <cell line: 5>()
      3 
      4 inputs = processor(images=raw_image, text=prompt, return_tensors="pt")
----> 5 generated_ids = model.generate(**inputs)
      6 generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()

11 frames
[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py](https://localhost:8080/#) in _conv_forward(self, input, weight, bias)
    457                             weight, bias, self.stride,
    458                             _pair(0), self.dilation, self.groups)
--> 459         return F.conv2d(input, weight, bias, self.stride,
    460                         self.padding, self.dilation, self.groups)
    461 

RuntimeError: Input type (float) and bias type (c10::Half) should be the same
"


Hi @joaopedrosdmm , I am unable to reproduce your error. Make sure that you have the latest libraries. Here's a quick snippet that worked for me. Let me know if it works on your side. 
```py
import torch
from transformers import Blip2ForConditionalGeneration, AutoProcessor, Blip2Processor, BitsAndBytesConfig
from PIL import Image
import requests
from transformers import Blip2Processor
    
nf4_config = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_quant_type="nf4",
  bnb_4bit_use_double_quant=True,
  bnb_4bit_compute_dtype=torch.bfloat16
)

def prepare_img():
    url = "https://huggingface.co/hf-internal-testing/blip-test-image/resolve/main/demo.jpg"
    image = Image.open(requests.get(url, stream=True).raw)
    return image

image = prepare_img()

processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b-coco")
inputs = processor(images=image, return_tensors="pt").to(0, dtype=torch.float16)
model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b-coco", device_map={"":0}, quantization_config=nf4_config)

generated_ids = model.generate(**inputs)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()
print(generated_text)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

in my case, i solved `FP4 quantization state not initialized` error by:

```diff
model = AutoModelForCausalLM.from_pretrained(
    ...
-   low_cpu_mem_usage=False,
)
```