Calling `to()` is not supported for `4-bit` quantized models with the installed version of bitsandbytes. The current device is `cuda:0`. If you intended to move the model, please install bitsandbytes >= 0.43.2.
Author: Kulbuntu
Pull request: no
Labels: name: Quantization

name: bug

### System Info

Transformers : version 4.48.0.dev0
OS : Arch Linux 6.12.6 Kernel 
Python version : 3.11.0
Rocm : 6.3.1 version 
GPU : AMD 6800 XT

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi guys i got a big problem with this error. Few hours ago i was able to use CXH Joy Caption for make a detailled prompt from an image. Sadly it don't work anymore and i don't know at all how i made it work for the first time
When i launch CXH Joy caption it give me this error even with the good bitsandbytes version.
i saw @younesbelkada made a script `from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer
import torch

model_path="tiiuae/falcon-40b-instruct"

config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, load_in_4bit=True, device_map="auto")

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-40b-instruct")

input_text = "Describe the solar system."
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids, max_length=10)
print(tokenizer.decode(outputs[0]))` 

But i don't know to use it and where to use it for solve this issue .
Here you'll see my terminal log from my last session with ComfyUI. 
If someone could help me it could be cool. 
I'm pretty in all this world. 

Here is the log. 
`(bit) [bryan@archlinux ComfyUI]$ python main.py 
[START] Security scan
DEPRECATION: Loading egg at /home/bryan/.pyenv/versions/3.11.1/envs/bit/lib/python3.11/site-packages/flash_attn-2.7.2.post1-py3.11.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
[DONE] Security scan
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2024-12-28 21:52:02.458268
** Platform: Linux
** Python version: 3.11.1 (main, Dec 26 2024, 18:00:13) [GCC 14.2.1 20240910]
** Python executable: /home/bryan/.pyenv/versions/bit/bin/python
** ComfyUI Path: /home/bryan/Documents/ComfyUI
** Log path: /home/bryan/Documents/ComfyUI/comfyui.log
DEPRECATION: Loading egg at /home/bryan/.pyenv/versions/3.11.1/envs/bit/lib/python3.11/site-packages/flash_attn-2.7.2.post1-py3.11.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
DEPRECATION: Loading egg at /home/bryan/.pyenv/versions/3.11.1/envs/bit/lib/python3.11/site-packages/flash_attn-2.7.2.post1-py3.11.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330

Prestartup times for custom nodes:
   0.9 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Manager

Total VRAM 16368 MB, total RAM 31903 MB
pytorch version: 2.5.1+rocm6.2
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.5.1+cu121 with CUDA 1201 (you have 2.5.1+rocm6.2)
    Python  3.11.11 (you have 3.11.1)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details
xformers version: 0.0.29
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon RX 6800 XT : native
Using sub quadratic optimization for attention, if you have memory or speed issues try using: --use-split-cross-attention
[Prompt Server] web root: /home/bryan/Documents/ComfyUI/web
Traceback (most recent call last):
  File "/home/bryan/Documents/ComfyUI/nodes.py", line 2089, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle/__init__.py", line 35, in <module>
    imported_module = importlib.import_module(".py.{}".format(name), __name__)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bryan/.pyenv/versions/3.11.1/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle/py/inner_shadow_v2.py", line 3, in <module>
    from .imagefunc import log, tensor2pil, pil2tensor, image2mask, shift_image, expand_mask, chop_image_v2, chop_mode_v2
  File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle/py/imagefunc.py", line 30, in <module>
    from skimage import img_as_float, img_as_ubyte
ModuleNotFoundError: No module named 'skimage'

Cannot import /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle module for custom nodes: No module named 'skimage'
Traceback (most recent call last):
  File "/home/bryan/Documents/ComfyUI/nodes.py", line 2089, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 936, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1073, in get_code
  File "<frozen importlib._bootstrap_external>", line 1130, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/bryan/Documents/ComfyUI/custom_nodes/joy-caption-batch/__init__.py'

Cannot import /home/bryan/Documents/ComfyUI/custom_nodes/joy-caption-batch module for custom nodes: [Errno 2] No such file or directory: '/home/bryan/Documents/ComfyUI/custom_nodes/joy-caption-batch/__init__.py'
[VideoHelperSuite] - WARNING - Failed to import imageio_ffmpeg
Traceback (most recent call last):
  File "/home/bryan/Documents/ComfyUI/nodes.py", line 2089, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance/__init__.py", line 35, in <module>
    imported_module = importlib.import_module(".py.{}".format(name), __name__)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bryan/.pyenv/versions/3.11.1/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance/py/get_color_tone.py", line 3, in <module>
    from .imagefunc import log, tensor2pil, gaussian_blur, get_image_color_tone, get_image_color_average, RGB_to_HSV, Hex_to_RGB
  File "/home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance/py/imagefunc.py", line 30, in <module>
    from skimage import img_as_float, img_as_ubyte
ModuleNotFoundError: No module named 'skimage'

Cannot import /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance module for custom nodes: No module named 'skimage'
Total VRAM 16368 MB, total RAM 31903 MB
pytorch version: 2.5.1+rocm6.2
xformers version: 0.0.29
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon RX 6800 XT : native
### Loading: ComfyUI-Manager (V2.55.5)
### ComfyUI Version: v0.3.10-4-g4b5bcd8 | Released on '2024-12-27'

Import times for custom nodes:
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/websocket_image_save.py
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Unload-Model
   0.0 seconds (IMPORT FAILED): /home/bryan/Documents/ComfyUI/custom_nodes/joy-caption-batch
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_JC2
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_joytag
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Miaoshouai-Tagger
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_SLK_joy_caption_two
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-GGUF
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts
   0.0 seconds (IMPORT FAILED): /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle_Advance
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/mikey_nodes
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-LTXTricks
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_joy-caption-alpha-two
   0.0 seconds (IMPORT FAILED): /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_LayerStyle
   0.0 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-Manager
   0.3 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_CXH_joy_caption
   0.3 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI_NYJY
   0.3 seconds: /home/bryan/Documents/ComfyUI/custom_nodes/ComfyUI-LTXVideo

Starting server

To see the GUI go to: http://127.0.0.1:8188
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
^Agot prompt
/home/bryan/Documents/ComfyUI/models/clip/siglip-so400m-patch14-384
Loading VLM's custom vision model
/home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_CXH_joy_caption/Joy_caption_alpha.py:237: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(clip_model_path, map_location='cpu')
/home/bryan/Documents/ComfyUI/models/LLM/Meta-Llama-3.1-8B-bnb-4bit
Successfully modified 'base_model_name_or_path' value in '/home/bryan/Documents/ComfyUI/models/Joy_caption_alpha/text_model/adapter_config.json'.
Loading tokenizer
Loading LLM
Loading VLM's custom text model
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
!!! Exception during processing !!! Calling `to()` is not supported for `4-bit` quantized models with the installed version of bitsandbytes. The current device is `cuda:0`. If you intended to move the model, please install bitsandbytes >= 0.43.2.
Traceback (most recent call last):
  File "/home/bryan/Documents/ComfyUI/execution.py", line 328, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bryan/Documents/ComfyUI/execution.py", line 203, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bryan/Documents/ComfyUI/execution.py", line 174, in _map_node_over_list
    process_inputs(input_dict, i)
  File "/home/bryan/Documents/ComfyUI/execution.py", line 163, in process_inputs
    results.append(getattr(obj, func)(**inputs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_CXH_joy_caption/Joy_caption_alpha.py", line 283, in gen
    self.loadCheckPoint()
  File "/home/bryan/Documents/ComfyUI/custom_nodes/Comfyui_CXH_joy_caption/Joy_caption_alpha.py", line 261, in loadCheckPoint
    text_model = AutoModelForCausalLM.from_pretrained(CHECKPOINT_PATH, device_map="auto",
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bryan/.pyenv/versions/bit/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bryan/.pyenv/versions/bit/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4288, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "/home/bryan/.pyenv/versions/bit/lib/python3.11/site-packages/accelerate/big_modeling.py", line 498, in dispatch_model
    model.to(device)
  File "/home/bryan/.pyenv/versions/bit/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3089, in to
    raise ValueError(
ValueError: Calling `to()` is not supported for `4-bit` quantized models with the installed version of bitsandbytes. The current device is `cuda:0`. If you intended to move the model, please install bitsandbytes >= 0.43.2.

Prompt executed in 3.92 seconds
^C
Stopped server
(bit) [bryan@archlinux ComfyUI]$ python -m bitsandbytes
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
CUDA specs: CUDASpecs(highest_compute_capability=(10, 3), cuda_version_string='62', cuda_version_tuple=(6, 2))
PyTorch settings found: CUDA_VERSION=62, Highest Compute Capability: (10, 3).
WARNING: CUDA versions lower than 11 are currently not supported for LLM.int8().
You will be only to use 8-bit optimizers and quantization routines!
To manually override the PyTorch CUDA version please see: https://github.com/TimDettmers/bitsandbytes/blob/main/docs/source/nonpytorchcuda.mdx
The directory listed in your path is found to be non-existent: local/archlinux
The directory listed in your path is found to be non-existent: @/tmp/.ICE-unix/935,unix/archlinux
The directory listed in your path is found to be non-existent: /usr/etc/pyenv.d
The directory listed in your path is found to be non-existent: /usr/local/etc/pyenv.d
The directory listed in your path is found to be non-existent: /etc/pyenv.d
The directory listed in your path is found to be non-existent: /usr/lib/pyenv/hooks
The directory listed in your path is found to be non-existent: /org/freedesktop/DisplayManager/Session1
The directory listed in your path is found to be non-existent: /etc/gtk/gtkrc
The directory listed in your path is found to be non-existent: /home/bryan/.gtkrc
The directory listed in your path is found to be non-existent: /etc/gtk-2.0/gtkrc
The directory listed in your path is found to be non-existent: /Sessions/1
The directory listed in your path is found to be non-existent: /org/freedesktop/DisplayManager/Seat0
The directory listed in your path is found to be non-existent: //debuginfod.archlinux.org 
The directory listed in your path is found to be non-existent: /Windows/1
CUDA SETUP: WARNING! CUDA runtime files not found in any environmental path.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Checking that the library is importable and CUDA is callable...
SUCCESS!
Installation was successful!
`

### Expected behavior

It should return a prompt for an image but it don't

Comments:
WDYT @SunMarc 

Can you check your version of bitsandbytes @Kulbuntu , the error happens because your version is too low. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.