I can't make sense of this works on Windows but not on Linux AutoModelForCausalLM.from_pretrained
Author: FurkanGozukara
Pull request: no
Labels: name: Quantization

name: bug


I am using below code with exactly same VENV on Windows and Linux and Windows works but Linux fails

error

```
1:50:23 - INFO - Attempting to load CogVLM2 model with quantization: 4 on device: cuda
01:50:23 - INFO - Loading tokenizer from: /home/Ubuntu/Downloads/upscale/STAR/models/cogvlm2-video-llama3-chat
01:50:23 - INFO - Tokenizer loaded successfully.
01:50:23 - INFO - Preparing to load model from: /home/Ubuntu/Downloads/upscale/STAR/models/cogvlm2-video-llama3-chat with quant: 4, dtype: torch.bfloat16, device: cuda, device_map: auto, low_cpu_mem: True
01:50:23 - INFO - Starting model loading - this operation cannot be interrupted once started
/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/torchvision/transforms/_functional_video.py:6: UserWarning: The 'torchvision.transforms._functional_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms.functional' module instead.
  warnings.warn(
/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/torchvision/transforms/_transforms_video.py:22: UserWarning: The 'torchvision.transforms._transforms_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms' module instead.
  warnings.warn(
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:08<00:00,  1.47s/steps]
01:50:33 - ERROR - Failed to load CogVLM2 model from path: /home/Ubuntu/Downloads/upscale/STAR/models/cogvlm2-video-llama3-chat
01:50:33 - ERROR - Exception type: ValueError
01:50:33 - ERROR - Exception details: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.

Traceback (most recent call last):
  File "/home/Ubuntu/Downloads/upscale/STAR/logic/cogvlm_utils.py", line 160, in load_cogvlm_model
    raise model_loading_result["error"]
  File "/home/Ubuntu/Downloads/upscale/STAR/logic/cogvlm_utils.py", line 122, in load_model_thread
    model = AutoModelForCausalLM.from_pretrained(
  File "/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained
    return model_class.from_pretrained(
  File "/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4000, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/accelerate/big_modeling.py", line 502, in dispatch_model
    model.to(device)
  File "/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2849, in to
    raise ValueError(
ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
01:50:33 - ERROR - Error during auto-captioning: 'Could not load CogVLM2 model (check logs for details): `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.'
Traceback (most recent call last):
  File "/home/Ubuntu/Downloads/upscale/STAR/logic/cogvlm_utils.py", line 160, in load_cogvlm_model
    raise model_loading_result["error"]
  File "/home/Ubuntu/Downloads/upscale/STAR/logic/cogvlm_utils.py", line 122, in load_model_thread
    model = AutoModelForCausalLM.from_pretrained(
  File "/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained
    return model_class.from_pretrained(
  File "/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4000, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/accelerate/big_modeling.py", line 502, in dispatch_model
    model.to(device)
  File "/home/Ubuntu/Downloads/upscale/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2849, in to
    raise ValueError(
ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/Ubuntu/Downloads/upscale/STAR/logic/cogvlm_utils.py", line 478, in auto_caption
    local_model_ref, local_tokenizer_ref = load_cogvlm_model(quantization, cogvlm_device, cog_vlm_model_path, logger)
  File "/home/Ubuntu/Downloads/upscale/STAR/logic/cogvlm_utils.py", line 243, in load_cogvlm_model
    raise gr.Error(f"Could not load CogVLM2 model (check logs for details): {str(e)[:200]}")
gradio.exceptions.Error: 'Could not load CogVLM2 model (check logs for details): `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.'
01:50:33 - INFO - Unloading CogVLM2 model with strategy: full
01:50:33 - INFO - CogVLM2 model/tokenizer not loaded or already unloaded.
```

entire pip freeze below



```
            bnb_config = None
            model_dtype = torch.bfloat16 if (device == 'cuda' and torch.cuda.is_available() and torch.cuda.get_device_capability()[0] >= 8) else torch.float16

            if BITSANDBYTES_AVAILABLE and device == 'cuda':
                if quantization == 4:
                    bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=model_dtype)
                elif quantization == 8:
                    bnb_config = BitsAndBytesConfig(load_in_8bit=True)
            elif quantization in [4, 8] and device != 'cuda':
                if logger:
                    logger.warning("BitsAndBytes quantization is only available on CUDA. Loading in FP16/BF16.")
                quantization = 0

            current_device_map = None
            if bnb_config and device == 'cuda':
                current_device_map = "auto"

            effective_low_cpu_mem_usage = True if bnb_config else False

            if logger:
                logger.info(f"Preparing to load model from: {cog_vlm_model_path} with quant: {quantization}, dtype: {model_dtype}, device: {device}, device_map: {current_device_map}, low_cpu_mem: {effective_low_cpu_mem_usage}")

            # Final check for cancellation before model loading (this is the longest operation)
            cancellation_manager.check_cancel("before model loading")

            # Note to user that model loading cannot be interrupted once started
            if logger:
                logger.info("Starting model loading - this operation cannot be interrupted once started")

            # Use a separate thread for model loading with periodic cancellation checks
            model_loading_result = {"model": None, "error": None, "cancelled_before_start": False}
            model_loading_complete = threading.Event()

            def load_model_thread():
                try:
                    # One final check right before starting the loading
                    if cancellation_manager.is_cancelled():
                        model_loading_result["cancelled_before_start"] = True
                        model_loading_complete.set()
                        return

                    ### START OF FIX ###
                    # Build arguments for from_pretrained conditionally
                    from_pretrained_kwargs = {
                        "trust_remote_code": True,
                        "quantization_config": bnb_config,
                        "low_cpu_mem_usage": effective_low_cpu_mem_usage,
                        "device_map": current_device_map
                    }

                    # Only specify torch_dtype for non-quantized models.
                    # For BNB models, the dtype is handled by the quantization_config
                    # and passing it explicitly can cause conflicts.
                    if not bnb_config:
                        from_pretrained_kwargs["torch_dtype"] = model_dtype if device == 'cuda' else torch.float32

                    model = AutoModelForCausalLM.from_pretrained(
                        cog_vlm_model_path,
                        **from_pretrained_kwargs
                    )
```

```absl-py==2.3.1
accelerate==1.8.1
addict==2.4.0
aiofiles==24.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.12.14
aiosignal==1.4.0
annotated-types==0.7.0
anyio==4.9.0
async-timeout==5.0.1
attrs==25.3.0
av==15.0.0
basicsr==1.4.2
bitsandbytes==0.46.1
Brotli==1.1.0
certifi==2025.7.14
charset-normalizer==3.4.2
click==8.2.1
decorator==5.2.1
decord==0.6.0
deepspeed==0.16.4
diffusers==0.34.0
easydict==1.13
einops==0.8.1
exceptiongroup==1.3.0
fairscale==0.4.13
fastapi==0.116.1
ffmpeg==1.4
ffmpy==0.6.0
filelock==3.18.0
flash-attn @ https://huggingface.co/MonsterMMORPG/SECourses_Premium_Flash_Attention/resolve/main/flash_attn-2.7.4.post1-cp310-cp310-linux_x86_64.whl#sha256=8978824a0affa29999428ee0cc87a4de3085e84ee91b612bf81ad5acd440ced7
frozenlist==1.7.0
fsspec==2025.7.0
ftfy==6.3.1
future==1.0.0
fvcore==0.1.5.post20221221
gradio==5.37.0
gradio_client==1.10.4
groovy==0.1.2
grpcio==1.73.1
h11==0.16.0
hf-xet==1.1.5
hf_transfer==0.1.9
hjson==3.1.0
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.33.4
idna==3.10
imageio==2.37.0
imageio-ffmpeg==0.6.0
importlib_metadata==8.7.0
iopath==0.1.10
Jinja2==3.1.6
lazy_loader==0.4
lightning-utilities==0.14.3
lmdb==1.7.2
lpips==0.1.4
Markdown==3.8.2
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mdurl==0.1.2
memory-efficient-attention-pytorch==0.1.6
moviepy==2.2.1
mpmath==1.3.0
msgpack==1.1.1
multidict==6.6.3
networkx==3.4.2
ninja==1.11.1.4
numpy==1.26.4
nvidia-cublas-cu12==12.8.3.14
nvidia-cuda-cupti-cu12==12.8.57
nvidia-cuda-nvrtc-cu12==12.8.61
nvidia-cuda-runtime-cu12==12.8.57
nvidia-cudnn-cu12==9.7.1.26
nvidia-cufft-cu12==11.3.3.41
nvidia-cufile-cu12==1.13.0.11
nvidia-curand-cu12==10.3.9.55
nvidia-cusolver-cu12==11.7.2.55
nvidia-cusparse-cu12==12.5.7.53
nvidia-cusparselt-cu12==0.6.3
nvidia-nccl-cu12==2.26.2
nvidia-nvjitlink-cu12==12.8.61
nvidia-nvtx-cu12==12.8.55
open-clip-torch==2.20.0
opencv-python==4.11.0.86
orjson==3.11.0
packaging==25.0
pandas==2.3.1
parameterized==0.9.0
pillow==11.3.0
platformdirs==4.3.8
portalocker==3.2.0
proglog==0.1.12
propcache==0.3.2
protobuf==3.20.3
psutil==7.0.0
py-cpuinfo==9.0.0
pydantic==2.11.7
pydantic_core==2.33.2
pydub==0.25.1
pyee==13.0.0
Pygments==2.19.2
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
python-ffmpeg==2.0.12
python-multipart==0.0.20
pytorch-lightning==2.5.2
pytorchvideo @ https://huggingface.co/MonsterMMORPG/SECourses_Premium_Flash_Attention/resolve/main/pytorchvideo-0.1.5-py3-none-any.whl#sha256=514a2ef4a68c5a7302e96b1d2b81ac0aeacbd1f63577ffa464fd145add1cc639
pytz==2025.2
PyYAML==6.0.2
regex==2024.11.6
requests==2.32.4
rich==14.0.0
ruff==0.12.3
safehttpx==0.1.6
safetensors==0.5.3
scenedetect @ git+https://github.com/Breakthrough/PySceneDetect.git@34dffabd8666bf4cbb94ff1995f74fcf593eb368
scikit-image==0.25.2
scipy==1.15.3
semantic-version==2.10.0
sentencepiece==0.2.0
shellingham==1.5.4
six==1.17.0
sk-video==1.1.10
sniffio==1.3.1
spandrel==0.4.1
starlette==0.47.1
sympy==1.14.0
tabulate==0.9.0
tb-nightly==2.20.0a20250716
tensorboard-data-server==0.7.2
termcolor==3.1.0
tifffile==2025.5.10
timm==1.0.17
tokenizers==0.19.1
tomli==2.2.1
tomlkit==0.13.3
torch==2.7.0+cu128
torchao==0.11.0+cu128
torchaudio==2.7.0+cu128
torchmetrics==1.7.4
torchsde==0.2.6
torchvision==0.22.0+cu128
tqdm==4.67.1
trampoline==0.1.2
transformers==4.43.4
triton==3.3.0
typer==0.16.0
typing-inspection==0.4.1
typing_extensions==4.14.1
tzdata==2025.2
urllib3==2.5.0
uvicorn==0.35.0
wcwidth==0.2.13
websockets==15.0.1
Werkzeug==3.1.3
xformers @ https://huggingface.co/MonsterMMORPG/SECourses_Premium_Flash_Attention/resolve/main/xformers-0.0.30+3abeaa9e.d20250427-cp310-cp310-linux_x86_64.whl#sha256=4ffe9a26923049b48076e4cd4c5004a2738e9539b2abdcf32775fcdbe3120c1d
yacs==0.1.8
yapf==0.43.0
yarl==1.20.1
zipp==3.23.0
``` 

### Who can help?

text models: @ArthurZucker
quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber



Comments:
Thanks for the report @FurkanGozukara , can you update your version of transformers and see if this still fails ? 

@SunMarc thank you . when i upgrade transformers i get this error i tried before :( just tested again with `Successfully installed tokenizers-0.21.2 transformers-4.53.2`

```
13:25:44 - ERROR - Error during auto-captioning: 'CogVLMVideoForCausalLM' object has no attribute '_extract_past_from_model_output'
Traceback (most recent call last):
  File "E:\Ultimate_Video_Processing_v1\STAR\logic\cogvlm_utils.py", line 684, in auto_caption
    outputs_tensor = local_model_ref.generate(**inputs_on_device, **gen_kwargs)
  File "E:\Ultimate_Video_Processing_v1\venv\lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "E:\Ultimate_Video_Processing_v1\venv\lib\site-packages\transformers\generation\utils.py", line 2625, in generate
    result = self._sample(
  File "E:\Ultimate_Video_Processing_v1\venv\lib\site-packages\transformers\generation\utils.py", line 3612, in _sample
    model_kwargs = self._update_model_kwargs_for_generation(
  File "E:\Ultimate_Video_Processing_v1\STAR\models\modules\transformers_modules\cogvlm2-video-llama3-chat\modeling_cogvlm.py", line 726, in _update_model_kwargs_for_generation
    cache_name, cache = self._extract_past_from_model_output(
  File "E:\Ultimate_Video_Processing_v1\venv\lib\site-packages\torch\nn\modules\module.py", line 1940, in __getattr__
    raise AttributeError(
AttributeError: 'CogVLMVideoForCausalLM' object has no attribute '_extract_past_from_model_output'
```


@SunMarc this is the file giving that error with latest version 

[modeling_cogvlm.py.txt](https://github.com/user-attachments/files/21293308/modeling_cogvlm.py.txt)



@SunMarc if you can tell me what to change i am gonna clone https://huggingface.co/THUDM/cogvlm2-video-llama3-chat and fix there

`_extract_past_from_model_output` was probably removed in the latest transformers and i'm not too familiar with the codebase. Tagging @gante as this is related to generation 

@SunMarc thank you. Gemini gave me an answer and on Windows worked. now testing on linux

fixed issue with latest transformers + cloning repo and fixing there myself