TypeError: GenerationMixin._extract_past_from_model_output() got an unexpected keyword argument 'standardize_cache_format'
Author: FurkanGozukara
Pull request: no
Labels: name: bug

I am using CogVLM2 video captioning model

It works latest with transformers==4.43.4

with transformers==4.44.0 and forward I get below error

but I need to use latest version of transformers since currently 4bit quantization fails on some gpus and platforms

how can i fix this issue?

`TypeError: GenerationMixin._extract_past_from_model_output() got an unexpected keyword argument 'standardize_cache_format'`

```
14:23:32 - INFO - Final video tensor shape for CogVLM processing: torch.Size([3, 24, 720, 1280])
14:23:35 - ERROR - Error during auto-captioning: GenerationMixin._extract_past_from_model_output() got an unexpected keyword argument 'standardize_cache_format'
Traceback (most recent call last):
  File "E:\Ultimate_Video_Processing_v1\STAR\logic\cogvlm_utils.py", line 679, in auto_caption
    outputs_tensor = local_model_ref.generate(**inputs_on_device, **gen_kwargs)
  File "E:\Ultimate_Video_Processing_v1\venv\lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "E:\Ultimate_Video_Processing_v1\venv\lib\site-packages\transformers\generation\utils.py", line 2024, in generate
    result = self._sample(
  File "E:\Ultimate_Video_Processing_v1\venv\lib\site-packages\transformers\generation\utils.py", line 3032, in _sample
    model_kwargs = self._update_model_kwargs_for_generation(
  File "E:\Ultimate_Video_Processing_v1\STAR\models\modules\transformers_modules\cogvlm2-video-llama3-chat\modeling_cogvlm.py", line 726, in _update_model_kwargs_for_generation
    cache_name, cache = self._extract_past_from_model_output(
TypeError: GenerationMixin._extract_past_from_model_output() got an unexpected keyword argument 'standardize_cache_format'
```

@amyeroberts, @qubvel @SunMarc @MekkCyber 

the error i am getting is below with 4.43.1 on B200 when doing 4bit quant. interesting same code same libraries on my rtx 5090 on windows working without errors

fp16 has no issues


```
11:45:10 - INFO - Preparing to load model from: /workspace/STAR/models/cogvlm2-video-llama3-chat with quant: 4, dtype: torch.bfloat16, device: cuda, device_map: auto, low_cpu_mem: True
11:45:10 - INFO - Starting model loading - this operation cannot be interrupted once started
/workspace/venv/lib/python3.10/site-packages/torchvision/transforms/_functional_video.py:6: UserWarning: The 'torchvision.transforms._functional_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms.functional' module instead.
  warnings.warn(
/workspace/venv/lib/python3.10/site-packages/torchvision/transforms/_transforms_video.py:22: UserWarning: The 'torchvision.transforms._transforms_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms' module instead.
  warnings.warn(
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████| 6/6 [01:18<00:00, 13.07s/steps]
11:46:30 - ERROR - Failed to load CogVLM2 model from path: /workspace/STAR/models/cogvlm2-video-llama3-chat
11:46:30 - ERROR - Exception type: ValueError
11:46:30 - ERROR - Exception details: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
Traceback (most recent call last):
  File "/workspace/STAR/logic/cogvlm_utils.py", line 160, in load_cogvlm_model
    raise model_loading_result["error"]
  File "/workspace/STAR/logic/cogvlm_utils.py", line 122, in load_model_thread
    model = AutoModelForCausalLM.from_pretrained(
  File "/workspace/venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained
    return model_class.from_pretrained(
  File "/workspace/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4000, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "/workspace/venv/lib/python3.10/site-packages/accelerate/big_modeling.py", line 502, in dispatch_model
    model.to(device)
  File "/workspace/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2849, in to
    raise ValueError(
ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
11:46:30 - ERROR - Error during auto-captioning: 'Could not load CogVLM2 model (check logs for details): `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.'
Traceback (most recent call last):
  File "/workspace/STAR/logic/cogvlm_utils.py", line 160, in load_cogvlm_model
    raise model_loading_result["error"]
  File "/workspace/STAR/logic/cogvlm_utils.py", line 122, in load_model_thread
    model = AutoModelForCausalLM.from_pretrained(
  File "/workspace/venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained
    return model_class.from_pretrained(
  File "/workspace/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4000, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "/workspace/venv/lib/python3.10/site-packages/accelerate/big_modeling.py", line 502, in dispatch_model
    model.to(device)
  File "/workspace/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2849, in to
    raise ValueError(
ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/STAR/logic/cogvlm_utils.py", line 478, in auto_caption
    local_model_ref, local_tokenizer_ref = load_cogvlm_model(quantization, cogvlm_device, cog_vlm_model_path, logger)
  File "/workspace/STAR/logic/cogvlm_utils.py", line 243, in load_cogvlm_model
    raise gr.Error(f"Could not load CogVLM2 model (check logs for details): {str(e)[:200]}")
gradio.exceptions.Error: 'Could not load CogVLM2 model (check logs for details): `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.'
11:46:30 - INFO - Unloading CogVLM2 model with strategy: full
```



Comments:
Hey @FurkanGozukara,

This model uses remote code that's stored on the Hub. When a model is released with remote code, it usually means it's not maintained by us, and can break with new versions of `transformers`.

I'd recommend opening an issue in the CogVLM hub repo so they can fix it in custom code. In the meantime, you can downgrade your transformers version to get it working

> Hey [@FurkanGozukara](https://github.com/FurkanGozukara),
> 
> This model uses remote code that's stored on the Hub. When a model is released with remote code, it usually means it's not maintained by us, and can break with new versions of `transformers`.
> 
> I'd recommend opening an issue in the CogVLM hub repo so they can fix it in custom code. In the meantime, you can downgrade your transformers version to get it working

sad that means it will never get fixed

what i wonder is why this was removed ? standardize_cache_format

It was removed since we are trying to clean-up generate and use a single cache format for all models. Thus no need for standardizing cache every step. Btw in the `main` version currently the `_extract_past_from_model_output` is removed totally

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.