ValueError: too many values to unpack (expected 2) when use glm4v or cogvlm2
Author: aihao2000
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.42.4
- Platform: Linux-5.4.119-19.0009.28-x86_64-with-glibc2.31
- Python version: 3.10.14
- Huggingface_hub version: 0.24.1
- Safetensors version: 0.4.3
- Accelerate version: 0.33.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: 1
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H800

### Who can help?

@amyeroberts
@ArthurZucker
@Narsil


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda"

tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-4v-9b", trust_remote_code=True)

query = '描述这张图片'
image = Image.open("your image").convert('RGB')
inputs = tokenizer.apply_chat_template([{"role": "user", "image": image, "content": query}],
                                       add_generation_prompt=True, tokenize=True, return_tensors="pt",
                                       return_dict=True)  # chat mode

inputs = inputs.to(device)
model = AutoModelForCausalLM.from_pretrained(
    "THUDM/glm-4v-9b",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True
).to(device).eval()

gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1}
with torch.no_grad():
    outputs = model.generate(**inputs, **gen_kwargs)
    outputs = outputs[:, inputs['input_ids'].shape[1]:]
    print(tokenizer.decode(outputs[0]))

```

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 38
     36 gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1}
     37 with torch.no_grad():
---> 38     outputs = model.generate(**inputs, **gen_kwargs)
     39     outputs = outputs[:, inputs["input_ids"].shape[1] :]
     40     print(tokenizer.decode(outputs[0]))

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/utils/_contextlib.py:116](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/utils/_contextlib.py:116), in context_decorator..decorate_context(*args, **kwargs)
    113 @functools.wraps(func)
    114 def decorate_context(*args, **kwargs):
    115     with ctx_factory():
--> 116         return func(*args, **kwargs)

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/transformers/generation/utils.py:1914](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/transformers/generation/utils.py:1914), in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
   1906     input_ids, model_kwargs = self._expand_inputs_for_generation(
   1907         input_ids=input_ids,
   1908         expand_size=generation_config.num_return_sequences,
   1909         is_encoder_decoder=self.config.is_encoder_decoder,
   1910         **model_kwargs,
   1911     )
   1913     # 13. run sample (it degenerates to greedy search when `generation_config.do_sample=False`)
-> 1914     result = self._sample(
   1915         input_ids,
   1916         logits_processor=prepared_logits_processor,
   1917         logits_warper=prepared_logits_warper,
   1918         stopping_criteria=prepared_stopping_criteria,
   1919         generation_config=generation_config,
   1920         synced_gpus=synced_gpus,
   1921         streamer=streamer,
   1922         **model_kwargs,
   1923     )
   1925 elif generation_mode in (GenerationMode.BEAM_SAMPLE, GenerationMode.BEAM_SEARCH):
   1926     # 11. prepare logits warper
   1927     prepared_logits_warper = (
   1928         self._get_logits_warper(generation_config, device=input_ids.device)
   1929         if generation_config.do_sample
   1930         else None
   1931     )

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/transformers/generation/utils.py:2651](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/transformers/generation/utils.py:2651), in GenerationMixin._sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, logits_warper, **model_kwargs)
   2648 model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
   2650 # forward pass to get next token
-> 2651 outputs = self(
   2652     **model_inputs,
   2653     return_dict=True,
   2654     output_attentions=output_attentions,
   2655     output_hidden_states=output_hidden_states,
   2656 )
   2658 if synced_gpus and this_peer_finished:
   2659     continue  # don't waste resources running the code we don't need

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:1173](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:1173), in ChatGLMForConditionalGeneration.forward(self, input_ids, images, position_ids, attention_mask, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict, return_last_logit)
   1170 use_cache = use_cache if use_cache is not None else self.config.use_cache
   1171 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-> 1173 transformer_outputs = self.transformer(
   1174     input_ids=input_ids,
   1175     images=images,
   1176     position_ids=position_ids,
   1177     attention_mask=attention_mask,
   1178     past_key_values=past_key_values,
   1179     inputs_embeds=inputs_embeds,
   1180     use_cache=use_cache,
   1181     output_hidden_states=output_hidden_states,
   1182     return_dict=return_dict,
   1183 )
   1185 hidden_states = transformer_outputs[0]
   1186 if return_last_logit:

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:1041](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:1041), in ChatGLMModel.forward(self, input_ids, images, position_ids, attention_mask, full_attention_mask, past_key_values, inputs_embeds, use_cache, output_hidden_states, return_dict)
   1038     rotary_pos_emb = rotary_pos_emb[None, :seq_length]
   1040 # Run encoder.
-> 1041 hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder(
   1042     inputs_embeds, full_attention_mask, rotary_pos_emb=rotary_pos_emb,
   1043     kv_caches=past_key_values, use_cache=use_cache, output_hidden_states=output_hidden_states
   1044 )
   1046 if not return_dict:
   1047     return tuple(v for v in [hidden_states, presents, all_hidden_states, all_self_attentions] if v is not None)

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:791](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:791), in GLMTransformer.forward(self, hidden_states, attention_mask, rotary_pos_emb, kv_caches, use_cache, output_hidden_states)
    781     layer_ret = torch.utils.checkpoint.checkpoint(
    782         layer,
    783         hidden_states,
   (...)
    788         use_reentrant=False
    789     )
    790 else:
--> 791     layer_ret = layer(
    792         hidden_states,
    793         attention_mask,
    794         rotary_pos_emb,
    795         kv_cache=kv_caches[index],
    796         use_cache=use_cache
    797     )
    798 hidden_states, kv_cache = layer_ret
    799 if use_cache:

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:694](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:694), in GLMBlock.forward(self, hidden_states, attention_mask, rotary_pos_emb, kv_cache, use_cache)
    692 layernorm_output = self.input_layernorm(hidden_states)
    693 # Self attention.
--> 694 attention_output, kv_cache = self.self_attention(
    695     layernorm_output,
    696     attention_mask,
    697     rotary_pos_emb,
    698     kv_cache=kv_cache,
    699     use_cache=use_cache
    700 )
    702 # Residual connection.
    703 if self.apply_residual_connection_post_layernorm:

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1553), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/miniconda3/envs/torch/lib/python3.10/site-packages/torch/nn/modules/module.py:1562), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:563](https://vscode-remote+qs-002d5930-002d228138-002etensorboard-002dnj-002edevops-002exiaohongshu-002ecom.vscode-resource.vscode-cdn.net/mnt/nj-aigc/usr/aihao/workspace/TheChosenOne/~/.cache/huggingface/modules/transformers_modules/glm-4v-9b/modeling_chatglm.py:563), in SelfAttention.forward(self, hidden_states, attention_mask, rotary_pos_emb, kv_cache, use_cache)
    561 # adjust key and value for inference
    562 if kv_cache is not None:
--> 563     cache_k, cache_v = kv_cache
    564     key_layer = torch.cat((cache_k, key_layer), dim=2)
    565     value_layer = torch.cat((cache_v, value_layer), dim=2)

ValueError: too many values to unpack (expected 2)
```

### Expected behavior

Successful operation

Comments:
Hi @aihao2000, thanks for raising an issue! 

The model in the example has its code on the hub and so is maintained by its repo owners. I'd suggest opening a discussion on [checkpoint's repo](https://huggingface.co/THUDM/glm-4v-9b) detailing this issue. 

Is this a behaviour that you've observed any regression with i.e. it used to work in earlier versions of transformers? 

@amyeroberts I downgraded transformers to 4.40.2 and now everything works fine.

hmmmm - might be a regression in the cache then, thanks for confirming! cc @ArthurZucker @gante