Repository: huggingface/diffusers

Issue:
IP adapter basic example causes ValueError: too many values to unpack (expected 3)
### Describe the bug

I tried to run the basic example from [this tutorial ](https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter?tasks=Inpainting). Building the model works, but it crashes on inference.

### Reproduction

```
from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch

pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to("cuda")
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name="ip-adapter_sdxl.bin")
pipeline.set_ip_adapter_scale(0.6)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_diner.png")
generator = torch.Generator(device="cpu").manual_seed(0)
images = pipeline(
    prompt="a polar bear sitting in a chair drinking a milkshake", 
    ip_adapter_image=image,
    negative_prompt="deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality",
    num_inference_steps=100, 
    generator=generator,
).images
```
Code is from [this tutorial](https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter?tasks=Inpainting)

### Logs

```shell
images = pipeline(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 1241, in __call__
    noise_pred = self.unet(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/unets/unet_2d_condition.py", line 1121, in forward
    sample, res_samples = downsample_block(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/unets/unet_2d_blocks.py", line 1199, in forward
    hidden_states = attn(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/transformers/transformer_2d.py", line 391, in forward
    hidden_states = block(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/attention.py", line 366, in forward
    attn_output = self.attn2(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 512, in forward
    return self.processor(
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 2195, in __call__
    ip_key = attn.head_to_batch_dim(ip_key)
  File "/home/ubuntu/anaconda3/envs/generative/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 551, in head_to_batch_dim
    batch_size, seq_len, dim = tensor.shape
ValueError: too many values to unpack (expected 3)
```


### System Info

- `diffusers` version: 0.26.3
- Platform: Linux-6.2.0-1018-aws-x86_64-with-glibc2.35
- Python version: 3.9.17
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Huggingface_hub version: 0.20.3
- Transformers version: 4.33.0.dev0
- Accelerate version: 0.21.0
- xFormers version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@DN6 @yiyixuxu


Discussion:
I installed Xformers. For some reason, that fixed it 🤷‍♂️
installing xFormers, pip uninstalled torch-1.13 and installed torch-2.2

the IP-Adapter attention processor for pytorch 2 is different, so you no longer have this problem

IMO, this issue should be fixed with a PR
@fabiorigano 
ohh our ip-adapter does not work with `IPAdapterAttnProcessor`??
hi @yiyixuxu 

The output tensors of `to_k_ip `and `to_v_ip` are 4D, but they are expected to be 3D (batch size, sequence length and embedding dimension) for head_to_batch_dim

the second dimension has always shape 1, so the tensors can be easily reshaped. this issue occurs only in IPAdapterAttnProcessor and not in IPAdapterAttnProcessor2_0

here it is the code to fix the isse
```py
# for ip-adapter
        for current_ip_hidden_states, scale, to_k_ip, to_v_ip, mask in zip(
            ip_hidden_states, self.scale, self.to_k_ip, self.to_v_ip, ip_adapter_masks
        ):
            ip_key = to_k_ip(current_ip_hidden_states)
            ip_value = to_v_ip(current_ip_hidden_states)

            ip_key = ip_key.reshape(ip_key.shape[0], ip_key.shape[-2], ip_key.shape[-1])  # Add this
            ip_value = ip_value.reshape(ip_value.shape[0], ip_value.shape[-2], ip_value.shape[-1]) # and this

            ip_key = attn.head_to_batch_dim(ip_key)
            ip_value = attn.head_to_batch_dim(ip_value)
```
@fabiorigano 
looks good! wanna open a PR?
sure!

Resolving patch:
diff --git a/src/diffusers/models/attention_processor.py b/src/diffusers/models/attention_processor.py
--- a/src/diffusers/models/attention_processor.py
+++ b/src/diffusers/models/attention_processor.py
@@ -559,12 +559,16 @@ def head_to_batch_dim(self, tensor: torch.Tensor, out_dim: int = 3) -> torch.Ten
             `torch.Tensor`: The reshaped tensor.
         """
         head_size = self.heads
-        batch_size, seq_len, dim = tensor.shape
-        tensor = tensor.reshape(batch_size, seq_len, head_size, dim // head_size)
+        if tensor.ndim == 3:
+            batch_size, seq_len, dim = tensor.shape
+            extra_dim = 1
+        else:
+            batch_size, extra_dim, seq_len, dim = tensor.shape
+        tensor = tensor.reshape(batch_size, seq_len * extra_dim, head_size, dim // head_size)
         tensor = tensor.permute(0, 2, 1, 3)
 
         if out_dim == 3:
-            tensor = tensor.reshape(batch_size * head_size, seq_len, dim // head_size)
+            tensor = tensor.reshape(batch_size * head_size, seq_len * extra_dim, dim // head_size)
 
         return tensor
 


Test patch: