Repository: huggingface/diffusers

Issue:
Running a batch of prompts and negative prompts in SemanticStableDiffusionPipeline causes tensor dimension mismatch
### Describe the bug

Hi, 
I noticed that running a batch of prompts and negative prompts with the SemanticStableDiffusionPipeline causes tensor dimension mismatch between the negative prompt embeddings and prompt embeddings.

I believe a fix to this would be in Line 485 of diffusers/pipelines/semantic_stable_diffusion/pipeline_semantic_stable_diffusion.py: uncond_embeddings = uncond_embeddings.repeat(batch_size, num_images_per_prompt, 1) to uncond_embeddings = uncond_embeddings.repeat(1, num_images_per_prompt, 1). I tested this with a batch of prompts and negative prompts, and also for a single case.

### Reproduction

```
from diffusers import SemanticStableDiffusionPipeline
import torch 
import numpy as np

device="cuda:0" # Change to your device

pipe=SemanticStableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5").to(device)
seeds=np.random.choice(range(1,1000),10,replace=False)
generator= [torch.Generator(device=device).manual_seed(int(seed))  for seed in seeds]
negative_prompts="""original character, children, cartoon"""
out = pipe(prompt=["Any string"]*len(generator), generator=generator, guidance_scale=7.5, 
            num_inference_steps=50, negative_prompt= [negative_prompts]*len(generator),
           editing_prompt=['sunglasses, wearing sunglasses']*len(generator),
           reverse_editing_direction=[False]*len(generator), 
           edit_warmup_steps=[10]*len(generator), 
           edit_guidance_scale=[7.5]*len(generator), 
           edit_threshold=[0.99]*len(generator), 
           edit_momentum_scale=0.3, 
           edit_mom_beta=0.6, 
        )
```

### Logs

```shell
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[1], line 11
      9 generator_i = [torch.Generator(device=device).manual_seed(int(seed))  for seed in seeds]
     10 negative_prompts="""original character, children, cartoon"""
---> 11 out = pipe(prompt=["Any string"]*len(generator_i), generator=generator_i, guidance_scale=7.5, 
     12             num_inference_steps=50, negative_prompt= [negative_prompts]*len(generator_i),
     13            editing_prompt=['sunglasses, wearing sunglasses']*len(generator_i),
     14            reverse_editing_direction=[False]*len(generator_i), 
     15            edit_warmup_steps=[10]*len(generator_i), 
     16            edit_guidance_scale=[7.5]*len(generator_i), 
     17            edit_threshold=[0.99]*len(generator_i), 
     18            edit_momentum_scale=0.3, 
     19            edit_mom_beta=0.6, 
     20         )

File ~/anaconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File ~/anaconda3/lib/python3.10/site-packages/diffusers/pipelines/semantic_stable_diffusion/pipeline_semantic_stable_diffusion.py:494, in SemanticStableDiffusionPipeline.__call__(self, prompt, height, width, num_inference_steps, guidance_scale, negative_prompt, num_images_per_prompt, eta, generator, latents, output_type, return_dict, callback, callback_steps, editing_prompt, editing_prompt_embeddings, reverse_editing_direction, edit_guidance_scale, edit_warmup_steps, edit_cooldown_steps, edit_threshold, edit_momentum_scale, edit_mom_beta, edit_weights, sem_guidance)
    492 if enable_edit_guidance:
    493     print(uncond_embeddings.shape, text_embeddings.shape, edit_concepts.shape)
--> 494     text_embeddings = torch.cat([uncond_embeddings, text_embeddings, edit_concepts])
    495 else:
    496     text_embeddings = torch.cat([uncond_embeddings, text_embeddings])

RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 7680 but got size 768 for tensor number 1 in the list.
```


### System Info

Python 3.10, 
Diffusers 0.17.1

### Who can help?

_No response_


Discussion:
Gently pinging @manuelbrack here :-) 
Makes sense that this went unnoticed as of yet. Using negative prompt in addition to SEGA seems like an edge use case anyways  :D
But we'll have look at it the proposed change looks reasonable, without having tested it yet. 
Also adding @kathath here.

Resolving patch:
diff --git a/src/diffusers/pipelines/semantic_stable_diffusion/pipeline_semantic_stable_diffusion.py b/src/diffusers/pipelines/semantic_stable_diffusion/pipeline_semantic_stable_diffusion.py
--- a/src/diffusers/pipelines/semantic_stable_diffusion/pipeline_semantic_stable_diffusion.py
+++ b/src/diffusers/pipelines/semantic_stable_diffusion/pipeline_semantic_stable_diffusion.py
@@ -442,7 +442,7 @@ def __call__(
         if do_classifier_free_guidance:
             uncond_tokens: List[str]
             if negative_prompt is None:
-                uncond_tokens = [""]
+                uncond_tokens = [""] * batch_size
             elif type(prompt) is not type(negative_prompt):
                 raise TypeError(
                     f"`negative_prompt` should be the same type to `prompt`, but got {type(negative_prompt)} !="
@@ -471,7 +471,7 @@ def __call__(
 
             # duplicate unconditional embeddings for each generation per prompt, using mps friendly method
             seq_len = uncond_embeddings.shape[1]
-            uncond_embeddings = uncond_embeddings.repeat(batch_size, num_images_per_prompt, 1)
+            uncond_embeddings = uncond_embeddings.repeat(1, num_images_per_prompt, 1)
             uncond_embeddings = uncond_embeddings.view(batch_size * num_images_per_prompt, seq_len, -1)
 
             # For classifier free guidance, we need to do two forward passes.


Test patch: