Repository: huggingface/diffusers

Issue:
Bug in StableDiffusionXLInpaintPipeline when strength==1
### Describe the bug

When you run inference with `strength==1` (is_max_strength set to True), you fail with the following error:

```
...
...
...
File /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py:1267, in StableDiffusionXLInpaintPipeline.__call__(self, prompt, prompt_2, image, mask_image, masked_image_latents, height, width, strength, num_inference_steps, denoising_start, denoising_end, guidance_scale, negative_prompt, negative_prompt_2, num_images_per_prompt, eta, generator, latents, prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds, output_type, return_dict, callback, callback_steps, cross_attention_kwargs, guidance_rescale, original_size, crops_coords_top_left, target_size, negative_original_size, negative_crops_coords_top_left, negative_target_size, aesthetic_score, negative_aesthetic_score, clip_skip)
   1264 return_image_latents = num_channels_unet == 4
   1266 add_noise = True if denoising_start is None else False
-> 1267 latents_outputs = self.prepare_latents(
   1268     batch_size * num_images_per_prompt,
   1269     num_channels_latents,
   1270     height,
   1271     width,
   1272     prompt_embeds.dtype,
   1273     device,
   1274     generator,
   1275     latents,
   1276     image=init_image,
   1277     timestep=latent_timestep,
   1278     is_strength_max=is_strength_max,
   1279     add_noise=add_noise,
   1280     return_noise=True,
   1281     return_image_latents=return_image_latents,
   1282 )
   1284 if return_image_latents:
   1285     latents, noise, image_latents = latents_outputs

File /usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py:733, in StableDiffusionXLInpaintPipeline.prepare_latents(self, batch_size, num_channels_latents, height, width, dtype, device, generator, latents, image, timestep, is_strength_max, add_noise, return_noise, return_image_latents)
    731     image = image.to(device=device, dtype=dtype)
    732     image_latents = self._encode_vae_image(image=image, generator=generator)
--> 733 image_latents = image_latents.repeat(batch_size // image_latents.shape[0], 1, 1, 1)
    735 if latents is None and add_noise:
    736     noise = randn_tensor(shape, generator=generator, device=device, dtype=dtype)

UnboundLocalError: local variable 'image_latents' referenced before assignment
```

### Reproduction

```py
import torch
from diffusers import StableDiffusionXLInpaintPipeline
from diffusers.utils import load_image

pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    # "stabilityai/stable-diffusion-xl-base-1.0",
    "diffusers/stable-diffusion-xl-1.0-inpainting-0.1",
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True,
)
pipe.to("cuda")

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = load_image(img_url).convert("RGB")
mask_image = load_image(mask_url).convert("RGB")

prompt = "A majestic tiger sitting on a bench"
image = pipe(
    prompt=prompt, image=init_image, mask_image=mask_image, num_inference_steps=50, strength=1
).images[0]
```

### Logs

_No response_

### System Info

- `diffusers` version: 0.22.0.dev0
- Platform: Linux-5.19.0-76051900-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Huggingface_hub version: 0.17.3
- Transformers version: 4.34.1
- Accelerate version: 0.23.0
- xFormers version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

@sayakpaul @yiyixuxu 


Discussion:
The error occurs due to not handling the case when `is_strength_max == True` and `latents is None`. In this case, the latent should be noise (which is correctly implemented).

Proposed fix:

https://github.com/huggingface/diffusers/blob/080081bdeda61a4723a61594f9f1c12909d695b7/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py#L725

```py
...
        if image.shape[1] == 4:
            image_latents = image.to(device=device, dtype=dtype)
            image_latents = image_latents.repeat(batch_size // image_latents.shape[0], 1, 1, 1)
        elif return_image_latents or (latents is None and not is_strength_max):
            image = image.to(device=device, dtype=dtype)
            image_latents = self._encode_vae_image(image=image, generator=generator)
            image_latents = image_latents.repeat(batch_size // image_latents.shape[0], 1, 1, 1)
...
```

Resolving patch:
diff --git a/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py b/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py
--- a/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py
+++ b/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py
@@ -741,10 +741,11 @@ def prepare_latents(
 
         if image.shape[1] == 4:
             image_latents = image.to(device=device, dtype=dtype)
+            image_latents = image_latents.repeat(batch_size // image_latents.shape[0], 1, 1, 1)
         elif return_image_latents or (latents is None and not is_strength_max):
             image = image.to(device=device, dtype=dtype)
             image_latents = self._encode_vae_image(image=image, generator=generator)
-        image_latents = image_latents.repeat(batch_size // image_latents.shape[0], 1, 1, 1)
+            image_latents = image_latents.repeat(batch_size // image_latents.shape[0], 1, 1, 1)
 
         if latents is None and add_noise:
             noise = randn_tensor(shape, generator=generator, device=device, dtype=dtype)


Test patch: