Repository: huggingface/diffusers

Issue:
New inpaint pipeline with parameter 'strength' generates poor results when using EulerAncestralDiscreteScheduler
### Describe the bug

Hi, I recently found the update for inpainting with strength, it's really cool. [code](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py#L660)
But when using EulerAncestralDiscreteScheduler, just simply set strength as 0.5 or other values<1, the inpaint pipeline fails. 

I'm not familiar with the math stuff in diffusion, but it seems that it's the scale process that leads to the problem.

When I rescale the latents back **after the preparing latents function** with `latents = latents / self.scheduler.init_noise_sigma` before the denoise process by unet, the results seems to be good.

```
    # for EulerAncestralDiscreteScheduler, the init_noise_sigma is 14.6146
    # for DPMSolverMultistepScheduler, the init_noise_sigma is 1.0
    latents = latents * self.scheduler.init_noise_sigma 

    return latents
```

# input and mask
![image](https://github.com/huggingface/diffusers/assets/72586584/efd7054a-dbce-4ef3-b855-a99fa7cbe17e)

![image](https://github.com/huggingface/diffusers/assets/72586584/bed00597-f8f2-4520-8044-ce22b7381a91)

# result
![image](https://github.com/huggingface/diffusers/assets/72586584/ef809efd-7e49-482e-b9a6-8c3232ba4c8c)

### Reproduction


```
import diffusers
import torch
from PIL import Image
from diffusers import StableDiffusionInpaintPipeline
pipe = StableDiffusionInpaintPipeline.from_pretrained('runwayml/stable-diffusion-inpainting', torch_dtype=torch.float16).to('cuda')
pipe.safety_checker = None
pipe.scheduler = diffusers.EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
image_pil = Image.open('./test/images/girl.png')
target_mask = Image.open('./test/images/girl_mask.png')
inpainted_image = pipe(
prompt="1girl, holding flowers, best quality, masterpiece, realistic, highres, ultral high details", negative_prompt="nsfw, breasts, nude, lowres, bad anatomy, bad hands, cropped, worst quality", image=image_pil, mask_image=target_mask,
height=image_pil.size[1], width=image_pil.size[0],strength=0.5,
num_inference_steps=30, guidance_scale=9,
).images[0]
```

### Logs

_No response_

### System Info

- `diffusers` version: 0.17.0.dev0
- Platform: Linux-5.19.0-35-generic-x86_64-with-glibc2.35
- Python version: 3.10.9
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Huggingface_hub version: 0.13.3
- Transformers version: 4.26.0
- Accelerate version: 0.18.0
- xFormers version: 0.0.15.dev+0bad001.d20230403
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
New inpaint pipeline with parameter 'strength' generates poor results when using EulerAncestralDiscreteScheduler
### Describe the bug

Hi, I recently found the update for inpainting with strength, it's really cool. [code](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py#L660)
But when using EulerAncestralDiscreteScheduler, just simply set strength as 0.5 or other values<1, the inpaint pipeline fails. 

I'm not familiar with the math stuff in diffusion, but it seems that it's the scale process that leads to the problem.

When I rescale the latents back **after the preparing latents function** with `latents = latents / self.scheduler.init_noise_sigma` before the denoise process by unet, the results seems to be good.

```
    # for EulerAncestralDiscreteScheduler, the init_noise_sigma is 14.6146
    # for DPMSolverMultistepScheduler, the init_noise_sigma is 1.0
    latents = latents * self.scheduler.init_noise_sigma 

    return latents
```

# input and mask
![image](https://github.com/huggingface/diffusers/assets/72586584/efd7054a-dbce-4ef3-b855-a99fa7cbe17e)

![image](https://github.com/huggingface/diffusers/assets/72586584/bed00597-f8f2-4520-8044-ce22b7381a91)

# result
![image](https://github.com/huggingface/diffusers/assets/72586584/ef809efd-7e49-482e-b9a6-8c3232ba4c8c)

### Reproduction


```
import diffusers
import torch
from PIL import Image
from diffusers import StableDiffusionInpaintPipeline
pipe = StableDiffusionInpaintPipeline.from_pretrained('runwayml/stable-diffusion-inpainting', torch_dtype=torch.float16).to('cuda')
pipe.safety_checker = None
pipe.scheduler = diffusers.EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
image_pil = Image.open('./test/images/girl.png')
target_mask = Image.open('./test/images/girl_mask.png')
inpainted_image = pipe(
prompt="1girl, holding flowers, best quality, masterpiece, realistic, highres, ultral high details", negative_prompt="nsfw, breasts, nude, lowres, bad anatomy, bad hands, cropped, worst quality", image=image_pil, mask_image=target_mask,
height=image_pil.size[1], width=image_pil.size[0],strength=0.5,
num_inference_steps=30, guidance_scale=9,
).images[0]
```

### Logs

_No response_

### System Info

- `diffusers` version: 0.17.0.dev0
- Platform: Linux-5.19.0-35-generic-x86_64-with-glibc2.35
- Python version: 3.10.9
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Huggingface_hub version: 0.13.3
- Transformers version: 4.26.0
- Accelerate version: 0.18.0
- xFormers version: 0.0.15.dev+0bad001.d20230403
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


Discussion:
@Magicalppj thanks for catching this! you are quite right it looks like the `init_noise_sigma` scaling factor for the latents breaks things when the initialised latents aren't pure noise (e.g. strength <1.) and the sigma is not 1. See:

![image](https://github.com/huggingface/diffusers/assets/71332436/7271c3ad-b78e-4199-ba1d-e07413a6f3a4)

After this small fix which means only the 'pure noise' when strength = 1 is scaled by the `init_noise_sigma`, e.g.
For `line 639` in `pipeline_stable_diffusion_inpaint.py`
```
if is_strength_max:
    # if strength is 100% then simply initialise the latents to noise
    latents = noise
    # scale the initial noise by the standard deviation required by the scheduler
    latents = latents * self.scheduler.init_noise_sigma
```

The results look like this for various samplers:

![image](https://github.com/huggingface/diffusers/assets/71332436/3c4ff88b-31b5-494f-8a44-ec08ee8b431a)

Problem solved! I will submit a PR soon with this fix. Possibly after the weekend.

@patrickvonplaten I also noticed that a strength of 0 can break things for other samplers and in general, the whole pipeline doesn't really make sense with inference steps < 1. Should I throw an error if this is the case to prompt the user to increase the strength so inference steps are more than or equal to 1 or should I just return the original image and skip the sampling loop altogether? Let me know the preference and I can put into a PR with the above when I find some time.
> @Magicalppj thanks for catching this! you are quite right it looks like the `init_noise_sigma` scaling factor for the latents breaks things when the initialised latents aren't pure noise (e.g. strength <1.) and the sigma is not 1. See:
> 
> ![image](https://user-images.githubusercontent.com/71332436/239652059-7271c3ad-b78e-4199-ba1d-e07413a6f3a4.png)
> 
> After this small fix which means only the 'pure noise' when strength = 1 is scaled by the `init_noise_sigma`, e.g. For `line 639` in `pipeline_stable_diffusion_inpaint.py`
> 
> ```
> if is_strength_max:
>     # if strength is 100% then simply initialise the latents to noise
>     latents = noise
>     # scale the initial noise by the standard deviation required by the scheduler
>     latents = latents * self.scheduler.init_noise_sigma
> ```

Thanks for reply! It's clear now the scale for latents should only be applied to pure noise, that indeed solves this problem, great! 
@rupertmenneer, it'd be great if you could send a PR! Agree that it doesn't make much sense if we have < 1 inference steps - happy to throw a nice error here. Also, it'd be very nice if you could open a PR to fix the strength :-) 
@Magicalppj thanks for catching this! you are quite right it looks like the `init_noise_sigma` scaling factor for the latents breaks things when the initialised latents aren't pure noise (e.g. strength <1.) and the sigma is not 1. See:

![image](https://github.com/huggingface/diffusers/assets/71332436/7271c3ad-b78e-4199-ba1d-e07413a6f3a4)

After this small fix which means only the 'pure noise' when strength = 1 is scaled by the `init_noise_sigma`, e.g.
For `line 639` in `pipeline_stable_diffusion_inpaint.py`
```
if is_strength_max:
    # if strength is 100% then simply initialise the latents to noise
    latents = noise
    # scale the initial noise by the standard deviation required by the scheduler
    latents = latents * self.scheduler.init_noise_sigma
```

The results look like this for various samplers:

![image](https://github.com/huggingface/diffusers/assets/71332436/3c4ff88b-31b5-494f-8a44-ec08ee8b431a)

Problem solved! I will submit a PR soon with this fix. Possibly after the weekend.

@patrickvonplaten I also noticed that a strength of 0 can break things for other samplers and in general, the whole pipeline doesn't really make sense with inference steps < 1. Should I throw an error if this is the case to prompt the user to increase the strength so inference steps are more than or equal to 1 or should I just return the original image and skip the sampling loop altogether? Let me know the preference and I can put into a PR with the above when I find some time.
> @Magicalppj thanks for catching this! you are quite right it looks like the `init_noise_sigma` scaling factor for the latents breaks things when the initialised latents aren't pure noise (e.g. strength <1.) and the sigma is not 1. See:
> 
> ![image](https://user-images.githubusercontent.com/71332436/239652059-7271c3ad-b78e-4199-ba1d-e07413a6f3a4.png)
> 
> After this small fix which means only the 'pure noise' when strength = 1 is scaled by the `init_noise_sigma`, e.g. For `line 639` in `pipeline_stable_diffusion_inpaint.py`
> 
> ```
> if is_strength_max:
>     # if strength is 100% then simply initialise the latents to noise
>     latents = noise
>     # scale the initial noise by the standard deviation required by the scheduler
>     latents = latents * self.scheduler.init_noise_sigma
> ```

Thanks for reply! It's clear now the scale for latents should only be applied to pure noise, that indeed solves this problem, great! 
@rupertmenneer, it'd be great if you could send a PR! Agree that it doesn't make much sense if we have < 1 inference steps - happy to throw a nice error here. Also, it'd be very nice if you could open a PR to fix the strength :-) 

Resolving patch:
diff --git a/src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py b/src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py
--- a/src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py
+++ b/src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py
@@ -863,12 +863,13 @@ def prepare_latents(
 
         if latents is None:
             noise = randn_tensor(shape, generator=generator, device=device, dtype=dtype)
+            # if strength is 1. then initialise the latents to noise, else initial to image + noise
             latents = noise if is_strength_max else self.scheduler.add_noise(image_latents, noise, timestep)
+            # if pure noise then scale the initial latents by the  Scheduler's init sigma
+            latents = latents * self.scheduler.init_noise_sigma if is_strength_max else latents
         else:
             latents = latents.to(device)
-
-        # scale the initial noise by the standard deviation required by the scheduler
-        latents = latents * self.scheduler.init_noise_sigma
+            latents = latents * self.scheduler.init_noise_sigma
 
         outputs = (latents,)
 
diff --git a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py
--- a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py
+++ b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py
@@ -648,12 +648,13 @@ def prepare_latents(
 
         if latents is None:
             noise = randn_tensor(shape, generator=generator, device=device, dtype=dtype)
+            # if strength is 1. then initialise the latents to noise, else initial to image + noise
             latents = noise if is_strength_max else self.scheduler.add_noise(image_latents, noise, timestep)
+            # if pure noise then scale the initial latents by the  Scheduler's init sigma
+            latents = latents * self.scheduler.init_noise_sigma if is_strength_max else latents
         else:
             latents = latents.to(device)
-
-        # scale the initial noise by the standard deviation required by the scheduler
-        latents = latents * self.scheduler.init_noise_sigma
+            latents = latents * self.scheduler.init_noise_sigma
 
         outputs = (latents,)
 
@@ -912,6 +913,12 @@ def __call__(
         timesteps, num_inference_steps = self.get_timesteps(
             num_inference_steps=num_inference_steps, strength=strength, device=device
         )
+        # check that number of inference steps is not < 1 - as this doesn't make sense
+        if num_inference_steps < 1:
+            raise ValueError(
+                f"After adjusting the num_inference_steps by strength parameter: {strength}, the number of pipeline"
+                f"steps is {num_inference_steps} which is < 1 and not appropriate for this pipeline."
+            )
         # at which timestep to set the initial noise (n.b. 50% if strength is 0.5)
         latent_timestep = timesteps[:1].repeat(batch_size * num_images_per_prompt)
         # create a boolean to check if the strength is set to 1. if so then initialise the latents with pure noise


Test patch:
diff --git a/tests/pipelines/stable_diffusion/test_stable_diffusion_inpaint.py b/tests/pipelines/stable_diffusion/test_stable_diffusion_inpaint.py
--- a/tests/pipelines/stable_diffusion/test_stable_diffusion_inpaint.py
+++ b/tests/pipelines/stable_diffusion/test_stable_diffusion_inpaint.py
@@ -231,6 +231,20 @@ def test_stable_diffusion_inpaint_lora(self):
     def test_inference_batch_single_identical(self):
         super().test_inference_batch_single_identical(expected_max_diff=3e-3)
 
+    def test_stable_diffusion_inpaint_strength_zero_test(self):
+        device = "cpu"  # ensure determinism for the device-dependent torch.Generator
+        components = self.get_dummy_components()
+        sd_pipe = StableDiffusionInpaintPipeline(**components)
+        sd_pipe = sd_pipe.to(device)
+        sd_pipe.set_progress_bar_config(disable=None)
+
+        inputs = self.get_dummy_inputs(device)
+
+        # check that the pipeline raises value error when num_inference_steps is < 1
+        inputs["strength"] = 0.01
+        with self.assertRaises(ValueError):
+            sd_pipe(**inputs).images
+
 
 class StableDiffusionSimpleInpaintPipelineFastTests(StableDiffusionInpaintPipelineFastTests):
     pipeline_class = StableDiffusionInpaintPipeline