Repository: huggingface/diffusers

Issue:
Visual artifacts when using `DPM++` schedulers and SDXL without the refiner model
### Describe the bug

All DPM++ schedulers are showing visual artifacts out of the base model when `denoising_end=1` (skipping the refiner). This effect is most notable with `DPM++ 2M SDE` configured using the flag [from the docs](https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/schedulers/overview.md).
![image](https://github.com/huggingface/diffusers/assets/8364947/91528205-71d8-4ab4-8d24-88c14944d20f)
These same artifacts are not seen when using SD1.5 with the same scheduler configuration.
![image](https://github.com/huggingface/diffusers/assets/8364947/26733953-8717-44d4-8b52-12d9e3be1a13)

### Reproduction

Intended to run in a notebook
```python3
import torch
from diffusers import StableDiffusionXLPipeline, StableDiffusionPipeline
from typing import cast
from diffusers import DPMSolverMultistepScheduler

sdxl_model = cast(StableDiffusionXLPipeline, StableDiffusionXLPipeline.from_pretrained(
    'stabilityai/stable-diffusion-xl-base-1.0',
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
)).to('cuda')
sd_model = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
    revision="fp16",
).to('cuda')

common_config = {'beta_start': 0.00085, 'beta_end': 0.012, 'beta_schedule': 'scaled_linear'}
dpmpp_2m_sde = DPMSolverMultistepScheduler(**common_config, algorithm_type="sde-dpmsolver++")
sdxl_model.scheduler = dpmpp_2m_sde
sd_model.scheduler = dpmpp_2m_sde

sdxl_model.watermark = None
generator = torch.Generator(device='cuda')
generator.manual_seed(12345)

params = {
    "prompt": ['a cat'],
    "num_inference_steps": 50,
    "height": 1024,
    "width": 1024,
    "guidance_scale": 7,
}

sdxl_res = sdxl_model(**params, denoising_end=1.0, generator=generator)
sdxl_img = sdxl_res.images[0]

generator.manual_seed(12345)
sd_res = sd_model(**params, generator=generator)
sd_img = sd_res.images[0]

display(sdxl_img)
display(sd_img)
```

### Logs

_No response_

### System Info

- `diffusers` version: 0.21.4
- Platform: Linux-5.4.0-163-generic-x86_64-with-glibc2.31
- Python version: 3.11.5
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Huggingface_hub version: 0.17.1
- Transformers version: 4.34.0
- Accelerate version: 0.22.0
- xFormers version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@yiyixuxu @patrickvonplaten


Discussion:
Would be good to get this one fixed, as it's been a problem since the SDXL launch.
@CodeCorrupt 

could you try use this script instead? a few notes:
1. you do not need to pass `denoising_end=1` if you want to skip refiner. Just leave it as its default value `None`- passing`denoising_end=1` is actually a no-op here 
https://github.com/huggingface/diffusers/blob/e5168588864d72a4dca37e90318c6b11da0eaaf1/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py#L905
2. we would want to use the same config as default scheduler when swapping out for a different scheduler  - in our case, the default config is a little bit different from the `common_config` you created, mainly `step_offset` and `timeatep_spaing`

I generated a few images, they looks fine to me but I might have less trained eyes. so let me know if the problem still persist 

```python
import torch
from diffusers import StableDiffusionXLPipeline, StableDiffusionPipeline
from typing import cast
from diffusers import DPMSolverMultistepScheduler

pipe = StableDiffusionXLPipeline.from_pretrained(
    'stabilityai/stable-diffusion-xl-base-1.0',
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
    add_watermarker=False)
pipe = pipe.to('cuda')

pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, algorithm_type="sde-dpmsolver++"
)

generator = torch.Generator(device='cuda').manual_seed(12345)

params = {
    "prompt": ['a cat'],
    "num_inference_steps": 50,
    "guidance_scale": 7,
}

sdxl_img = pipe(**params, generator=generator).images[0]
sdxl_img.save(f"sdxl_dpm_out.png")
```
@CodeCorrupt 
actually I think I still see them with the script I just gave you .....:cold_face:
![yiyi_test_4_out_0](https://github.com/huggingface/diffusers/assets/12631849/1fb438ca-e78b-4ca5-b3c6-c4febfb685cf)

can you try this? I think they are gone for real this time but let me know if it's not......

```python
import torch
from diffusers import StableDiffusionXLPipeline, StableDiffusionPipeline
from typing import cast
from diffusers import DPMSolverMultistepScheduler

pipe = StableDiffusionXLPipeline.from_pretrained(
    'stabilityai/stable-diffusion-xl-base-1.0',
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
    add_watermarker=False)
pipe = pipe.to('cuda')

pipe.scheduler  = DPMSolverMultistepScheduler.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    subfolder="scheduler")


seed = 1
generator = torch.Generator(device='cuda').manual_seed(seed)

params = {
    "prompt": ['a cat'],
    "num_inference_steps": 50,
    "guidance_scale": 7,
}

sdxl_img = pipe(**params, generator=generator).images[0]
sdxl_img.save(f"out_{seed}.png")
```

![yiyi_test_4_out_1](https://github.com/huggingface/diffusers/assets/12631849/a56e2232-4dd3-45b5-a393-ca68830546a6)
Hey @yiyixuxu Looks like it's not using the right `algorithm_type` in the most recent example. I made a test script to compare.

```python
import torch
from diffusers import StableDiffusionXLPipeline
from diffusers import DPMSolverMultistepScheduler, DPMSolverSinglestepScheduler

pipe = StableDiffusionXLPipeline.from_pretrained(
    'stabilityai/stable-diffusion-xl-base-1.0',
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
    add_watermarker=False)
pipe = pipe.to('cuda')

common_config = {'beta_start': 0.00085, 'beta_end': 0.012, 'beta_schedule': 'scaled_linear'}
schedulers = {
    "DPMPP_2M": (DPMSolverMultistepScheduler, {}),
    "DPMPP_2M_K": (DPMSolverMultistepScheduler, {"use_karras_sigmas": True}),
    "DPMPP_2M_SDE": (DPMSolverMultistepScheduler, {"algorithm_type": "sde-dpmsolver++"}),
    "DPMPP_2M_SDE_K": (DPMSolverMultistepScheduler, {"use_karras_sigmas": True, "algorithm_type": "sde-dpmsolver++"}),
    "DPMPP_SDE": (DPMSolverSinglestepScheduler, {}),
    "DPMPP_SDE_K": (DPMSolverSinglestepScheduler, {"use_karras_sigmas": True}),
}

selected_scheduler = 'DPMPP_2M_SDE'
scheduler_old = schedulers[selected_scheduler][0](**common_config, **schedulers[selected_scheduler][1])
scheduler_new = schedulers[selected_scheduler][0].from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    subfolder="scheduler",
    **schedulers[selected_scheduler][1],
)

params = {
    "prompt": ['a cat'],
    "num_inference_steps": 50,
    "guidance_scale": 7,
}
for s in [scheduler_new, scheduler_old]:
    seed = 12345
    generator = torch.Generator(device='cuda').manual_seed(seed)

    pipe.scheduler = s
    sdxl_img = pipe(**params, generator=generator).images[0]
    display(sdxl_img)
```

I'm still seeing the same artifacts when using the `.from_pretrained()`
![image](https://github.com/huggingface/diffusers/assets/8364947/71bfe6f8-f8c4-4848-a132-80acc60fd213)

@CodeCorrupt 
ahhh really sorry you're absolutely right - my results was actually based on `dpmsolver +++` :pleading_face:

@LuChengTHU can you aso take a look into this? I can reproduce this bug 
@CodeCorrupt 
actually i saw same artifacts in k-diffusion too (with auto1111)
![image](https://github.com/huggingface/diffusers/assets/12631849/2c131a40-845c-4620-9a58-f5aa86cf9738)

here is my setting - do you see same thing in automatic1111 as well? 

![Screenshot 2023-10-19 at 11 20 42 AM](https://github.com/huggingface/diffusers/assets/12631849/6b6f05c5-6c47-4df7-8d40-5f528ce74532)

@CodeCorrupt 
btw the artifacts would go away if you increase the number of inference steps

below, num_inference_steps = 60,70,80,100. It gradually reduce and I think completely disappeared at 100 steps 

![yiyi_test_4_out_seed_12345_steps_60](https://github.com/huggingface/diffusers/assets/12631849/1c976a54-b92f-4d8d-af42-1d406ca1c779)
![yiyi_test_4_out_seed_12345_steps_70](https://github.com/huggingface/diffusers/assets/12631849/af4d23c7-b24f-434f-8e6c-946e371ad3de)
![yiyi_test_4_out_seed_12345_steps_80](https://github.com/huggingface/diffusers/assets/12631849/27527df8-c6ea-460a-8045-223f3feffb0b)
![yiyi_test_4_out_seed_12345_steps_100](https://github.com/huggingface/diffusers/assets/12631849/95ff2e4b-2bb7-4a04-855c-1ce48e16a232)


Since the same issue also present in k-diffusion/auto1111, and also because of the the fact that it went away when we increase the number of inference steps,  I think this is probably not a bug in the implementation. It could be just this scheduler does not work well with SDXL. I would be curious to understand why and hopefully @LuChengTHU have some insights to share soon :) but I think there is not much action for us to take here

also cc @patrickvonplaten here, let me know if we should investigate this further 








Have we confirmed that swapping the default VAE to use [this one](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix) doesn't help at all? 
@sayakpaul
> Have we confirmed that swapping the default VAE to use [this one](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix) doesn't help at all?
 
confirmed 
DPM++ scheduler is known to not work super well for SDXL. Euler is usually the better choice
@CodeCorrupt 

let us know if there is anything you want us to investigate more:) 
> DPM++ scheduler is known to not work super well for SDXL. Euler is usually the better choice

DPM++ cleans up on FID when comparing it to other methods like Euler for the same number of steps. It would be nice if we could get it working because we would save compute when doing inference.
@AmericanPresidentJimmyCarter  Noob question - what is "FID"?
Never mind - found it! https://en.wikipedia.org/wiki/Fr%C3%A9chet_inception_distance
Yeah, when training new text to image models we always use thing sampler because it produces the best (lowest) FID values. It works very well for SD1.x/2, so it would be good to figure out what is causing the issue with SDXL.
Hey @yiyixuxu It would be great if we could find the root cause and get the DPM++ schedulers to "work super great" on SDXL :smile:. I'm doing some research myself, but I imagine it would be far more efficient if you and the team could look into it since you have the domain knowledge here.
My biggest question still is, "Why don't DPM++ schedulers work well?".
> DPM++ scheduler is known to not work super well for SDXL. Euler is usually the better choice

Hi @patrickvonplaten , is there any more examples / findings about this conclusion? I will try to figure out the reason :)
> > DPM++ scheduler is known to not work super well for SDXL. Euler is usually the better choice
> 
> Hi @patrickvonplaten , is there any more examples / findings about this conclusion? I will try to figure out the reason :)

I'm not fully sure why it happens. The effect seems to go away a bit when using higher inference step sizes (e.g. 50), but with just 25 steps there seem to always be some artifacts. It would be incredible if you could dive a bit deeper here :heart: 

Resolving patch:
diff --git a/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py b/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
--- a/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
+++ b/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
@@ -117,9 +117,17 @@ class DPMSolverMultistepScheduler(SchedulerMixin, ConfigMixin):
         lower_order_final (`bool`, defaults to `True`):
             Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. This can
             stabilize the sampling of DPMSolver for steps < 15, especially for steps <= 10.
+        euler_at_final (`bool`, defaults to `False`):
+            Whether to use Euler's method in the final step. It is a trade-off between numerical stability and detail
+            richness. This can stabilize the sampling of the SDE variant of DPMSolver for small number of inference
+            steps, but sometimes may result in blurring.
         use_karras_sigmas (`bool`, *optional*, defaults to `False`):
             Whether to use Karras sigmas for step sizes in the noise schedule during the sampling process. If `True`,
             the sigmas are determined according to a sequence of noise levels {σi}.
+        use_lu_lambdas (`bool`, *optional*, defaults to `False`):
+            Whether to use the uniform-logSNR for step sizes proposed by Lu's DPM-Solver in the noise schedule during
+            the sampling process. If `True`, the sigmas and time steps are determined according to a sequence of
+            `lambda(t)`.
         lambda_min_clipped (`float`, defaults to `-inf`):
             Clipping threshold for the minimum value of `lambda(t)` for numerical stability. This is critical for the
             cosine (`squaredcos_cap_v2`) noise schedule.
@@ -154,7 +162,9 @@ def __init__(
         algorithm_type: str = "dpmsolver++",
         solver_type: str = "midpoint",
         lower_order_final: bool = True,
+        euler_at_final: bool = False,
         use_karras_sigmas: Optional[bool] = False,
+        use_lu_lambdas: Optional[bool] = False,
         lambda_min_clipped: float = -float("inf"),
         variance_type: Optional[str] = None,
         timestep_spacing: str = "linspace",
@@ -258,6 +268,12 @@ def set_timesteps(self, num_inference_steps: int = None, device: Union[str, torc
             sigmas = self._convert_to_karras(in_sigmas=sigmas, num_inference_steps=num_inference_steps)
             timesteps = np.array([self._sigma_to_t(sigma, log_sigmas) for sigma in sigmas]).round()
             sigmas = np.concatenate([sigmas, sigmas[-1:]]).astype(np.float32)
+        elif self.config.use_lu_lambdas:
+            lambdas = np.flip(log_sigmas.copy())
+            lambdas = self._convert_to_lu(in_lambdas=lambdas, num_inference_steps=num_inference_steps)
+            sigmas = np.exp(lambdas)
+            timesteps = np.array([self._sigma_to_t(sigma, log_sigmas) for sigma in sigmas]).round()
+            sigmas = np.concatenate([sigmas, sigmas[-1:]]).astype(np.float32)
         else:
             sigmas = np.interp(timesteps, np.arange(0, len(sigmas)), sigmas)
             sigma_last = ((1 - self.alphas_cumprod[0]) / self.alphas_cumprod[0]) ** 0.5
@@ -354,6 +370,19 @@ def _convert_to_karras(self, in_sigmas: torch.FloatTensor, num_inference_steps)
         sigmas = (max_inv_rho + ramp * (min_inv_rho - max_inv_rho)) ** rho
         return sigmas
 
+    def _convert_to_lu(self, in_lambdas: torch.FloatTensor, num_inference_steps) -> torch.FloatTensor:
+        """Constructs the noise schedule of Lu et al. (2022)."""
+
+        lambda_min: float = in_lambdas[-1].item()
+        lambda_max: float = in_lambdas[0].item()
+
+        rho = 1.0  # 1.0 is the value used in the paper
+        ramp = np.linspace(0, 1, num_inference_steps)
+        min_inv_rho = lambda_min ** (1 / rho)
+        max_inv_rho = lambda_max ** (1 / rho)
+        lambdas = (max_inv_rho + ramp * (min_inv_rho - max_inv_rho)) ** rho
+        return lambdas
+
     def convert_model_output(
         self,
         model_output: torch.FloatTensor,
@@ -787,8 +816,9 @@ def step(
         if self.step_index is None:
             self._init_step_index(timestep)
 
-        lower_order_final = (
-            (self.step_index == len(self.timesteps) - 1) and self.config.lower_order_final and len(self.timesteps) < 15
+        # Improve numerical stability for small number of steps
+        lower_order_final = (self.step_index == len(self.timesteps) - 1) and (
+            self.config.euler_at_final or (self.config.lower_order_final and len(self.timesteps) < 15)
         )
         lower_order_second = (
             (self.step_index == len(self.timesteps) - 2) and self.config.lower_order_final and len(self.timesteps) < 15
diff --git a/src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py b/src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py
--- a/src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py
+++ b/src/diffusers/schedulers/scheduling_dpmsolver_multistep_inverse.py
@@ -117,6 +117,10 @@ class DPMSolverMultistepInverseScheduler(SchedulerMixin, ConfigMixin):
         lower_order_final (`bool`, defaults to `True`):
             Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. This can
             stabilize the sampling of DPMSolver for steps < 15, especially for steps <= 10.
+        euler_at_final (`bool`, defaults to `False`):
+            Whether to use Euler's method in the final step. It is a trade-off between numerical stability and detail
+            richness. This can stabilize the sampling of the SDE variant of DPMSolver for small number of inference
+            steps, but sometimes may result in blurring.
         use_karras_sigmas (`bool`, *optional*, defaults to `False`):
             Whether to use Karras sigmas for step sizes in the noise schedule during the sampling process. If `True`,
             the sigmas are determined according to a sequence of noise levels {σi}.
@@ -154,6 +158,7 @@ def __init__(
         algorithm_type: str = "dpmsolver++",
         solver_type: str = "midpoint",
         lower_order_final: bool = True,
+        euler_at_final: bool = False,
         use_karras_sigmas: Optional[bool] = False,
         lambda_min_clipped: float = -float("inf"),
         variance_type: Optional[str] = None,
@@ -804,8 +809,9 @@ def step(
         if self.step_index is None:
             self._init_step_index(timestep)
 
-        lower_order_final = (
-            (self.step_index == len(self.timesteps) - 1) and self.config.lower_order_final and len(self.timesteps) < 15
+        # Improve numerical stability for small number of steps
+        lower_order_final = (self.step_index == len(self.timesteps) - 1) and (
+            self.config.euler_at_final or (self.config.lower_order_final and len(self.timesteps) < 15)
         )
         lower_order_second = (
             (self.step_index == len(self.timesteps) - 2) and self.config.lower_order_final and len(self.timesteps) < 15


Test patch:
diff --git a/tests/schedulers/test_scheduler_dpm_multi.py b/tests/schedulers/test_scheduler_dpm_multi.py
--- a/tests/schedulers/test_scheduler_dpm_multi.py
+++ b/tests/schedulers/test_scheduler_dpm_multi.py
@@ -29,6 +29,7 @@ def get_scheduler_config(self, **kwargs):
             "algorithm_type": "dpmsolver++",
             "solver_type": "midpoint",
             "lower_order_final": False,
+            "euler_at_final": False,
             "lambda_min_clipped": -float("inf"),
             "variance_type": None,
         }
@@ -195,6 +196,10 @@ def test_lower_order_final(self):
         self.check_over_configs(lower_order_final=True)
         self.check_over_configs(lower_order_final=False)
 
+    def test_euler_at_final(self):
+        self.check_over_configs(euler_at_final=True)
+        self.check_over_configs(euler_at_final=False)
+
     def test_lambda_min_clipped(self):
         self.check_over_configs(lambda_min_clipped=-float("inf"))
         self.check_over_configs(lambda_min_clipped=-5.1)
@@ -258,6 +263,12 @@ def test_full_loop_with_karras_and_v_prediction(self):
 
         assert abs(result_mean.item() - 0.2096) < 1e-3
 
+    def test_full_loop_with_lu_and_v_prediction(self):
+        sample = self.full_loop(prediction_type="v_prediction", use_lu_lambdas=True)
+        result_mean = torch.mean(torch.abs(sample))
+
+        assert abs(result_mean.item() - 0.1554) < 1e-3
+
     def test_switch(self):
         # make sure that iterating over schedulers with same config names gives same results
         # for defaults