Repository: huggingface/diffusers
Pull request: [SDXL] Partial diffusion support for Text2Img and Img2Img Pipelines
# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #4003 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md)?
- [x] Did you read our [philosophy doc](https://github.com/huggingface/diffusers/blob/main/PHILOSOPHY.md) (important for complex PRs)?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/diffusers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Core library:

- Schedulers: @williamberman and @patrickvonplaten
- Pipelines:  @patrickvonplaten and @sayakpaul
- Training examples: @sayakpaul and @patrickvonplaten
- Docs: @stevenliu and @yiyixu
- JAX and MPS: @pcuenca
- Audio: @sanchit-gandhi
- General functionalities: @patrickvonplaten and @sayakpaul

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- transformers: [different repo](https://github.com/huggingface/transformers)
- safetensors: [different repo](https://github.com/huggingface/safetensors)

-->
- General functionalities: @patrickvonplaten and @sayakpaul


Problem statement:
[SD-XL] Enhanced inference control
## Feature Request: Enhanced Control of the Inference Process in SDXL Pipeline
We are seeking to enhance the control over the denoising process within the SDXL pipeline. The objective is to better handle the generation and refinement of images through fine-tuning of noise application and control of the inference process.

**Describe the solution you'd like**

1. **Introduce a new parameter, `max_inference_steps`**: This new parameter, optional, will control the number of steps in the inference process before returning the outputs. This provides an early exit option during the denoising process. For instance, if `num_inference_steps=20` and `max_inference_steps=10`, the pipeline will output a latent that is half denoised. 

2. **Introduce a new parameter, `add_noise`**: This optional parameter, defaulting to `True` for backward compatibility, controls the addition of noise to the input in the SDXL Img2Img pipeline. When set to `False`, this parameter will prevent further noise from being added to the inputs. We need this, so that the details from the base image are not overwritten by the refiner, which does not have great composition in its data distribution.

3. **Introduce a new parameter, `first_inference_step`**: This optional parameter, defaulting to `None` for backward compatibility, is intended for the SDXL Img2Img pipeline. When set, the pipeline will not use the `strength` parameter to reduce `num_inference_steps`, and early timesteps will be skipped, allowing alignment and reducing artifacts.

**Describe alternatives you've considered**

Using the callback function from the base pipeline to retrieve the latents halfway through, and pass them in as `latents` to the Refiner model is a hack approach, but it's the only other one there is.

**Intended outcome**

These enhancements will allow for more nuanced and precise control over the image generation and refining process within the SDXL pipeline, leading to more effective and desired results. 

Kindly review and consider this proposal. I can begin this work, if it is accepted.

Hints from issue discussion:
cc @patrickvonplaten @sayakpaul 
Seems like interesting additions! Thanks for being so detailed. Do you have some results for us to take a look at with these changes? 
Hello, I am Sytan, the creator of said workflow. I have been working with Comfy himself to make an optimized SDXL workflow that is not only faster than the traditionally shared img2img workflow, but also higher quality. All of this is being run locally on a 3080 10GB GPU. I can attach some example images below.

From Left to Right: Punk

- Img2img (24 seconds)
- Mine (15 seconds)
- Base only (18 seconds)

![Comfy Punk](https://github.com/huggingface/diffusers/assets/122675732/84a238d0-ca6d-4363-8b54-d05d7700148b)
From Left to Right: Forest

- Base (19 seconds)
- Img2Img (25 seconds)
- Mine (15 seconds)

![Comfy Spring Forest](https://github.com/huggingface/diffusers/assets/122675732/e0b9967a-c399-4906-a262-f9750cefc548)
From Left to Right: Statue

- Img2Img (23 seconds)
- Base (18 seconds)
- Mine (14 seconds)

![Comfy Statue](https://github.com/huggingface/diffusers/assets/122675732/bbf5442f-b421-4af5-95c1-b1ad00cebfa0)

I hope these results and time savings are satisfactory. I will be posting a live 10 image comparison vote to Reddit soon to make sure that people enjoy my results blindly.

If any of you have questions about the workflow or other details, please refer back to @bghira
Oh wow. The results are so amazing! 🤯

While reporting the timings, could you also shed some details on what do you mean by "Img2Img" and "Base")?
I have just talked to Joe Penna from SAI, and he has given me the clear to share my information. I will be posting more info tomorrow when I have it all collected!
img2img refers to the standard diffusers example for the refiner where all timesteps are completed by the sdxl base model.

base is just sdxl without refiner.
Interesting thread! Let's wait until @SytanSD's answer then and see how we can most elegantly add the improved workflow? 
@SytanSD and I have been working together on this enhancement and the Diffusers work will be done by myself, with the background and majority of the research having been done by him. My explanation above is from him :) 
Superb! Would be down to review the PRs! 
> I have just talked to Joe Penna from SAI, and he has given me the clear to share my information. I will be posting more info tomorrow when I have it all collected!

are you ScythSergal from Reddit?

Commit message: diffusers#4003 Do not add noise during img2img when we have a defined first timestep

Patch to review:
diff --git a/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py b/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
index 265e325a45e0..4f0854c3c142 100644
--- a/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
+++ b/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
@@ -545,6 +545,7 @@ def __call__(
         height: Optional[int] = None,
         width: Optional[int] = None,
         num_inference_steps: int = 50,
+        max_inference_steps: Optional[int] = None,
         guidance_scale: float = 5.0,
         negative_prompt: Optional[Union[str, List[str]]] = None,
         num_images_per_prompt: Optional[int] = 1,
@@ -579,6 +580,9 @@ def __call__(
             num_inference_steps (`int`, *optional*, defaults to 50):
                 The number of denoising steps. More denoising steps usually lead to a higher quality image at the
                 expense of slower inference.
+            max_inference_steps (`int`, *optional*):
+                Instead of completing the backwards pass entirely, stop and return the output after this many steps.
+                Can be useful with `output_type="latent"` and an img2img pipeline, possibly with better fine detail.
             guidance_scale (`float`, *optional*, defaults to 7.5):
                 Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598).
                 `guidance_scale` is defined as `w` of equation 2. of [Imagen
@@ -782,6 +786,9 @@ def __call__(
                     progress_bar.update()
                     if callback is not None and i % callback_steps == 0:
                         callback(i, t, latents)
+                if max_inference_steps is not None and i >= max_inference_steps:
+                    logger.debug(f'Breaking inference loop at step {i} as we have reached max_inference_steps={max_inference_steps}')
+                    break
 
         # make sure the VAE is in float32 mode, as it overflows in float16
         self.vae.to(dtype=torch.float32)
diff --git a/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py b/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
index 1618d9d04a74..dec3b1ad37e1 100644
--- a/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
+++ b/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
@@ -455,11 +455,27 @@ def prepare_extra_step_kwargs(self, generator, eta):
         return extra_step_kwargs
 
     def check_inputs(
-        self, prompt, strength, callback_steps, negative_prompt=None, prompt_embeds=None, negative_prompt_embeds=None
+        self, prompt, strength, num_inference_steps, first_inference_step, callback_steps, negative_prompt=None, prompt_embeds=None, negative_prompt_embeds=None
     ):
         if strength < 0 or strength > 1:
             raise ValueError(f"The value of strength should in [0.0, 1.0] but is {strength}")
-
+        if num_inference_steps is None:
+            raise ValueError("`num_inference_steps` cannot be None.")
+        elif not isinstance(num_inference_steps, int) or num_inference_steps <= 0:
+            raise ValueError(
+                f"`num_inference_steps` has to be a positive integer but is {num_inference_steps} of type"
+                f" {type(num_inference_steps)}."
+            )
+        if first_inference_step is not None and (not isinstance(first_inference_step, int) or first_inference_step <= 0):
+            raise ValueError(
+                f"`first_inference_step` has to be a positive integer but is {first_inference_step} of type"
+                f" {type(first_inference_step)}."
+            )
+        if first_inference_step is not None and first_inference_step > num_inference_steps:
+            raise ValueError(
+                f"`first_inference_step` has to be smaller than `num_inference_steps` but is {first_inference_step} and"
+                f" `num_inference_steps` is {num_inference_steps}."
+            )
         if (callback_steps is None) or (
             callback_steps is not None and (not isinstance(callback_steps, int) or callback_steps <= 0)
         ):
@@ -494,16 +510,19 @@ def check_inputs(
                     f" {negative_prompt_embeds.shape}."
                 )
 
-    def get_timesteps(self, num_inference_steps, strength, device):
+    def get_timesteps(self, num_inference_steps, first_inference_step, strength, device):
         # get the original timestep using init_timestep
-        init_timestep = min(int(num_inference_steps * strength), num_inference_steps)
+        if first_inference_step is None:
+            init_timestep = min(int(num_inference_steps * strength), num_inference_steps)
+        else:
+            init_timestep = first_inference_step - 1
 
         t_start = max(num_inference_steps - init_timestep, 0)
         timesteps = self.scheduler.timesteps[t_start * self.scheduler.order :]
 
         return timesteps, num_inference_steps - t_start
 
-    def prepare_latents(self, image, timestep, batch_size, num_images_per_prompt, dtype, device, generator=None):
+    def prepare_latents(self, image, timestep, batch_size, num_images_per_prompt, dtype, device, generator=None, add_noise=True):
         if not isinstance(image, (torch.Tensor, PIL.Image.Image, list)):
             raise ValueError(
                 f"`image` has to be of type `torch.Tensor`, `PIL.Image.Image` or list but is {type(image)}"
@@ -555,12 +574,12 @@ def prepare_latents(self, image, timestep, batch_size, num_images_per_prompt, dt
             )
         else:
             init_latents = torch.cat([init_latents], dim=0)
+        if add_noise:
+            shape = init_latents.shape
+            noise = randn_tensor(shape, generator=generator, device=device, dtype=dtype)
 
-        shape = init_latents.shape
-        noise = randn_tensor(shape, generator=generator, device=device, dtype=dtype)
-
-        # get latents
-        init_latents = self.scheduler.add_noise(init_latents, noise, timestep)
+            # get latents
+            init_latents = self.scheduler.add_noise(init_latents, noise, timestep)
         latents = init_latents
 
         return latents
@@ -619,6 +638,8 @@ def __call__(
         ] = None,
         strength: float = 0.3,
         num_inference_steps: int = 50,
+        max_inference_steps: Optional[int] = None,
+        first_inference_step: Optional[int] = None,
         guidance_scale: float = 5.0,
         negative_prompt: Optional[Union[str, List[str]]] = None,
         num_images_per_prompt: Optional[int] = 1,
@@ -650,7 +671,7 @@ def __call__(
                 instead.
             image (`torch.FloatTensor` or `PIL.Image.Image` or `np.ndarray` or `List[torch.FloatTensor]` or `List[PIL.Image.Image]` or `List[np.ndarray]`):
                 The image(s) to modify with the pipeline.
-            strength (`float`, *optional*, defaults to 0.8):
+            strength (`float`, *optional*, defaults to 0.3):
                 Conceptually, indicates how much to transform the reference `image`. Must be between 0 and 1. `image`
                 will be used as a starting point, adding more noise to it the larger the `strength`. The number of
                 denoising steps depends on the amount of noise initially added. When `strength` is 1, added noise will
@@ -659,6 +680,12 @@ def __call__(
             num_inference_steps (`int`, *optional*, defaults to 50):
                 The number of denoising steps. More denoising steps usually lead to a higher quality image at the
                 expense of slower inference.
+            max_inference_steps (`int`, *optional*):
+                Instead of completing the backwards pass entirely, stop and return the output after this many steps.
+                Can be useful with `output_type="latent"` and an img2img pipeline, possibly with better fine detail.
+            first_inference_step (`int`, *optional*):
+                Ignore the first steps of the denoising process, and start from here.
+                Useful if the input is a latent tensor that still has residual noise, eg. using `max_inference_steps`.
             guidance_scale (`float`, *optional*, defaults to 7.5):
                 Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598).
                 `guidance_scale` is defined as `w` of equation 2. of [Imagen
@@ -737,7 +764,7 @@ def __call__(
             "not-safe-for-work" (nsfw) content, according to the `safety_checker`.
         """
         # 1. Check inputs. Raise error if not correct
-        self.check_inputs(prompt, strength, callback_steps, negative_prompt, prompt_embeds, negative_prompt_embeds)
+        self.check_inputs(prompt, strength, num_inference_steps, first_inference_step, callback_steps, negative_prompt, prompt_embeds, negative_prompt_embeds)
 
         # 2. Define call parameters
         if prompt is not None and isinstance(prompt, str):
@@ -781,12 +808,14 @@ def __call__(
 
         # 5. Prepare timesteps
         self.scheduler.set_timesteps(num_inference_steps, device=device)
-        timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, strength, device)
+        timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, first_inference_step, strength, device)
         latent_timestep = timesteps[:1].repeat(batch_size * num_images_per_prompt)
-
+        add_noise = True
+        if first_inference_step is not None:
+            add_noise = False
         # 6. Prepare latent variables
         latents = self.prepare_latents(
-            image, latent_timestep, batch_size, num_images_per_prompt, prompt_embeds.dtype, device, generator
+            image, latent_timestep, batch_size, num_images_per_prompt, prompt_embeds.dtype, device, generator, add_noise
         )
         # 7. Prepare extra step kwargs.
         extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)
@@ -855,6 +884,9 @@ def __call__(
                     progress_bar.update()
                     if callback is not None and i % callback_steps == 0:
                         callback(i, t, latents)
+                if max_inference_steps is not None and i >= max_inference_steps:
+                    logger.debug(f'Breaking inference loop at step {i} as we have reached max_inference_steps={max_inference_steps}')
+                    break
 
         # make sure the VAE is in float32 mode, as it overflows in float16
         self.vae.to(dtype=torch.float32)


Code review comments:
diff_hunk: @@ -579,6 +580,9 @@ def __call__(
             num_inference_steps (`int`, *optional*, defaults to 50):
                 The number of denoising steps. More denoising steps usually lead to a higher quality image at the
                 expense of slower inference.
+            max_inference_steps (`int`, *optional*):
+                Instead of completing the backwards pass entirely, stop and return the output after this many steps.
path: src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
text: @user1:
Let's maybe use an alterative term for "backwards pass" so that people don't confuse this with backdrop. 

@user1:
I would also include the example you gave me here so that people get the idea immediately: https://github.com/huggingface/diffusers/issues/4003#issue-1795176432. 

WDYT? 

@author:
I hesitated because I wasn't sure how busy these can be. but I agree.

diff_hunk: @@ -781,12 +808,14 @@ def __call__(
 
         # 5. Prepare timesteps
         self.scheduler.set_timesteps(num_inference_steps, device=device)
-        timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, strength, device)
+        timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, first_inference_step, strength, device)
path: src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
text: @user1:
```suggestion
        if denoising_start > 0.0:
            logger.info(f"`strength`: {strength} is overwritten with `denoising_start`: {denoising_start}.")
            strength = denoising_start
            
        timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, strength, device)
```

@author:
actually, inside get_timesteps, the strength value is ignored, when the pipeline is starting on a particular step.

diff_hunk: @@ -781,12 +808,14 @@ def __call__(
 
         # 5. Prepare timesteps
         self.scheduler.set_timesteps(num_inference_steps, device=device)
-        timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, strength, device)
+        timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, first_inference_step, strength, device)
         latent_timestep = timesteps[:1].repeat(batch_size * num_images_per_prompt)
-
+        add_noise = True
+        if first_inference_step is not None:
path: src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
text: @user1:
```suggestion
        if denoising_start > 0.0
```

diff_hunk: @@ -737,7 +764,7 @@ def __call__(
             "not-safe-for-work" (nsfw) content, according to the `safety_checker`.
         """
         # 1. Check inputs. Raise error if not correct
-        self.check_inputs(prompt, strength, callback_steps, negative_prompt, prompt_embeds, negative_prompt_embeds)
+        self.check_inputs(prompt, strength, num_inference_steps, first_inference_step, callback_steps, negative_prompt, prompt_embeds, negative_prompt_embeds)
path: src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
text: @user1:
```suggestion
        self.check_inputs(prompt, strength, num_inference_steps, first_inference_step, callback_steps, negative_prompt, prompt_embeds, negative_prompt_embeds)
```

I think it would be nicer if we could use `denoising_end` here

diff_hunk: @@ -781,12 +808,14 @@ def __call__(
 
         # 5. Prepare timesteps
         self.scheduler.set_timesteps(num_inference_steps, device=device)
-        timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, strength, device)
+        timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, first_inference_step, strength, device)
         latent_timestep = timesteps[:1].repeat(batch_size * num_images_per_prompt)
-
+        add_noise = True
+        if first_inference_step is not None:
+            add_noise = False
path: src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
text: @user1:
```suggestion
        add_noise = True
        if denoising_start > 0.0:
            add_noise = False
```

I like this mechanism for now! Nice


@author:
I have altered this to take an explicitly-defined parameter, so that noise may optionally be added for more creative/variable outputs while retaining the ability to properly align timesteps from a previous instance.

diff_hunk: @@ -545,6 +545,7 @@ def __call__(
         height: Optional[int] = None,
         width: Optional[int] = None,
         num_inference_steps: int = 50,
+        max_inference_steps: Optional[int] = None,
path: src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
text: @user1:
```suggestion
        denoising_end: Optional[float] = None,
```

I think it would be a bit nicer if we could represent numbers as % here similar to how we do it for controlnet: https://huggingface.co/docs/diffusers/api/pipelines/controlnet#diffusers.StableDiffusionControlNetPipeline.__call__.control_guidance_start

Also, if a user decides to go with "partial support" / "eDiffi" approach for SD-XL then it's arguably nicer to only once set `denoising_end=0.6` and control everything just as before with `num_inference_steps`

@author:
```python
    def timesteps_from_strength(
        self, strength: float, num_inference_steps: int
    ):
        """Retrieve values for `final_inference_step` and `begin_inference_step` from `strength`, `num_inference_steps`

        Args:
            strength (float): A traditional img2img strength between 0.0 and 1.0, with higher values resulting in greater
                                influence from the img2img model and lower values, more influence from the base model.
            num_inference_steps (int): The total number of inference steps to be taken.
        """
        # We need to invert the percentage. A strength of 0.0 should result in 100% of the inference steps.
        inverse_strength = 1.0 - strength
        final_inference_step = int(num_inference_steps * inverse_strength)
        begin_inference_step = final_inference_step
        return final_inference_step, begin_inference_step
```

here is a middle-ground i have struck, to allow for this rounding error to be handled in advance.

diff_hunk: @@ -619,6 +638,8 @@ def __call__(
         ] = None,
         strength: float = 0.3,
         num_inference_steps: int = 50,
+        max_inference_steps: Optional[int] = None,
+        first_inference_step: Optional[int] = None,
path: src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
text: @user1:
```suggestion
        denoising_start: Optional[float] = None,
```

diff_hunk: @@ -619,6 +638,8 @@ def __call__(
         ] = None,
         strength: float = 0.3,
         num_inference_steps: int = 50,
+        max_inference_steps: Optional[int] = None,
path: src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
text: @user1:
```suggestion
        denoising_end: Optional[float] = None,
```

Same as: https://github.com/huggingface/diffusers/pull/4015/files#r1257982804 => can we switch to % here maybe? 

@author:
so if you use percentages the accuracy of the calculation and converting to integer means you might misalign timesteps, which results in artifacting

@user2:
Fair point! 

@author:
i have an idea i will implement a helper method to return a pair of values from two inputs.

@user1:
Hmm can you explain how you might accidentally misalign? 

@user1:
Actually I think this can't really happen if we always round, e.g.:

```py
max_inference_steps = int(round(denoising_end * num_inference_steps))
first_inference_steps = int(round(denoising_start * num_inference_Steps))
```

`denoising_end` should always be equal to `1 - denoising_start` and in this case we will always have:

```py
num_inference_steps = max_inference_steps + first_inference_steps
```

no? 

@author:
it happens when you have precise values for `strength` that result in remainders in the `int()` casting that become discarded. i have tested it:
![1689000316 7576849ca8f41e6b56b1ff29f093680e55fbc6](https://github.com/huggingface/diffusers/assets/59658056/727c7404-8103-4b3c-866a-91af6d57791a)

In this test, `num_inference_steps` is set to `27` and `strength` is set to `0.73` - the calculation says that the model should run for about `19.71` steps, but since we can't run a fractional step, this would be truncated by `int()` to `19`, but the value for `begin_inference_step` is `7.27` which truncates to `7`.

This results in a misalignment of one step, which is, apparently, enough to disrupt the output altogether.

We can use round(), but instead, I have opted for what I've considered a more robust approach.

Furthermore, there are some prompts/seeds/subject material where these artifacts prove useful in making the output more variable and creative. Having the option to set those and override our best judgement seems like one of the things that makes Diffusers great, and I wanted to retain that.




@user3:
+1 on @user1 suggestion on using a percentage 

It seems to me we will just have to set  `denoising_end = denoising_start`, and default it to be `(1- strength)`? 

then,
`final_inference_steps = int(denoising_end * num_inference_steps)`
`begin_inference_steps = int(denoising_start * num_inference_steps)`

so in the example `num_inference_step = 27`, `strength= 0.73`,  we will have `final_inference_steps = begin_inference_steps = 7`, i.e  run `7` steps in base model and `20` in refiner - is this what's intended? 


@author:
ive tested this, and while it seems like it should work, it has really broken the results for me, overall. i am digging into it.