Repository: huggingface/diffusers
Pull request: flux controlnet control_guidance_start and control_guidance_end implement
# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9555 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md)?
- [x] Did you read our [philosophy doc](https://github.com/huggingface/diffusers/blob/main/PHILOSOPHY.md) (important for complex PRs)?
- [ ] Was this discussed/approved via a GitHub issue or the [forum](https://discuss.huggingface.co/c/discussion-related-to-httpsgithubcomhuggingfacediffusers/63)? Please add a link to it if that's the case.
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/diffusers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/diffusers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @.

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Core library:

- Schedulers: @yiyixuxu
- Pipelines and pipeline callbacks: @yiyixuxu and @asomoza
- Training examples: @sayakpaul
- Docs: @stevhliu and @sayakpaul
- JAX and MPS: @pcuenca
- Audio: @sanchit-gandhi
- General functionalities: @sayakpaul @yiyixuxu @DN6

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc
- PEFT: @sayakpaul @BenjaminBossan

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- transformers: [different repo](https://github.com/huggingface/transformers)
- safetensors: [different repo](https://github.com/huggingface/safetensors)

-->


Problem statement:
[Flux Controlnet] Add control_guidance_start and control_guidance_end
It'd be nice to have `control_guidance_start` and `control_guidance_start` parameters added to flux Controlnet and Controlnet Inpainting pipelines.

I'm currently making experiments with Flux Controlnet Inpainting but the results are poor even with a `controlnet_conditioning_scale` set to 0.6. 

I have to set `controlnet_conditioning_scale` to 0.4 to have non broken results.

Maybe giving more control with the guidance start and end would help reach better results ?

Hints from issue discussion:
cc @asomoza 
Flux doesn't have the behavior of the the other models because we have the distilled version of the model, so this is just my speculation but, for the other inpainting/tile controlnets, you must have the `control_guidance_end` to `1.0` or you'll get a different image at the end which defeats the purpose of a pure inpainting controlnet.

Of course you can still play with the end result if you want a somewhat similar image, but that also can be achieved with plain img2img or you can do a mix of techniques to achieve a good results so having `control_guidance_start` and `control_guidance_start` would be a nice thing to have.

I'm not sure it'd help Flux to be good at inpainting though, so better if you keep you expectations low about this. Maybe you can test it with ComfyUI if it has it and post the results?
In my case I'm not using inpainting conventionnaly to inpaint small areas and add or modify elements. (Like the dog on the bench example)

I'm using it to modify 90% of the image, and preserve stuff like like outdoor view from an interior photo. 
And then I add ControlNets to keep the structure of the room intact with Depth or MLSD etc. and transform the style of the image. (Example in this video: https://www.youtube.com/watch?v=3E5V27jLHxU)

I saw a picture of a ComfyUI workflow on Twitter showing start_percent: 0.000 and end_percent: 0.3000 which I assume is the equivalent to `control_guidance_start` and `end` ? And I see the result is really good (except it doesn't use inpainting in the example to prevent modifying the window). 

![image](https://github.com/user-attachments/assets/c8a043d6-ef0d-49c4-84ed-e9bd3664d3d0)

That's what led me to think those options might be helpful in my case..

> start_percent: 0.000 and end_percent: 0.3000 which I assume is the equivalent to control_guidance_start and end ?

you're correct, and yes, if you don't care about the final result and use the image as a guide, you can use the `control_guidance_end`.

If you look at the image though, the controlnet isn't doing that much , you can probably achieve the same result with img2img and prompting. The only similarity with the control image is that there's a room, a window and a bed, it's missing a whole window and the bookshelf, and the shapes aren't even the same for the other ones.

When you say 'preserve the outdoor view' of the scene, you mean 100%? If you lower the guidance end that won't be the result though.

Just so you know, I'm not against this feature, just giving my opinion. If no one else from the community does it, I can allocate some time soon and implement this feature and test it. I use this option a lot with SDXL so I can see the benefit of it.
> When you say 'preserve the outdoor view' of the scene, you mean 100%? If you lower the guidance end that won't be the result though.

Yes 100%. However the rest of the room will be modified with a strength between 0.7 and 1.

My goal is too use a high`controlnet_conditioning_scale` like between 0.6 and 0.8. But at the same time a `control_guidance_end` that ends pretty soon like 0.3 or 0.4. So that the base structure of the output respects well the different ControlNets, and the rest of the diffusion process is more "free" so that Flux can do its magic without control.

I think that's worth trying honestly, Diffusers library is much more developer friendly than ComfyUI, and with Flux being very "strict" with its parameters range when combined with ControlNet + Inpainting, I think it may help reach more consistent results.

If you need examples, here are the input, mask and output I usually get With SD 1.5 with some ControlNets:
![image](https://github.com/user-attachments/assets/4c21fc70-6f30-4422-abe9-a9cbc6d57fa5)
![image](https://github.com/user-attachments/assets/e3ea6b72-103d-494f-8c0f-619ba16ee9d9)
![image](https://github.com/user-attachments/assets/4953e807-c239-4827-9590-c229fc7f3b7f)

With Flux I'm nowhere near that result yet.
alright, I added the labels and let's see if someone from the community is willing to do this, if not, when I have the time I'll implement it.
Thanks a lot @asomoza 🙏
@asomoza I have added the control_guidance_start and end for all three flux controlnet files.

Commit message: Merge branch 'main' into feature/flux_control_guidance

Patch to review:
diff --git a/src/diffusers/pipelines/flux/pipeline_flux_controlnet.py b/src/diffusers/pipelines/flux/pipeline_flux_controlnet.py
index 6c072c482020..2c52d311287e 100644
--- a/src/diffusers/pipelines/flux/pipeline_flux_controlnet.py
+++ b/src/diffusers/pipelines/flux/pipeline_flux_controlnet.py
@@ -72,7 +72,9 @@
         >>> image = pipe(
         ...     prompt,
         ...     control_image=control_image,
-        ...     controlnet_conditioning_scale=0.6,
+        ...     control_guidance_start=0.2,
+        ...     control_guidance_end=0.8,
+        ...     controlnet_conditioning_scale=1.0,
         ...     num_inference_steps=28,
         ...     guidance_scale=3.5,
         ... ).images[0]
@@ -572,6 +574,8 @@ def __call__(
         num_inference_steps: int = 28,
         timesteps: List[int] = None,
         guidance_scale: float = 7.0,
+        control_guidance_start: Union[float, List[float]] = 0.0,
+        control_guidance_end: Union[float, List[float]] = 1.0,
         control_image: PipelineImageInput = None,
         control_mode: Optional[Union[int, List[int]]] = None,
         controlnet_conditioning_scale: Union[float, List[float]] = 1.0,
@@ -674,6 +678,17 @@ def __call__(
         height = height or self.default_sample_size * self.vae_scale_factor
         width = width or self.default_sample_size * self.vae_scale_factor
 
+        if not isinstance(control_guidance_start, list) and isinstance(control_guidance_end, list):
+            control_guidance_start = len(control_guidance_end) * [control_guidance_start]
+        elif not isinstance(control_guidance_end, list) and isinstance(control_guidance_start, list):
+            control_guidance_end = len(control_guidance_start) * [control_guidance_end]
+        elif not isinstance(control_guidance_start, list) and not isinstance(control_guidance_end, list):
+            mult = len(self.controlnet.nets) if isinstance(self.controlnet, FluxMultiControlNetModel) else 1
+            control_guidance_start, control_guidance_end = (
+                mult * [control_guidance_start],
+                mult * [control_guidance_end],
+            )
+
         # 1. Check inputs. Raise error if not correct
         self.check_inputs(
             prompt,
@@ -839,7 +854,16 @@ def __call__(
         num_warmup_steps = max(len(timesteps) - num_inference_steps * self.scheduler.order, 0)
         self._num_timesteps = len(timesteps)
 
-        # 6. Denoising loop
+        # 6. Create tensor stating which controlnets to keep
+        controlnet_keep = []
+        for i in range(len(timesteps)):
+            keeps = [
+                1.0 - float(i / len(timesteps) < s or (i + 1) / len(timesteps) > e)
+                for s, e in zip(control_guidance_start, control_guidance_end)
+            ]
+            controlnet_keep.append(keeps[0] if isinstance(self.controlnet, FluxControlNetModel) else keeps)
+
+        # 7. Denoising loop
         with self.progress_bar(total=num_inference_steps) as progress_bar:
             for i, t in enumerate(timesteps):
                 if self.interrupt:
@@ -856,12 +880,19 @@ def __call__(
                 guidance = torch.tensor([guidance_scale], device=device) if use_guidance else None
                 guidance = guidance.expand(latents.shape[0]) if guidance is not None else None
 
+                if isinstance(controlnet_keep[i], list):
+                    current_controlnet_conditioning_scale = [
+                        c * s for c, s in zip(controlnet_conditioning_scale, controlnet_keep[i])
+                    ]
+                else:
+                    current_controlnet_conditioning_scale = controlnet_conditioning_scale * controlnet_keep[i]
+
                 # controlnet
                 controlnet_block_samples, controlnet_single_block_samples = self.controlnet(
                     hidden_states=latents,
                     controlnet_cond=control_image,
                     controlnet_mode=control_mode,
-                    conditioning_scale=controlnet_conditioning_scale,
+                    conditioning_scale=current_controlnet_conditioning_scale,
                     timestep=timestep / 1000,
                     guidance=guidance,
                     pooled_projections=pooled_prompt_embeds,
diff --git a/src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py b/src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py
index deeb9e3f546a..09b4c41cbea9 100644
--- a/src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py
+++ b/src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py
@@ -69,7 +69,9 @@
         ...     prompt,
         ...     image=init_image,
         ...     control_image=control_image,
-        ...     controlnet_conditioning_scale=0.6,
+        ...     control_guidance_start=0.2,
+        ...     control_guidance_end=0.8,
+        ...     controlnet_conditioning_scale=1.0,
         ...     strength=0.7,
         ...     num_inference_steps=2,
         ...     guidance_scale=3.5,
@@ -631,6 +633,8 @@ def __call__(
         num_inference_steps: int = 28,
         timesteps: List[int] = None,
         guidance_scale: float = 7.0,
+        control_guidance_start: Union[float, List[float]] = 0.0,
+        control_guidance_end: Union[float, List[float]] = 1.0,
         control_mode: Optional[Union[int, List[int]]] = None,
         controlnet_conditioning_scale: Union[float, List[float]] = 1.0,
         num_images_per_prompt: Optional[int] = 1,
@@ -710,6 +714,17 @@ def __call__(
         height = height or self.default_sample_size * self.vae_scale_factor
         width = width or self.default_sample_size * self.vae_scale_factor
 
+        if not isinstance(control_guidance_start, list) and isinstance(control_guidance_end, list):
+            control_guidance_start = len(control_guidance_end) * [control_guidance_start]
+        elif not isinstance(control_guidance_end, list) and isinstance(control_guidance_start, list):
+            control_guidance_end = len(control_guidance_start) * [control_guidance_end]
+        elif not isinstance(control_guidance_start, list) and not isinstance(control_guidance_end, list):
+            mult = len(self.controlnet.nets) if isinstance(self.controlnet, FluxMultiControlNetModel) else 1
+            control_guidance_start, control_guidance_end = (
+                mult * [control_guidance_start],
+                mult * [control_guidance_end],
+            )
+
         self.check_inputs(
             prompt,
             prompt_2,
@@ -862,6 +877,14 @@ def __call__(
             latents,
         )
 
+        controlnet_keep = []
+        for i in range(len(timesteps)):
+            keeps = [
+                1.0 - float(i / len(timesteps) < s or (i + 1) / len(timesteps) > e)
+                for s, e in zip(control_guidance_start, control_guidance_end)
+            ]
+            controlnet_keep.append(keeps[0] if isinstance(self.controlnet, FluxControlNetModel) else keeps)
+
         num_warmup_steps = max(len(timesteps) - num_inference_steps * self.scheduler.order, 0)
         self._num_timesteps = len(timesteps)
 
@@ -877,11 +900,18 @@ def __call__(
                 )
                 guidance = guidance.expand(latents.shape[0]) if guidance is not None else None
 
+                if isinstance(controlnet_keep[i], list):
+                    current_controlnet_conditioning_scale = [
+                        c * s for c, s in zip(controlnet_conditioning_scale, controlnet_keep[i])
+                    ]
+                else:
+                    current_controlnet_conditioning_scale = controlnet_conditioning_scale * controlnet_keep[i]
+
                 controlnet_block_samples, controlnet_single_block_samples = self.controlnet(
                     hidden_states=latents,
                     controlnet_cond=control_image,
                     controlnet_mode=control_mode,
-                    conditioning_scale=controlnet_conditioning_scale,
+                    conditioning_scale=current_controlnet_conditioning_scale,
                     timestep=timestep / 1000,
                     guidance=guidance,
                     pooled_projections=pooled_prompt_embeds,
diff --git a/src/diffusers/pipelines/flux/pipeline_flux_controlnet_inpainting.py b/src/diffusers/pipelines/flux/pipeline_flux_controlnet_inpainting.py
index e763200155f6..4707bca741e3 100644
--- a/src/diffusers/pipelines/flux/pipeline_flux_controlnet_inpainting.py
+++ b/src/diffusers/pipelines/flux/pipeline_flux_controlnet_inpainting.py
@@ -71,6 +71,8 @@
         ...     image=init_image,
         ...     mask_image=mask_image,
         ...     control_image=control_image,
+        ...     control_guidance_start=0.2,
+        ...     control_guidance_end=0.8,
         ...     controlnet_conditioning_scale=0.7,
         ...     strength=0.7,
         ...     num_inference_steps=28,
@@ -737,6 +739,8 @@ def __call__(
         timesteps: List[int] = None,
         num_inference_steps: int = 28,
         guidance_scale: float = 7.0,
+        control_guidance_start: Union[float, List[float]] = 0.0,
+        control_guidance_end: Union[float, List[float]] = 1.0,
         control_mode: Optional[Union[int, List[int]]] = None,
         controlnet_conditioning_scale: Union[float, List[float]] = 1.0,
         num_images_per_prompt: Optional[int] = 1,
@@ -826,6 +830,17 @@ def __call__(
         global_height = height
         global_width = width
 
+        if not isinstance(control_guidance_start, list) and isinstance(control_guidance_end, list):
+            control_guidance_start = len(control_guidance_end) * [control_guidance_start]
+        elif not isinstance(control_guidance_end, list) and isinstance(control_guidance_start, list):
+            control_guidance_end = len(control_guidance_start) * [control_guidance_end]
+        elif not isinstance(control_guidance_start, list) and not isinstance(control_guidance_end, list):
+            mult = len(self.controlnet.nets) if isinstance(self.controlnet, FluxMultiControlNetModel) else 1
+            control_guidance_start, control_guidance_end = (
+                mult * [control_guidance_start],
+                mult * [control_guidance_end],
+            )
+
         # 1. Check inputs
         self.check_inputs(
             prompt,
@@ -1031,6 +1046,14 @@ def __call__(
             generator,
         )
 
+        controlnet_keep = []
+        for i in range(len(timesteps)):
+            keeps = [
+                1.0 - float(i / len(timesteps) < s or (i + 1) / len(timesteps) > e)
+                for s, e in zip(control_guidance_start, control_guidance_end)
+            ]
+            controlnet_keep.append(keeps[0] if isinstance(self.controlnet, FluxControlNetModel) else keeps)
+
         # 9. Denoising loop
         num_warmup_steps = max(len(timesteps) - num_inference_steps * self.scheduler.order, 0)
         self._num_timesteps = len(timesteps)
@@ -1049,11 +1072,18 @@ def __call__(
                 else:
                     guidance = None
 
+                if isinstance(controlnet_keep[i], list):
+                    current_controlnet_conditioning_scale = [
+                        c * s for c, s in zip(controlnet_conditioning_scale, controlnet_keep[i])
+                    ]
+                else:
+                    current_controlnet_conditioning_scale = controlnet_conditioning_scale * controlnet_keep[i]
+
                 controlnet_block_samples, controlnet_single_block_samples = self.controlnet(
                     hidden_states=latents,
                     controlnet_cond=control_image,
                     controlnet_mode=control_mode,
-                    conditioning_scale=controlnet_conditioning_scale,
+                    conditioning_scale=current_controlnet_conditioning_scale,
                     timestep=timestep / 1000,
                     guidance=guidance,
                     pooled_projections=pooled_prompt_embeds,


Code review comments:
diff_hunk: @@ -856,12 +880,19 @@ def __call__(
                 guidance = torch.tensor([guidance_scale], device=device) if use_guidance else None
                 guidance = guidance.expand(latents.shape[0]) if guidance is not None else None
 
+                if isinstance(controlnet_keep[i], list):
+                    current_controlnet_conditioning_scale = [
+                        c * s for c, s in zip(controlnet_conditioning_scale, controlnet_keep[i])
+                    ]
+                else:
+                    current_controlnet_conditioning_scale = controlnet_conditioning_scale * controlnet_keep[i]
path: src/diffusers/pipelines/flux/pipeline_flux_controlnet.py
text: @user1:
we should probably stay consistent with the other implementations, is there a reason why you changed this part to be different from the SD3 and SDXL ones?

@author:
yeah its a mistake, controlnet_conditioning_scale could be a list as well, will make changes