Repository: huggingface/diffusers
Pull request: update StableDiffusion3Img2ImgPipeline.add image size validation
# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #10153 
Add support for height and width parameters in the __call__() method of the StableDiffusion3Img2ImgPipeline. Users can specify the output image size by providing these parameters. Additionally, implement input validation for height and width.

Referencing StableDiffusion3Pipeline, if height and width are not provided, default sizes should be calculated as:

```python
height = height or self.default_sample_size * self.vae_scale_factor
width = width or self.default_sample_size * self.vae_scale_factor
```
instead of using the dimensions of the input image by default. This prevents errors in vector calculations caused by incompatible height and width values from the input image.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md)?
- [ ] Did you read our [philosophy doc](https://github.com/huggingface/diffusers/blob/main/PHILOSOPHY.md) (important for complex PRs)?
- [&#x2714;] Was this discussed/approved via a GitHub issue or the [forum](https://discuss.huggingface.co/c/discussion-related-to-httpsgithubcomhuggingfacediffusers/63)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/diffusers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/diffusers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @.

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Core library:

- Schedulers: @yiyixuxu
- Pipelines and pipeline callbacks: @yiyixuxu and @asomoza
- Training examples: @sayakpaul
- Docs: @stevhliu and @sayakpaul
- JAX and MPS: @pcuenca
- Audio: @sanchit-gandhi
- General functionalities: @sayakpaul @yiyixuxu @DN6

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc
- PEFT: @sayakpaul @BenjaminBossan

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- transformers: [different repo](https://github.com/huggingface/transformers)
- safetensors: [different repo](https://github.com/huggingface/safetensors)

-->


Problem statement:
In the StableDiffusion3Img2ImgPipeline, the height dimension of latents is odd, which may cause a mismatch with the height dimension of noise_pred.
### Describe the bug

In the `StableDiffusion3Img2ImgPipeline`, the height dimension of latents is odd, which may cause a mismatch with the height dimension of noise_pred.

During the process of using the `StableDiffusion3Img2ImgPipeline`, I input an image of shape (1, 3, 856, 1088). After passing through the `self.prepare_latents` function in the pipeline, the resulting latents have a shape of (1, 16, 107, 136), where the height dimension is 107 (an odd number). This causes a mismatch with the height dimension of noise_pred, which is produced later in `self.transforme`r. In `self.transformer`, the height is computed as `height // patch_size`, where patch_size is presumably set to 2 by default, resulting in a final height of 106 for `noise_pred`. This dimension mismatch causes issues in the calculation within `self.scheduler.step`, preventing the computation from proceeding correctly.

### Reproduction

```python
class TextToImagePipeline:
    pipeline: StableDiffusion3Pipeline = None
    device: str = None

    def start(self):
        if torch.cuda.is_available():
            # model_path = os.getenv("MODEL_PATH", "stabilityai/stable-diffusion-3.5-large")
            model_path = settings.SD_LARGE_DIR
            api_logger.info("Loading CUDA")
            self.device = "cuda"
            self.pipeline = StableDiffusion3Pipeline.from_pretrained(
                model_path,
                torch_dtype=torch.bfloat16,
            ).to(device=self.device)
        else:
            raise Exception("No CUDA device available")
shared_pipeline = TextToImagePipeline()
def image2image(shared_pipe,
                prompt,
                prompt_3,
                negative_prompt,
                strength,
                num_inference_steps,
                guidance_scale,
                generator,
                image):
    scheduler = shared_pipe.pipeline.scheduler.from_config(shared_pipe.pipeline.scheduler.config)
    pipeline = AutoPipelineForImage2Image.from_pipe(shared_pipe.pipeline, scheduler=scheduler)
    # pipeline.enable_model_cpu_offload()
    init_image = load_image(image)
    image = pipeline(prompt=prompt,
                     prompt_3=prompt_3,
                     strength=strength,
                     negative_prompt=negative_prompt,
                     image=init_image,
                     num_inference_steps=num_inference_steps,
                     guidance_scale=guidance_scale,
                     generator=generator,
                     max_sequence_length=512
                     ).images
    api_logger.info(f"output:{image}")
    return image[0]

    - ps:
    The input image has a pixel ratio of 861x1089.
    After processing with image = self.image_processor.preprocess(image), the image becomes (1, 3, 856, 1088).

### Logs

_No response_

### System Info

Linux 24.04.1
GPU RTX5880 48G
python 3.10
pytorch 2.5.1

### Who can help?

_No response_

Hints from issue discussion:
![image](https://github.com/user-attachments/assets/42634de7-7109-499b-9efd-3907beba1597)

I eventually added a check for the evenness of the height and width dimensions of the latents. If they are odd, I adjust them to the largest even number within the range. I'm not sure if this method is advisable.
cc @sayakpaul @yiyixuxu SD3 resolutions must be factor of 16?
Based on the implementation, I think yes.

https://github.com/huggingface/diffusers/blob/6131a93b969f87d171148bd367fd9990d5a49b6b/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py#L531-L538
Thanks @sayakpaul 

@ZHJ19970917 This check is missing from `StableDiffusion3Img2ImgPipeline`, would you like to add it?
Sure, I can add the missing check.

Commit message: make style

Patch to review:
diff --git a/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_img2img.py b/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_img2img.py
index c91b4ee80eaa..7a8c602ccea5 100644
--- a/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_img2img.py
+++ b/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_img2img.py
@@ -218,6 +218,9 @@ def __init__(
         )
         self.tokenizer_max_length = self.tokenizer.model_max_length
         self.default_sample_size = self.transformer.config.sample_size
+        self.patch_size = (
+            self.transformer.config.patch_size if hasattr(self, "transformer") and self.transformer is not None else 2
+        )
 
     # Copied from diffusers.pipelines.stable_diffusion_3.pipeline_stable_diffusion_3.StableDiffusion3Pipeline._get_t5_prompt_embeds
     def _get_t5_prompt_embeds(
@@ -531,6 +534,8 @@ def check_inputs(
         prompt,
         prompt_2,
         prompt_3,
+        height,
+        width,
         strength,
         negative_prompt=None,
         negative_prompt_2=None,
@@ -542,6 +547,15 @@ def check_inputs(
         callback_on_step_end_tensor_inputs=None,
         max_sequence_length=None,
     ):
+        if (
+            height % (self.vae_scale_factor * self.patch_size) != 0
+            or width % (self.vae_scale_factor * self.patch_size) != 0
+        ):
+            raise ValueError(
+                f"`height` and `width` have to be divisible by {self.vae_scale_factor * self.patch_size} but are {height} and {width}."
+                f"You can use height {height - height % (self.vae_scale_factor * self.patch_size)} and width {width - width % (self.vae_scale_factor * self.patch_size)}."
+            )
+
         if strength < 0 or strength > 1:
             raise ValueError(f"The value of strength should in [0.0, 1.0] but is {strength}")
 
@@ -710,6 +724,8 @@ def __call__(
         prompt: Union[str, List[str]] = None,
         prompt_2: Optional[Union[str, List[str]]] = None,
         prompt_3: Optional[Union[str, List[str]]] = None,
+        height: Optional[int] = None,
+        width: Optional[int] = None,
         image: PipelineImageInput = None,
         strength: float = 0.6,
         num_inference_steps: int = 50,
@@ -824,12 +840,16 @@ def __call__(
             [`~pipelines.stable_diffusion_3.StableDiffusion3PipelineOutput`] if `return_dict` is True, otherwise a
             `tuple`. When returning a tuple, the first element is a list with the generated images.
         """
+        height = height or self.default_sample_size * self.vae_scale_factor
+        width = width or self.default_sample_size * self.vae_scale_factor
 
         # 1. Check inputs. Raise error if not correct
         self.check_inputs(
             prompt,
             prompt_2,
             prompt_3,
+            height,
+            width,
             strength,
             negative_prompt=negative_prompt,
             negative_prompt_2=negative_prompt_2,
@@ -890,7 +910,7 @@ def __call__(
             pooled_prompt_embeds = torch.cat([negative_pooled_prompt_embeds, pooled_prompt_embeds], dim=0)
 
         # 3. Preprocess image
-        image = self.image_processor.preprocess(image)
+        image = self.image_processor.preprocess(image, height, width)
 
         # 4. Prepare timesteps
         timesteps, num_inference_steps = retrieve_timesteps(self.scheduler, num_inference_steps, device, sigmas=sigmas)


Code review comments:
diff_hunk: @@ -890,7 +910,7 @@ def __call__(
             pooled_prompt_embeds = torch.cat([negative_pooled_prompt_embeds, pooled_prompt_embeds], dim=0)
 
         # 3. Preprocess image
-        image = self.image_processor.preprocess(image)
+        image = self.image_processor.preprocess(image, height, width)
path: src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_img2img.py
text: @user1:
```suggestion
        image = self.image_processor.preprocess(image, height=height, width=width)
```