Repository: huggingface/diffusers
Pull request: Make T2I-Adapter downscale padding match the UNet
# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #5377 

This PR updates the padding behavior of the T2I-Adapter down blocks to more closely mirror the padding behavior of the UNet model.

Prior to this change, the T2I-Adapter pipelines only worked with input/output image dimensions that were multiples of the T2I-Adapter's total downscale factor (64 for SD1 and 32 for SDXL). After this change, the same pipelines can be run with input/output dimensions that are multiples of the T2I-Adapter's initial pixel unshuffling downscale factor (8 for SD1 and 16 for SDXL).

# Manual Testing

The change is covered by unit tests, but the following script can be used as a visual regression test for this feature. This script is adapted from [this comment on the original issue](https://github.com/huggingface/diffusers/issues/5377#issuecomment-1764510314).

```python
import torch
from PIL import Image
from controlnet_aux import PidiNetDetector
import numpy as np
from diffusers import T2IAdapter, StableDiffusionAdapterPipeline

in_dims = (680, 384)
#in_dims = (640, 384)

image = Image.open("dog.png")
processor = PidiNetDetector.from_pretrained("lllyasviel/Annotators")
sketch_image = processor(image).resize(in_dims).convert("L")
tensor_img = (torch.from_numpy(np.array(sketch_image).astype(np.float32) / 255.0) > 0.5).float()
sketch_image = tensor_img.numpy()
sketch_image = (sketch_image * 255).astype(np.uint8)
sketch_image = Image.fromarray(sketch_image)
sketch_image.save("sketch.png")
adapter = T2IAdapter.from_pretrained("TencentARC/t2iadapter_sketch_sd15v2", torch_dtype=torch.float16)
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", adapter=adapter, safety_checker=None, torch_dtype=torch.float16, variant="fp16"
)
pipe.to("cuda")
generator = torch.Generator().manual_seed(0)
sketch_image_out = pipe(prompt="a dog", image=sketch_image, generator=generator).images[0]
sketch_image_out.save("sketch_image_out.png")

print(f"In dims: {in_dims}, Out dims: {(sketch_image_out.width, sketch_image_out.height)}")
```

Input `dog.png`:
![image](https://github.com/huggingface/diffusers/assets/14897797/9701ed4e-c312-4589-ab3d-d9516cd50d5f)

## Before this change, `in_dims=(680, 384)`

Note that output dimensions do not match input dimensions.
`In dims: (680, 384), Out dims: (640, 384)`
![image](https://github.com/huggingface/diffusers/assets/14897797/1f5c2b04-8261-4060-a053-32ead9872375)

## Before this change, `in_dims=(640, 384)`
`In dims: (640, 384), Out dims: (640, 384)`
![image](https://github.com/huggingface/diffusers/assets/14897797/898037b1-2f6d-440b-9d86-f8ad53dab762)

## After this change, `in_dims=(680, 384)`

The output dimensions match the input dimensions, as expected. The change in the output's appearance is expected, because the latent dimension has changed, so the same random seed no longer produces the same noise.
`In dims: (680, 384), Out dims: (680, 384)`

![image](https://github.com/huggingface/diffusers/assets/14897797/b6cc7538-bced-4e47-a70f-36567595b1f7)

## After this change, `in_dims=(640, 384)`

The output image for dimensions that already 'worked' is identical to before the change.
`In dims: (640, 384), Out dims: (640, 384)`

![image](https://github.com/huggingface/diffusers/assets/14897797/2259ffc1-bea2-4c0a-a190-bdd09887c7d1)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md)?
- [x] Did you read our [philosophy doc](https://github.com/huggingface/diffusers/blob/main/PHILOSOPHY.md) (important for complex PRs)?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/diffusers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Core library:

- Schedulers: @williamberman and @patrickvonplaten
- Pipelines:  @patrickvonplaten and @sayakpaul
- Training examples: @sayakpaul and @patrickvonplaten
- Docs: @stevhliu and @yiyixuxu
- JAX and MPS: @pcuenca
- Audio: @sanchit-gandhi
- General functionalities: @patrickvonplaten and @sayakpaul

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- transformers: [different repo](https://github.com/huggingface/transformers)
- safetensors: [different repo](https://github.com/huggingface/safetensors)

-->


Problem statement:
Make T2I-Adapter downscale padding match the UNet
# Background

T2I-Adapter models produce residual maps that must match the corresponding UNet hidden state dimensions at each downscaling level. This works as expected when the T2I-Adapter's input image is evenly divisible by the T2I-Adapter's `total_downscale_factor`.

For example, a `FullAdapter` used with an SD1.5 UNet and a 1280 x 640 image produces feature maps with the following dimensions:
- `[1, 320, 80, 160]`
- `[1, 640, 40, 80]`
- `[1. 1280, 20, 40]`
- `[1. 1280, 10, 20]`

# Problem

When an input image is **not** evenly divisible by the T2I-Adapter's `total_downscale_factor`, the T2I-Adapter produces residual maps with different dimensions than the UNet model's hidden states.

For example, consider a `FullAdapter` used with an SD1.5 UNet and a 680 x 384 image. The UNet produces hidden states with the following dimensions:
- `[1, 320, 48, 85]`
- `[1, 640, 24, 43]`
- `[1, 1280, 12, 22]`
- `[1, 1280, 6, 11]`

Meanwhile, the T2I-Adapter produces residual maps with the following dimensions:
- `[1, 320, 48, 85]`
- `[1, 640, 24, 42]`
- `[1, 1280, 12, 21]`
- `[1, 1280, 6, 10]`

The UNet and T2I-Adapter dimensions do not match, resulting in a torch exception.

# Why should we solve this?

It would be nice for T2I-Adapters to work with any input latent noise dimension that is supported by the UNet. For context, I am an InvokeAI contributor and would rather not restrict output image dimensions to multiples of 64 when a T2I-Adapter is applied.

# Root cause

The differences in downscaled feature map dimensions between the UNet and T2I-Adapter are caused by differences in padding behavior between their downscaling operations.

The UNet downscales with a `torch.nn.Conv2d` layer (inside `Downsample2D`), initialized as follows:
```python
torch.nn.Conv2d(..., kernel_size=3, stride=2, padding=1)
```

The T2I-Adapter downscales with a `torch.nn.AvgPool2d` layer (also inside `Downsample2D`), initialized as follows:
```python
torch.nn.AvgPool2d(kernel_size=2, stride=2)
```

# Proposed Solution

The T2I-Adapter can be modified to match the UNet's downscaling padding behavior by using `ceil_mode=True`:
```python
torch.nn.AvgPool2d(kernel_size=2, stride=2, ceil_mode=True)
```

This change would have no impact on input images with dimensions that already work.

# Questions for the diffusers team:

1. Do you agree that this is worth fixing? If yes, I'm happy to put up a PR.
2. Are there any objections to the proposed solution?
3. Should this be fixed within `Downsample2D` or just for T2I-Adapters? I'm leaning towards just fixing it for T2I-Adapters, because `Downsample2D` is used in many places and I'm wary of unintended side effects from changing it.

# Aside: Alignment offset

(This issue is more relevant to the original model creators.)

The padding discrepancy points to an underlying alignment issue in the model that might be worth fixing.

In the current configuration, the T2I Adapter feature maps and the UNet2D feature maps are always offset by half a pixel with respect to the previous feature map. It's difficult to estimate how much impact this would actually have in the trained model, but it may be hindering the T2I Adapter's ability to strictly adhere to the conditioning image's structure. As a rough calculation, in an SD1.5 T2I-Adapter the 0.5 pixel offset in the last downscale operation corresponds to a 16-pixel offset in the original image (`0.5 pixel offset` x `downscale factor at the second-lowest resolution`). Not huge, but might have an impact.

This alignment issue could be fixed in the T2I-Adapter by downscaling with a `torch.nn.AvgPool2d` layer initialized as follows (or by downscaling with a conv layer):
```python
torch.nn.AvgPool2d(kernel_size=3, stride=2, padding=1)
```

This change would alter the characteristics of the T2I-Adapter feature maps, so would warrant re-training the T2I-Adapters. **To be clear, I'm just saying that this might be an interesting experiment.** This configuration would have a blurring effect on the feature maps and may actually worsen performance.

Hints from issue discussion:
Cc @MC-E here as well. 
@sayakpaul @yiyixuxu can we try to look into this ourselves and try to solve it? 
Sorry for getting to this a bit late. 

Excellent thread here @RyanJDick! Thanks for detailing about the problem!

From my side:

* It's definitely worth fixing! And then it would be interesting to see the impact of this fix on the generation quality of the outputs. For example, if we make the input image divisible w.r.t the current form -- is that better / worse than having the proposed fix. 
* re

> Should this be fixed within Downsample2D or just for T2I-Adapters? I'm leaning towards just fixing it for T2I-Adapters, because Downsample2D is used in many places and I'm wary of unintended side effects from changing it.

💯 agree here. 

Regarding "Aside: Alignment offset", you probably could build on top of our [SDXL T2I Adapter training script](https://github.com/huggingface/diffusers/tree/main/examples/t2i_adapter) and study the consequences. Let me know if that makes sense. 

> It's definitely worth fixing! And then it would be interesting to see the impact of this fix on the generation quality of the outputs. For example, if we make the input image divisible w.r.t the current form -- is that better / worse than having the proposed fix.

Sounds good! I'll put up a PR, including some experiment results to show how it affects generation.

> Regarding "Aside: Alignment offset", you probably could build on top of our [SDXL T2I Adapter training script](https://github.com/huggingface/diffusers/tree/main/examples/t2i_adapter) and study the consequences. Let me know if that makes sense.

Cool. I won't get to this right away, but would be interested in giving it a try at some point. (I'd also be happy to see someone else give it a shot!)
Hi @RyanJDick, I tried your input but couldn't reproduce the error on the diffusers (T2I-Adapter-1.4/1.5). Can you provide more details?
 
![sketch (1)](https://github.com/huggingface/diffusers/assets/54032224/d6bb4899-4aca-422c-ba50-3fbafd3f2861)
![sketch_image_out (13)](https://github.com/huggingface/diffusers/assets/54032224/2b7a0703-a853-4f5c-896e-2c78a50a9bde)

```bash
import torch
from PIL import Image
from controlnet_aux import PidiNetDetector
import numpy as np
from diffusers import (
    T2IAdapter,
    StableDiffusionAdapterPipeline
)
image = Image.open('dog.png')
processor = PidiNetDetector.from_pretrained('lllyasviel/Annotators')
sketch_image = processor(image).resize((680, 384)).convert('L')
tensor_img = (torch.from_numpy(np.array(sketch_image).astype(np.float32)/255.)>0.5).float()
sketch_image = tensor_img.numpy()
sketch_image = (sketch_image * 255).astype(np.uint8)
sketch_image = Image.fromarray(sketch_image)
sketch_image.save('sketch.png')
adapter = T2IAdapter.from_pretrained("TencentARC/t2iadapter_sketch_sd15v2", torch_dtype=torch.float16)
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", adapter=adapter, safety_checker=None, torch_dtype=torch.float16, variant="fp16"
)
[pipe.to](http://pipe.to/)('cuda')
generator = torch.Generator().manual_seed(0)
sketch_image_out = pipe(prompt="a dog", image=sketch_image, generator=generator).images[0]
sketch_image_out.save('sketch_image_out.png')
```

> Hi @RyanJDick, I tried your input but couldn't reproduce the error on the diffusers (T2I-Adapter-1.4/1.5). Can you provide more details?

The `StableDiffusionAdapterPipeline` has a workaround that adjusts the height/width to work: https://github.com/huggingface/diffusers/blob/cc12f3ec929d0359080cb6233c598e4f6d1f4f69/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py#L558-L583

So, this is only a problem when calling the model directly. I'll include a unit test that triggers this case in my PR. 
Okay:) thanks for your contribution!

Commit message: Update the T2I-Adapter down blocks to better match the padding behavior of the UNet.

Patch to review:
diff --git a/src/diffusers/models/adapter.py b/src/diffusers/models/adapter.py
index 64d64d07bf77..6e1918334f13 100644
--- a/src/diffusers/models/adapter.py
+++ b/src/diffusers/models/adapter.py
@@ -51,24 +51,26 @@ def __init__(self, adapters: List["T2IAdapter"]):
         if len(adapters) == 1:
             raise ValueError("For a single adapter, please use the `T2IAdapter` class instead of `MultiAdapter`")
 
-        # The outputs from each adapter are added together with a weight
-        # This means that the change in dimenstions from downsampling must
-        # be the same for all adapters. Inductively, it also means the total
-        # downscale factor must also be the same for all adapters.
-
-        first_adapter_total_downscale_factor = adapters[0].total_downscale_factor
-
+        # The outputs from each adapter are added together with a weight.
+        # This means that the change in dimensions from downsampling must
+        # be the same for all adapters. Inductively, it also means the
+        # downscale_factor and total_downscale_factor must be the same for all
+        # adapters.
         for idx in range(1, len(adapters)):
-            adapter_idx_total_downscale_factor = adapters[idx].total_downscale_factor
-
-            if adapter_idx_total_downscale_factor != first_adapter_total_downscale_factor:
+            if (
+                adapters[idx].total_downscale_factor != adapters[0].total_downscale_factor
+                or adapters[idx].downscale_factor != adapters[0].downscale_factor
+            ):
                 raise ValueError(
-                    f"Expecting all adapters to have the same total_downscale_factor, "
-                    f"but got adapters[0].total_downscale_factor={first_adapter_total_downscale_factor} and "
-                    f"adapter[`{idx}`]={adapter_idx_total_downscale_factor}"
+                    f"Expecting all adapters to have the same downscaling behavior, but got:\n"
+                    f"adapters[0].total_downscale_factor={adapters[0].total_downscale_factor}\n"
+                    f"adapters[0].downscale_factor={adapters[0].downscale_factor}\n"
+                    f"adapter[`{idx}`].total_downscale_factor={adapters[idx].total_downscale_factor}\n"
+                    f"adapter[`{idx}`].downscale_factor={adapters[idx].downscale_factor}"
                 )
 
         self.total_downscale_factor = adapters[0].total_downscale_factor
+        self.downscale_factor = adapters[0].downscale_factor
 
     def forward(self, xs: torch.Tensor, adapter_weights: Optional[List[float]] = None) -> List[torch.Tensor]:
         r"""
@@ -274,6 +276,14 @@ def forward(self, x: torch.Tensor) -> List[torch.Tensor]:
     def total_downscale_factor(self):
         return self.adapter.total_downscale_factor
 
+    @property
+    def downscale_factor(self):
+        """The downscale factor applied in the T2I-Adapter's initial pixel
+        unshuffle operation. If an input image's dimensions are not evenly
+        divisible by the downscale_factor then an exception will be raised.
+        """
+        return self.adapter.unshuffle.downscale_factor
+
 
 # full adapter
 
@@ -399,7 +409,7 @@ def __init__(self, in_channels: int, out_channels: int, num_res_blocks: int, dow
 
         self.downsample = None
         if down:
-            self.downsample = Downsample2D(in_channels)
+            self.downsample = nn.AvgPool2d(kernel_size=2, stride=2, ceil_mode=True)
 
         self.in_conv = None
         if in_channels != out_channels:
@@ -526,7 +536,7 @@ def __init__(self, in_channels: int, out_channels: int, num_res_blocks: int, dow
 
         self.downsample = None
         if down:
-            self.downsample = Downsample2D(in_channels)
+            self.downsample = nn.AvgPool2d(kernel_size=2, stride=2, ceil_mode=True)
 
         self.in_conv = nn.Conv2d(in_channels, mid_channels, kernel_size=1)
         self.resnets = nn.Sequential(*[LightAdapterResnetBlock(mid_channels) for _ in range(num_res_blocks)])
diff --git a/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py b/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py
index dca9e5fc3de2..d5916e649ee3 100644
--- a/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py
+++ b/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_adapter.py
@@ -568,8 +568,8 @@ def _default_height_width(self, height, width, image):
             elif isinstance(image, torch.Tensor):
                 height = image.shape[-2]
 
-            # round down to nearest multiple of `self.adapter.total_downscale_factor`
-            height = (height // self.adapter.total_downscale_factor) * self.adapter.total_downscale_factor
+            # round down to nearest multiple of `self.adapter.downscale_factor`
+            height = (height // self.adapter.downscale_factor) * self.adapter.downscale_factor
 
         if width is None:
             if isinstance(image, PIL.Image.Image):
@@ -577,8 +577,8 @@ def _default_height_width(self, height, width, image):
             elif isinstance(image, torch.Tensor):
                 width = image.shape[-1]
 
-            # round down to nearest multiple of `self.adapter.total_downscale_factor`
-            width = (width // self.adapter.total_downscale_factor) * self.adapter.total_downscale_factor
+            # round down to nearest multiple of `self.adapter.downscale_factor`
+            width = (width // self.adapter.downscale_factor) * self.adapter.downscale_factor
 
         return height, width
 
diff --git a/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py b/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py
index d4272696c23b..585dd0339a1c 100644
--- a/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py
+++ b/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py
@@ -600,8 +600,8 @@ def _default_height_width(self, height, width, image):
             elif isinstance(image, torch.Tensor):
                 height = image.shape[-2]
 
-            # round down to nearest multiple of `self.adapter.total_downscale_factor`
-            height = (height // self.adapter.total_downscale_factor) * self.adapter.total_downscale_factor
+            # round down to nearest multiple of `self.adapter.downscale_factor`
+            height = (height // self.adapter.downscale_factor) * self.adapter.downscale_factor
 
         if width is None:
             if isinstance(image, PIL.Image.Image):
@@ -609,8 +609,8 @@ def _default_height_width(self, height, width, image):
             elif isinstance(image, torch.Tensor):
                 width = image.shape[-1]
 
-            # round down to nearest multiple of `self.adapter.total_downscale_factor`
-            width = (width // self.adapter.total_downscale_factor) * self.adapter.total_downscale_factor
+            # round down to nearest multiple of `self.adapter.downscale_factor`
+            width = (width // self.adapter.downscale_factor) * self.adapter.downscale_factor
 
         return height, width
 
diff --git a/tests/pipelines/stable_diffusion/test_stable_diffusion_adapter.py b/tests/pipelines/stable_diffusion/test_stable_diffusion_adapter.py
index d48175a7789b..640ed8aaab00 100644
--- a/tests/pipelines/stable_diffusion/test_stable_diffusion_adapter.py
+++ b/tests/pipelines/stable_diffusion/test_stable_diffusion_adapter.py
@@ -18,6 +18,7 @@
 import unittest
 
 import numpy as np
+from parameterized import parameterized
 import torch
 from transformers import CLIPTextConfig, CLIPTextModel, CLIPTokenizer
 
@@ -57,23 +58,24 @@ class AdapterTests:
     def get_dummy_components(self, adapter_type):
         torch.manual_seed(0)
         unet = UNet2DConditionModel(
-            block_out_channels=(32, 64),
+            block_out_channels=(32, 32, 32, 64),
             layers_per_block=2,
             sample_size=32,
             in_channels=4,
             out_channels=4,
-            down_block_types=("CrossAttnDownBlock2D", "DownBlock2D"),
-            up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
+            # Test with all 4 down blocks to ensure that the T2I-Adapter downscaling gets fully exercised in the tests.
+            down_block_types=("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D"),
+            up_block_types=("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D"),
             cross_attention_dim=32,
         )
         scheduler = PNDMScheduler(skip_prk_steps=True)
         torch.manual_seed(0)
         vae = AutoencoderKL(
-            block_out_channels=[32, 64],
+            block_out_channels=[32, 32, 32, 64],
             in_channels=3,
             out_channels=3,
-            down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
-            up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"],
+            down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D", "DownEncoderBlock2D", "DownEncoderBlock2D"],
+            up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D", "UpDecoderBlock2D", "UpDecoderBlock2D"],
             latent_channels=4,
         )
         torch.manual_seed(0)
@@ -96,9 +98,9 @@ def get_dummy_components(self, adapter_type):
         if adapter_type == "full_adapter" or adapter_type == "light_adapter":
             adapter = T2IAdapter(
                 in_channels=3,
-                channels=[32, 64],
+                channels=[32, 32, 32, 64],
                 num_res_blocks=2,
-                downscale_factor=2,
+                downscale_factor=8,
                 adapter_type=adapter_type,
             )
         elif adapter_type == "multi_adapter":
@@ -106,16 +108,16 @@ def get_dummy_components(self, adapter_type):
                 [
                     T2IAdapter(
                         in_channels=3,
-                        channels=[32, 64],
+                        channels=[32, 32, 32, 64],
                         num_res_blocks=2,
-                        downscale_factor=2,
+                        downscale_factor=8,
                         adapter_type="full_adapter",
                     ),
                     T2IAdapter(
                         in_channels=3,
-                        channels=[32, 64],
+                        channels=[32, 32, 32, 64],
                         num_res_blocks=2,
-                        downscale_factor=2,
+                        downscale_factor=8,
                         adapter_type="full_adapter",
                     ),
                 ]
@@ -137,11 +139,13 @@ def get_dummy_components(self, adapter_type):
         }
         return components
 
-    def get_dummy_inputs(self, device, seed=0, num_images=1):
+    def get_dummy_inputs(self, device, seed=0, height=128, width=128, num_images=1):
         if num_images == 1:
-            image = floats_tensor((1, 3, 64, 64), rng=random.Random(seed)).to(device)
+            image = floats_tensor((1, 3, height, width), rng=random.Random(seed)).to(device)
         else:
-            image = [floats_tensor((1, 3, 64, 64), rng=random.Random(seed)).to(device) for _ in range(num_images)]
+            image = [
+                floats_tensor((1, 3, height, width), rng=random.Random(seed)).to(device) for _ in range(num_images)
+            ]
 
         if str(device).startswith("mps"):
             generator = torch.manual_seed(seed)
@@ -171,6 +175,36 @@ def test_inference_batch_single_identical(self):
         self._test_inference_batch_single_identical(expected_max_diff=2e-3)
 
 
+    def test_image_dimensions(self):
+        """Test that the T2I-Adapter pipeline supports any input dimension that
+        is divisible by the adapter's `downscale_factor`. This test was added in
+        response to an issue where the T2I Adapter's downscaling padding
+        behavior did not match the UNet's behavior.
+        """
+        components = self.get_dummy_components()
+        sd_pipe = StableDiffusionAdapterPipeline(**components)
+        sd_pipe = sd_pipe.to(torch_device)
+        sd_pipe.set_progress_bar_config(disable=None)
+
+        # Populate a list of test_dims to test odd dimensions at every T2I down
+        # block.
+        test_dims = []
+        base_dim = 4
+        factor = components["adapter"].downscale_factor
+        total_downscale_factor = components["adapter"].total_downscale_factor
+        while factor <= total_downscale_factor:
+            # Add a test dimension such that the internal T2I feature map will
+            # have an odd shape after downscaling by `factor`.
+            test_dims.append((base_dim * total_downscale_factor // factor + 1) * factor)
+            factor *= 2
+
+        for dim in test_dims:
+            inputs = self.get_dummy_inputs(torch_device, height=dim, width=dim)
+            image = sd_pipe(**inputs).images
+
+            assert image.shape == (1, dim, dim, 3)
+
+
 class StableDiffusionFullAdapterPipelineFastTests(AdapterTests, PipelineTesterMixin, unittest.TestCase):
     def get_dummy_components(self):
         return super().get_dummy_components("full_adapter")
@@ -186,8 +220,10 @@ def test_stable_diffusion_adapter_default_case(self):
         image = sd_pipe(**inputs).images
         image_slice = image[0, -3:, -3:, -1]
 
-        assert image.shape == (1, 64, 64, 3)
-        expected_slice = np.array([0.4858, 0.5500, 0.4278, 0.4669, 0.6184, 0.4322, 0.5010, 0.5033, 0.4746])
+        assert image.shape == (1, 128, 128, 3)
+        expected_slice = np.array(
+            [0.27978146, 0.36439905, 0.3206715, 0.29253614, 0.36390454, 0.3165658, 0.4384598, 0.43083128, 0.38120443]
+        )
         assert np.abs(image_slice.flatten() - expected_slice).max() < 5e-3
 
 
@@ -206,8 +242,10 @@ def test_stable_diffusion_adapter_default_case(self):
         image = sd_pipe(**inputs).images
         image_slice = image[0, -3:, -3:, -1]
 
-        assert image.shape == (1, 64, 64, 3)
-        expected_slice = np.array([0.4965, 0.5548, 0.4330, 0.4771, 0.6226, 0.4382, 0.5037, 0.5071, 0.4782])
+        assert image.shape == (1, 128, 128, 3)
+        expected_slice = np.array(
+            [0.27612323, 0.3632322, 0.31936637, 0.2896598, 0.3640582, 0.31550938, 0.438073, 0.43133593, 0.38108835]
+        )
         assert np.abs(image_slice.flatten() - expected_slice).max() < 5e-3
 
 
@@ -215,8 +253,8 @@ class StableDiffusionMultiAdapterPipelineFastTests(AdapterTests, PipelineTesterM
     def get_dummy_components(self):
         return super().get_dummy_components("multi_adapter")
 
-    def get_dummy_inputs(self, device, seed=0):
-        inputs = super().get_dummy_inputs(device, seed, num_images=2)
+    def get_dummy_inputs(self, device, height=128, width=128, seed=0):
+        inputs = super().get_dummy_inputs(device, seed, height=height, width=width, num_images=2)
         inputs["adapter_conditioning_scale"] = [0.5, 0.5]
         return inputs
 
@@ -231,8 +269,10 @@ def test_stable_diffusion_adapter_default_case(self):
         image = sd_pipe(**inputs).images
         image_slice = image[0, -3:, -3:, -1]
 
-        assert image.shape == (1, 64, 64, 3)
-        expected_slice = np.array([0.4902, 0.5539, 0.4317, 0.4682, 0.6190, 0.4351, 0.5018, 0.5046, 0.4772])
+        assert image.shape == (1, 128, 128, 3)
+        expected_slice = np.array(
+            [0.27745497, 0.36333847, 0.32152444, 0.29158655, 0.3639205, 0.31645983, 0.4382596, 0.43064785, 0.3810274]
+        )
         assert np.abs(image_slice.flatten() - expected_slice).max() < 5e-3
 
     def test_inference_batch_consistent(
diff --git a/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py b/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py
index 92c22ca2c34c..8b818599fee4 100644
--- a/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py
+++ b/tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl_adapter.py
@@ -48,19 +48,20 @@ class StableDiffusionXLAdapterPipelineFastTests(PipelineTesterMixin, unittest.Te
     def get_dummy_components(self, adapter_type="full_adapter_xl"):
         torch.manual_seed(0)
         unet = UNet2DConditionModel(
-            block_out_channels=(32, 64),
+            block_out_channels=(32, 32, 64),
             layers_per_block=2,
             sample_size=32,
             in_channels=4,
             out_channels=4,
-            down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
-            up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
+            # Test with all 3 down blocks to ensure that the T2I-Adapter downscaling gets fully exercised in the tests.
+            down_block_types=("DownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D"),
+            up_block_types=("CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "UpBlock2D"),
             # SD2-specific config below
-            attention_head_dim=(2, 4),
+            attention_head_dim=2,
             use_linear_projection=True,
             addition_embed_type="text_time",
             addition_time_embed_dim=8,
-            transformer_layers_per_block=(1, 2),
+            transformer_layers_per_block=1,
             projection_class_embeddings_input_dim=80,  # 6 * 8 + 32
             cross_attention_dim=64,
         )
@@ -73,11 +74,11 @@ def get_dummy_components(self, adapter_type="full_adapter_xl"):
         )
         torch.manual_seed(0)
         vae = AutoencoderKL(
-            block_out_channels=[32, 64],
+            block_out_channels=[32, 32, 32, 64],
             in_channels=3,
             out_channels=3,
-            down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"],
-            up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"],
+            down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D", "DownEncoderBlock2D", "DownEncoderBlock2D"],
+            up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D", "UpDecoderBlock2D", "UpDecoderBlock2D"],
             latent_channels=4,
             sample_size=128,
         )
@@ -104,9 +105,9 @@ def get_dummy_components(self, adapter_type="full_adapter_xl"):
         if adapter_type == "full_adapter_xl":
             adapter = T2IAdapter(
                 in_channels=3,
-                channels=[32, 64],
+                channels=[32, 32, 64],
                 num_res_blocks=2,
-                downscale_factor=4,
+                downscale_factor=16,
                 adapter_type=adapter_type,
             )
         elif adapter_type == "multi_adapter":
@@ -114,16 +115,16 @@ def get_dummy_components(self, adapter_type="full_adapter_xl"):
                 [
                     T2IAdapter(
                         in_channels=3,
-                        channels=[32, 64],
+                        channels=[32, 32, 64],
                         num_res_blocks=2,
-                        downscale_factor=4,
+                        downscale_factor=16,
                         adapter_type="full_adapter_xl",
                     ),
                     T2IAdapter(
                         in_channels=3,
-                        channels=[32, 64],
+                        channels=[32, 32, 64],
                         num_res_blocks=2,
-                        downscale_factor=4,
+                        downscale_factor=16,
                         adapter_type="full_adapter_xl",
                     ),
                 ]
@@ -147,11 +148,13 @@ def get_dummy_components(self, adapter_type="full_adapter_xl"):
         }
         return components
 
-    def get_dummy_inputs(self, device, seed=0, num_images=1):
+    def get_dummy_inputs(self, device, seed=0, height=64, width=64, num_images=1):
         if num_images == 1:
-            image = floats_tensor((1, 3, 64, 64), rng=random.Random(seed)).to(device)
+            image = floats_tensor((1, 3, height, width), rng=random.Random(seed)).to(device)
         else:
-            image = [floats_tensor((1, 3, 64, 64), rng=random.Random(seed)).to(device) for _ in range(num_images)]
+            image = [
+                floats_tensor((1, 3, height, width), rng=random.Random(seed)).to(device) for _ in range(num_images)
+            ]
 
         if str(device).startswith("mps"):
             generator = torch.manual_seed(seed)
@@ -180,10 +183,39 @@ def test_stable_diffusion_adapter_default_case(self):
 
         assert image.shape == (1, 64, 64, 3)
         expected_slice = np.array(
-            [0.5752919, 0.6022097, 0.4728038, 0.49861962, 0.57084894, 0.4644975, 0.5193715, 0.5133664, 0.4729858]
+            [0.54066867, 0.5574119, 0.47273698, 0.62617165, 0.576637, 0.50059223, 0.5860178, 0.5637511, 0.5143911]
         )
         assert np.abs(image_slice.flatten() - expected_slice).max() < 5e-3
 
+    def test_image_dimensions(self):
+        """Test that the T2I-Adapter pipeline supports any input dimension that
+        is divisible by the adapter's `downscale_factor`. This test was added in
+        response to an issue where the T2I Adapter's downscaling padding
+        behavior did not match the UNet's behavior.
+        """
+        components = self.get_dummy_components()
+        sd_pipe = StableDiffusionXLAdapterPipeline(**components)
+        sd_pipe = sd_pipe.to(torch_device)
+        sd_pipe.set_progress_bar_config(disable=None)
+
+        # Populate a list of test_dims to test odd dimensions at every T2I down
+        # block.
+        test_dims = []
+        base_dim = 4
+        factor = components["adapter"].downscale_factor
+        total_downscale_factor = components["adapter"].total_downscale_factor
+        while factor <= total_downscale_factor:
+            # Add a test dimension such that the internal T2I feature map will
+            # have an odd shape after downscaling by `factor`.
+            test_dims.append((base_dim * total_downscale_factor // factor + 1) * factor)
+            factor *= 2
+
+        for dim in test_dims:
+            inputs = self.get_dummy_inputs(torch_device, height=dim, width=dim)
+            image = sd_pipe(**inputs).images
+
+            assert image.shape == (1, dim, dim, 3)
+
     @parameterized.expand(["full_adapter", "full_adapter_xl", "light_adapter"])
     def test_total_downscale_factor(self, adapter_type):
         """Test that the T2IAdapter correctly reports its total_downscale_factor."""
@@ -222,8 +254,8 @@ class StableDiffusionXLMultiAdapterPipelineFastTests(
     def get_dummy_components(self):
         return super().get_dummy_components("multi_adapter")
 
-    def get_dummy_inputs(self, device, seed=0):
-        inputs = super().get_dummy_inputs(device, seed, num_images=2)
+    def get_dummy_inputs(self, device, seed=0, height=64, width=64):
+        inputs = super().get_dummy_inputs(device, seed, height, width, num_images=2)
         inputs["adapter_conditioning_scale"] = [0.5, 0.5]
         return inputs
 
@@ -240,7 +272,7 @@ def test_stable_diffusion_adapter_default_case(self):
 
         assert image.shape == (1, 64, 64, 3)
         expected_slice = np.array(
-            [0.5813032, 0.60995954, 0.47563356, 0.5056669, 0.57199144, 0.4631841, 0.5176794, 0.51252556, 0.47183886]
+            [0.54532427, 0.5617021, 0.46638602, 0.6338569, 0.5826053, 0.5019801, 0.59624064, 0.5663944, 0.5151665]
         )
         assert np.abs(image_slice.flatten() - expected_slice).max() < 5e-3
 


Code review comments:
diff_hunk: @@ -186,8 +220,10 @@ def test_stable_diffusion_adapter_default_case(self):
         image = sd_pipe(**inputs).images
         image_slice = image[0, -3:, -3:, -1]
 
-        assert image.shape == (1, 64, 64, 3)
-        expected_slice = np.array([0.4858, 0.5500, 0.4278, 0.4669, 0.6184, 0.4322, 0.5010, 0.5033, 0.4746])
+        assert image.shape == (1, 128, 128, 3)
+        expected_slice = np.array(
+            [0.27978146, 0.36439905, 0.3206715, 0.29253614, 0.36390454, 0.3165658, 0.4384598, 0.43083128, 0.38120443]
+        )
path: tests/pipelines/stable_diffusion/test_stable_diffusion_adapter.py
text: @author:
Note to reviewers:

You'll notice several `expected_slice` values changing in this PR. These values changed as a result of changes to `get_dummy_components(...)` in [6d4a060](https://github.com/huggingface/diffusers/pull/5435/commits/6d4a060a34415ec973a252944216f4fb8b9926cd). The changes to adapter.py **did not** require changes to `expected_slice`. I was careful to keep these changes in separate commits so that you can verify this, if desired.

@user1:
I still prefer to keep the default tests unchanged and instead add new tests. If possible, could we do that? Applies to all the tests below. 

@author:
Done. See my comment here: https://github.com/huggingface/diffusers/pull/5435#discussion_r1363960973

(My original comment in this thread no longer applies.)

diff_hunk: @@ -171,6 +175,36 @@ def test_inference_batch_single_identical(self):
         self._test_inference_batch_single_identical(expected_max_diff=2e-3)
 
 
+    def test_image_dimensions(self):
+        """Test that the T2I-Adapter pipeline supports any input dimension that
+        is divisible by the adapter's `downscale_factor`. This test was added in
+        response to an issue where the T2I Adapter's downscaling padding
+        behavior did not match the UNet's behavior.
+        """
+        components = self.get_dummy_components()
+        sd_pipe = StableDiffusionAdapterPipeline(**components)
+        sd_pipe = sd_pipe.to(torch_device)
+        sd_pipe.set_progress_bar_config(disable=None)
+
+        # Populate a list of test_dims to test odd dimensions at every T2I down
+        # block.
+        test_dims = []
+        base_dim = 4
+        factor = components["adapter"].downscale_factor
+        total_downscale_factor = components["adapter"].total_downscale_factor
+        while factor <= total_downscale_factor:
+            # Add a test dimension such that the internal T2I feature map will
+            # have an odd shape after downscaling by `factor`.
+            test_dims.append((base_dim * total_downscale_factor // factor + 1) * factor)
+            factor *= 2
+
+        for dim in test_dims:
+            inputs = self.get_dummy_inputs(torch_device, height=dim, width=dim)
+            image = sd_pipe(**inputs).images
+
+            assert image.shape == (1, dim, dim, 3)
path: tests/pipelines/stable_diffusion/test_stable_diffusion_adapter.py
text: @user1:
I think it might be better to leverage `parameterized` here and to be more explicit in terms of defining these dimensions. Helps us instantly know what we're testing for. 

@author:
Done in [18e2d4b](https://github.com/huggingface/diffusers/pull/5435/commits/18e2d4b6b20d0463771d1bdf2a4f3c74d8f5a78c)

diff_hunk: @@ -51,24 +51,26 @@ def __init__(self, adapters: List["T2IAdapter"]):
         if len(adapters) == 1:
             raise ValueError("For a single adapter, please use the `T2IAdapter` class instead of `MultiAdapter`")
 
-        # The outputs from each adapter are added together with a weight
-        # This means that the change in dimenstions from downsampling must
-        # be the same for all adapters. Inductively, it also means the total
-        # downscale factor must also be the same for all adapters.
-
-        first_adapter_total_downscale_factor = adapters[0].total_downscale_factor
-
+        # The outputs from each adapter are added together with a weight.
+        # This means that the change in dimensions from downsampling must
+        # be the same for all adapters. Inductively, it also means the
+        # downscale_factor and total_downscale_factor must be the same for all
+        # adapters.
         for idx in range(1, len(adapters)):
-            adapter_idx_total_downscale_factor = adapters[idx].total_downscale_factor
-
-            if adapter_idx_total_downscale_factor != first_adapter_total_downscale_factor:
+            if (
+                adapters[idx].total_downscale_factor != adapters[0].total_downscale_factor
path: src/diffusers/models/adapter.py
text: @user1:
Could we try to assign `adapters[0].total_downscale_factor` in a new variable as was before? I think that slightly improves on the readability front. 

@author:
Cleaned up in [1ca2532](https://github.com/huggingface/diffusers/pull/5435/commits/1ca2532b940f6e8f87e224325093d1ef0278065f)