Repository: huggingface/diffusers
Pull request: [Training] Better image interpolation in training scripts
# What does this PR do?

As discussed [here](https://github.com/huggingface/diffusers/issues/6397), this PR will add LANCZOS as a default interpolation mode for the image resizing in the training scripts and if the users prefers can choose BILINEAR.

I'll add this to the most popular recent ones and leave the rest to the community if they want to add them to other training scripts.

I'll do some training runs first to test if I can see the difference, but still, I already know that LANCZOS is better and that the models can pick subtle details that the human eye can't.

<!-- Remove if not applicable -->

Fixes #6397

## Who can review?

@bghira @linoytsaban @sayakpaul 

## Additional

Here's a little script if you want to test and try to see the difference:

```python
import matplotlib.pyplot as plt
from PIL import Image
from torchvision import transforms

from diffusers.utils import load_image


image = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/training/circle.png")

# Define the resize transforms for 256x256
resize_bilinear_256 = transforms.Resize((256, 256), interpolation=transforms.InterpolationMode.BILINEAR)
resize_lanczos_256 = transforms.Resize((256, 256), interpolation=transforms.InterpolationMode.LANCZOS)

# Apply the 256x256 transforms
image_bilinear_256 = resize_bilinear_256(image)
image_lanczos_256 = resize_lanczos_256(image)
image_pil_lanczos_256 = image.resize((256, 256), Image.Resampling.LANCZOS)

# Define the resize transforms for 32x32
resize_bilinear_32 = transforms.Resize((32, 32), interpolation=transforms.InterpolationMode.BILINEAR)
resize_lanczos_32 = transforms.Resize((32, 32), interpolation=transforms.InterpolationMode.LANCZOS)

# Apply the 32x32 transforms
image_bilinear_32 = resize_bilinear_32(image)
image_lanczos_32 = resize_lanczos_32(image)
image_pil_lanczos_32 = image.resize((32, 32), Image.Resampling.LANCZOS)

# Display the results
plt.figure(figsize=(16, 12))

# First row - 256x256 resizing
plt.subplot(2, 4, 1)
plt.title("Original Image")
plt.imshow(image)
plt.axis("off")

plt.subplot(2, 4, 2)
plt.title("Bilinear (256x256)")
plt.imshow(image_bilinear_256)
plt.axis("off")

plt.subplot(2, 4, 3)
plt.title("torchvision Lanczos (256x256)")
plt.imshow(image_lanczos_256)
plt.axis("off")

plt.subplot(2, 4, 4)
plt.title("PIL Lanczos (256x256)")
plt.imshow(image_pil_lanczos_256)
plt.axis("off")

# Second row - 32x32 resizing
plt.subplot(2, 4, 5)
plt.title("Original Image")
plt.imshow(image)
plt.axis("off")

plt.subplot(2, 4, 6)
plt.title("Bilinear (32x32)")
plt.imshow(image_bilinear_32)
plt.axis("off")

plt.subplot(2, 4, 7)
plt.title("torchvision Lanczos (32x32)")
plt.imshow(image_lanczos_32)
plt.axis("off")

plt.subplot(2, 4, 8)
plt.title("PIL Lanczos (32x32)")
plt.imshow(image_pil_lanczos_32)
plt.axis("off")

plt.tight_layout()
plt.show()
```

Problem statement:
Add a section about how training data resizing might affect the quality of the end models
Have been chatting with @bghira on our Discord forum about the negative effects of not carefully resizing the training data. 

@bghira noticed that the way we resize the images in our training examples can introduce unwanted artifacts in the samples generated by the end fine-tuned model. In short, it ruins the fidelity of the generated samples. 

We don't want to make the data input pipelines in our training examples super sophisticated, guaranteeing SoTA results. But that said, I think it's best in the community's interest if we made a note about this phenomenon in the READMEs and just added a comment about it in the pipelines. 

Curious to know what @patil-suraj @apolinario think about this. Personally, I think it makes sense. 

I will let @bghira share some more insights as well.

Hints from issue discussion:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Not stale.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md) are likely to be ignored.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md) are likely to be ignored.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md) are likely to be ignored.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md) are likely to be ignored.
is this still an issue @sayakpaul? I think the community is already aware of this and as always, the best solution is to have a ready made dataset with the optimal resolutions for training instead of doing the resize on the fly while training.

Also newer models don't seem to be affected by this that much or at least that's what I have experienced and saw, some people are even training Flux with 512px images.
resizing artifacts are learnt by Flux as well, and nearest neighbour resize done automatically is still the way the training examples here run so they're not setting a good example for others' best practices, this is still something to address
you do a lot of training so I'll believe you, I'll try to address this in the training examples then by adding a note.
i think a while back the image interpolation was going to be made configurable with the LANCZOS as the default instead, if you prefer

Commit message: initial

Patch to review:
diff --git a/examples/dreambooth/train_dreambooth_lora_sdxl.py b/examples/dreambooth/train_dreambooth_lora_sdxl.py
index f0d993ad9bbc..aa63c96f1764 100644
--- a/examples/dreambooth/train_dreambooth_lora_sdxl.py
+++ b/examples/dreambooth/train_dreambooth_lora_sdxl.py
@@ -669,6 +669,13 @@ def parse_args(input_args=None):
         ),
     )
 
+    parser.add_argument(
+        "--image_interpolation_mode",
+        type=str,
+        default="lanczos",
+        help="The image interpolation method to use for resizing images.",
+    )
+
     if input_args is not None:
         args = parser.parse_args(input_args)
     else:
@@ -790,7 +797,12 @@ def __init__(
         self.original_sizes = []
         self.crop_top_lefts = []
         self.pixel_values = []
-        train_resize = transforms.Resize(size, interpolation=transforms.InterpolationMode.BILINEAR)
+
+        if args.image_interpolation_mode == "bilinear":
+            train_resize = transforms.Resize(size, interpolation=transforms.InterpolationMode.BILINEAR)
+        else:
+            train_resize = transforms.Resize(size, interpolation=transforms.InterpolationMode.LANCZOS)
+
         train_crop = transforms.CenterCrop(size) if center_crop else transforms.RandomCrop(size)
         train_flip = transforms.RandomHorizontalFlip(p=1.0)
         train_transforms = transforms.Compose(


Code review comments:
diff_hunk: @@ -790,7 +797,12 @@ def __init__(
         self.original_sizes = []
         self.crop_top_lefts = []
         self.pixel_values = []
-        train_resize = transforms.Resize(size, interpolation=transforms.InterpolationMode.BILINEAR)
+
+        if args.image_interpolation_mode == "bilinear":
+            train_resize = transforms.Resize(size, interpolation=transforms.InterpolationMode.BILINEAR)
+        else:
+            train_resize = transforms.Resize(size, interpolation=transforms.InterpolationMode.LANCZOS)
path: examples/dreambooth/train_dreambooth_lora_sdxl.py
text: @user1:
```suggestion
        interpolation = getattr(transforms.InterpolationMode, args.image_interpolation_mode.upper(), None)
        if interpolation is None:
            raise ValueError(f"Unsupported interpolation mode.")
        train_resize = transforms.Resize(size, interpolation=interpolation)
```

Could we do something like this?

@author:
yeah, not sure about the other interpolation modes but I guess if the user uses another one it's because they know what they're doing

@user2:
yeah since we have the default as lanczos I also think it's ok to assume the user knows what their doing if they use a mode other than lanczos/bilinear

@user1:
Just cleaner code and preemptively solves the inevitable issue report: "Setting `image_interpolation_mode` other than `bilinear` uses `lanczos`"

@user3:
Let's go with @user1's suggestion. It's quite nice!