internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/diffusers

Issue:
Comments swapped in Stable Diffusion depth2img
### Describe the bug

In [pipeline_stable_diffusion_depth2img.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_depth2img.py), lines 541 and 531 have swapped labels in the comments.

### Reproduction

 -, mistake in the code

### Logs

_No response_

### System Info

N/A


Discussion:
Hey @neverix, 

Thanks a lot for the issue!
Would you like to open a PR to fix it? :-) 

Resolving patch:
diff --git a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_depth2img.py b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_depth2img.py
--- a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_depth2img.py
+++ b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_depth2img.py
@@ -551,7 +551,7 @@ def __call__(
             prompt, device, num_images_per_prompt, do_classifier_free_guidance, negative_prompt
         )
 
-        # 4. Preprocess image
+        # 4. Prepare depth mask
         depth_mask = self.prepare_depth_map(
             image,
             depth_map,
@@ -561,10 +561,10 @@ def __call__(
             device,
         )
 
-        # 5. Prepare depth mask
+        # 5. Preprocess image
         image = preprocess(image)
 
-        # 6. set timesteps
+        # 6. Set timesteps
         self.scheduler.set_timesteps(num_inference_steps, device=device)
         timesteps, num_inference_steps = self.get_timesteps(num_inference_steps, strength, device)
         latent_timestep = timesteps[:1].repeat(batch_size * num_images_per_prompt)


Test patch:
Repository: huggingface/diffusers

Issue:
Fix PNDMScheduler docstring 
In the PNDM docstring it is saying that the `prediction_type` argument could be `epsilon` , `sample` or `v_prediction`.

https://github.com/huggingface/diffusers/blob/247b5feea125c6933384c59dcfd6d286c71dd8ba/src/diffusers/schedulers/scheduling_pndm.py#L87

But in the step method, it raises an error if we use `sample`

https://github.com/huggingface/diffusers/blob/247b5feea125c6933384c59dcfd6d286c71dd8ba/src/diffusers/schedulers/scheduling_pndm.py#L376

and says that it can only use epsilon or v_prediction

https://github.com/huggingface/diffusers/blob/247b5feea125c6933384c59dcfd6d286c71dd8ba/src/diffusers/schedulers/scheduling_pndm.py#L380

Consider remove the `sample` mention in the class docstring


Discussion:
Hey @Warvito,

Thanks for the issue, you're 100% correct. Would you mind opening an issue for it? :-) 

Resolving patch:
diff --git a/src/diffusers/schedulers/scheduling_pndm.py b/src/diffusers/schedulers/scheduling_pndm.py
--- a/src/diffusers/schedulers/scheduling_pndm.py
+++ b/src/diffusers/schedulers/scheduling_pndm.py
@@ -84,8 +84,7 @@ class PNDMScheduler(SchedulerMixin, ConfigMixin):
             otherwise it uses the value of alpha at step 0.
         prediction_type (`str`, default `epsilon`, optional):
             prediction type of the scheduler function, one of `epsilon` (predicting the noise of the diffusion
-            process), `sample` (directly predicting the noisy sample`) or `v_prediction` (see section 2.4
-            https://imagen.research.google/video/paper.pdf)
+            process) or `v_prediction` (see section 2.4 https://imagen.research.google/video/paper.pdf)
         steps_offset (`int`, default `0`):
             an offset added to the inference steps. You can use a combination of `offset=1` and
             `set_alpha_to_one=False`, to make the last step use step 0 for the previous alpha product, as done in


Test patch:
Repository: huggingface/diffusers

Issue:
Failing test of `examples/train_unconditional.py`
Reference: https://github.com/huggingface/diffusers/actions/runs/4245740195/jobs/7381643637

I could verify that this happens when installing `accelerate` from `main`, but not in the current release. I'll investigate if it's a regression or an API change.


Discussion:

Resolving patch:
diff --git a/examples/unconditional_image_generation/train_unconditional.py b/examples/unconditional_image_generation/train_unconditional.py
--- a/examples/unconditional_image_generation/train_unconditional.py
+++ b/examples/unconditional_image_generation/train_unconditional.py
@@ -23,7 +23,7 @@
 from diffusers import DDPMPipeline, DDPMScheduler, UNet2DModel
 from diffusers.optimization import get_scheduler
 from diffusers.training_utils import EMAModel
-from diffusers.utils import check_min_version, is_tensorboard_available, is_wandb_available
+from diffusers.utils import check_min_version, is_accelerate_version, is_tensorboard_available, is_wandb_available
 
 
 # Will error if the minimal version of diffusers is not installed. Remove at your own risks.
@@ -628,10 +628,13 @@ def transform_images(examples):
                 images_processed = (images * 255).round().astype("uint8")
 
                 if args.logger == "tensorboard":
-                    accelerator.get_tracker("tensorboard").add_images(
-                        "test_samples", images_processed.transpose(0, 3, 1, 2), epoch
-                    )
+                    if is_accelerate_version(">=", "0.17.0.dev0"):
+                        tracker = accelerator.get_tracker("tensorboard", unwrap=True)
+                    else:
+                        tracker = accelerator.get_tracker()
+                    tracker.add_images("test_samples", images_processed.transpose(0, 3, 1, 2), epoch)
                 elif args.logger == "wandb":
+                    # Upcoming `log_images` helper coming in https://github.com/huggingface/accelerate/pull/962/files
                     accelerator.get_tracker("wandb").log(
                         {"test_samples": [wandb.Image(img) for img in images_processed], "epoch": epoch},
                         step=global_step,


Test patch:
Repository: huggingface/diffusers

Issue:
Runtime error when trying to explicitly pass image embeddings to StableUnCLIPImg2ImgPipeline
### Describe the bug

I'm trying to use StableUnCLIPImg2ImgPipeline with modified image embeddings. When trying to pass embeddings explicitly instead of image, the error is raised.

The fix is simple, just check `image_embeds is None` instead of casting `image_embeds` to bool. I'll create a pull request with the fix.

### Reproduction

```python
import torch
from diffusers import StableUnCLIPImg2ImgPipeline

pipe = StableUnCLIPImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-1-unclip", torch_dtype=torch.float16, variation="fp16"
)
pipe.to("cuda")

embedding = torch.randn(1, 1024, device="cuda", dtype=torch.float16)

pipe(image_embeds=embedding)
```

### Logs

```shell
Traceback (most recent call last):
  File "/home/experiments/remix/stable-diffusion-remix/bug.py", line 12, in <module>
    pipe(image_embeds=embedding)
  File "/home/experiments/remix/stable-diffusion-remix/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/experiments/remix/stable-diffusion-remix/.venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py", line 750, in __call__
    image_embeds = self._encode_image(
  File "/home/experiments/remix/stable-diffusion-remix/.venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py", line 391, in _encode_image
    if not image_embeds:
RuntimeError: Boolean value of Tensor with more than one value is ambiguous
```
```


### System Info

- `diffusers` version: 0.15.0.dev0
- Platform: Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-glibc2.27
- Python version: 3.10.8
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Huggingface_hub version: 0.13.3
- Transformers version: 4.27.3
- Accelerate version: 0.18.0
- xFormers version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Discussion:

Resolving patch:
diff --git a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py
--- a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py
+++ b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img.py
@@ -388,7 +388,7 @@ def _encode_image(
             # what the expected dimensions of inputs should be and how we handle the encoding.
             repeat_by = num_images_per_prompt
 
-        if not image_embeds:
+        if image_embeds is None:
             if not isinstance(image, torch.Tensor):
                 image = self.feature_extractor(images=image, return_tensors="pt").pixel_values
 


Test patch:
Repository: huggingface/diffusers

Issue:
Is this docstring of `encoder_hidden_states` argument in `Transformer2DModel` correct?
https://github.com/huggingface/diffusers/blob/e607a582cfaa7dfaf7913fc3bb54c35eceee583c/src/diffusers/models/transformer_2d.py#L228-L230

The shape is `(batch_size, seq_length, embedding_dim)`, isn't it? Also it's supposed to be float tensor. Or maybe it supports both long and float?


Discussion:
You're right it should be a float tensor and the shape should be `(batch_size, seq_length, embedding_dim)` - mind opening a PR here? :-) 

Resolving patch:
diff --git a/src/diffusers/models/transformer_2d.py b/src/diffusers/models/transformer_2d.py
--- a/src/diffusers/models/transformer_2d.py
+++ b/src/diffusers/models/transformer_2d.py
@@ -225,7 +225,7 @@ def forward(
             hidden_states ( When discrete, `torch.LongTensor` of shape `(batch size, num latent pixels)`.
                 When continuous, `torch.FloatTensor` of shape `(batch size, channel, height, width)`): Input
                 hidden_states
-            encoder_hidden_states ( `torch.LongTensor` of shape `(batch size, encoder_hidden_states dim)`, *optional*):
+            encoder_hidden_states ( `torch.FloatTensor` of shape `(batch size, sequence len, embed dims)`, *optional*):
                 Conditional embeddings for cross attention layer. If not given, cross-attention defaults to
                 self-attention.
             timestep ( `torch.long`, *optional*):


Test patch:
Repository: huggingface/diffusers

Issue:
Return cache folder removed as argument in 0.15.0
### Describe the bug

`return_cached_folder` argument removed from `from_pretrained`

### Reproduction

```py
from diffusers import DiffusionPipeline
import torch

pipe, cache_folder = DiffusionPipeline.from_pretrained(
            "runwayml/stable-diffusion-v1-5",
            torch_dtype=torch.float16,
            return_cached_folder = True,
        )
```

### Logs

```shell
Keyword arguments {'return_cached_folder': True} are not expected by StableDiffusionPipeline and will be ignored.
```


### System Info

- `diffusers` version: 0.15.0
- Platform: Linux-5.15.0-1031-aws-x86_64-with-glibc2.31
- Python version: 3.9.5
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Huggingface_hub version: 0.13.3
- Transformers version: 4.25.1
- Accelerate version: 0.18.0
- xFormers version: 0.0.16rc425


Discussion:

Resolving patch:
diff --git a/src/diffusers/pipelines/pipeline_utils.py b/src/diffusers/pipelines/pipeline_utils.py
--- a/src/diffusers/pipelines/pipeline_utils.py
+++ b/src/diffusers/pipelines/pipeline_utils.py
@@ -1059,7 +1059,7 @@ def load_module(name, value):
         return_cached_folder = kwargs.pop("return_cached_folder", False)
         if return_cached_folder:
             message = f"Passing `return_cached_folder=True` is deprecated and will be removed in `diffusers=0.17.0`. Please do the following instead: \n 1. Load the cached_folder via `cached_folder={cls}.download({pretrained_model_name_or_path})`. \n 2. Load the pipeline by loading from the cached folder: `pipeline={cls}.from_pretrained(cached_folder)`."
-            deprecate("return_cached_folder", "0.17.0", message, take_from=kwargs)
+            deprecate("return_cached_folder", "0.17.0", message)
             return model, cached_folder
 
         return model


Test patch:
Repository: huggingface/diffusers

Issue:
from_ckpt not working properly on windows
### Describe the bug

When attempting to use the `StableDiffusionPipeline.from_ckpt()` with HF checkpoint link on Windows OS, a `HFValidationError` exception is raised. This issue stems from how the Path is processed in the code [here](https://github.com/huggingface/diffusers/blob/67cf0445ef48b1f913b90ce0025ac0c75673e32e/src/diffusers/loaders.py#L1366). On Windows, the `repo_id` and `file_path` will contain backslashes instead of forward slashes, resulting in an improper repo_id and file_path format. E.g., repo_id = 'runwayml\\\stable-diffusion-v1-5', file_path = 'blob\\\main\\\v1-5-pruned-emaonly.ckpt'
This issue is not present on Linux systems.

### Reproduction

```
from diffusers import StableDiffusionPipeline
StableDiffusionPipeline.from_ckpt('https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/v1-5-pruned-emaonly.ckpt')
```

### Logs

_No response_

### System Info

- Platform: Windows-10-10.0.19045-SP0
- `diffusers` version: 0.16.1
- Python version: 3.10.6
- Huggingface_hub version: 0.14.1

   <br>
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- `diffusers` version: 0.16.1
- Python version: 3.10.6
- Huggingface_hub version: 0.14.1




Discussion:
Hey @LyubimovVladislav,

Thanks a mille for the issue! Would you like to open a pull request that makes sure the line: https://github.com/huggingface/diffusers/blob/67cf0445ef48b1f913b90ce0025ac0c75673e32e/src/diffusers/loaders.py#L1366 and following work both on Linux and Windows? 

I sadly don't have a Windows computer to test this 

Resolving patch:
diff --git a/src/diffusers/loaders.py b/src/diffusers/loaders.py
--- a/src/diffusers/loaders.py
+++ b/src/diffusers/loaders.py
@@ -1447,8 +1447,8 @@ def from_ckpt(cls, pretrained_model_link_or_path, **kwargs):
         ckpt_path = Path(pretrained_model_link_or_path)
         if not ckpt_path.is_file():
             # get repo_id and (potentially nested) file path of ckpt in repo
-            repo_id = str(Path().joinpath(*ckpt_path.parts[:2]))
-            file_path = str(Path().joinpath(*ckpt_path.parts[2:]))
+            repo_id = "/".join(ckpt_path.parts[:2])
+            file_path = "/".join(ckpt_path.parts[2:])
 
             if file_path.startswith("blob/"):
                 file_path = file_path[len("blob/") :]


Test patch:
Repository: huggingface/diffusers

Issue:
Watchdog caught collective operation timeout when training ddpm using train_unconditional.py
### Describe the bug

When using mutil-gpu,and the data set is large or the resolution is high(need more time to inference), problems arise when saving the image.This is due to timeout in NCCL(1800s,30min)，it can be solved by the following code.
    kwargs = InitProcessGroupKwargs(timeout=timedelta(seconds=7200))
    accelerator = Accelerator(
        gradient_accumulation_steps=args.gradient_accumulation_steps,
        mixed_precision=args.mixed_precision,
        log_with=args.logger,
        project_config=accelerator_project_config,
        kwargs_handlers=[kwargs],
    )
Maybe you can keep the default time, but add the code for people who need it

### Reproduction

accelerate launch --mixed_precision="fp16" --multi_gpu train_unconditional.py   --dataset_name=""   --resolution=512--output_dir=""   --train_batch_size=1   --num_epochs=100   --gradient_accumulation_steps=1   --learning_rate=1e-5   --lr_warmup_steps=500   --mixed_precision="fp16"


### Logs

_No response_

### System Info

- `diffusers` version: 0.16.1
- Platform: Linux-4.4.0-210-generic-x86_64-with-glibc2.27
- Python version: 3.10.11
- PyTorch version (GPU?): 1.12.1 (True)
- Huggingface_hub version: 0.14.1
- Transformers version: 4.31.0.dev0
- Accelerate version: 0.19.0
- xFormers version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_


Discussion:
cc @sayakpaul 
Thanks for reporting this! Would you maybe like to contribute a PR for this? 

Also cc: @muellerzr for the bit on `accelerate`. 

Resolving patch:
diff --git a/examples/unconditional_image_generation/train_unconditional.py b/examples/unconditional_image_generation/train_unconditional.py
--- a/examples/unconditional_image_generation/train_unconditional.py
+++ b/examples/unconditional_image_generation/train_unconditional.py
@@ -4,6 +4,7 @@
 import math
 import os
 import shutil
+from datetime import timedelta
 from pathlib import Path
 from typing import Optional
 
@@ -11,7 +12,7 @@
 import datasets
 import torch
 import torch.nn.functional as F
-from accelerate import Accelerator
+from accelerate import Accelerator, InitProcessGroupKwargs
 from accelerate.logging import get_logger
 from accelerate.utils import ProjectConfiguration
 from datasets import load_dataset
@@ -286,11 +287,13 @@ def main(args):
     logging_dir = os.path.join(args.output_dir, args.logging_dir)
     accelerator_project_config = ProjectConfiguration(project_dir=args.output_dir, logging_dir=logging_dir)
 
+    kwargs = InitProcessGroupKwargs(timeout=timedelta(seconds=7200))#a big number for high resolution or big dataset
     accelerator = Accelerator(
         gradient_accumulation_steps=args.gradient_accumulation_steps,
         mixed_precision=args.mixed_precision,
         log_with=args.logger,
         project_config=accelerator_project_config,
+        kwargs_handlers=[kwargs],
     )
 
     if args.logger == "tensorboard":


Test patch:
Repository: huggingface/diffusers

Issue:
Ability to customize network rank LoRA dreambooth training
**Is your feature request related to a problem? Please describe.**
For LoRA training, custom UIs like A1111 or kohya SD allow customization of the LoRA network rank which can make the results look much nicer but with the side effect of making LoRAs larger.

**Describe the solution you'd like**
Allowing configuration of LoRA network rank on the training dreambooth script (https://github.com/isidentical/diffusers/blob/4aca45500cf601b2bb4769651d3081cac845312d/examples/dreambooth/train_dreambooth_lora.py)

**Describe alternatives you've considered**
N/A

**Additional context**
N/A



Discussion:

Resolving patch:
diff --git a/examples/dreambooth/train_dreambooth_lora.py b/examples/dreambooth/train_dreambooth_lora.py
--- a/examples/dreambooth/train_dreambooth_lora.py
+++ b/examples/dreambooth/train_dreambooth_lora.py
@@ -436,6 +436,12 @@ def parse_args(input_args=None):
         default=None,
         help="The optional `class_label` conditioning to pass to the unet, available values are `timesteps`.",
     )
+    parser.add_argument(
+        "--rank",
+        type=int,
+        default=4,
+        help=("The dimension of the LoRA update matrices."),
+    )
 
     if input_args is not None:
         args = parser.parse_args(input_args)
@@ -845,7 +851,9 @@ def main(args):
                 LoRAAttnProcessor2_0 if hasattr(F, "scaled_dot_product_attention") else LoRAAttnProcessor
             )
         unet_lora_attn_procs[name] = lora_attn_processor_class(
-            hidden_size=hidden_size, cross_attention_dim=cross_attention_dim
+            hidden_size=hidden_size,
+            cross_attention_dim=cross_attention_dim,
+            rank=args.rank,
         )
 
     unet.set_attn_processor(unet_lora_attn_procs)
@@ -860,7 +868,9 @@ def main(args):
         for name, module in text_encoder.named_modules():
             if name.endswith(TEXT_ENCODER_ATTN_MODULE):
                 text_lora_attn_procs[name] = LoRAAttnProcessor(
-                    hidden_size=module.out_proj.out_features, cross_attention_dim=None
+                    hidden_size=module.out_proj.out_features,
+                    cross_attention_dim=None,
+                    rank=args.rank,
                 )
         text_encoder_lora_layers = AttnProcsLayers(text_lora_attn_procs)
         temp_pipeline = DiffusionPipeline.from_pretrained(


Test patch:
Repository: huggingface/diffusers

Issue:
Default args in `StableDiffusionControlNetPipeline.check_inputs` causes TypeError
### Describe the bug

Default value of `control_guidance_start` and `control_guidance_end` in `StableDiffusionControlNetPipeline.check_inputs` function causes `TypeError: object of type 'float' has no len()`

### Reproduction

```
from diffusers import (
   StableDiffusionControlNetPipeline,
   ControlNetModel
)
import torch
from PIL import Image

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained('runwayml/stable-diffusion-v1-5', controlnet=controlnet)

pipe.check_inputs('', Image.new('RGB',(512,512)), callback_steps=1)

```

### Logs

```shell
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
Traceback (most recent call last):
  File "bug.py", line 11, in <module>
    pipe.check_inputs('',Image.new('L',(512,512)),callback_steps=1)
  File "/home/neil/anaconda3/envs/reference_only/lib/python3.8/site-packages/diffusers/pipelines/controlnet/pipeline_controlnet.py", line 598, in check_inputs
    if len(control_guidance_start) != len(control_guidance_end):
TypeError: object of type 'float' has no len()
```


### System Info

- `diffusers` version: 0.18.2
- Platform: Linux-5.15.0-1037-gcp-x86_64-with-glibc2.17
- Python version: 3.8.16
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Huggingface_hub version: 0.16.4
- Transformers version: 4.30.2
- Accelerate version: 0.18.0
- xFormers version: 0.0.17
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@patrickvonplaten


Discussion:

Resolving patch:
diff --git a/src/diffusers/pipelines/controlnet/pipeline_controlnet.py b/src/diffusers/pipelines/controlnet/pipeline_controlnet.py
--- a/src/diffusers/pipelines/controlnet/pipeline_controlnet.py
+++ b/src/diffusers/pipelines/controlnet/pipeline_controlnet.py
@@ -554,6 +554,12 @@ def check_inputs(
         else:
             assert False
 
+        if not isinstance(control_guidance_start, (tuple, list)):
+            control_guidance_start = [control_guidance_start]
+            
+        if not isinstance(control_guidance_end, (tuple, list)):
+            control_guidance_end = [control_guidance_end]
+            
         if len(control_guidance_start) != len(control_guidance_end):
             raise ValueError(
                 f"`control_guidance_start` has {len(control_guidance_start)} elements, but `control_guidance_end` has {len(control_guidance_end)} elements. Make sure to provide the same number of elements to each list."


Test patch: