internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/diffusers

Issue:
Error occurs when using validation option of train_dreambooth.py
### Describe the bug

Training with validation option on failed for train_dreambooth.py and train_dreambooth_lora.py. After specify `--validation_prompt`, the program exits at validation stage, complaining about missing attribute of Namespace.

Turns out that vars() functions added in #4248 only do shallow copy of attributes, so the pop() function will actually remove `validation_images` attribute from `args`, causing the error. A prove in the following image, I did two prints before and after the pop:

![image](https://github.com/huggingface/diffusers/assets/16359093/50c93ec0-f95a-4248-a91d-435adc264016)

### Reproduction

Just try the given example:

```
accelerate launch train_dreambooth_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --instance_prompt="a photo of sks dog" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --checkpointing_steps=100 \
  --learning_rate=1e-4 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=500 \
  --validation_prompt="A photo of sks dog in a bucket" \
  --validation_epochs=50 \
  --seed="0" \
  --push_to_hub
```

### Logs

_No response_

### System Info

python 3.8

### Who can help?

_No response_


Discussion:

Resolving patch:
diff --git a/examples/dreambooth/train_dreambooth.py b/examples/dreambooth/train_dreambooth.py
--- a/examples/dreambooth/train_dreambooth.py
+++ b/examples/dreambooth/train_dreambooth.py
@@ -14,6 +14,7 @@
 # See the License for the specific language governing permissions and
 
 import argparse
+import copy
 import gc
 import hashlib
 import itertools
@@ -1116,7 +1117,7 @@ def compute_text_embeddings(prompt):
     # We need to initialize the trackers we use, and also store our configuration.
     # The trackers initializes automatically on the main process.
     if accelerator.is_main_process:
-        tracker_config = vars(args)
+        tracker_config = vars(copy.deepcopy(args))
         tracker_config.pop("validation_images")
         accelerator.init_trackers("dreambooth", config=tracker_config)
 
diff --git a/examples/dreambooth/train_dreambooth_lora.py b/examples/dreambooth/train_dreambooth_lora.py
--- a/examples/dreambooth/train_dreambooth_lora.py
+++ b/examples/dreambooth/train_dreambooth_lora.py
@@ -14,6 +14,7 @@
 # See the License for the specific language governing permissions and
 
 import argparse
+import copy
 import gc
 import hashlib
 import itertools
@@ -1067,7 +1068,7 @@ def compute_text_embeddings(prompt):
     # We need to initialize the trackers we use, and also store our configuration.
     # The trackers initializes automatically on the main process.
     if accelerator.is_main_process:
-        tracker_config = vars(args)
+        tracker_config = vars(copy.deepcopy(args))
         tracker_config.pop("validation_images")
         accelerator.init_trackers("dreambooth-lora", config=tracker_config)
 


Test patch:
Repository: huggingface/diffusers

Issue:
Wrong default value (or documentation) in StableDiffusionXLPipeline
### Describe the bug

According to the documentation(1), the default value for guidance_scale is 7.5. After rendering a couple of thousand images, I set this value explicitly, only to find out that in fact, it defaults to 5.0. This is unlikely to be a documentation error, as 7.5 is the most sensible value, also used in previous versions of SD.

(1) https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl#diffusers.StableDiffusionXLPipeline

### Reproduction

Use pipeline without passing guidance_scale arg, then with 7.5, then with 5.0.

### Logs

_No response_

### System Info

all versions

### Who can help?

_No response_


Discussion:

Resolving patch:
diff --git a/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py b/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
--- a/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
+++ b/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
@@ -608,7 +608,7 @@ def __call__(
                 scheduler. The denoising_end parameter should ideally be utilized when this pipeline forms a part of a
                 "Mixture of Denoisers" multi-pipeline setup, as elaborated in [**Refining the Image
                 Output**](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl#refining-the-image-output)
-            guidance_scale (`float`, *optional*, defaults to 7.5):
+            guidance_scale (`float`, *optional*, defaults to 5.0):
                 Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598).
                 `guidance_scale` is defined as `w` of equation 2. of [Imagen
                 Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale >


Test patch:
Repository: huggingface/diffusers

Issue:
example/texual_inversion/   multi text inversion is actually not working.
### Describe the bug

https://github.com/huggingface/diffusers/blob/4c4fe042a7f718f30f6bd897b967ff800e035347/examples/textual_inversion/textual_inversion.py#L629C1-L648
https://github.com/huggingface/diffusers/blob/4c4fe042a7f718f30f6bd897b967ff800e035347/examples/textual_inversion/textual_inversion.py#L706-L716https://github.com/huggingface/diffusers/blob/4c4fe042a7f718f30f6bd897b967ff800e035347/examples/textual_inversion/textual_inversion.py#L521C1-L537

code makes multiple placeholder and embeddings but the training text is using only placeholder token



### Reproduction

export MODEL_NAME="runwayml/stable-diffusion-v1-5"&&
export DATA_DIR="./cat"&&
accelerate launch textual_inversion.py \
	--num_vectors 4\
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$DATA_DIR \
  --learnable_property="object" \
  --placeholder_token="<cat>" --initializer_token="cat" \
	--output_dir="basic_multi_4_vectors_cat"\
	--validation_prompt="A photo of <cat>"\
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --max_train_steps=3000 \
  --learning_rate=5.0e-04 --scale_lr \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --save_as_full_pipeline \
	--num_validation_images=3\
	--validation_steps=100


### Logs

_No response_

### System Info

- `diffusers` version: 0.20.0.dev0
- Platform: Linux-5.4.0-1098-oracle-x86_64-with-debian-buster-sid
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.0 (True)
- Huggingface_hub version: 0.16.4
- Transformers version: 4.30.2
- Accelerate version: 0.20.3
- xFormers version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_


Discussion:

Resolving patch:
diff --git a/examples/textual_inversion/textual_inversion.py b/examples/textual_inversion/textual_inversion.py
--- a/examples/textual_inversion/textual_inversion.py
+++ b/examples/textual_inversion/textual_inversion.py
@@ -708,7 +708,7 @@ def main():
         data_root=args.train_data_dir,
         tokenizer=tokenizer,
         size=args.resolution,
-        placeholder_token=args.placeholder_token,
+        placeholder_token=(" ".join(tokenizer.convert_ids_to_tokens(placeholder_token_ids))),
         repeats=args.repeats,
         learnable_property=args.learnable_property,
         center_crop=args.center_crop,


Test patch:
Repository: huggingface/diffusers

Issue:
vae is not use in the doc
### Describe the bug

https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0?fbclid=IwAR3CTWQ3hLP1VxTpgFHVP82zhkYm7rnlKJ4HPq32rPAI9ltOWPbIcA2PFys
![image](https://github.com/huggingface/diffusers/assets/18206728/183e7371-d865-4d85-b12e-3accade005a0)


https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl
![image](https://github.com/huggingface/diffusers/assets/18206728/6fe59ec6-3734-48c2-a64a-c320754a6729)


### Reproduction

web

### Logs

_No response_

### System Info

web

### Who can help?

@stevhliu @yiyixuxu 


Discussion:

Resolving patch:
diff --git a/src/diffusers/pipelines/controlnet/pipeline_controlnet_sd_xl.py b/src/diffusers/pipelines/controlnet/pipeline_controlnet_sd_xl.py
--- a/src/diffusers/pipelines/controlnet/pipeline_controlnet_sd_xl.py
+++ b/src/diffusers/pipelines/controlnet/pipeline_controlnet_sd_xl.py
@@ -81,8 +81,8 @@
         ...     "diffusers/controlnet-canny-sdxl-1.0", torch_dtype=torch.float16
         ... )
         >>> vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
-        >>> pipe = StableDiffusionControlNetPipeline.from_pretrained(
-        ...     "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnet, torch_dtype=torch.float16
+        >>> pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
+        ...     "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnet, vae=vae, torch_dtype=torch.float16
         ... )
         >>> pipe.enable_model_cpu_offload()
 


Test patch:
Repository: huggingface/diffusers

Issue:
LoraLoaderMixin | RuntimeError: dictionary changed size during iteration
### Describe the bug

Related to the fix for #4797 

```
File "/workspace/sandbox/test.py", line 18, in <module>
    pipe.load_lora_weights(
  File "/home/user/mambaforge/envs/tensorml/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 904, in load_lora_weights
    state_dict, network_alphas = self.lora_state_dict(
  File "/home/user/mambaforge/envs/tensorml/lib/python3.10/site-packages/diffusers/loaders.py", line 1087, in lora_state_dict
    state_dict = cls._maybe_map_sgm_blocks_to_diffusers(state_dict, unet_config)
  File "/home/user/mambaforge/envs/tensorml/lib/python3.10/site-packages/diffusers/loaders.py", line 1146, in _maybe_map_sgm_blocks_to_diffusers
    for layer in all_keys:
RuntimeError: dictionary changed size during iteration
```

The bug in the original code is caused by modifying a dictionary while iterating over its keys. This leads to a "RuntimeError: dictionary changed size during iteration." when loading some community created LoRAs. The fix I applied, using `list(all_keys)` to create a static snapshot of keys, prevents the dictionary from changing in size during iteration and resolves the issue.

I intend to submit a pull request for this issue.

### Reproduction

```
from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True,
)

pipe.to("cuda")

pipe.load_lora_weights(
    "chillpixel/blacklight-makeup-sdxl-lora",
    weight_name="blacklight_makeup_v1.safetensors",
)

image = pipe(
    prompt="portrait photo of a kitty cat with blacklight makeup",
).images[0]
```

### Logs

_No response_

### System Info

Ubuntu 20.04.6 LTS
Python 3.10.12
diffusers 0.21.0.dev0

### Who can help?

@patrickvonplaten #4797


Discussion:

Resolving patch:
diff --git a/src/diffusers/loaders.py b/src/diffusers/loaders.py
--- a/src/diffusers/loaders.py
+++ b/src/diffusers/loaders.py
@@ -1123,7 +1123,7 @@ def _best_guess_weight_name(cls, pretrained_model_name_or_path_or_dict, file_ext
     @classmethod
     def _maybe_map_sgm_blocks_to_diffusers(cls, state_dict, unet_config, delimiter="_", block_slice_pos=5):
         # 1. get all state_dict_keys
-        all_keys = state_dict.keys()
+        all_keys = list(state_dict.keys())
         sgm_patterns = ["input_blocks", "middle_block", "output_blocks"]
 
         # 2. check if needs remapping, if not return original dict


Test patch:
Repository: huggingface/diffusers

Issue:
convert_original_stable_diffison script have issue
https://github.com/huggingface/diffusers/blob/8263cf00f832399bca215e29fa7572e0b0bde4da/scripts/convert_original_stable_diffusion_to_diffusers.py#L157C9-L157C40

config_files command line argment is not include in this script.
I think we need to add this code

parser.add_argument(
        "--config_files",
        default=None,
        type=str,
        help="The YAML config file corresponding to the original architecture.",
    )


Discussion:

Resolving patch:
diff --git a/scripts/convert_original_stable_diffusion_to_diffusers.py b/scripts/convert_original_stable_diffusion_to_diffusers.py
--- a/scripts/convert_original_stable_diffusion_to_diffusers.py
+++ b/scripts/convert_original_stable_diffusion_to_diffusers.py
@@ -35,6 +35,12 @@
         type=str,
         help="The YAML config file corresponding to the original architecture.",
     )
+    parser.add_argument(
+        "--config_files",
+        default=None,
+        type=str,
+        help="The YAML config file corresponding to the architecture.",
+    )
     parser.add_argument(
         "--num_in_channels",
         default=None,


Test patch:
Repository: huggingface/diffusers

Issue:
[HacktoberFest][Missing Docstrings] Autoencoder `encode`
This feature request is included in this years Diffusers' Hacktoberfest 👨‍💻! 

Hacktoberfest's purpose is to give upcoming open-source developers the opportunity to learn about open-source and programming by fixes simple issues (like this one).

**How to solve the issue**:

- 1. Every issue has a description further below explaining the problem. First, you should try to understand the issue.
- 2. Second, follow the [contribution guide](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md) to learn how to contribute to Diffusers.\
- 3. Claim this issue for you by opening a PR that links to this issue and writing "I am solving this issue in `<link-to-pr>`"


**Issue**

The [encode method of AutoencoderKL](https://github.com/huggingface/diffusers/blob/78a78515d64736469742e5081337dbcf60482750/src/diffusers/models/autoencoder_kl.py#L252) is completely missing docstrings, can you add them?





Discussion:

Resolving patch:
diff --git a/src/diffusers/models/autoencoder_kl.py b/src/diffusers/models/autoencoder_kl.py
--- a/src/diffusers/models/autoencoder_kl.py
+++ b/src/diffusers/models/autoencoder_kl.py
@@ -249,7 +249,21 @@ def set_default_attn_processor(self):
         self.set_attn_processor(processor, _remove_lora=True)
 
     @apply_forward_hook
-    def encode(self, x: torch.FloatTensor, return_dict: bool = True) -> AutoencoderKLOutput:
+    def encode(
+        self, x: torch.FloatTensor, return_dict: bool = True
+    ) -> Union[AutoencoderKLOutput, Tuple[DiagonalGaussianDistribution]]:
+        """
+        Encode a batch of images into latents.
+
+        Args:
+            x (`torch.FloatTensor`): Input batch of images.
+            return_dict (`bool`, *optional*, defaults to `True`):
+                Whether to return a [`~models.autoencoder_kl.AutoencoderKLOutput`] instead of a plain tuple.
+
+        Returns:
+                The latent representations of the encoded images. If `return_dict` is True, a
+                [`~models.autoencoder_kl.AutoencoderKLOutput`] is returned, otherwise a plain `tuple` is returned.
+        """
         if self.use_tiling and (x.shape[-1] > self.tile_sample_min_size or x.shape[-2] > self.tile_sample_min_size):
             return self.tiled_encode(x, return_dict=return_dict)
 


Test patch:
Repository: huggingface/diffusers

Issue:
[HacktoberFest][Missing Docstrings] Autoencoder `decode`
This feature request is included in this years Diffusers' Hacktoberfest 👨‍💻! 

Hacktoberfest's purpose is to give upcoming open-source developers the opportunity to learn about open-source and programming by fixes simple issues (like this one).

**How to solve the issue**:

- 1. Every issue has a description further below explaining the problem. First, you should try to understand the issue.
- 2. Second, follow the [contribution guide](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md) to learn how to contribute to Diffusers.\
- 3. Claim this issue for you by opening a PR that links to this issue and writing "I am solving this issue in `<link-to-pr>`"


**Issue**

The [decode method of AutoencoderKL](https://github.com/huggingface/diffusers/blob/78a78515d64736469742e5081337dbcf60482750/src/diffusers/models/autoencoder_kl.py#L283) is completely missing docstrings, can you add them?





Discussion:

Resolving patch:
diff --git a/src/diffusers/models/autoencoder_kl.py b/src/diffusers/models/autoencoder_kl.py
--- a/src/diffusers/models/autoencoder_kl.py
+++ b/src/diffusers/models/autoencoder_kl.py
@@ -281,6 +281,20 @@ def _decode(self, z: torch.FloatTensor, return_dict: bool = True) -> Union[Decod
 
     @apply_forward_hook
     def decode(self, z: torch.FloatTensor, return_dict: bool = True) -> Union[DecoderOutput, torch.FloatTensor]:
+        """
+        Decode a batch of images.
+
+        Args:
+            z (`torch.FloatTensor`): Input batch of latent vectors.
+            return_dict (`bool`, *optional*, defaults to `True`):
+                Whether to return a [`~models.vae.DecoderOutput`] instead of a plain tuple.
+
+        Returns:
+            [`~models.vae.DecoderOutput`] or `tuple`:
+                If return_dict is True, a [`~models.vae.DecoderOutput`] is returned, otherwise a plain `tuple` is
+                returned.
+
+        """
         if self.use_slicing and z.shape[0] > 1:
             decoded_slices = [self._decode(z_slice).sample for z_slice in z.split(1)]
             decoded = torch.cat(decoded_slices)


Test patch:
Repository: huggingface/diffusers

Issue:
Missing docs in multiple diffusers/models files
Multiple classes such as [this](https://github.com/huggingface/diffusers/blob/78a78515d64736469742e5081337dbcf60482750/src/diffusers/models/unet_2d_blocks.py#L1114) in the models directory are missing documentation or an explanation for what the different parameters mean, and sometimes type hints. Has it been done this way by choice to allow for a clean and direct code view? if not, I would be happy to take this up as an issue and write the docs.


Discussion:
I think it was on purpose in some sense because these blocks are usually modified to accommodate different features. But maybe @patrickvonplaten can shed more light! 
Hey @a-r-r-o-w, 

Good point! Actually it would be amazing if all these blocks would have better type hints and doc strings - we simply forgot to add those when the library was created 
Got it @patrickvonplaten! Will work on this and open a PR as soon as possible :rocket:

Resolving patch:
diff --git a/src/diffusers/models/unet_2d_blocks.py b/src/diffusers/models/unet_2d_blocks.py
--- a/src/diffusers/models/unet_2d_blocks.py
+++ b/src/diffusers/models/unet_2d_blocks.py
@@ -466,6 +466,18 @@ def get_up_block(
 
 
 class AutoencoderTinyBlock(nn.Module):
+    """
+    Tiny Autoencoder block used in [`AutoencoderTiny`]. It is a mini residual module consisting of plain conv + ReLU blocks.
+
+    Args:
+        in_channels (`int`): The number of input channels.
+        out_channels (`int`): The number of output channels.
+        act_fn (`str`):` The activation function to use. Supported values are `"swish"`, `"mish"`, `"gelu"`, and `"relu"`.
+
+    Returns:
+        `torch.FloatTensor`: A tensor with the same shape as the input tensor, but with the number of channels equal to `out_channels`.
+    """
+
     def __init__(self, in_channels: int, out_channels: int, act_fn: str):
         super().__init__()
         act_fn = get_activation(act_fn)


Test patch:
Repository: huggingface/diffusers

Issue:
Packaging Type Information Not Distributed
### Describe the bug

Diffusers does not appear to adhere to [PEP 561 – Distributing and Packaging Type Information](https://peps.python.org/pep-0561/). As a result, static type checkers, such as mypy or pyright, report incorrect errors.

According to the docs, this can be remedied by
1. Creating a blank `py.typed` in the package; note, that it applies recursively
2. Add the line `package_data = {'diffusers': ['py.typed']}` to setup.py

In searching for a related issue I found https://github.com/huggingface/diffusers/issues/5249, which indicates to me that type hints are highly desired in this project.

### Reproduction

Anything that imports `diffusers` will cause the problem with a static type checker. I used this file:

```python
from diffusers import AutoPipelineForText2Image

# https://huggingface.co/runwayml/stable-diffusion-v1-5
diffusion_model: str = "runwayml/stable-diffusion-v1-5"

# The auto pipeline automatically detects and loads the model
pipeline = AutoPipelineForText2Image.from_pretrained(
    diffusion_model, use_safetensors=True
).to("cpu")

# Generate the image based on the prompt
prompt: str = "A python is sitting on a desk, and the snake is wearing glasses."
image = pipeline(prompt, num_inference_steps=25).images[0]
image.show()

```

#### mypy

> error: Skipping analyzing "diffusers": module is installed, but missing library stubs or py.typed marker  [import]
> note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports


#### pyright
`--verifytypes diffusers`
> error: No py.typed file found



### Logs

_No response_

### System Info

- `diffusers` version: 0.21.4
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- PyTorch version (GPU?): 2.1.0+cu121 (False)
- Huggingface_hub version: 0.17.3
- Transformers version: 4.34.0
- Accelerate version: 0.23.0
- xFormers version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
- mypy 1.5.1 (compiled: yes)
- pyright 1.1.329

### Who can help?

_No response_


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -255,6 +255,7 @@ def run(self):
     url="https://github.com/huggingface/diffusers",
     package_dir={"": "src"},
     packages=find_packages("src"),
+    package_data={"diffusers": ["py.typed"]},
     include_package_data=True,
     python_requires=">=3.8.0",
     install_requires=list(install_requires),


Test patch: