internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/diffusers

Issue:
Question about flax model output classes
For example here, it is using `from dataclasses import dataclass`
https://github.com/huggingface/diffusers/blob/d8b0e4f433778f9d4555cfab642e43db3aff4833/src/diffusers/models/unet_2d_condition_flax.py#L22-L23

But transformers equivalents use `@flax.struct.dataclass`. For example [here](https://github.com/huggingface/transformers/blob/21f6f58721dd9154357576be6de54eefef1f1818/src/transformers/models/bert/modeling_flax_bert.py#L63-L64)
```py
@flax.struct.dataclass
class FlaxBertForPreTrainingOutput(ModelOutput):
```

The benefit of using `@flax.struct.dataclass` over naive `python dataclass` is that: [`jax.jit` can consume @flax.struct.dataclass](https://github.com/huggingface/transformers/pull/11886)

So the question is: should we use `@flax.struct.dataclass`  on `diffusers` as well ?


Discussion:
Good catch @mishig25 !

We should indeed use `@flax.struct.dataclass` here for flax outputs as you said it works well with JAX.
My mistake, I used `flax.struct.dataclass` for the schedulers but blindly copied the code here from the PyTorch version. Thanks a lot @mishig25, I'll fix it :)
perfect!

Resolving patch:
diff --git a/src/diffusers/models/unet_2d_condition_flax.py b/src/diffusers/models/unet_2d_condition_flax.py
--- a/src/diffusers/models/unet_2d_condition_flax.py
+++ b/src/diffusers/models/unet_2d_condition_flax.py
@@ -1,6 +1,6 @@
-from dataclasses import dataclass
 from typing import Tuple, Union
 
+import flax
 import flax.linen as nn
 import jax
 import jax.numpy as jnp
@@ -19,7 +19,7 @@
 )
 
 
-@dataclass
+@flax.struct.dataclass
 class FlaxUNet2DConditionOutput(BaseOutput):
     """
     Args:


Test patch:
Repository: huggingface/diffusers

Issue:
Update Pillow library
### Describe the bug

Compatibility error when installing Pillow library > 9.1

|Deprecated | Use Instead|
|----------------|----------|
|Image.LANCZOS | Image.Resampling.LANCZOS|
|Image.NEAREST | Image.Resampling.NEAREST|
|Image.BILINEAR| Image.Resampling.BILINEAR|

Here the [link](https://pillow.readthedocs.io/en/stable/deprecations.html)

### Reproduction

pip install pillow==9.1
```python
from diffusers import AutoencoderKL, UNet2DConditionModel
```

### Logs

_No response_

### System Info

Compatibility error when installing Pillow library > 9.1

```console
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

[/usr/local/lib/python3.7/dist-packages/transformers/utils/import_utils.py](https://localhost:8080/#) in _get_module(self, module_name)
   1030         try:
-> 1031             return importlib.import_module("." + module_name, self.__name__)
   1032         except Exception as e:

19 frames

AttributeError: module 'PIL.Image' has no attribute 'Resampling'


The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)

[/usr/local/lib/python3.7/dist-packages/transformers/utils/import_utils.py](https://localhost:8080/#) in _get_module(self, module_name)
   1034                 f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
   1035                 f" traceback):\n{e}"
-> 1036             ) from e
   1037 
   1038     def __reduce__(self):

RuntimeError: Failed to import transformers.models.clip.feature_extraction_clip because of the following error (look up to see its traceback):
module 'PIL.Image' has no attribute 'Resampling'
```


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -77,7 +77,7 @@
 # 1. all dependencies should be listed here with their version requirements if any
 # 2. once modified, run: `make deps_table_update` to update src/diffusers/dependency_versions_table.py
 _deps = [
-    "Pillow",
+    "Pillow<10.0",  # keep the PIL.Image.Resampling deprecation away
     "accelerate>=0.11.0",
     "black==22.8",
     "datasets",
diff --git a/src/diffusers/dependency_versions_table.py b/src/diffusers/dependency_versions_table.py
--- a/src/diffusers/dependency_versions_table.py
+++ b/src/diffusers/dependency_versions_table.py
@@ -2,7 +2,7 @@
 # 1. modify the `_deps` dict in setup.py
 # 2. run `make deps_table_update``
 deps = {
-    "Pillow": "Pillow",
+    "Pillow": "Pillow<10.0",
     "accelerate": "accelerate>=0.11.0",
     "black": "black==22.8",
     "datasets": "datasets",


Test patch:
Repository: huggingface/diffusers

Issue:
LDM Bert `config.json` path
### Describe the bug

### Problem
There is a reference to an LDM Bert that 404's
```bash
src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py:    "ldm-bert": "https://huggingface.co/ldm-bert/resolve/main/config.json",
```

I was able to locate a `config.json` at `https://huggingface.co/valhalla/ldm-bert/blob/main/config.json`

Is this the correct `config.json`? 

#### Notes for reviewer
Happy to send a PR if needed to update, feel free to do on your own if it's faster/easier :)

### Reproduction

na

### Logs

```shell
na
```


### System Info

na


Discussion:
Yes this seems wrong, let's update it (cc @patil-suraj )

Resolving patch:
diff --git a/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py b/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py
--- a/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py
+++ b/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py
@@ -192,7 +192,7 @@ def __call__(
 
 
 LDMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
-    "ldm-bert": "https://huggingface.co/ldm-bert/resolve/main/config.json",
+    "ldm-bert": "https://huggingface.co/valhalla/ldm-bert/blob/main/config.json",
 }
 
 


Test patch:
Repository: huggingface/diffusers

Issue:
[dreambooth] RuntimeError when trying to Train on a Tesla T4 on Colab.
### Describe the bug

just right when it starts running the training it crashes with out of memory error. When i set it to fp16, i get a unscale_() was already called for error.. I'm not sure how to run this on a 16gb GPU? 

### Reproduction

Tesla T4, using 12 training images and i set the step count 1200 and max 100 regulation images. 

### Logs

```shell
RuntimeError                              Traceback (most recent call last)
<ipython-input-24-c6e3ce5f5a40> in <module>
      1 #@title Run training
      2 import accelerate
----> 3 accelerate.notebook_launcher(training_function, args=(text_encoder, vae, unet))
      4 with torch.no_grad():
      5     torch.cuda.empty_cache()

4 frames
/usr/local/lib/python3.7/dist-packages/torch/cuda/amp/grad_scaler.py in unscale_(self, optimizer)
    268 
    269         if optimizer_state["stage"] is OptState.UNSCALED:
--> 270             raise RuntimeError("unscale_() has already been called on this optimizer since the last update().")
    271         elif optimizer_state["stage"] is OptState.STEPPED:
    272             raise RuntimeError("unscale_() is being called after step().")

RuntimeError: unscale_() has already been called on this optimizer since the last update().
```


### System Info

DreamBooth Diffuser. 


Discussion:
Heey @Maki9009,

Thanks for raising the issue. Gently pinging @patil-suraj here
Thanks for the issue, did you use `mixed_precision` for training ?

Resolving patch:
diff --git a/examples/dreambooth/train_dreambooth.py b/examples/dreambooth/train_dreambooth.py
--- a/examples/dreambooth/train_dreambooth.py
+++ b/examples/dreambooth/train_dreambooth.py
@@ -566,7 +566,8 @@ def collate_fn(examples):
                     loss = F.mse_loss(noise_pred, noise, reduction="none").mean([1, 2, 3]).mean()
 
                 accelerator.backward(loss)
-                accelerator.clip_grad_norm_(unet.parameters(), args.max_grad_norm)
+                if accelerator.sync_gradients:
+                    accelerator.clip_grad_norm_(unet.parameters(), args.max_grad_norm)
                 optimizer.step()
                 lr_scheduler.step()
                 optimizer.zero_grad()


Test patch:
Repository: huggingface/diffusers

Issue:
[Documentation] "r" as method docstring for forward
### Describe the bug

As seen [here](https://github.com/huggingface/diffusers/blame/688031c592a08832387761971f1e6ca504a5900b/src/diffusers/models/unet_2d.py#L173), the forward docstrings of the UNet classes have "r" as the method description. Why is this?

https://github.com/huggingface/diffusers/blob/688031c592a08832387761971f1e6ca504a5900b/src/diffusers/models/unet_2d.py#L173-L180

I think @kashif you added these?

### Reproduction

_No response_

### Logs

_No response_

### System Info

Not needed, documentation issue.


Discussion:
@natolambert my bad I believe co-pilot is to blame! Let me fix it.

Resolving patch:
diff --git a/src/diffusers/models/unet_2d.py b/src/diffusers/models/unet_2d.py
--- a/src/diffusers/models/unet_2d.py
+++ b/src/diffusers/models/unet_2d.py
@@ -170,7 +170,7 @@ def forward(
         timestep: Union[torch.Tensor, float, int],
         return_dict: bool = True,
     ) -> Union[UNet2DOutput, Tuple]:
-        """r
+        r"""
         Args:
             sample (`torch.FloatTensor`): (batch, channel, height, width) noisy inputs tensor
             timestep (`torch.FloatTensor` or `float` or `int): (batch) timesteps
diff --git a/src/diffusers/models/unet_2d_condition_flax.py b/src/diffusers/models/unet_2d_condition_flax.py
--- a/src/diffusers/models/unet_2d_condition_flax.py
+++ b/src/diffusers/models/unet_2d_condition_flax.py
@@ -215,7 +215,7 @@ def __call__(
         return_dict: bool = True,
         train: bool = False,
     ) -> Union[FlaxUNet2DConditionOutput, Tuple]:
-        """r
+        r"""
         Args:
             sample (`jnp.ndarray`): (channel, height, width) noisy inputs tensor
             timestep (`jnp.ndarray` or `float` or `int`): timesteps


Test patch:
Repository: huggingface/diffusers

Issue:
Parallel dataloader for unconditional diffusion example
**Is your feature request related to a problem? Please describe.**
The loading of the samples is sequential and without prefetching n the `train_unconditional.py` script.
This results in an underutilization of the GPU if large images and/or "slow" IO are used.

**Describe the solution you'd like**
Adding a parameter, which sets the `num_workers` parameter of the PyTorch dataloader, solves this issue by loading the samples in parallel using multiple workers.

**Describe alternatives you've considered**
- Leave it as it is and accept a slower training
- I am not that familiar with `accelerate`, but maybe we can get the number of workers from there instead of introducing a new parameter.

**Additional context**
- Tested locally (1x 2080 ti + 32 core Threadripper) using our own dataset (https://github.com/bit-bots/TORSO_21_dataset). Resulted in 140% speedup as well as a higher GPU utilization in `nvtop`.



Discussion:
cc @anton-l 
Any updates here @anton-l ? 
Thanks for the feedback @Flova! Added the parameter in https://github.com/huggingface/diffusers/pull/1027

Resolving patch:
diff --git a/examples/unconditional_image_generation/train_unconditional.py b/examples/unconditional_image_generation/train_unconditional.py
--- a/examples/unconditional_image_generation/train_unconditional.py
+++ b/examples/unconditional_image_generation/train_unconditional.py
@@ -83,7 +83,16 @@ def parse_args():
         "--train_batch_size", type=int, default=16, help="Batch size (per device) for the training dataloader."
     )
     parser.add_argument(
-        "--eval_batch_size", type=int, default=16, help="Batch size (per device) for the eval dataloader."
+        "--eval_batch_size", type=int, default=16, help="The number of images to generate for evaluation."
+    )
+    parser.add_argument(
+        "--dataloader_num_workers",
+        type=int,
+        default=0,
+        help=(
+            "The number of subprocesses to use for data loading. 0 means that the data will be loaded in the main"
+            " process."
+        ),
     )
     parser.add_argument("--num_epochs", type=int, default=100)
     parser.add_argument("--save_images_epochs", type=int, default=10, help="How often to save images during training.")
@@ -249,7 +258,9 @@ def transforms(examples):
         return {"input": images}
 
     dataset.set_transform(transforms)
-    train_dataloader = torch.utils.data.DataLoader(dataset, batch_size=args.train_batch_size, shuffle=True)
+    train_dataloader = torch.utils.data.DataLoader(
+        dataset, batch_size=args.train_batch_size, shuffle=True, num_workers=args.dataloader_num_workers
+    )
 
     lr_scheduler = get_scheduler(
         args.lr_scheduler,


Test patch:
Repository: huggingface/diffusers

Issue:
[Flax] 🚨 0.7.0 not working 🚨
### Describe the bug

![Screenshot 2022-11-05 083749-x](https://user-images.githubusercontent.com/54370274/200104662-894bbbce-b435-48fc-af87-b539595fc898.jpg)


### Reproduction

_No response_

### Logs

_No response_

### System Info

TPU v3-8


Discussion:
I am still testing...
`0.7.0 🦆 💀  😢 `

![Screenshot 2022-11-05 092325](https://user-images.githubusercontent.com/54370274/200106625-49093699-8181-4648-97eb-1100cb33547e.jpg)


Maybe related 17 hours ago [Jaxlib release v0.3.24](https://github.com/google/jax/releases/tag/jaxlib-v0.3.24) also this new version using more memory 😭
Thanks for notifying us here so quickly! I'm not on my computer at the moment - but will try to fix it by tonight! 

Could you double check at which commit of main results start to become incorrect? 
I suspect that we messed up something between 0.6.0 and 0.7.0 here - also @pcuenca @patil-suraj here
accelerate sus
Thanks @camenduru! I'm not at my computer either, but will look into it later today. 
🚓🚔🚨 https://github.com/huggingface/diffusers/commit/0b61cea347e9b464fb03506cb78a49d38e1c74ee 🚨🚔🚓
Thanks, @camenduru, fix incoming!

Resolving patch:
diff --git a/src/diffusers/models/embeddings_flax.py b/src/diffusers/models/embeddings_flax.py
--- a/src/diffusers/models/embeddings_flax.py
+++ b/src/diffusers/models/embeddings_flax.py
@@ -88,4 +88,6 @@ class FlaxTimesteps(nn.Module):
 
     @nn.compact
     def __call__(self, timesteps):
-        return get_sinusoidal_embeddings(timesteps, embedding_dim=self.dim, freq_shift=self.freq_shift)
+        return get_sinusoidal_embeddings(
+            timesteps, embedding_dim=self.dim, freq_shift=self.freq_shift, flip_sin_to_cos=True
+        )


Test patch:
Repository: huggingface/diffusers

Issue:
Memory efficient attention not working with fp16 weights
### Describe the bug

Following the example code with available in the `0.7.0` release 
```py
from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    revision="fp16",
    torch_dtype=torch.float16,
).to("cuda")

pipe.enable_xformers_memory_efficient_attention()

with torch.inference_mode():
    sample = pipe("a small cat")
```

I'm getting the following error: 
```py
RuntimeError: expected scalar type Half but found Float
```

### System Info

diffusers==0.7.2
pytorch==1.12.1


Discussion:
The memory efficient attention mechanism seems to be working just fine - if I do a 
`with torch.autocast("cuda")` instead of `with torch.inference_mode()` it works, and performs faster with `enable_xformers_memory_efficient_attention` as opposed to disabled.

So it seems like a casting issue indeed as described
Could it be related to this issue? https://github.com/huggingface/diffusers/pull/532#issuecomment-1290278995 
Can you try updating the xformers library?

Resolving patch:
diff --git a/src/diffusers/models/attention.py b/src/diffusers/models/attention.py
--- a/src/diffusers/models/attention.py
+++ b/src/diffusers/models/attention.py
@@ -492,6 +492,8 @@ def forward(self, hidden_states, context=None, mask=None):
         # attention, what we cannot get enough of
         if self._use_memory_efficient_attention_xformers:
             hidden_states = self._memory_efficient_attention_xformers(query, key, value)
+            # Some versions of xformers return output in fp32, cast it back to the dtype of the input
+            hidden_states = hidden_states.to(query.dtype)
         else:
             if self._slice_size is None or query.shape[0] // self._slice_size == 1:
                 hidden_states = self._attention(query, key, value)


Test patch:
Repository: huggingface/diffusers

Issue:
It is not possible to disable telemetry
### Describe the bug

Although the DISABLE_TELEMETRY and HF_HUB_OFFLINE env variables exist, setting them doesn't actually seem to disable telemetry.

https://github.com/huggingface/diffusers/blob/main/src/diffusers/hub_utils.py#L87

Pass is used, and /api/telemetry is prepended to the huggingface endpoint.

### Reproduction

- set DISABLE_TELEMETRY to True
- run stablediffusion from pretrained
- see `/api/telemetry/diffusers_from_pretrained` in the requests made to huggingface

### Logs

_No response_

### System Info

main branch


Discussion:

Resolving patch:
diff --git a/src/diffusers/hub_utils.py b/src/diffusers/hub_utils.py
--- a/src/diffusers/hub_utils.py
+++ b/src/diffusers/hub_utils.py
@@ -84,7 +84,7 @@ def send_telemetry(data: Dict, name: str):
         name: a unique name to differentiate the telemetry logs, e.g. "diffusers_examples" or "diffusers_notebooks"
     """
     if DISABLE_TELEMETRY or HF_HUB_OFFLINE:
-        pass
+        return
 
     headers = {"user-agent": http_user_agent(data)}
     endpoint = HUGGINGFACE_CO_TELEMETRY + name


Test patch:
Repository: huggingface/diffusers

Issue:
dreambooth (minor): command line handling: accelerate logger used before being initialized
### Describe the bug

As I was working on another feature, I realized that parse_args uses `logger`. But `logger` isn't ready until after parse_args returns, as it depends on a configured `Accelerator`.


### Reproduction

Passing `--class_data_dir` but not `--with_prior_preservation` tries to write a warning, but the logger isn't ready yet:

```
$ python train_dreambooth.py --pretrained_model_name_or_path repro \
  --instance_data_dir repro --instance_prompt repro --class_data_dir repro
Traceback (most recent call last):
  File ".../diffusers/examples/dreambooth/train_dreambooth.py", line 748, in <module>
    args = parse_args()
  File ".../diffusers/examples/dreambooth/train_dreambooth.py", line 244, in parse_args
    logger.warning("You need not use --class_data_dir without --with_prior_preservation.")
  File ".../diffusers/lib/python3.10/logging/__init__.py", line 1847, in warning
    self.log(WARNING, msg, *args, **kwargs)
  File ".../diffusers/lib/python3.10/site-packages/accelerate/logging.py", line 48, in log
    if self.isEnabledFor(level) and self._should_log(main_process_only):
  File ".../diffusers/lib/python3.10/site-packages/accelerate/logging.py", line 33, in _should_log
    state = AcceleratorState()
  File ".../diffusers/lib/python3.10/site-packages/accelerate/state.py", line 86, in __init__
    raise ValueError(
ValueError: Please make sure to properly initialize your accelerator via `accelerator = Accelerator()` before using any functionality from the `accelerate` library.
```


### Logs

_No response_

### System Info

```
$ git log --format=oneline|head -1
69de9b2eaa3a9047e0074b49753eb7a71aec4a5d [Textual Inversion] Do not update other embeddings (#1665)
```



Discussion:

Resolving patch:
diff --git a/examples/dreambooth/train_dreambooth.py b/examples/dreambooth/train_dreambooth.py
--- a/examples/dreambooth/train_dreambooth.py
+++ b/examples/dreambooth/train_dreambooth.py
@@ -3,6 +3,7 @@
 import itertools
 import math
 import os
+import warnings
 from pathlib import Path
 from typing import Optional
 
@@ -255,10 +256,11 @@ def parse_args(input_args=None):
         if args.class_prompt is None:
             raise ValueError("You must specify prompt for class images.")
     else:
+        # logger is not available yet
         if args.class_data_dir is not None:
-            logger.warning("You need not use --class_data_dir without --with_prior_preservation.")
+            warnings.warn("You need not use --class_data_dir without --with_prior_preservation.")
         if args.class_prompt is not None:
-            logger.warning("You need not use --class_prompt without --with_prior_preservation.")
+            warnings.warn("You need not use --class_prompt without --with_prior_preservation.")
 
     return args
 


Test patch:
Repository: huggingface/diffusers

Issue:
OOM when using PyTorch, if Flax/JAX are installed
Bisected to f106ab4


Discussion:
Repro: install JAX for CUDA + Flax, then attempt to load any stable diffusion pipeline. JAX must be mistakenly initialized and reserves most of the GPU memory.

Resolving patch:
diff --git a/src/diffusers/schedulers/scheduling_ddpm_flax.py b/src/diffusers/schedulers/scheduling_ddpm_flax.py
--- a/src/diffusers/schedulers/scheduling_ddpm_flax.py
+++ b/src/diffusers/schedulers/scheduling_ddpm_flax.py
@@ -208,7 +208,7 @@ def step(
         model_output: jnp.ndarray,
         timestep: int,
         sample: jnp.ndarray,
-        key: jax.random.KeyArray = jax.random.PRNGKey(0),
+        key: jax.random.KeyArray,
         return_dict: bool = True,
     ) -> Union[FlaxDDPMSchedulerOutput, Tuple]:
         """


Test patch:
Repository: huggingface/diffusers

Issue:
EMA model in Stable Diffusion training example is wrong
### Describe the bug

The step call is updating self.decay, which will then be used to compute the step-dependent-decay in the next steps, resulting in a bit of a mess that makes EMA not really work as it should.

https://github.com/huggingface/diffusers/blob/ab0e92fdc8b42e1739840d10bb0b1dc7edaf2ff3/examples/text_to_image/train_text_to_image.py#L281-L293

The issue seems to be with the `self.decay = ...` which should be `decay = ...`  in order not to affect the original decay value.

Edit: Just checked the original implementation, this does seem to be a bug/typo indeed: https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/ema.py

### Reproduction

A replication of the algorithm currently in there shows that it makes little sense:
```
decay = 0.9999
for i in range(10000):
    value = (1 + i) / (10 + i)
    decay = 1 - min(decay, value)
    print(f"{i:07d} {decay:.6f}")

Step        Decay
0000000 0.900000
0000001 0.818182
0000002 0.750000
0000003 0.692308
0000004 0.642857
0000005 0.600000
0000006 0.562500
0000007 0.529412
0000008 0.500000
0000009 0.500000
```

### Logs

_No response_

### System Info

0.12 dev


Discussion:
Hey, @jorgemcgomes  great catch! This is indeed a bug.

Resolving patch:
diff --git a/examples/text_to_image/train_text_to_image.py b/examples/text_to_image/train_text_to_image.py
--- a/examples/text_to_image/train_text_to_image.py
+++ b/examples/text_to_image/train_text_to_image.py
@@ -278,24 +278,19 @@ def __init__(self, parameters: Iterable[torch.nn.Parameter], decay=0.9999):
         self.decay = decay
         self.optimization_step = 0
 
-    def get_decay(self, optimization_step):
-        """
-        Compute the decay factor for the exponential moving average.
-        """
-        value = (1 + optimization_step) / (10 + optimization_step)
-        return 1 - min(self.decay, value)
-
     @torch.no_grad()
     def step(self, parameters):
         parameters = list(parameters)
 
         self.optimization_step += 1
-        self.decay = self.get_decay(self.optimization_step)
+
+        # Compute the decay factor for the exponential moving average.
+        value = (1 + self.optimization_step) / (10 + self.optimization_step)
+        one_minus_decay = 1 - min(self.decay, value)
 
         for s_param, param in zip(self.shadow_params, parameters):
             if param.requires_grad:
-                tmp = self.decay * (s_param - param)
-                s_param.sub_(tmp)
+                s_param.sub_(one_minus_decay * (s_param - param))
             else:
                 s_param.copy_(param)
 


Test patch: