internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/diffusers

Issue:
Wrong Text-encoder Grad Setting in Custom_Diffusion
### Describe the bug

According to [Custom Diffusion](https://arxiv.org/pdf/2212.04488.pdf), It only trains modified tokens. In implementation, it set the grad of non-modified tokens to zero. But there is a bug in this implementation.

https://github.com/huggingface/diffusers/blob/531e719163d2d7cf0d725bb685c1e8fe3393b9da/examples/custom_diffusion/train_custom_diffusion.py#L1180-L1187

In 

https://github.com/huggingface/diffusers/blob/531e719163d2d7cf0d725bb685c1e8fe3393b9da/examples/custom_diffusion/train_custom_diffusion.py#L1183

it should be 

`torch.arange(len(tokenizer)) != modifier_token_id[i + 1]`

as the variant 'i' starts from 0.

Consequently, the last concept would not be trained if you want to train multiple concepts(It works well when single concept training).

Should I open a PR to fix that? 

### Reproduction

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export OUTPUT_DIR="./ckpt"

accelerate launch train_custom_diffusion.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --output_dir=$OUTPUT_DIR \
  --concepts_list=./concept.json \
  --resolution=512  \
  --train_batch_size=2  \
  --learning_rate=1e-5  \
  --lr_warmup_steps=0 \
  --max_train_steps=500 \
  --scale_lr \
  --hflip  \
  --modifier_token "<new1>+<new2>" \
  --no_safe_serialization \

### Logs

_No response_

### System Info

diffusers :0.27.0.dev0

### Who can help?

@sayakpaul @DN6 @yiyixuxu 


Discussion:
Thanks for the detailed thread. Happy to review a fix PR. 

Also cc: @nupurkmr9 

Resolving patch:
diff --git a/examples/custom_diffusion/train_custom_diffusion.py b/examples/custom_diffusion/train_custom_diffusion.py
--- a/examples/custom_diffusion/train_custom_diffusion.py
+++ b/examples/custom_diffusion/train_custom_diffusion.py
@@ -1178,7 +1178,7 @@ def main(args):
                         grads_text_encoder = text_encoder.get_input_embeddings().weight.grad
                     # Get the index for tokens that we want to zero the grads for
                     index_grads_to_zero = torch.arange(len(tokenizer)) != modifier_token_id[0]
-                    for i in range(len(modifier_token_id[1:])):
+                    for i in range(1, len(modifier_token_id)):
                         index_grads_to_zero = index_grads_to_zero & (
                             torch.arange(len(tokenizer)) != modifier_token_id[i]
                         )


Test patch:
Repository: huggingface/diffusers

Issue:
Missing raise statement
### Describe the bug

https://github.com/huggingface/diffusers/blob/363699044e365ef977a7646b500402fa585e1b6b/src/diffusers/pipelines/stable_diffusion_gligen/pipeline_stable_diffusion_gligen.py#L471-L475

### Reproduction

There seems to be a missing raise statement in the gligen code (the `ValueError` is not raised). I can help fix that. @yiyi

### Logs

_No response_

### System Info

This is still in the main branch.

### Who can help?

_No response_
Missing raise statement
### Describe the bug

https://github.com/huggingface/diffusers/blob/363699044e365ef977a7646b500402fa585e1b6b/src/diffusers/pipelines/stable_diffusion_gligen/pipeline_stable_diffusion_gligen.py#L471-L475

### Reproduction

There seems to be a missing raise statement in the gligen code (the `ValueError` is not raised). I can help fix that. @yiyi

### Logs

_No response_

### System Info

This is still in the main branch.

### Who can help?

_No response_


Discussion:
Good catch! I searched in the repo and found [another one](https://github.com/huggingface/diffusers/blob/363699044e365ef977a7646b500402fa585e1b6b/examples/community/llm_grounded_diffusion.py#L533). Could you add this to the PR as well if it is confirmed?
Sure. Happy to review PRs.
Good catch! I searched in the repo and found [another one](https://github.com/huggingface/diffusers/blob/363699044e365ef977a7646b500402fa585e1b6b/examples/community/llm_grounded_diffusion.py#L533). Could you add this to the PR as well if it is confirmed?
Sure. Happy to review PRs.

Resolving patch:
diff --git a/examples/community/llm_grounded_diffusion.py b/examples/community/llm_grounded_diffusion.py
--- a/examples/community/llm_grounded_diffusion.py
+++ b/examples/community/llm_grounded_diffusion.py
@@ -530,7 +530,7 @@ def check_inputs(
                 )
 
         if len(phrases) != len(boxes):
-            ValueError(
+            raise ValueError(
                 "length of `phrases` and `boxes` has to be same, but"
                 f" got: `phrases` {len(phrases)} != `boxes` {len(boxes)}"
             )
diff --git a/src/diffusers/pipelines/stable_diffusion_gligen/pipeline_stable_diffusion_gligen.py b/src/diffusers/pipelines/stable_diffusion_gligen/pipeline_stable_diffusion_gligen.py
--- a/src/diffusers/pipelines/stable_diffusion_gligen/pipeline_stable_diffusion_gligen.py
+++ b/src/diffusers/pipelines/stable_diffusion_gligen/pipeline_stable_diffusion_gligen.py
@@ -469,7 +469,7 @@ def check_inputs(
                 )
 
         if len(gligen_phrases) != len(gligen_boxes):
-            ValueError(
+            raise ValueError(
                 "length of `gligen_phrases` and `gligen_boxes` has to be same, but"
                 f" got: `gligen_phrases` {len(gligen_phrases)} != `gligen_boxes` {len(gligen_boxes)}"
             )


Test patch:
Repository: huggingface/diffusers

Issue:
Potential UNet EMA bug in `train_text_to_image_sdxl`
### Describe the bug

Hi, 

Am I seeing a bug here? https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py#L1198-L1201

At the end of each epoch we perform a validation step which involves having `ema_unet` store `unet`'s params temporarily, and then loading `ema_unet`'s params into `unet` since the latter is what is used in the `pipeline` class to perform validation. However, when that is done there is no call to `ema_unet.restore(unet.parameters())` in order to restore the original UNet's parameters. I would have expected that method call to be made at the end of this if block: 

https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py#L1192-L1253

Does that sound about right?

Thanks,
Chris

### Reproduction

N/A

### Logs

```shell
N/A
```


### System Info

N/A (reading off Github code here)

### Who can help?

_No response_


Discussion:
i don't think this is a bug, but it might be undocumented/unexpected behaviour.

the whole time, the validations are done on the EMA weights. but at the end, the `save_pretrained` is run after setting it up so that the unet is the EMA weights.

the way this trainer functions, the non-EMA weights are thrown away at the end - they could be placed as an 'noema' variant, but that would increase complexity a bit.
Hi,

> the whole time, the validations are done on the EMA weights.

Yes, which is fine, but unless I'm misreading the code here, the validation is done _per epoch_ (i.e. each time the data loader has been exhausted). Therefore, in the next epoch, the UNet still has EMA weights applied to it when it should have been restored to its original weights.

As for the final validation:

https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py#L1256-L1305

 there is also a `ema_unet.copy_to(unet.parameters())` call which is completely fine, because as you said, the non-EMA weights are going to be discarded when training terminates.
oh, i see what you mean now. i'm sorry, i thought we were looking only at the final validations. that does indeed appear to be an issue. would you like to fix it?
Yep I'll write one up

Resolving patch:
diff --git a/examples/text_to_image/train_text_to_image_sdxl.py b/examples/text_to_image/train_text_to_image_sdxl.py
--- a/examples/text_to_image/train_text_to_image_sdxl.py
+++ b/examples/text_to_image/train_text_to_image_sdxl.py
@@ -1252,6 +1252,10 @@ def compute_time_ids(original_size, crops_coords_top_left):
                 del pipeline
                 torch.cuda.empty_cache()
 
+                if args.use_ema:
+                    # Switch back to the original UNet parameters.
+                    ema_unet.restore(unet.parameters())
+
     accelerator.wait_for_everyone()
     if accelerator.is_main_process:
         unet = unwrap_model(unet)


Test patch:
Repository: huggingface/diffusers

Issue:
pretrained_model_or_path variable name in AutoPipeline
### Describe the bug

In DiffusionPipeline.from_pretrained() the first parameter is called `pretrained_model_name_or_path` whereas in the Autopipelines, it is called `pretrained_model_or_path` (despite the class docs calling it `pretrained_model_name_or_path`  here too).

Don't know if this was intended, just a bit of a minor issue when trying to instantiate the pipelines with kwargs

### Reproduction

N/A

### Logs

_No response_

### System Info

N/A

### Who can help?

@stevhliu


Discussion:
Hi!
feel free to open a PR
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Resolving patch:
diff --git a/src/diffusers/pipelines/auto_pipeline.py b/src/diffusers/pipelines/auto_pipeline.py
--- a/src/diffusers/pipelines/auto_pipeline.py
+++ b/src/diffusers/pipelines/auto_pipeline.py
@@ -216,7 +216,7 @@ def from_pretrained(cls, pretrained_model_or_path, **kwargs):
         ```
 
         Parameters:
-            pretrained_model_name_or_path (`str` or `os.PathLike`, *optional*):
+            pretrained_model_or_path (`str` or `os.PathLike`, *optional*):
                 Can be either:
 
                     - A string, the *repo id* (for example `CompVis/ldm-text2im-large-256`) of a pretrained pipeline
@@ -489,7 +489,7 @@ def from_pretrained(cls, pretrained_model_or_path, **kwargs):
         ```
 
         Parameters:
-            pretrained_model_name_or_path (`str` or `os.PathLike`, *optional*):
+            pretrained_model_or_path (`str` or `os.PathLike`, *optional*):
                 Can be either:
 
                     - A string, the *repo id* (for example `CompVis/ldm-text2im-large-256`) of a pretrained pipeline
@@ -765,7 +765,7 @@ def from_pretrained(cls, pretrained_model_or_path, **kwargs):
         ```
 
         Parameters:
-            pretrained_model_name_or_path (`str` or `os.PathLike`, *optional*):
+            pretrained_model_or_path (`str` or `os.PathLike`, *optional*):
                 Can be either:
 
                     - A string, the *repo id* (for example `CompVis/ldm-text2im-large-256`) of a pretrained pipeline


Test patch:
Repository: huggingface/diffusers

Issue:
Duplicate code: SD3 JointAttnProcessor2_0
**What API design would you like to have changed or added to the library? Why?**

SD3's JointAttnProcessor2_0. It has a duplicate code.

**What use case would this enable or better enable? Can you give us a code example?**

In 1135 of diffusers.models.attention_processor.py
https://github.com/huggingface/diffusers/blob/7f51f286a5397cb3e5c5a25693681aa4955e6241/src/diffusers/models/attention_processor.py

Before:
```python
hidden_states = hidden_states = F.scaled_dot_product_attention(
        query, key, value, dropout_p=0.0, is_causal=False
)
```

After:
```python
hidden_states = F.scaled_dot_product_attention(
        query, key, value, dropout_p=0.0, is_causal=False
)
```


Discussion:

Resolving patch:
diff --git a/src/diffusers/models/attention_processor.py b/src/diffusers/models/attention_processor.py
--- a/src/diffusers/models/attention_processor.py
+++ b/src/diffusers/models/attention_processor.py
@@ -1132,9 +1132,7 @@ def __call__(
         key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
         value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
 
-        hidden_states = hidden_states = F.scaled_dot_product_attention(
-            query, key, value, dropout_p=0.0, is_causal=False
-        )
+        hidden_states = F.scaled_dot_product_attention(query, key, value, dropout_p=0.0, is_causal=False)
         hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)
         hidden_states = hidden_states.to(query.dtype)
 
@@ -1406,7 +1404,6 @@ def __call__(
 
 
 class AttnProcessorNPU:
-
     r"""
     Processor for implementing flash attention using torch_npu. Torch_npu supports only fp16 and bf16 data types. If
     fp32 is used, F.scaled_dot_product_attention will be used for computation, but the acceleration effect on NPU is
diff --git a/utils/update_metadata.py b/utils/update_metadata.py
--- a/utils/update_metadata.py
+++ b/utils/update_metadata.py
@@ -24,6 +24,7 @@
 Script modified from:
 https://github.com/huggingface/transformers/blob/main/utils/update_metadata.py
 """
+
 import argparse
 import os
 import tempfile


Test patch:
Repository: openmc-dev/openmc

Issue:
Writing/reading HDF5 state points in parallel
Currently the HDF5 state points will not work in parallel because only the source bank from processor 0 is written to the state point. This should be changed to use parallel HDF5 functionality and write/read the source bank collectively.



Discussion:

Resolving patch:
diff --git a/src/utils/statepoint.py b/src/utils/statepoint.py
--- a/src/utils/statepoint.py
+++ b/src/utils/statepoint.py
@@ -147,8 +147,11 @@ def __init__(self, filename):
         self._read_metadata()
 
     def _read_metadata(self):
+        # Read filetype
+        self.filetype = self._get_int(path='filetype')[0]
+
         # Read statepoint revision
-        self.revision = self._get_int(path='revision_statepoint')[0]
+        self.revision = self._get_int(path='revision')[0]
 
         # Read OpenMC version
         if self._hdf5:
@@ -179,7 +182,8 @@ def _read_metadata(self):
         if self.run_mode == 2:
             self.n_inactive = self._get_int(path='n_inactive')[0]
             self.gen_per_batch = self._get_int(path='gen_per_batch')[0]
-            self.k_batch = self._get_double(self.current_batch, path='k_batch')
+            self.k_batch = self._get_double(
+                self.current_batch*self.gen_per_batch, path='k_generation')
             self.entropy = self._get_double(
                 self.current_batch*self.gen_per_batch, path='entropy')
             if self.revision >= 8:


Test patch:
Repository: openmc-dev/openmc

Issue:
Python API errors out when trying to import state points with old specifications
When using the latest version of the python API with state point files generated with older versions of OpenMC, the Python API breaks when trying to import the state point file. As the error below shows, the numpy.ndarray filetype cannot be decoded so the API errors out before the revision check:

```
        94         # Ensure filetype and revision are correct
        95         if 'filetype' not in self._f or self._f[
 --->   96                 'filetype'].value.decode() != 'statepoint':
        97             raise IOError('{} is not a statepoint file.'.format(filename))
        98         if self._f['revision'].value != 14:

        AttributeError: 'numpy.ndarray' object has no attribute 'decode'
```

Also, as an aside, the state point revision number on the documentation on github [(link)](https://github.com/mit-crpg/openmc/blob/af6c1412bb110a2d2e88005508725a3d6a68f129/docs/source/usersguide/output/statepoint.rst) says 13, but I think it should be 14 as it differs from the state point specification 13 on the openmc website documentation.



Discussion:
Thanks for pointing this out @samuelshaner. I'll put in place some kind of better check there. You're right aobut the revision number; I'll fix that too.


Resolving patch:
diff --git a/openmc/statepoint.py b/openmc/statepoint.py
--- a/openmc/statepoint.py
+++ b/openmc/statepoint.py
@@ -92,9 +92,14 @@ def __init__(self, filename):
         self._f = h5py.File(filename, 'r')
 
         # Ensure filetype and revision are correct
-        if 'filetype' not in self._f or self._f[
-                'filetype'].value.decode() != 'statepoint':
-            raise IOError('{} is not a statepoint file.'.format(filename))
+        try:
+            if 'filetype' not in self._f or self._f[
+                    'filetype'].value.decode() != 'statepoint':
+                raise IOError('{} is not a statepoint file.'.format(filename))
+        except AttributeError:
+            raise IOError('Could not read statepoint file. This most likely '
+                          'means the statepoint file was produced by a different '
+                          'version of OpenMC than the one you are using.')
         if self._f['revision'].value != 14:
             raise IOError('Statepoint file has a file revision of {} '
                           'which is not consistent with the revision this '


Test patch:
Repository: openmc-dev/openmc

Issue:
Unable to run examples/python/pincell
Hi,

After generating the xml files and trying to `openmc` I get the following error:
```
Reading tallies XML file...
ERROR: Two or more meshes use the same unique ID: 1
```


Discussion:

Resolving patch:
diff --git a/examples/python/pincell/build-xml.py b/examples/python/pincell/build-xml.py
--- a/examples/python/pincell/build-xml.py
+++ b/examples/python/pincell/build-xml.py
@@ -119,7 +119,7 @@
 ###############################################################################
 
 # Instantiate a tally mesh
-mesh = openmc.Mesh(mesh_id=1)
+mesh = openmc.Mesh()
 mesh.type = 'regular'
 mesh.dimension = [100, 100, 1]
 mesh.lower_left = [-0.62992, -0.62992, -1.e50]


Test patch:
Repository: openmc-dev/openmc

Issue:
Consistent normalization of Legendre and Zernike polys
Currently, users have to normalize by (2n + 1) / 2 for Legendre coefficients and similarly by 1 / pi for Zernikes. If we're going to do this, then I think we should also remove the Zernike code that multiplies the Zernike polynomials by √((2n + 2) / epsilon). We should decide: either the user has to apply the integration factors consistently or they don't.


Discussion:

Resolving patch:
diff --git a/openmc/filter_expansion.py b/openmc/filter_expansion.py
--- a/openmc/filter_expansion.py
+++ b/openmc/filter_expansion.py
@@ -318,15 +318,15 @@ class ZernikeFilter(ExpansionFilter):
 
     This filter allows scores to be multiplied by Zernike polynomials of the
     particle's position normalized to a given unit circle, up to a
-    user-specified order. The Zernike polynomials follow the definition by `Noll
-    <https://doi.org/10.1364/JOSA.66.000207>`_ and are defined as
+    user-specified order. The standard Zernike polynomials follow the definition by
+    Born and Wolf, *Principles of Optics* and are defined as
 
     .. math::
-        Z_n^m(\rho, \theta) = \sqrt{2n + 2} R_n^m(\rho) \cos (m\theta), \quad m > 0
+        Z_n^m(\rho, \theta) = R_n^m(\rho) \cos (m\theta), \quad m > 0
 
-        Z_n^{m}(\rho, \theta) = \sqrt{2n + 2} R_n^{m}(\rho) \sin (m\theta), \quad m < 0
+        Z_n^{m}(\rho, \theta) = R_n^{m}(\rho) \sin (m\theta), \quad m < 0
 
-        Z_n^{m}(\rho, \theta) = \sqrt{n + 1} R_n^{m}(\rho), \quad m = 0
+        Z_n^{m}(\rho, \theta) = R_n^{m}(\rho), \quad m = 0
 
     where the radial polynomials are
 
@@ -335,7 +335,8 @@ class ZernikeFilter(ExpansionFilter):
         \frac{n+m}{2} - k)! (\frac{n-m}{2} - k)!} \rho^{n-2k}.
 
     With this definition, the integral of :math:`(Z_n^m)^2` over the unit disk
-    is exactly :math:`\pi` for each polynomial.
+    is :math:`\frac{\epsilon_m\pi}{2n+2}` for each polynomial where :math:`\epsilon_m` is
+    2 if :math:`m` equals 0 and 1 otherwise.
 
     Specifying a filter with order N tallies moments for all :math:`n` from 0 to
     N and each value of :math:`m`. The ordering of the Zernike polynomial


Test patch:
Repository: openmc-dev/openmc

Issue:
Check for derived in tally sum but not in sum_sq
Is there any good reason why we check `self.derived` [here](https://github.com/mit-crpg/openmc/blob/develop/openmc/tallies.py#L211)? We don't perform that check for `mean` or `sum_sq`. This is troublesome when creating a new tally with `get_slice` for example.


Discussion:
I think the logic behind it is that only real tallies have a `sum` attribute because that it corresponds directly to what OpenMC is actually accumulating during transport. For a derived tally, it doesn't make sense to have a "sum" because it doesn't reflect a tally for which realizations were actually accumulated. However, a derived tally can have a mean that is determined in relation to the original tallies it was derived from. I'm not sure why `sum_sq` doesn't have a similar check because the same argument applies as for `sum`.

What is the specific problem that you are running into with `get_slice`?
"troublesome" was the wrong word to use... I wasn't thinking completely about my problem; `mean` contains all the information I need.

The logic makes sense to me. Maybe I'll submit a PR with the `derived` check added for `sum_sq` and then consider this a closed issue.

Resolving patch:
diff --git a/openmc/tallies.py b/openmc/tallies.py
--- a/openmc/tallies.py
+++ b/openmc/tallies.py
@@ -248,7 +248,7 @@ def sum(self):
 
     @property
     def sum_sq(self):
-        if not self._sp_filename:
+        if not self._sp_filename or self.derived:
             return None
 
         if not self._results_read:


Test patch:
Repository: openmc-dev/openmc

Issue:
Update atomic mass evaluation to AME 2016
The atomic mass data we use for `openmc.data.atomic_mass(...)` is from the 2012 atomic mass evaluation. There is an updated evaluated from 2016 that is [available](https://www-nds.iaea.org/amdc/).
Update atomic mass evaluation to AME 2016
The atomic mass data we use for `openmc.data.atomic_mass(...)` is from the 2012 atomic mass evaluation. There is an updated evaluated from 2016 that is [available](https://www-nds.iaea.org/amdc/).


Discussion:

Resolving patch:
diff --git a/openmc/data/data.py b/openmc/data/data.py
--- a/openmc/data/data.py
+++ b/openmc/data/data.py
@@ -141,8 +141,8 @@
 def atomic_mass(isotope):
     """Return atomic mass of isotope in atomic mass units.
 
-    Atomic mass data comes from the `Atomic Mass Evaluation 2012
-    <https://www-nds.iaea.org/amdc/ame2012/AME2012-1.pdf>`_.
+    Atomic mass data comes from the `Atomic Mass Evaluation 2016
+    <https://www-nds.iaea.org/amdc/ame2016/AME2016-a.pdf>`_.
 
     Parameters
     ----------
@@ -157,8 +157,8 @@ def atomic_mass(isotope):
     """
     if not _ATOMIC_MASS:
 
-        # Load data from AME2012 file
-        mass_file = os.path.join(os.path.dirname(__file__), 'mass.mas12')
+        # Load data from AME2016 file
+        mass_file = os.path.join(os.path.dirname(__file__), 'mass16.txt')
         with open(mass_file, 'r') as ame:
             # Read lines in file starting at line 40
             for line in itertools.islice(ame, 39, None):
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -32,7 +32,7 @@
     # Data files and librarries
     'package_data': {
         'openmc.capi': ['libopenmc.{}'.format(suffix)],
-        'openmc.data': ['mass.mas12', 'BREMX.DAT', '*.h5']
+        'openmc.data': ['mass16.txt', 'BREMX.DAT', '*.h5']
     },
 
     # Metadata


Test patch:
diff --git a/tests/unit_tests/test_data_misc.py b/tests/unit_tests/test_data_misc.py
--- a/tests/unit_tests/test_data_misc.py
+++ b/tests/unit_tests/test_data_misc.py
@@ -51,14 +51,14 @@ def test_thin():
 
 
 def test_atomic_mass():
-    assert openmc.data.atomic_mass('H1') == 1.00782503223
-    assert openmc.data.atomic_mass('U235') == 235.043930131
+    assert openmc.data.atomic_mass('H1') == 1.00782503224
+    assert openmc.data.atomic_mass('U235') == 235.04392819
     with pytest.raises(KeyError):
         openmc.data.atomic_mass('U100')
 
 
 def test_atomic_weight():
-    assert openmc.data.atomic_weight('C') == 12.011115164862904
+    assert openmc.data.atomic_weight('C') == 12.011115164864455
     with pytest.raises(ValueError):
         openmc.data.atomic_weight('Qt')