noamaanMulla-03/transformers-issues huggingface dataset:
[tests] make `test_sdpa_can_compile_dynamic` device-agnostic
Author: faaany
Pull request: yes
Labels: name: run-slow

## What does this PR do?
`test_sdpa_can_compile_dynamic` should not only work on GPU, it also works on XPU, below is an example:

```bash
PASSED tests/models/hubert/test_modeling_hubert.py::HubertModelTest::test_sdpa_can_compile_dynamic
```

@amyeroberts and @ydshieh 

Comments:
Hi @amyeroberts , thanks for the review, undrafted! 

can we merge this PR as well? Thx a lot! 

Hi @faaany, apologies for the delay. I can see that adding the `[run_all]` tag didn't run the sdpa tests -- @ydshieh how can we trigger these?

Hi @amyeroberts This test is only for A10 or A100.

```
            if not torch.version.cuda or major < 8:
                self.skipTest(reason="This test requires an NVIDIA GPU with compute capability >= 8.0")
```

And so far, `.github/workflows/self-pr-slow-ci.yml` is only written to run on a T4.

The quick but manual way is to launch a A10 SSH runner

https://github.com/huggingface/transformers/actions/workflows/ssh-runner.yml

and run the tests manually.

Hi @amyeroberts , can we also proceed with merging of this PR? 

Hi @faaany, yes, apologies for the delay. As this isn't changing any core logic it shouldn't really affect the slow tests so I think we can merge anyway
Input data format
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds the `input_data_format` argument to all of the image processor methods. 

This allows for passing in of images with an unusual number of channels, or ones where it's difficult to infer because of ambiguity e.g size (3, 3, 3).

This is an alternative to #24577

Fixes issues like: 
* https://github.com/huggingface/transformers/issues/21981
* https://github.com/huggingface/transformers/issues/21638
* https://github.com/huggingface/transformers/issues/22577


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Has 'input_data_format' argument been added to Mask2FormerImageProcessor? I am getting the following warning when used:
/usr/local/lib/python3.11/dist-packages/transformers/utils/deprecation.py:172: UserWarning: The following named arguments are not valid for `Mask2FormerImageProcessor.__init__` and were ignored: 'input_data_format'
  return func(*args, **kwargs)
  cc @amyeroberts , please confirm.
Need guidance to modify number of cores/threads settings when inferencing under transformers framework
Author: luentong
Pull request: no

In my project I need to run a LlavaLlamaForCausalLM instance inference under transformers framework, which is basically a subclass of nn.Module. I want to modify number of cores/threads settings when doing inferencing, in a machine with 160-core cpus and no gpu. Which folder in the source code of transformers/pytorch is responsible for this? Can anyone point to a specific location? Thanks a lot

Comments:
This question is probably more suited to the Hugging Face forums or discord, since it isn't actually a bug or feature request in the framework itself!
Fix/attention mask mismatch
Author: Pradnyanand09
Pull request: yes

##Fix custom 4D attention mask shape mismatch for packed inputs (#35290)

When packing samples using a custom attention mask, a shape mismatch error occurs:
RuntimeError: The size of tensor a (324) must match the size of tensor b (18) at non-singleton dimension 3

This error happens because the custom 4D mask created from a packed 2D mask (with a total length larger than the per-sample query length) is not block-diagonal. The Transformers utility expects a block-diagonal 4D mask where each block corresponds to an individual sequence (e.g., each block is 18×18 if each sample has 18 tokens), but a single contiguous mask is produced instead.

Description of the Fix:

Detection of Packed Masks:
The updated to_4d method now checks if the provided 2D mask is "packed" (i.e., the total length is a multiple of the individual sequence length but not equal to it).
Block-Diagonal Reconstruction:
When a packed mask is detected, the 2D mask is reshaped into blocks and reassembled into a block-diagonal mask. This ensures that each block corresponds exactly to one sample's tokens.
Causal Mask Application:
For causal masks, the fix applies the causal mask block-by-block. Specifically, for each block, only the last query_length columns of the generated causal mask (of shape [batch_size, 1, query_length, query_length + past_key_values_length]) are extracted. This sub-mask (of shape [batch_size, 1, query_length, query_length]) is then applied to the corresponding block in the packed mask, ensuring that the shapes match and the causal masking is correctly applied.
References:

Original Issue: [#35290](https://github.com/huggingface/transformers/issues/35290)
Related PR: [#35517](https://github.com/huggingface/transformers/pull/35517)
Testing Instructions:

A new test file has been added at tests/test_attention_mask_converter.py to verify the behavior in both packed and non-packed scenarios.
pytest tests/test_attention_mask_converter.py -v
Additionally, run your reproduction script to ensure that the shape mismatch error is resolved when using a custom attention mask for packed inputs.
Additional Notes:

This fix is one possible solution to handle packed attention masks by restructuring them into block-diagonal form. Further refinements might be needed for additional edge cases such as sliding window support or when past key values are used.
Feedback and suggestions are welcome.


Comments:
I think this is a duplicate of #35517

I'll review #35517 and take the necessary actions.
Move audio top_k tests to the right file and add slow decorator
Author: Rocketknight1
Pull request: yes

I missed this when reviewing the last PR - the tests should have been in the audio classification pipeline file, and we should have a `@slow` decorator because the model is ~300mb. @sambhavnoobcoder it might be better make a `tiny-random` model instead!

Comments:
(Failing tests are unrelated)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36072). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks @Rocketknight1 , i'll pay special attention to this next time as well .
[docs] fix model checkpoint name
Author: faaany
Pull request: yes

## What does this PR do?
The code snippet doesn't work because `distilbert_base_uncased_squad` doesn't exist in HuggingFace model hub. The correct one is `distilbert/distilbert-base-uncased-distilled-squad`.

Documentation: @stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36075). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] fix typo
Author: faaany
Pull request: yes

## What does this PR do?
This PR fixes a typo in the doc. 

Documentation: @stevhliu



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36080). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] fix not-working example code in `perf_infer_gpu_one.md`
Author: faaany
Pull request: yes

## What does this PR do?
When running through the code examples in `GPU inference` doc, I found the following issues:
1. hard-coded "cuda" device
2. "bigscience/bloom-2b5" doesn't exist anymore in HF model hub
3. the memory limitation set for "bigscience/bloom-3b" is too small, which causes the code to crash.
4. typo fix 
5. `model.to_bettertransformer()` is out-dated

Documentation: @stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36087). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
The JUKEBOX model does not have the test_modeling_jukebox.py file.
Author: 1hb6s7t
Pull request: no
Labels: name: Feature request

### Feature request

- `transformers` version: 4.44.2
- Platform: Windows-10-10.0.26100-SP0
- Python version: 3.9.19
- Huggingface_hub version: 0.25.2
- Safetensors version: 0.4.4
- Accelerate version: 0.34.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Motivation

During the process of transferring the JUKEBOX model, I planned to evaluate its transfer effect. However, a problem occurred: I couldn't find the test_modeling_jukebox.py file for testing.

### Your contribution

...

Comments:
멜론으로 음악이 나오던데요? 4D 얘기할때는 무서운 귀신노래나왔어요

Hi @1hb6s7t, the jukebox model has been deprecated, which means we are no longer actively supporting or testing it in the CI. The modeling files still exist and can be found in `models/deprecated/jukebox`.
Training issues latest version 
Author: WoutDeRijck
Pull request: no
Labels: name: trainer

name: bug

### System Info

- `transformers` version: 4.48.0.dev0
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.11.11
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 3060 Laptop GPU

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:
1. Clone ModernBert repo
2. Install latest transformers version (4.48.0-dev0)
3. Run examples/train_st.py to finetune modernbert.

### Expected behavior

I would expect no errors.
However when building transformers from an earlier commit it works. 
> `pip install git+https://github.com/huggingface/transformers.git@f42084e6411c39b74309af4a7d6ed640c01a4c9e`

So I think something broke during the latest commits.

Comments:
Greetings from Greece!
I would like to express my interest in contributing to this project. I believe I can provide valuable assistance to the team.Look forward to the opportunity to collaborate and contribute.

hi @WoutDeRijck, you'll have to give us some specifics about the error message for us to be able to diagnose this one!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Set output_attentions=True for model.geneate
Author: yiyexy
Pull request: no
Labels: name: bug

name: Generation

### System Info

transformers==4.44.0
Ubuntu 22.04.2 LTS  
python==3.10.13

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. I used the LLaVA arch with LlavaQwen2ForCausalLM.
2. I will get right response with output_attentions=False in model.generate()
3. The response are not expected and some nan in scores with output_attentions=True.

### Expected behavior

I expected get normal response with attention score.

Comments:
```
output_ids = model.generate(
                input_ids,
                images=images,
                attention_mask=None,
                do_sample=False,
                max_new_tokens=1024,
                use_cache=True,
                stopping_criteria=[stopping_criteria],
                output_attentions=True,
                output_scores=True,
                return_dict_in_generate=True,
                )
```

![image](https://github.com/user-attachments/assets/94d2805a-2059-4848-b858-a666ac1910cf)


Could you provide a minimal reproducible code?

Seems to be same as https://github.com/huggingface/transformers/issues/35270. I verified that the NaNs have been there from the release time, so I don't think there is anything fundamentally wrong in the code

There could be a workaround as suggested in the last comment, I will take a look later and see if we need it

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
'do_sample' model default cannot be overridden
Author: Zoher15
Pull request: no
Labels: name: bug

### System Info

transformers 4.47.1, python 3.10.

Basically while using Qwen-2-VL-Instruct (default config sets do_sample=True), if I set the model_kwargs with do_sample=False, I am unable to override the model_config. I had to make changes to the generations/utils.py to override....

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Use a model like Qwen2-VL-Instruct with default config using do_sample = True
2. Try to override the behavior through kwargs used during model.generate

### Expected behavior

If I set do_sample=False during generation, the default config of the model should be overridden

Comments:
cc @zucchini-nlp @gante 

@Zoher15 hey!

The default parameters should be overriden when you pass them as kwargs in the `generate`. Can you share how exactly you are using `generate` as for me `model.generate(inputs, do_sample=False)` works well ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Whisper] Fix num_return_sequences behavior
Author: eustlb
Pull request: yes

# What does this PR do?

Fixes #32373 #33445


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35939). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @eustlb, thank you for focusing on this issue! As I tried the fixed codes, it seems that the issue is still not fixed for me, as the generate function is still output k same hypotheses... Could you please help me out? Thank you.
Loss.. should be specified as either training loss or validation loss
Author: FlogramMatt
Pull request: no
Labels: name: bug

### System Info

Doesn't apply.. using runpod

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

As you fine tune a new model, it shows a loss curve and numbers as it runs.. is that training loss or validation loss? Should be labelled

If training loss.. how can I view validation loss? I set 'val size', it should exist somewhere

### Expected behavior

Should be labeled more clearly

Comments:
Could you please elaborate more or provide a script to replicate this?

I did figure it out, it is training loss.

 I mean when fine tuning the model, you have a 'loss' graph in bottom right and same for outputs in terminal and it's not obvious that it's training loss, not validation loss.  Would be nice if labelled better.

I mean if your code doesn't specify to plot a graph then it will not show any graph.
And if you are plotting the graph then you can label it correctly

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix MllamaVisionAttention typehint
Author: kylesayrs
Pull request: yes

## What does this PR do? ##
* Fix type hint in `MllamaVisionAttention.forward`

Comments:
@kylesayrs can you run `make fix-copies` to make CI happy?

Edit: rebasing should solve it now

@zucchini-nlp Sorry for the delay here, thanks for your help

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35975). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Processors: allow tuples of images when checking
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36055

For the rest of models, we don't have a helper to `flatten_and_maybe_load_images` so I just modified code within `processing_xxx.py`. I don't think such a helper is a need for now, we prefer to not allow users to load images like that, and promote proper usage of chat templates

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36084). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
PaliGemma processor should also accept tuples in addition to lists
Author: doctorpangloss
Pull request: no
Labels: name: VLM

https://github.com/huggingface/transformers/blob/0de15c988b0d27758ce360adb2627e9ea99e91b3/src/transformers/models/paligemma/processing_paligemma.py#L263

tuples are also valid here, the validation code is quite aggressive

Comments:
@doctorpangloss right, actually the check for images should be done using an existing helper fn from below. Would you like to open a PR? Otherwise I can fix it :)

https://github.com/huggingface/transformers/blob/0de15c988b0d27758ce360adb2627e9ea99e91b3/src/transformers/image_utils.py#L269

go for it I am already using your branch for testing
Chat template: update for processor
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Prerequisite: we need images as batched list support for all VLMs, which is now done thanks to Yoni 💛 

This PR adds:
- Support for batched inputs, which I forgot to add the last time
- Tiny hack for `add_special_tokens` as per @Rocketknight1's request
- More tests for chat templates in the Mixin, so we don't write the same test per model. Note that for test to run, we need to have the correct `chat_template` in processor, not a dummy one
- Video loading that can use `video_fps` which is claimed to be better than sampling N frames uniformly, especially for longer videos. NOTE: that will result in videos of different `num_frames` and can lead to errors when batching, so should be used only when `bs=1`. We don't support videos of different frame counts, yet
- Ability to load from list of image frames saved as image files

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35953). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool, I will merge it then. Core modeling is not touched and code owners review should be enough

Looking forward to this landing on a release so we can show one-step processor snippets for chat VLMs! 🔥
Fix OS err
Author: muellerzr
Pull request: yes

# What does this PR do?

There's an issue with https://github.com/huggingface/transformers/pull/35580 where if we have too many GPUs, we hit race conditions while trying to do the `os.rename`/move. Since we only need this once per node, really it just needs to happen under `args.should_save` (since that will make it happen only once per node or per instance as a whole if not doing save-on-each-node). 

This PR fixes this race condition.

I tested it on the cluster on 8 GPUs, which replicated the issue

Fixes https://github.com/huggingface/transformers/issues/36076


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36094). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@muellerzr im still facing the race condition even after this fix. because i think possible that the below happens

```python 
     if os.path.exists(output_dir): # two processes come here and both check (this is not atomic)
                    try:
                        os.renames(output_dir, checkpoint_dir) # first process comes in to do the rename (succeeds), second process comes in but fails (because not output_dir) is gone
                    except OSError as e:
                        if e.errno in [errno.ENOTEMPTY, errno.EEXIST]:  # Directory/File already exists
                            shutil.rmtree(checkpoint_dir)
                            os.renames(output_dir, checkpoint_dir)
                        else:
                            raise
```
Abit curious why this never happened before
Paligemma: revert #36084
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

cc @ydshieh 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36113). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.